There was a course at the Met Museum given by a gal who had a PhD in math, had been a mathematician for IBM when she decided to chuck it and get a PhD in Art History. "Art in Context".
So one Saturday morning she lectured on Gothic architecture starting with St. Denis and moving onto Amiens. Amiens was interesting because Columbia University's Architecture School had just put on the web "The Amiens Project" and it took forever to just download one drawing! I recall that she said "The architects of Amiens understood the calculus, they just couldn't codify it." Meaning I guess that the language of mathematics hadn't progressed much beyond Euclid.
Amiens was built, what, 500 years before Newton, 200 years before Galileo?
Good luck, I'm trying to get the same information for about two years now.
And yes, quoting out of context is a method of choice. There's nowhere in the existing standards that require a "positive control" for completing a credible ABX test.
Last edited:
Really I just keep asking outline the process and propose these "anchors" for this specific DAC test. I mean a detailed description of what is involved and what is supposed to happen. Why do I need to read the ITU spec for the purpose of this test I accept it based on its provenance (I know members of various standards committees)? Where did I ever disagree with it? I'm sure it covers no peeking or cheats and I'm happy with that.
These thing are work and they are not fun.
Let me repeat it , it is amazing how strong bias works.
(SCR)I´m not sure mmerill99 stated it the way that you claim - strong bias leads imo to less positive interpretation (or stronger to more negative interpretation) of written statements of a member seen as opponent - but the goal of using a positive control is to show that a test works in the way it should work - and i´m pretty sure that mmerill99 did already write that.
And please think about it and tell me your thoughts; if an ABX test is used to test a given fixed sensory difference but the proportion of correct answers in the ABX is significantly lower than in an A/B test used to test the same given fixed sensory difference, could it mean (everything else equal, especially the number of trials) that the ABX has a hidden flaw?
I guess you don´t like to read the ITU-R BS.1116-3?
I assume you want to test the two DACs with music samples; so for the normal trials you use the same music sample as stimulus for both DACs, but for the control trials you prepare music samples with a slightly higher level (around 0.2 dB and lower) which you are using in the control trials. One DAC gets fed with the music sample of the normal level while the other DAC gets fed with the music sample of slightly higher level. But your listeners will still answer the same question as before.
There are two possibilities to choose from; as usual both have their advantages and disadvantages but principially you can use control trials within your normal test routine (requires to use more trials overall) or you can run a completely seperated test (with the same number of trials as you normally would use) with the control stimulus/stimuli against the normal/unaltered stimulus.
Last edited:
Not the stats, which would be a separate issue if one wanted to get into that. They did and presumably still do describe what the displayed stats mean about the chances of someone guessing. The description is incorrect, but it is not big deal.
The story I was referring to is Foobar could print out your score with a checksum included. If you feed the whole printout into a website they put up, it could verify that the score numbers were not altered by recalculating and verifying the checksum. This was believed to be of value to prevent cheating when reporting scores. However, cheating was still quite possible which I generally alluded to but didn't describe in any detail. A diyaudio moderator then said I would be doing a service to the community if I would disclose details of the potential cheat, because he said, they suspected there had been some cheating going on and they didn't know how it was possible. So, I complied with the request.
Last edited:
Ok, I’ll bite.
Assuming a separate control test, the combination of the two tests have four possible outcomes. Please state the interpretation and the actions to follow for each of the four outcomes.
That leaves me out from the start. BTW I don't see a 0.2dB level change as fitting on their 5 point annoyance scale. In fact the whole document seems centered around CODEC's and how much information can you remove before annoyance sets it. As I said their reproduction system has very modest requirements and no qualification other than THD is applied to the distortion spectra.
Is the concept of abstraction really totally outdated?
We are/were talking about sensory tests in general and about certain test protocols in detail.
The ITU-Recommendations do explain a lot of thing that are in general important for "double blind listening tests" that should be able to detect not so obvious or big differences.
That includes to emphasize the usage of controls (their anchors) and the importance to check for possible factors that might lead to lower sensitivity.
And some of the points are direct related to their specific test protocol that was especially designed for their needs in testing codecs.
Could we use the ABC/HR protocol for other EUTs (than codecs)? Of course we can, but might do some adaptions, it is not mandatory to use a grading scale that includes terms like "annoying" we could use another attribute that fits better to our demands. But the principial requirements for controls (their anchors) and checks for sufficient sensitivity are still the same. The same holds true for the training and accommodation time.
Do we need to use the ABC/HR method? No of course not, there are other protocols that we could choose if we need no grading. Use for example the A/B paired comparison method.
Do we still have to use controls (their anchors)? Of course as the underlying principles are still the same. Do we still have to check for sufficient sensitivity? Of coures we have to.
It´s probably the 1000th repetition, but any test has to be objective, valid and reliable.
Validity means that the test really tests what you (the experimenter) pretends it does. If it should test if a difference between two EUTs exists it should _really_ test that, it shall not test (hidden because the experimenter is ignorant of the fact) instead if your detectors were desensibilised by the specific test conditions. T ofind out if this desensibilisation happens, a smart and honest experimenter (who wants to know if his test is a good one) use positive controls (and for completeness negative controls as well).
Just as a reminder, if the same test person gives a lower percentage of correct answers in an ABX than in an A/B test when testing the exact same sensory difference, then the validity of your test is in question when you are using an ABX without guards against the drawbacks of this test protocol.
Last edited:
I wouldn't agree - the highlighted test is pre-training & not a hidden anchor or control
I maintain that the hidden anchor should be within the normal test in a random number of trials.
If it is included as a separate test then the participant is aware & this awareness may effect the attention, awakeness, etc.
When included as random trials within the actual test the results can be analyzed & if the participant failed to score correctly these control trials (some agreed level maybe 4 or 5 correct out f 6 control trials) then the that participant's complete results are discarded. If this is happening with many participants then the test itself has to be examined as regards fit for purpose
Last edited:
As usual it depends, if the listeners don´t know about the EUT they only know that doing two runs of trials are requested.
A random number of control trials is questionable as a minimum set is required due to statistical reasons. Further you have to carefully check if the control trials itself lead to distraction as in these trials the differences might be suddenly ... different (assuming that a real audible difference between the DUTs exists).
Edit: So i would choose to include controls when sort of (hedonic) rating is used and several stimuli will be rated, but use seperated tests when only two items are under test and the same stimulus is used in every trial.
If different stimuli are used, the controls can be used again within the "normal" trials.
Last edited:
Yes, for sure, it gets complicated for those that are interested in conducting valid tests but for others, who claim it's just a simple test, close your eyes, no peeking, blah, blah. As long as they are getting null results sure what could possibly be the problem
Yes, for sure, it gets complicated for those that are interested in conducting valid tests but for others, who claim it's just a simple test, close your eyes, no peeking, blah, blah. As long as they are getting null results sure what could possibly be the problem
Let´s see if in the end something will have changed.....
Will you guys please stop the word salad and propose something tangible and in detail. In case you have not figured it out no one is arguing anymore. You can discard my results ahead of time I'm only interested in seeing others claim's substantiated, I'll even let you set the rules.
Last edited:
Yes you are still arguing & after interminable posts misinterpreting what I clearly said & arguing as a result, you are now moving into stage 2 argumentum
If you're not interested (& this is plainly obvious) why not leave the thread?
- Status
- Not open for further replies.