There were several issues pointed out earlier in the thread that may have alluded you.
The OP claimed that his ABX test is valid because it is the same approach used by the FDA in clinical trials. This is complete nonsense.
The OP actually believes that his test can prove that two devices sound the same. He held this position after several people pointed out it wasn’t the case.
The OP has a history of presupposing the result even before he has begun the test.
The first thing a scientist must to do, when considering the results of a test, is to make sure that the particular test methodology and implementation actually generate valid results.
Even if you believe in the validity of ABX testing in general, it seems that you have uncritically accepted the results of this test – I am guessing the results suited your world view so you deemed the test valid even without any knowledge of the actual implementation.
You have taken more of a philosophical than a scientific position, but, if it saves you money on your DAC…
And ?
That would be of great importance if all sighted listening tests by audiophiles and hifi critics were centered around the overall feeling of pleasure and involvement one gets from music through a particular system. But many of these tests also involve analytical listening as they focus on particular aspects (imaging, details, bass response and so on). Are you going to throw all that out ?
Planet10 imo just points to the fact that external validity is always questionable with this sort of tests.
Not at all, i was just asking related questions.
Overall our discussion styles unfortunately seem to be sufficiently incompatible.
So, is that a "yes" or a "no"?
Surely you must have some idea of how ridiculous these comments are, in the context of academic discourse?
Have you got any evidence for that statement in bold? I mean statistical, scientific evidence?
Is the word of mouth anecdote not usually followed up by a person doing their own sighted listening & is are false positive rates for this inordinately high? Just wondering if that is what you mean?
Last edited:
Something that keeps cropping up is the problem of short term audio memory, which appears to mean that only suitably trained or experienced people are able to do a meaningful ABX test. For the rest of us normal folk who like to compare in a more relaxed way similar to how we would listen to music for pleasure, how valid is extended listening over days between changes? I appreciate the problem of things sounding better merely because they're different.
A recent article in NYT How the Shape of Your Ears Affects What You Hear - The New York Times might clarify some aspects of auditory perception i.e. it uses a model-based approach - we compare what we hear/see against previously stored abstracted aspects or models. In this case a change in the shape of the ear disturbs our perception of elevation location of the source of what is being heard. Just how large the difference in the soundfield impinging on the tympanic membrane from this change in ear shape, I don't know but it makes for an interesting question.
What this research shows is that auditory perception uses a model based approach - we locate the elevation of a sound based on the pattern of frequencies that we have learned correlate with the actual location we can see or sense in other ways. A small change in this pattern breaks this correlation but the interesting thing is that we adapt to changes after about a week & then have the same location capabilities as our previous perception gave.
This means that we learn/lay down new aspects/models by correlation between our senses & our understanding of how the world works within that perceptual framework.
The same happens with visual perception so prism glasses that flip the image falling on the Fovea into the correct, upright orientation is disorienting at first but after about a week the visual world is no longer perceived as flipped but is correctly orientated.
Here's the really interesting part - taking off the glasses & we immediately? adjust to the image on the fovea being upside down & again perceive it in the correct orientation. The same applies with ears - slightly change the shape of the ears pinna & we find it difficult to localise height of sounds but readjusts over a week of use. Changing between the original pinna shape & new shape is then instantly adapted to rather than taking a week. We seem to retain the new perceptual model learned & can instantly(?) flip between them.
So just to bring this to our hobby. Do we 'learn' the sound of our playback system & putting in a new device allows us to perceive it as a change in some aspect of the sound? It might not be an very obvious aspect - could be a small change in our perception of elevation/soundstage/individualisation of the performers, etc.
So as is being asked here - what is the value of sighted listening Vs ABX blind testing? Maybe some aspects of our auditory perception are not instant A/B differences but rather more subtle differences which are teased out over time. Being able to A/B these subtle differences may well require training/experience to be able to differentiate
Retiring to my bunker now with hard-hat in place
What this research shows is that auditory perception uses a model based approach - we locate the elevation of a sound based on the pattern of frequencies that we have learned correlate with the actual location we can see or sense in other ways. A small change in this pattern breaks this correlation but the interesting thing is that we adapt to changes after about a week & then have the same location capabilities as our previous perception gave.
This means that we learn/lay down new aspects/models by correlation between our senses & our understanding of how the world works within that perceptual framework.
The same happens with visual perception so prism glasses that flip the image falling on the Fovea into the correct, upright orientation is disorienting at first but after about a week the visual world is no longer perceived as flipped but is correctly orientated.
Here's the really interesting part - taking off the glasses & we immediately? adjust to the image on the fovea being upside down & again perceive it in the correct orientation. The same applies with ears - slightly change the shape of the ears pinna & we find it difficult to localise height of sounds but readjusts over a week of use. Changing between the original pinna shape & new shape is then instantly adapted to rather than taking a week. We seem to retain the new perceptual model learned & can instantly(?) flip between them.
So just to bring this to our hobby. Do we 'learn' the sound of our playback system & putting in a new device allows us to perceive it as a change in some aspect of the sound? It might not be an very obvious aspect - could be a small change in our perception of elevation/soundstage/individualisation of the performers, etc.
So as is being asked here - what is the value of sighted listening Vs ABX blind testing? Maybe some aspects of our auditory perception are not instant A/B differences but rather more subtle differences which are teased out over time. Being able to A/B these subtle differences may well require training/experience to be able to differentiate
Retiring to my bunker now with hard-hat in place
The matter of the short auditory memory wrt our discussion is a controversial one. If taken at face value the statement is true as the auditory memory is the echoic memory that appears to be a real short term memory (accordng to the literature the time span is within a range of at max a couple of seconds).
In the broader sense it is a bit misleading as auditory memory means (should mean) the ability to memorize certain properties of original sound events and to retrieve such information from working memory and long term storage.
As we have often pointed out in the past -and mmerill99 did again describing the NYT article - we deal with models about our senses. The term "working memory" came out from one specific memory model, it means that sounds are transferred from the echoic memory to another storage area where processing of these sounds begins. Unprocessed sound seem to fade away in quite short time while sound information that is used may last for minutes to hours.
According to the model processing and categorization of sounds allows transfer to long term memory where the information may last for years (forever?).
Mark4 pointed out that it is our understanding that remembering might not only retrieve information from long term storage but also (?ever?) tends to alter the information and writing back a (at least) slightly different version.
Wrt multidimensional stimuli afaik there is not so much published material available that deals with degrees of difference and the according ability to remember the various differences.
What we know so far seems to provide evidence that on average participants that are not used to do ABX tests do experience more difficulties compared for example doing an A/B test.
But, in my experience, on average people are experiencing difficulties when doing A/B as well, if they were not used to do such kind of tests.
So anyway people should practise as usually adapting to unusal conditions takes some time.
As some people describe getting good results with ABX tests, it might still be a useful tool.
As a reminder, the ABX test is a test for difference (only) and should only be used if the knowing about a difference (alone) is really an important information.(It usually is at certain stages of a development or altering process; but wrt the questions that we discuss i´d say it isn´t that important)
Postscript: ABX is similar to other protocols like triangle (and others) tests, because participants have to memorize three items but does not have the advantage of enhance efficiency.
In the case of a "sort the odd one out" test, participants probe "AAB" , "ABA", "BAB" and "BBA" in the trials and should be able to identify the different sample. As chance to get the correct answer by random guessing is only 1/3, less trials are needed to get the same socalled "level of significance" .
In an ABX the chance to get a correct answer by random guessing is still 1/2 as it is in the case of an A/B test.
A/B tests are more similar to our usual habits for comparison of two "things" , so might have a genuine advantage, but the statistical needs for more trials and the somewhat artifical "blinding" cause usually (on average) that some accomodation/training time is still needed.
Listening over prolonged time spans do not harm generally but it depends on the question that you trying to explore and the specific way to assess it.
Last edited:
Member
Joined 2009
Paid Member
This, at least, is lucid. It's a commonly-made mistake.
Take 4 DACs: call them A, B, C and D. As that act of definition already implies, they are individually different. They are not A, A, A and A. Each is a unique combination of hundreds of components, very few of which - perhaps none at all - are shared. That's a good starting point.
A blind audio test doesn't address the reality of differences between A, B, C and D; it determines whether a given listener can distinguish between them in a specific, uncomfortable, mode of perception. More on this shortly.
Conversely, placebo-filtering drug trials aim to separate the somatic effect of a drug from the psycho-somatic impact of expectation bias. A clear window is needed on how medication impacts the unconscious systems of the body without the influence of the mind. For the purpose of such testing, the brain is an inconvenience, contaminating the data. Note that the drug is the object of such a test: it's efficacy is in question. A wide sample of reactions in the bodies of its subjects validates or invalidates it.
When contriving a test aimed at determining listener acuity, we have to be mindful that it's a test of perception: the brain is not an inconvenient obstacle in such a test: it is the very subject of it. It would be absurd to blindfold those taking part in an eye test. 'Sciency' (but inapplicable) methodology might give a spurious gloss to a perception experiment in disguise, but it's a parlour trick: watch the subject, not the object.
The central problem is modeling. The brain needs a roughly pegged out guideline to begin building a model from external stimuli. It might be a memory, an expectation - even a name, colour or smell. Over time, the absorption of sense information fills in that picture - sometimes eventually erasing the original outline impression with a more comprehensive pattern of data. Sensory deprivation creates altered mental states because our psychology destabilises in the absence of information. So, yes, knowing what you're listening to creates overwhelming expectation bias. Equally, not knowing what you're listening to is disorientating and stressful. Many studies confirm that cognitive function - particular fine-grained perceptive ability - is profoundly impaired by such stress.
What makes blind testing feel so much more like a parlour trick is an insistence by certain advocates on rapid switching, or quick-fire judgment. Modeling takes time: it's a gathering and organising of subtle stimuli. But this slow/steady accretion builds to something remarkably solid. As a phrase 'audio memory' is misapplied: literally it's zero: you have no apparatus to recreate a specific airborne vibration: when it's gone, it's gone. However, a characteristic sound you have modeled over time might stay with you for a lifetime: your partner's voice; a pair of speakers sold twenty years ago.
I was reminded of this some years ago, talking to my daughter in a crowded, noisy supermarket aisle. Some distance away (perhaps 10 metres) amid the hubbub, I heard someone drop their keys. I immediately walked over to a guilty-looking couple and asked them to return them. The sonic imprimatur of that peculiar combination of keys on my fob, modeled by years of acquaintance, was unmistakable. Unknown to me (unheard!), they had dropped from my pocket earlier. It was sheer chance that I bumped into the thieves before they completed their shopping.
The only valid conclusion to be drawn from blind testing audio equipment is that it's the wrong methodology to tell us anything useful about audio equipment. It does, however, tell us something interesting about perception: differences vanish when 'blind', and reappear when sighted: rather like holding your nose when tasting cinnamon.
Last edited:
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.