To accept that failing in ABX tests is due to expectation bias might be a result of strong biases (not only expectation) as well. 
Perceptual Evaluation of multidimensional stimuli is more difficult than testing a onedimensional difference. As we already know that the ABX method is more mentally involving (and most likely leads to a different internal judgement process in comparison to an A/B test) an experimenter should not underestimate the difficulties this specific test conditions presents to participants.
There are remedies, an experimenter just have to provide accomodation time and must use positive controls to check if sufficient sensitivity is reached.
Caveat: if an experimenter doesn´t know what the difference is, the "sufficient" condition might be hard to evaluate......
Perceptual Evaluation of multidimensional stimuli is more difficult than testing a onedimensional difference. As we already know that the ABX method is more mentally involving (and most likely leads to a different internal judgement process in comparison to an A/B test) an experimenter should not underestimate the difficulties this specific test conditions presents to participants.
There are remedies, an experimenter just have to provide accomodation time and must use positive controls to check if sufficient sensitivity is reached.
Caveat: if an experimenter doesn´t know what the difference is, the "sufficient" condition might be hard to evaluate......
To accept that failing in ABX tests is due to expectation bias might be a result of strong biases (not only expectation) as well.
Not hearing a difference is only a failure if the difference really exists. Failing in ABX tests is not due to expectation bias. On the contrary, the purpose of ABX testing is to remove expectation bias from the equation. Are you suggesting that people expect to not hear a difference under ABX conditions?
ABX testing doesn't "fail" if people can't hear the difference, for whatever reason. It only achieves a positive result when someone can hear the difference, and the null hypothesis can be rejected..
Your argument relies on the assumption that the differences you can hear are real. You are not entitled to do that without first ruling out the possible effects of expectation bias. The only way that can be done is by having someone demonstrate that they can hear the difference in a controlled situation.
On one side I'm a fan of abx and other dbt variants for determining whether things are clearly audible, but as Jacob writes, one must expect that sensitivity drops with the various stresses involved with such a test. With it, we gain a huge amount of specificity.
There are ways to mitigate many of the stresses of dbt protocols, but that does require degrees of training and familiarization, which again makes the test longer.
Likewise positive controls can show some degree of the listener's sensitivity to the test. With digital signals, its not too hard to imagine how one could add an effect at a specific level (Or ladder) to find a cutoff.
This is all basic experimental design and stuff generally useful no matter the discipline, albeit with different methodologies.
There are ways to mitigate many of the stresses of dbt protocols, but that does require degrees of training and familiarization, which again makes the test longer.
Likewise positive controls can show some degree of the listener's sensitivity to the test. With digital signals, its not too hard to imagine how one could add an effect at a specific level (Or ladder) to find a cutoff.
This is all basic experimental design and stuff generally useful no matter the discipline, albeit with different methodologies.
Last edited:
There are many biases now known. Some differ in subtle ways, but can be experimentally shown as distinct. Here is the list:
List of cognitive biases - Wikipedia
You might want to take a look at the definition for expectation bias. You sure it is the best fit for what goes on in sighted listening?
No. Ego is a Freudian concept, now mostly deprecated. Even Jung is pretty passe. Back in their time psychology was highly theoretical. For at least the last 20 years or so psychology has gone experimental. The new models are essentially science-based, that is to say, they follow from application of the scientific method.
The term "ego stroking" does not refer to a specific psychological concept. It is a general language expression, and everyone knows what it means.
Seems to me it is loose slang that doesn't belong in a technical discussion. Maybe in the lounge would be a better fit. In addition to being loose slang, its connotations are dripping with judgement and bias. Look, we learn how to talk in an informed way about electronics. It would be great if we could try to speak accurately about the subject of bias and other mental processes if we are going to have a serious conversation about ABX and related topics.
Freud formulated a conceptual model of the mind, and ego was one of the components he hypothesized. However, he did not use it followed by the word 'stroking.' Once we are at the point of talking about ego stroking we have departed from Freud's serious efforts to treat severe mental illness and entered into pop psychology, at best.
Unfortunately, discussion of ABX testing and what people can and can't hear has a history of devolving into heated arguments with little in the way of sober scientific perspective. Jakob2 has repeatedly tried to refer to relevant research and patiently explained over and over again where this science is for this subject area. Nobody listens, remembers, cares, they just keep going back to their same old tribal camps because they have already made up their minds on what they think about the subject, not that many are particularly well informed on it.
Ironic? All humans are biased all the time, it is unavoidable. Hopefully, we can try to put some constraint on it and do our best to maintain a scientific attitude.
I think I will leave this thread for now. Maybe take a look at some point later. If is is a serious discussion about the science, great. If not, see around somewhere else.
Last edited:
No, but I did, and it was obvious what I meant, to you included apparently, after you'd felt compelled to correct me, perhaps you are too intellectual
A the technical university I attended they made a test where some test persons should listen to some speakers. They were told that the x-over filter would be changed between listening sessions (it was not) and the front cloth color was changed to make it easier to talk about the different filters. Note that only the color if the front changed - not the speaker and not the filter.
The test result showed clearly that green speakers are best....
The test result showed clearly that green speakers are best....
You can say that again. I'm as subject to it as anyone though I don't do much critical listening I could share stories from blind/non-blind wine tastings that could make anyone laugh. There's a whole "real" people dynamic involved that includes size of group, economic diversity, how well they are acquainted, degree of social obligation to be respectful, etc. I don't know how you sort this out.
Remember I was the one that memorized the clicks at the AES ABX demo and pooped the party.
It was meant a bit "tongue in cheek", so sorry it obviously is dangerous to try it in a foreign language.
But is is following your reasoning as you demanded "proof" (just as a reminder as formally proving something with empirical tests isn´t possible) for sighted listening impression by doing an "ABX" .
In that sense a failed ABX is an ABX where rejection of the null hypothesis wasn´t possible.
You asserted before:
That triggered my comment that to easily attribute the reason for a failed "ABX" to expectation bias (because no real difference exists) could be a matter of bias itsself.
Please remember my post that you answered, in which i wrote that we usually don´t know about the reasons for a negative test result and so only are able to report that the null-hypothesis couldn´t be rejected.
An ABX - test can´t remove expectation bias, it only is able to avoid that expectation bias leads to false positives (alpha errors).
Without doing additional questionaires or tests we routinely don´t know what participants expect, but from other sensory tests where those questionaires were included, we know that the participants expected a lot of (often quite different) things about to happen.
Therefore sometimes tests are used where the listeners don´t get informations about the specific effects under test. Although that most probably will not avoid any expectation (and therefore bias) it avoids a specific kind of bias.
No, you just misinterpretated what i´ve wrote.
The irony here is that in attempting to describe others you have accurately described yourself!
Unlike you, I don't have a vested interest in the outcome. I didn't pay over $2000 for a DAC. I just want to know the truth. Are the differences you claim to hear real? I'd be happy with the outcome either way.
Of course you want them to be real, maybe you even need them to be real, but until you demonstrate in a controlled ABX test, that they are real, no-one is obliged to believe you.
The frustrating thing is that all it would take is one person to demonstrate that they can hear the difference in an unsighted situation. Why doesn't this happen? Instead of trying to find or invent faults in the experimental method, why not just set up a test, as best you can, and see what happens. Are the differences so minuscule that that they are that easily hidden?
To do a definitive test with extremely good positive predictive value and translation to how we listen (I'm critical of samples less than 10 or so seconds on account of it no longer representing the complexity of listening to music, but this depends on use case) requires extremely good controls to ensure parity of testing and a very high number of tests (I haven't run the math to get a decent end criteria, but suffice it to say that you need extra replications to push the experimental noise down for such subtle changes), likely spread over numerous days to avoid exhaustion.
Who the heck wants to do that? Especially when it won't change the language of the discussion one iota and also puts the tester/tested at risk of finding out something against their hypothesis, much less blowback from publication?
Who the heck wants to do that? Especially when it won't change the language of the discussion one iota and also puts the tester/tested at risk of finding out something against their hypothesis, much less blowback from publication?
In the years I spent studying for a Science Degree, no-one ever reminded me of that! In fact the very purpose of empirical tests is to prove hypotheses. Without them science couldn't function.
Exactly! Failing to reject the null hypothesis doesn't "prove" anything. ABX tests can only prove something when the null hypothesis is rejected, and in the situation we are considering that would require someone (just one person!) to demonstrate that they can hear the difference under controlled conditions.
Proof is proof, and its proof that changes the language of discussion.
The problem is, and has always been, that "believers" expect others to accept their observations without providing and evidence or proof. Of course people will doubt them, and why shouldn't they?
At the end of the day, if the differences, if they exist, can only be reliably heard with such extreme difficulty, then what's the point?
Jacob is entirely correct, and I'm hoping it's a misunderstanding of how you respectively frame the language. Without testing every possible permutation of the universe (multiverse?) we are instead saying that, according to the tests we've run, this is what we've observed and projecting that prior testing onto future tests.
When n observations get so huge and the confidence in the observation so high, then it's practically proven, albeit not definitively. Sort of like American physical society has something like a 6 sigma confidence interval to publish a discovery. That's incredibly high, but still infinitely far away from definitive.
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.