Technically, you can only say, that with a very large number of trials, the average will converge towards 50%. There's no 'dictating' that it must be 50% with anything less than an infinite number of trials.
Not to split hairs but I thought to reinforce what you already know ;-)
Jan
Yep, I'm sick of the very common audiophile forum posts saying power cord A is night and day better than B, and cord A is $2000. 😱
In reality, it is common that the difference in sound between A and B is below the threshold that ABX test can resolve if it's is placebo or not. By ABX test, we can see if his claim "night and day" is exaggerated or not, but we can't tell if he could really hear the difference between them.
Last edited:
I think we can. When there is a sufficient large number of trials and the outcome is very close to 50% we can say with very high certainty that there is no audible difference.
We should not demand 100% certainty from any test, that will not happen.
Jan
Yes, but no one would participate with the ABX tests with "a sufficient large number of trials" in reality. For audiophile parts test, most of the time, the difference is way too small for the "normal" ABX test, and that is what I'm saying.
Yes, except you may have left out one part: ...for ABX testing that occurs after the test subject has received extensive training and practice. (If the research Jakob2 alluded to for small differences in sound is correct, which I think it might be; don't know enough to say for sure).
Mark consistently presupposes that nulls in listening tests are false-negatives and if we did the test "just so" it'd return a non-null result. I was wondering why we were looking at things so clearly different before I connected those dots.
If not obvious, my training has taught me largely the opposite -- most things are going to turn out to be nulls. Also using older, simplistic, statistical models (student t test), especially on low N trials is a sure fire way to get things wrong. P values are too easy to manipulate.
Last edited:
At the same time, if the effect is so small it requires heroic efforts to "hear", then how much should we really care?
In the mind of this person the $2000 power chord IS sounding better.
But when confronted with solid science that doesn't fit this persons belief system/world view, cognitive dissonance kicks is. This is an extremely powerful psychological mechanism, I suffer from it, most people (if not all, in one way or an other) suffer from it. Resulting in the claim that science must be wrong.
There are 2 steps needed to overcome this cognitive dissonance phenomenon.
- Accept that you as a person have flaw's.
- Realize that changing your belief system/world view doesn't actually change the world.
Yes, I was trying to keep things simple and intuitive. I already know quite a bit more, but thank you anyway for your helpful intentions.
Around here if one says too much it may be inferred that one is trying to show off or thinks others are ignorant. And if one keeps things simple, it may be inferred that one doesn't know much.
Sometimes, no mater what is said, it gets misinterpreted. When that happens instead of asking for clarification people often jump to some conclusion or other.
That's human one aspect of human nature that I wish more people were interested in improving their understanding of.
In that regard, it has been good to hear your descriptions of some of the books you have been reading about neuroscience and cognitive psychology. Debiasing is an important area of research right now. I wish more people took an interest in some of those areas. Certainly, it's being taught more and more in business schools. Eventually, it will probably be added to other curriculums. Be interesting to see if engineering is one of them.
Last edited:
Yep, he spend $2000 for the psychological effect, cable itself is a free giveaway. Maybe...
If only it were that easy. People are very complicated, and cognitive dissonance as a theory doesn't offer much depth of understanding most of the time. More useful conceptual theories have come along since Festinger came up with that idea. Kahneman and Tversky opened up almost a whole new world. Highly recommended to take a look.
I actually want to pull back and be far less un-charitable to Mark. Obviously a null test is not a proof of a negative, but it does tighten the noose around the potential effect size. And I, like the rest of humanity, have a hard time letting go of something I hope is true. I've just had enough experiments beat me down from keeping my hopes up for too long. 😉
Or you can just end up completely skint so you can't afford expensive cable 🙂 . It is nice when you can pop out the other side. Took me nearly 30 years.
The ABX test only has three results: the difference can be reliably heard, the difference cannot be reliably heard, insufficient runs have been done to form a conclusion. You cannot say that some of these three results are meaningless. Please note the word "reliably". Sometimes hearing the difference and sometimes not will emerge in the results.plasnu said:
You use the terms "pass" and "fail", not me. It is certainly possible that someone can reliably hear a difference on one day and not on another day. ABX will show this. ABX simply shows whether, when the test was run, a person can hear a difference. Other people, or other times, may give a different result.
ABX tests how reliably a difference can be heard. If it is an obvious difference then a score near 100% would be obtained.Markw4 said:
Nonsense! Until you have run lots of tests there is a good chance that any particular score is just a random result. The only other explanation is that the person is deliberately trying to undermine the test by choosing the wrong result. So do more tests, or get the person to affirm that he is not cheating. Even after many tests there is still a non-zero chance that the result is unreliable. There are only two choices: imperfect tests, no tests.
It takes a minimum of 30 trials per sample to approximate a normal distribution and to say anything signifant statistically about this test. I didn't see instruction addressing that point in the test. Each person should take the test a minimum of thirt times and then find the sample mean ... unless you want argue how this test follows a normal distribution.
It is an interesting test but no statistically significant information can be taken from it from what I can tell.
You're not wrong at all--between the lack of experimental control and the very large N you'd want for multivariate analysis, it's a complete crapshoot.
It's greatest utility is to each individual who ran the test to get their own impressions about what's important.
- Status
- Not open for further replies.