A few remarks wrt to controlled tests.
1.) Planet10 pov is correct when poiting out that a negative test result does not establish that no difference between the DUTs exists. It isn´t a flaw of the ABX test it is just something common for the usual NHST (i.e. N ull H ypothesis S ignificance T est) .
The null hypothesis states (in our usual listening tests) that the results could have occurred due to random guesses and we analyze the results under the premise that the null hypothesis is true, which means we calculate the probability that the given results could occur due to chance.
If the calculated probability is lower than a predefined decision criterion we conclude that the null hypothesis can be rejected.
If the calculated probability is higher than the predefined decision criterion we can´t reject the null hypothesis, but we also can´t conclude that the null hypothesis is true, because we haven´t really examined the reason for the specific test results.
P(D l Ho) isn´t the same as P(Ho l D) which means probability of the data (if Ho is true) isn´t the same as the probability of Ho given the data.
Ho = null hypothesis
D = data
2.) It is known since the ~1960s that the mental processes are more involving when doing ABX tests in comparison to A/B tests. It doesn´t mean that ABX tests are invalid but it demands that an experimenter take it into consideration
3.) Although an preference test is not a test for difference it nevertheless can help as an established preference means that a difference exists.
It is sometimes difficult because not every kind of difference causes a preference (similar degree of dislike is possible) and if testing with a group the preference might be segmented even equally.
4.) Training and accomodation are very important. As abraxalito already pointed out, listening under test conditions is different from "normal/casual" listening.
And as JonBocani said any such test is as much a test of the participant as it is a test of the actuall DUT, so it is important to check the listeners sensitivity via proper positive controls (and to check the environment/setup by using a negative control)
5.) It is important to remember that any such test at first only delivers the result "null hypothesis can be rejected" or "null hypothesis can not be rejected". Further conclusions are only allowed if it can be showed that the test was objective, valid and reliable.
1.) Planet10 pov is correct when poiting out that a negative test result does not establish that no difference between the DUTs exists. It isn´t a flaw of the ABX test it is just something common for the usual NHST (i.e. N ull H ypothesis S ignificance T est) .
The null hypothesis states (in our usual listening tests) that the results could have occurred due to random guesses and we analyze the results under the premise that the null hypothesis is true, which means we calculate the probability that the given results could occur due to chance.
If the calculated probability is lower than a predefined decision criterion we conclude that the null hypothesis can be rejected.
If the calculated probability is higher than the predefined decision criterion we can´t reject the null hypothesis, but we also can´t conclude that the null hypothesis is true, because we haven´t really examined the reason for the specific test results.
P(D l Ho) isn´t the same as P(Ho l D) which means probability of the data (if Ho is true) isn´t the same as the probability of Ho given the data.
Ho = null hypothesis
D = data
2.) It is known since the ~1960s that the mental processes are more involving when doing ABX tests in comparison to A/B tests. It doesn´t mean that ABX tests are invalid but it demands that an experimenter take it into consideration
3.) Although an preference test is not a test for difference it nevertheless can help as an established preference means that a difference exists.
It is sometimes difficult because not every kind of difference causes a preference (similar degree of dislike is possible) and if testing with a group the preference might be segmented even equally.
4.) Training and accomodation are very important. As abraxalito already pointed out, listening under test conditions is different from "normal/casual" listening.
And as JonBocani said any such test is as much a test of the participant as it is a test of the actuall DUT, so it is important to check the listeners sensitivity via proper positive controls (and to check the environment/setup by using a negative control)
5.) It is important to remember that any such test at first only delivers the result "null hypothesis can be rejected" or "null hypothesis can not be rejected". Further conclusions are only allowed if it can be showed that the test was objective, valid and reliable.
Don;t know what you're talking about? How can blind testing have "a risk of preference affecting the result" - this is really is nonsensical
Round & round it goes - where it stops nobody knows. Yes, folks, we seem to be now at the circus.
seems you never mentioned what your amp is
i think its extremely likely that your amp is the bottleneck, which makes your test system flawed
i think its extremely likely that your amp is the bottleneck, which makes your test system flawed
the OP gave fairly detailed info on speakers so its assumed he is well aware of speaker performance
I'm thinking you are wondering if distortions would be masking any perceived difference? The speakers, room and position would have more than any half decent amp
the room look quite nice in the picture actually
in simple language the test setup is not capable of revealing the diffs
in simple language the test setup is not capable of revealing the diffs
Amps can make a lot of difference too, since they produce higher order distortion than speakers usually do. A lot of the differences between DACs also involves higher order distortion which could be masked by amplifier distortion.
Having had the experience of comparing various amps, they can make a lot of difference or not depending on how good/bad they are and the level of distortion that is trying to be detected.
Having had the experience of comparing various amps, they can make a lot of difference or not depending on how good/bad they are and the level of distortion that is trying to be detected.
You are probably right, I think he should seriously consider using headphones and a good headphone amp
especially for TUBE guys, i think they are the main group that always propose all dacs sound the same
DAC shoot out
There are many many people whose sole education in audio comes direct from the pages of Stereo-pile and other fantasy based magazines who believe the resolution of human hearing is better than any modern instrument can measure and any engineer can ever comprehend. They wax poetic using tens of undefinable terms "slam" and "pace" and the like. To them the scientific method of the ABX is always invalid due to some super natural uncontrollable flaw in any setup you can imagine or ever build. Since these folks are a member of a religion, where any excuse they dream up is valid, they will present no scientifically valid evidence for their opinion and will accept none from you. Save your breath.
Magicians and audiophile sales people fool customers every day into thinking they just witnessed the impossible. There are thresholds for human perception, different for sure from person to person. So it seems obvious that once that threshold is exceeded no human will be able to tell the difference.
I am headed out to an informal DAC shoot out later today. We are going to do a blind comparison of two modern multi-thousand dollar DACs with a really simple DIY 8 bit dac we built using a $4 micro processor and a few resistors and caps. It has some dither so it measures about like a 12 bit unit. I hope it loses, but who knows. At the age of 55, I doubt I can hear the difference between a true 12 bit DAC with proper filtering and a perfect dac.
It is going to be a great afternoon.
There are many many people whose sole education in audio comes direct from the pages of Stereo-pile and other fantasy based magazines who believe the resolution of human hearing is better than any modern instrument can measure and any engineer can ever comprehend. They wax poetic using tens of undefinable terms "slam" and "pace" and the like. To them the scientific method of the ABX is always invalid due to some super natural uncontrollable flaw in any setup you can imagine or ever build. Since these folks are a member of a religion, where any excuse they dream up is valid, they will present no scientifically valid evidence for their opinion and will accept none from you. Save your breath.
Magicians and audiophile sales people fool customers every day into thinking they just witnessed the impossible. There are thresholds for human perception, different for sure from person to person. So it seems obvious that once that threshold is exceeded no human will be able to tell the difference.
I am headed out to an informal DAC shoot out later today. We are going to do a blind comparison of two modern multi-thousand dollar DACs with a really simple DIY 8 bit dac we built using a $4 micro processor and a few resistors and caps. It has some dither so it measures about like a 12 bit unit. I hope it loses, but who knows. At the age of 55, I doubt I can hear the difference between a true 12 bit DAC with proper filtering and a perfect dac.
It is going to be a great afternoon.
People often don't understand what are the strengths & weaknesses of our perceptions & often make mistaken comparisons.
Yes, our auditory perception has been refined to do a highly specialised task - make sense of the auditory world using the large amount of pressure waves impinging the eardrum & yet usually not enough information in these pressure waves to definitively determine all the auditory objects in the soundfield. It is a wonderfully complex task that it achieves in real-time, moment to moment. Our interaction with the world doesn't rely on the ability to differentiate fine audio differences but it does rely on quick determination about sounds & the objects from which they have arisen & the many other aspects of that sound which are important. it's based on a best guess & usually fairly accurate when it can use other information, like vision, knowledge of the behaviour of sound from particular objects, in particular environments, etc
Subtle differences are pretty meaningless in that scenario, by & large.
A jetfighter is far more powerful than a humming bird & yet it can't hover over a flower & sip nectar.
Yes, our auditory perception has been refined to do a highly specialised task - make sense of the auditory world using the large amount of pressure waves impinging the eardrum & yet usually not enough information in these pressure waves to definitively determine all the auditory objects in the soundfield. It is a wonderfully complex task that it achieves in real-time, moment to moment. Our interaction with the world doesn't rely on the ability to differentiate fine audio differences but it does rely on quick determination about sounds & the objects from which they have arisen & the many other aspects of that sound which are important. it's based on a best guess & usually fairly accurate when it can use other information, like vision, knowledge of the behaviour of sound from particular objects, in particular environments, etc
Subtle differences are pretty meaningless in that scenario, by & large.
A jetfighter is far more powerful than a humming bird & yet it can't hover over a flower & sip nectar.
Last edited:
A man and his friends do some searching one day and conclude that they cannot see any fairies at the bottom of his garden. Others seem unsurprised at this, as they do not expect to see fairies at the bottom of any garden.
However, the Fairy Appreciation Society is alarmed and comes up with reasons why no fairies were seen:
1. the man and his friends were not trained so didn't know what they were looking for
2. they were wearing the wrong kind of glasses
3. fairies are quite small so you have to have good eyesight and look in the right place
4. it is just possible that this particular garden has no fairies, but this result should not be generalised
5. the search may need to be repeated on another day, just in case the fairies were away for a while
6. if you want to find fairies don't use amateurs to look for them
7. you can only see fairies if you believe in them
8. looking for fairies is not sufficient, you have to listen for them too
Just a silly story, quite irrelevant to anything else.
However, the Fairy Appreciation Society is alarmed and comes up with reasons why no fairies were seen:
1. the man and his friends were not trained so didn't know what they were looking for
2. they were wearing the wrong kind of glasses
3. fairies are quite small so you have to have good eyesight and look in the right place
4. it is just possible that this particular garden has no fairies, but this result should not be generalised
5. the search may need to be repeated on another day, just in case the fairies were away for a while
6. if you want to find fairies don't use amateurs to look for them
7. you can only see fairies if you believe in them
8. looking for fairies is not sufficient, you have to listen for them too
Just a silly story, quite irrelevant to anything else.
Do you believe that an OTL tube amp and a cheap Pioneer receiver sound the same?
What is the basis for your binary condition of “good enough” or “not good enough”?
How do you define or specify “good enough”?
My offence is that we live, increasingly, in a world where narrative fallacies have overtaken evidence and logic in shaping peoples’ views.
For example, instead of trying to understand the critiques of the ABX approach on their merits (or lack of merits), based on evidence and logic, you have imagined that I am part of a snake oil conspiracy, thus explaining why I am offended by someone like the OP exposing the “truth”. You have substituted your imagined story for the objective reality.
I believe that there are questionable claims at the fringes. To be much wider, however, would require collusion and coordination among an increasingly larger group of manufacturers, suppliers, and sellers, as well as ever-larger groups of buyers who are easily bamboozled.
I would like to hear your evidence for and explanation of how this can be the case.
Were evidence and logic ever in the lead, or do you live in a different world?
It seems to me, that any difference that can only be heard/shown with that level of equipment and training, will in any case be completely irrelevant in any practical scenario...
Doppler,
Let me give you the stock answer to your first question which is already on this thread:
- all competently design amplifiers, operating within their linear area, etc, will sound the same
Now where this answer becomes the Salem witch hunt is when you ask the follow up question - how do you know an amp is meeting the criteria stated - answer: because it will sound the same in a blind test.
In other words the 'test' defines the criteria - it's no longer a test about differences, it has now become the definition of 'competence'.
Logic plays no part in this
Let me give you the stock answer to your first question which is already on this thread:
- all competently design amplifiers, operating within their linear area, etc, will sound the same
Now where this answer becomes the Salem witch hunt is when you ask the follow up question - how do you know an amp is meeting the criteria stated - answer: because it will sound the same in a blind test.
In other words the 'test' defines the criteria - it's no longer a test about differences, it has now become the definition of 'competence'.
Logic plays no part in this
Last edited: