DAC blind test: NO audible difference whatsoever

Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.
frugal-phile™
Joined 2001
Paid Member
Does Mac Mini work like Windows, performing any SRC for a shared sound device?

Not in general, but sometimes. 3rd party add-ons to iTunes like PureMusic & Ammara, do things like take over complete control of the CORE sound routines, load tune off disk into RAM & other tricks that do produce better DDR. I could see subtle differences in DACs being buried by ITunes used native. Optical on the macMii (all Macs) is not 1st rate and needs serious work to be cleaned up.

I use FireWire and was very surprised that a difference similar to adding PureMusic to iTunes, but of lessor magnitude, could be had by replacing the generic $5 Firewire cable with one optimized for audio quality.

dave
 
Results cannot be used for a general statement.

I beg to differ.

Only 4 participants, no. But i conducted a blind test with 100+ participants and the trend was clear before reaching 10... No surprise ''golden ears'' of any kind or a sudden change of trend in the middle.

A hard-to-spot difference is pretty much universal, i would dare to say. Unless you're very close to a threshold, which i explained earlier.

So, yes, given more participants and more rounds, i would use the results to make a general statement. I would go as far as being loud about it, even.. :)

But just not now. Not with just 4 participants and not before giving a shot with the ribbon tweeter + nearfield set-up.
 
Not in general, but sometimes. 3rd party add-ons to iTunes like PureMusic & Ammara, do things like take over complete control of the CORE sound routines, load tune off disk into RAM & other tricks that do produce better DDR. I could see subtle differences in DACs being buried by ITunes used native. Optical on the macMii (all Macs) is not 1st rate and needs serious work to be cleaned up.

I use FireWire and was very surprised that a difference similar to adding PureMusic to iTunes, but of lessor magnitude, could be had by replacing the generic $5 Firewire cable with one optimized for audio quality.

dave

Oh boy!

The cat is out of the bag, Dave... We're really talking digital cables mumbo-jumbo now? :confused:

I mean... Really?
 
None of that matters. ABX is statistically unable to determine whether 2 devices are the same, only if they ar edifferent.

dave

Dave: If it pleases you, maybe a different and more productive approach would be to suggest how he might design and conduct additional tests that would try to prove the hypothesis that the DUTs sound the same?

Everyone else: In this thread Dave is like a fly buzzing around in the room and saying: "The conclusion is invalid! The conclusion is invalid! The conclusion is invalid!". Just ignore him.
 
Last edited:
It is true that an ABX test can only prove that two DUTs are different, and it cannot prove that they are the same. The reason has to do with the math of statistics. Statistical tests can either "reject" a null hypothesis or "fail to reject" a null hypothesis. They cannot prove a null hypothesis. In this case, the null hypothesis would be that the DUTs are the same. So the ABX test can show that they are not the same, or fail to show that they are not the same. This second conclusion is not the same as showing they are the same. Sounds like double-talk, but it's not. A common explanation used is this: a jury either finds a defendant Guilty or Not Guilty. Not Guilty is different from Innocent. Not Guilty means you couldn't find enough evidence to support Guilty. But it doesn't conclude innocence. Really it leaves the question sort of undetermined.


So how do you apply this to ABX testing of DACs. The OP's ABX test did not / cannot show that the DUTs are the same. But it did fail to show they are different. I guess that might mean for example that there could be a difference, and if you improved the test somehow (different tracks, different lengths, who knows), then you still might have a chance of finding a difference. But even as the test stands, failing to show they are different, in a scenario where we were expecting to easily find a difference, is still quite a valuable outcome.


Thank you dc3 for the explanation, it's appreciated. I see the point now.

I think the whole objective of the test is missed here, though.
The ABX tests, at least the way i'm conducted it, is not really about the components tested per se. No, it's more about the participants sensorial capacities. Components being proven identical or not is pretty much irrelevant here. What really matters is the participant's ability to spot a potential difference between A and B. That's it. Nothing more.

The spotlight is on the human capacities here, not the components.

In the absolute, YES, i agree, there is no possible proof, with an ABX, that 2 tested components are the same. But that's irrelevant. To say that bluntly: we just don't give a damn about that.
What we seek, as audiophiles, as DIYers, as curious people... is answers. Answers that can lead to decisions, which means purchases. And time invested in our hobby. Answers that are rare in this very subjective yet very technical/complicated domain.

Bottomline? The results have statistical value.
No doubt about it.

I'd say that is very valuable information: ''4 guys did a blind test and were not able to spot a low-cost DAC from a very expensive one''. Unless the test was obviously flawed, biased or downright dishonest, i would consider it as a valuable piece of information. The absolute truth? Of course not. Nothing is, anyway.

It shows that a certain number of individuals failed to identify A from B, within a specific context. Nothing more, nothing less. It gives perspective on a potential differential between A, B and the human capacities (on a certain limited sample).
If something changes suddenly with more participants -a bigger sample- i'll be sure to let you know. But for now it doesn't look that way. At all.

On a scientific point of view, and that's very important to understand, what becomes interesting is the repeatability. More participants, different participants with different background, age, sex, etc... Also different tunes, different lenghts, different speakers, etc... Even the whole test duplicated by other people, anywhere in the world, with different of everything mentioned above...

That's only the start of it, but i sure won't spend 10 years on the topic trying to get 50,000 participants just to make the proof irrefutable, nothing will anyway... Especially for people who thinks ABX is a waste of time (mostly because against their idea of their truth).

4 individuals (seasoned audiophiles/DIYers, i must point out...) and, so far, it's ''bad news''... Bad news in the way that hopes are LOW to find any noticeable differences in a ''normal'' set-up with active listeners. Hopes are next to NON-EXISTENT in passive unfocused listening mode, with poor equipement, in a noisy environment.

So basically, here and now, i can make that statement:

We are not talking about day & night kind of audible difference. If any.
 
Last edited:
Everyone else: In this thread Dave is like a fly buzzing around in the room and saying: "The conclusion is invalid! The conclusion is invalid! The conclusion is invalid!". Just ignore him.


I like Dave's input.

As my signature points out, i strongly believe in contrarian point of views. That's the very basis of questionning the unquestionned/unquestionnable.

:)
 
When forced to make a choice… that is a key to the strengths/weaknesses of an ABX test. The forced choice almost guarantees that small differences will be missed (ref the Wikipedia aricle you linked too).

dave


Dave, the ''Forced choice'' idea, while sounding dramatic to you, doesn't hold water, as explained in previous pages.

What you guys see here is the blunt ''NO'' outcome of this test, but i noticed the process of it as well. Regardless of the ''no, nobody can notice any difference'' answer, there was also the level of confidence in each answers, from each participants. Obviously, we were not close to a threshold. On the contrary, we were struggling to find any hint that could lead to a positive identification.


At first, the brains tells you ''Yes, you can! Concentrate, concentrate more!'' and after few rounds of unsatisfying results, your brains tells you '' Damn, i'm not sure anymore.. Concentrate on what? '' and, finally, you're losing hope. Even worst: the brains sensations you FELT before, kind of disappear... Pretty much like if previous perceptions were altered. Very difficult to explain, really, but this is how it felt for me and that's my observations.
 
Yeah yeah yeah whatever. Every single audio component has an ABX mob that says 'there is no difference'. Speakers, DACs, Amps, Cables, on and on. Hell even turntables (TechnicsTechnicTechnics blah blah blah).

And yet none of these people seem even remotely aware of each other, and none of them ever advocate just going online and buying a cheap receiver and speaker box set. Because based on their criteria, that's exactly all you need and nothing more.

In my ten years of this hobby I've never seen it suggested, despite it being the most logical (by their criteria) outcome of their assessment. Instead we somehow find ourselves being recommended to some product that surprise surprise isn't very cheap and whose only talking point is that the company surprise surprise expresses a viewpoint aligned with the skeptic.

So getting dragged into some myopic spat based around one component is a red herring, a canard, a MacGuffin. If you can't also address this bizarre consistency of skepticism across ALL components, you're just wasting people's time with dogwhistles.
 
Last edited:
Yeah yeah yeah whatever. Every single audio component has an ABX mob that says 'there is no difference'. Speakers, DACs, Amps, Cables, on and on.

And yet none of these people seem even remotely aware of each other, and none of them ever advocate just going online and buying a cheap receiver and speaker box set. Because based on their criteria, that's exactly all you need and nothing more.

In my ten years of this hobby I've never seen it suggested, despite it being the most logical (by their criteria) outcome of their assessment. Instead we somehow find ourselves being recommended to some product that surprise surprise isn't very cheap and whose only talking point is that the company surprise surprise expresses a viewpoint aligned with the skeptic.

That's completely untrue in my case.

I had only lossless/uncompressed files and the day after my MP3/AAC/CD/HD2496 blind test i started to purchase iTunes files and stopped ripping CDs, while completely ignoring high def craze...

The day after i failed to spot an EQd midrange within 400-7.2khz, i installed a pair of 30 bucks drivers instead of the expensive ones i had.

Today, i'm listening to a cheap DAC without any urge to get anything ''better''.

In my case: instant-changes. My ''audiophile life'' is changed permanently, unless something else changes my mind!
 
wushuliu, i get your point but you must also consider the human/emotion/ego factors.

Yesterday, i felt DUMB.

Talking about DACs all those years? How better that one was over that one? :(

It's difficult to swallow, really. Difficult to accept, even. I sure can understand people who would mindblock that kind of information. It's very human.
 
wushuliu, i get your point but you must also consider the human/emotion/ego factors.

Yesterday, i felt DUMB.

Talking about DACs all those years? How better that one was over that one? :(

It's difficult to swallow, really. Difficult to accept, even. I sure can understand people who would mindblock that kind of information. It's very human.

Or you can take into account the probability, yes the PROBABILITY, that some of us can discern the differences.

The DAC argument gets even dicier for you because you have the engineers, yes THE ENGINEERS from different companies like ESS and AKM who have spoken about how they specifically tailor their designs. So apparently you have seen through their veils as well. Feel free to watch the ESS lecture video on youtube or google the AKM interview.

Of course maybe they were forced at gunpoint by marketing. Who knows.
 
In another thread, someone said Macs do auto-SRC just like Window does unless care is taken to set system sample rate to source sample rate. Real-time SRC usually has quality limitations. In the testing done for this thread, IIRC, both 16/44 and 16/48 source material was used. Was the OS reconfigured as necessary to match source sample rate without needing to use any SRC?


DF96: I have reservations about ABX as we have it now for detecting very small differences. In particular, I don't think Foobar ABX is ideal for that purpose as currently implemented. If modest improvements were made to Foobar ABX to make it less distracting to use, I think we might see some improvement in scores for detecting low level distortion. In addition, I have no problem with blind or double-blind testing, in fact I think we need to have it. It is very clear that humans can be easily convinced they hear things that are not real, but rather an illusion probably mostly arising from the mis-focusing of attention in conscious awareness. That being said, I'm not convinced ABX as commonly implemented is probably the most sensitive blind test possible. It is a test we have now, much like we had THD for distortion evaluation many years ago. We now know that not all THD is perceptually the same. We don't yet have research to move forward with more sensitive blind testing protocols, or to make a reasonably convincing case that ABX as we have it now is probably as sensitive as we can practically achieve. For detecting differences requiring less than a very high level of concentration ABX seems to work pretty well. Based on what I just said, am I an "ABX objector" in your estimation?
 
Or you can take into account the probability, yes the PROBABILITY, that some of us can discern the differences.

The DAC argument gets even dicier for you because you have the engineers, yes THE ENGINEERS from different companies like ESS and AKM who have spoken about how they specifically tailor their designs. So apparently you have seen through their veils as well. Feel free to watch the ESS lecture video on youtube or google the AKM interview.

Of course maybe they were forced at gunpoint by marketing. Who knows.


I don't think audiophiles, engineers or even companies are dishonest.

I think they really believe what they believe. In good faith. Maybe a small percentage don't believe what they say and keeps saying it anyway, but i'm not really worried about them...

What worries me, though, is the lack of questionning the basic elements. Especially among engineers and companies. Of course, they're human. They can make mistakes. They can be misleaded by anything, like we are. But, still... Not much was made in regards of what humans can really hear and feel, and that's not good in a hobby where the very idea is to please the ears/brains.

I'm a wine aficionado as well and blind testing is a widely accepted method of tasting and comparison. I'm doing wine blind tests on a very regular basis and i can assure you that a 5$ bottle couldn't hide from a 500$ wine, from 4 seasoned tasters... I have seen very often overpriced wines being humiliated by cheaper wines, but, overall, the market is pretty much balanced, with the usual diminishing returns law that kicks in along the luxury way...

The same applies regarding sport cars. There is no way an entry-level Toyota could be a match with a Lamborghini.

No, the audiophile market is pretty much unique: A very unbalanced market filled with unsatisfactory and unproven (misleading) solutions.
 
The only audio-related ABX, etc. research I find credible is the work done by Sean Olive/Toole. That's because he has a multi-million dollar facility to conduct listening tests, and even more importantly, get this, he also dedicates resources to training people how to LISTEN.

So if he says there's no difference blah blah on something I take pause and consider. If the usual suspects on the internet, do, I'd like them to show us the intensive research they've done. I say that in reference to those who make their arguments as facts, which they often do. Everyone of course is entitled to their own opinion.
 
Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.