DAC blind test: NO audible difference whatsoever

Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.
(UPDATES BELOW)


That blind test was long due, it's now done.

Started with a 30$ unit (Fiio) against a 3000$ one (Forssell) and once SPL-matched (massive gain difference), no one could tell the difference in a ABX test.

Then, we switched to a different set-up, using a pair of B&W CM9 speakers and the Forssell against a Eximus DP1 (3500$ or so). Same result: impossible to spot them in a ABX.

We were only 4 participants, but regardless it didn't feel like day & night difference to start with... ''Eyes opened'' we FELT differences, but couldn't prove it in the ABX.

Cables, amplifiers, Lossy v.s. Lossless/HD, EQ'd mid drivers, DAC... Nope. Nothing is passing a ABX blind test.

I'm pretty sure, now, that the human auditory capacities are very, very, overestimated. :(

The good news is: we can probably save a LOT of money.


-------------------------UPDATE 14th Nov 2017------------------------

Here is some pictures, including the whole room while we change set-ups (the 3rd one is on his way...)

An externally hosted image should be here but it was not working when we last tested it.

An externally hosted image should be here but it was not working when we last tested it.

An externally hosted image should be here but it was not working when we last tested it.

An externally hosted image should be here but it was not working when we last tested it.





-------------------UPDATE 16th Nov 2017-------------------------------



System set-ups #1 & 2 were very useful to spot the possible weaknesses of the whole process and equipement. So today was the Day 1 of the system set-up #3, which includes the following changes, based on observations and also comments and thoughts on this very thread:

.: New DIY speakers including Faital Pro 18FH500 and ribbon tweeters RAAL 140-15D.
.: High quality Solen passive crossover components, including dual Film & Foil capacitors for the tweeters.
.: Instead of iTunes and a computer, we used a Astell & Kern AK300 as the source (Toslink direct line out).
.: No switchbox anymore, a manual switch is made with each pair of RCA cables, in a way the participant cannot know.
.: Larger selection of music to choose from.
.: Shorter AND/OR longer music excerpts duration, whichever makes the participant more comfortable.
.: Uncompressed and also some 24/96 music files.
.: Closer distance to the speakers, from a previous 3,12m to 2,25m.
.: Better SPL-matching, using pure tones and double-checked with 2 differents mic.

Basically, to our actual knowledge, everything was carefully made to optimize the chances of a positive identification in that 3rd and final attempt. That test will continue to run for few days, we welcome participants from DIYaudio. Here are the first results:

3rd set-up - DAY 1

First participant was there last sunday (set-ups #1 & 2) and, sighted, was optimistic about his chances. Felt confident that changes made was for the best. FAILED to identify on every music excerpts.

Second participant, non-audiophile, male 34yo, and did not participate previous rounds. FAILED to identify on every music excerpts.

Comments collected afterwards was unanimous: even sighted, the potential audible differences were extremely thin. They both felt they could grasp some hints, but they didn't made it, through the test. They were the most confident with the 24/96 music file Angel of Harlem from The Persuasions and the 16/44 Ungear Moi from CoH, but it didn't change the results whatsoever compared to other music files.

Conclusions of Day 1: From a very promising 3rd set-up, the hopes are now getting extremely low to find a participant who will be able to spot the 19,99$ DAC from the 3000$ one. We already know that 100% of the population is now an impossible target, we do hope that in the next days we'll find new participants that will get better results.


3rd set-up - DAY 2

3rd participant, non-audiophile, male 35yo, and did not participate previous rounds. FAILED to identify on every music excerpts.

4th participant, audiophile background, female 42yo, and did not participate previous rounds. FAILED to identify on every music excerpts.

5th participant, non-audiophile, male 45yo, and did not participate previous rounds. FAILED to identify on every music excerpts.

6th participant, non-audiophile, female 40yo, and did not participate previous rounds. FAILED to identify on every music excerpts.

7th participant, audiophile +DIY background, male 40yo, just passed audiogram (8dBHL, symmetric). FAILED to identify on every music excerpts.
 
Last edited:
The only DAC that I could really dare to say I heard a difference was a (fake?) TDA1543 DAC.
And forget hearing, it was measurable. (HF rolloff)

But after modding it and fixing the design error (*not really but long story) it sounded the same as the not-NOS DACs.
 
Unless my test was flawed somehow, as far as i'm concerned the best solution would be to use the built-in converters on the DSP of your choosing.

From my pov your test isn't so much 'flawed' rather the wrong test. I listen to music for how it makes me feel. So I'll continue with equipment which delivers the emotional goods. I'm happy that you've found a super-cheap solution which meets your needs :)
 
given that modern DACs are now highly evolved with extremely high performance I see no reason why they should sound differently unless implementation issues have not been addressed, or as is common in audio, we simply prefer a little flavour and sometimes an old wine is best

I plan to save a lot of time and money myself :D
 
Or it speaks to how good an ABX test is. BTW the ABX is incapable of statisically deciding that 2 DUT are the same. Your results are only applicable to your text on that day.

dave


ABX testing method is widely used by multi-billions pharmaceutical companies for decades. In fact, it's a sina que non condition for FDA approval.

My methodology for this particular test may be flawed, but i don't think the ABX method is.

Or prove me wrong. ;)
 
From my pov your test isn't so much 'flawed' rather the wrong test. I listen to music for how it makes me feel. So I'll continue with equipment which delivers the emotional goods. I'm happy that you've found a super-cheap solution which meets your needs :)

Based on that comment, you don't understand how an ABX test works.

It's not about ''feeling'' or tastes, it's about being able to identify A from B.

Basically, when done correctly, it's the nuclear option against snake-oil mumbo-jumbo. :cool:
 
frugal-phile™
Joined 2001
Paid Member
ABX testing method is widely used by multi-billions pharmaceutical companies for decades. In fact, it's a sina que non condition for FDA approval.

But it does not rely on human perception, has far larger sample sizes, and they are interested in drugs that make a difference. All things that do no apply to your ABX test.

And are drug trials really ABX tests?

dave
 
But it does not rely on human perception, has far larger sample sizes, and they are interested in drugs that make a difference. All things that do no apply to your ABX test.

And are drug trials really ABX tests?

dave

You need to avoid the ''placebo effect'' in order to prove the efficiency of a drug.

The placebo effect is proven to be effective (same as the hifi boutique salesperson's speech) so the way to deal with that: they split the test in two groups: one that will take the real drug and the other one with some sugar pill that doesn't make any effect (besides placebo).

At the end, most of the time the two groups will have positive effects and the real drug must prove itself more efficient than the sugar pill (placebo) by a certain margin.

I'm not sure about the details, but i think most drugs needs to pass phase I, II and III, which may require few of these tests, in order to be FDA approved.
 
9lzh4p3t0at1.gif



200.gif
 
Status
This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.