Blind DAC Public Listening Test Results

[RocketScientist]

I conducted a public blind listening test by making high quality recordings of the outputs of 4 different USB DACs using their line outputs and also the headphone outputs loaded with real headphones. The music excerpts were available for download in FLAC format and 20 people voted on at least some of the trials. The DACs are:

• Behringer UCA202 ($29)
• Modified UCA202 (above + $5 worth of parts)
• NuForce uDAC-2 ($130)
• Benchmark DAC1 Pre ($1600)

This was only an informal listening test, so don't expect rigorous methods or all the results to be statistically valid, but there were some interesting clear preferences. The link below has the full results:

NwAvGuy: DAC Listening Challenge Results

[Colin]

Nice work. Interesting blog and shame you got booted off another forum because one advertiser couldn't address the technical limitations of their product.

[RocketScientist]

Thanks. I haven't quite been fully booted (yet at least) off Head-Fi but they've succeeded in making it much more difficult to contribute there. It's especially a shame because the majority of participants in the listening tests came from Head-Fi and I, and others, have been unable to even indicate in those Head-Fi threads the results have been published. Any such indications are immediately deleted. One really has to wonder where Head-Fi's priorities are?

Fortunately there are plenty of more rational forums--such as this one! And I'm more after quality than quantity anyway.

[SY]

There's things to agree with and disagree with, there's things that could be done differently, but... you recognized what you did for what it is, there are some clever things said, you made some very trenchant observations, you qualified your comments, and I'm delighted you posted that link here.

[SoNic_real_one]

That concept of testing is based on:

It's somewhat like looking at 3 different shirts at an online retailer. You can tell easily how they're different from each other by looking at the pictures of each shirt using your PC. You don't need a perfect computer display to tell that one might be a slightly darker shade of blue than another one. Most any computer display is good enough to show the differences. But you can't be sure of the exact shade of blue until you have the real shirt in front of you.

See, that assumption is false... Because many people cannot discern between subtle shades of blue. So, even if different, their eyes will still see them identical because of their poor vision (or poor PC monitor). And they would guess.

Your tests, run on a good equipment will reveal the differences. Run on a system that outputs tones of c**p, that c**p will MASK the real differences. A PC speaker system that has a bandwidth of 60-15kHz and a SNR/Dynamic of 60dB will NOT reveal nothing that happens out of those limits.
Maybe you should ask the testers to reveal their listening equipment and age.

[RocketScientist]

Thanks SY. The "trenchant" remarks are mostly related to the outright attacks NuForce has tried to make on my review and even me personally. I've tried to remain as objective as possible regarding the measurements, results, etc. But I'm sure some of my commentary is tainted by my negative experiences with NuForce. Still, I've tried to compliment their product where compliments are due. It's not all bad.

And if I do this again, I'd very much appreciate input on ways to do a better job.

[RocketScientist]

Originally Posted by SoNic_real_one

That concept of testing is based on:
See, that assumption is false... Because many people cannot discern between subtle shades of blue. So, even if different, their eyes will still see them identical because of their poor vision (or poor PC monitor). And they would guess.

Your tests, run on a good equipment will reveal the differences. Run on a system that outputs tones of c**p, that c**p will MASK the real differences. A PC speaker system that has a bandwidth of 60-15kHz and a SNR/Dynamic of 60dB will NOT reveal nothing that happens out of those limits.

I should probably be more clear about a "reasonably decent" computer display or listening gear. Obviously a set of $20 PC speakers isn't going to cut it. In reality, most of the participants used headphones in the $100 -$400 range and many were driven from relatively expensive dedicated headphone sources. A few used the headphone jack on a MacBook Pro, etc. but they still used fairly well respected cans.

I agree the lower you go fidelity wise, the more there's a potential for masking. But it's interesting the subtle advantages of the high-end Benchmark stood out even when listening to the files on much less expensive hardware with far inferior specs including the SNR, etc.

And short of having the listeners go through an interview/selection process, I can't do anything about their "poor vision" or "poor hearing". I think it's fair to just include those interested in headphone listening as they're the typical customer of the products being compared.

[SY]

Originally Posted by RocketScientist

And if I do this again, I'd very much appreciate input on ways to do a better job.

I'd be delighted to help.

[azneinstein]

Rocket, I thought your blind test was great and it shows what audio is about. Clearly, the assumption is as price increases- we get better fidelity, so we expect which one should be better until/or we hit personal preferences or "incorrect" pairing of sounds. I'm thinking of doing a similiar test of straight 1-10 scores on 10 or so 8" fullrangers soon with RS drivers, hopefully the Burros, random ebay's to the Visatons B200 and it'll be all by ear and friends which I expect, similiar results where the Visatons being the only audiophile driver to top and have everything else follow.

But I definitely think DACs are one of the harder equipment to judge which is why people expect the need for all the components to be audiophile. I tried an A/B/C a while back with an oldschool monster branded dac, audio authority, and another friends dac where we could barely pick out differences but in blind tests and different songs, we probably couldn't pick out accurately which dac was playing what, just which one sounds better.

[RocketScientist]

@azneinstein I agree DAC differences are generally "non obvious". When I first conceived the test, my greatest concern was there wouldn't be any consistent outcome. But I did careful ABX trials on my own (using Foobar) of the first few recordings and was somewhat surprised I scored very well on some of them. But that only identified there was a difference. It was harder to pick a favorite when I didn't know which file was which. So I really didn't know what to expect from the "public".

I've seen enough solid evidence to be convinced that some people are much better "listeners" than others including in blind tests. I think it's partly just human genetics--we all have things we're predisposed to be better and worse at--and partly a matter of training one's ears and brain to hear small differences. So just because I can or can't hear something, I don't assume the same will be true for everyone.

HydrogenAudio has done some interesting listening tests using Foobar ABX of things like different re-sampling algorithms. And, with some, many couldn't hear a reliable difference while a select few could. There was even a blind cable test where among a panel of many listeners, only Michael Fremer from Stereophile (if I remember correctly) could reliably identify one of the cables.

So that's partly why I did this test. I thought it would be fun to see how my listening abilities stacked up to whoever else participated. Electronics lend themselves well to this sort of test where anyone can use Foobar ABX to compare the test files. Speakers and headphones, however, are very different.

I know Harman International has a really expensive and advanced listening room with a theatrical-like "stage" that can automatically at the push-of-a-button reposition speakers in the same spot for blind testing behind an acoustically transparent but visually opaque "screen". Even something as minor as trying to compare two speakers side-by-side or stacked on top of each other can mess up the comparison. Sean Olive has written about all that on his blog. He's perhaps most famous for his excellent article about all the problems with "sighted" (versus blind) listening:

Dishonesty of Sighted Listening

The "sighted" bias came out in my listening tests in that one person flagged early on one of the reference tracks (that hadn't been through any of the DAC's) as being "clearly the worst" and 5 more people promptly "heard" the same thing. Some of those who used blind ABX, however, picked the same track as their favorite. So it's easy to see how even a single positive or negative subjective comment on any product in the forums can severely bias many others who evaluate the same thing.

And I also think it's significant that a lot of manufactures "design by ear" but they use sighted listening. Given all the obvious and well proven bias with sighted listening, is it really valid to use it as the primary criteria for product design? Sean Olive does a better job than most of trying to grapple with that issue.

And as for price, that was partly why I chose the lowly $29 Behringer to include for comparison. It's like including a $3 wine in an expensive wine tasting. I love it when the low priced entry turns out to be a well liked bargain in a blind test! I this case it didn't, but some still liked it.