Blind Listening Tests & Amplifiers
I'll probably be outcast here, but what are everyone's thoughts about blind listening tests of amplifiers?
I've built several of my own amps (some of my own design and some from others) and spent a lot of time listening to high-end gear (which I used to sell) from the likes of Audio Research, PS-Audio and others. I firmly believed there were significant audible differences between amps as I'd heard them with my own ears.
Then I started to hear about some convincing blind tests and finally conducted my own. I was stunned at the results. I couldn't tell a $300 amp from a $3000 in the store I was working at. Neither could anyone else who worked there. It was a major blow to my audio belief system. I'd always thought Julian Hirsch (main reviewer for Stereo Review) was an idiot for arguing all high quality amps, kept within their power limits, sound the same. It turns out he was right!
Since then, I've read Douglas Self's book on amplifier design where he says the same thing--high quality amps with flat frequency response, low noise and low distortion all sound pretty much the same if kept within their power and current capabilities.
I can certainly hear the difference between a class A single ended amp (which measure very poorly) and a conventional amp in a blind test. I can even tell the digital amps I've been involved with (Tripath Class T and various Class D designs) from a class AB amp in a blind test but they also have some measurable problems. But, between high quality low distortion conventional amps that measure well, I flunk the test and so does everyone else I know.
For a recent non-believer, I used an Onkyo SR500 Dolby Digital receiver--purchased reconditioned for $200 (they're $250 - $300 new) against some well regarded separates. It's rated at 65 watts x 2 stereo per the FTC guidelines into 8 ohms. Distortion is 0.08% from 20-20k from 1 watt to 65 watts into 8 ohms. It has a "direct" bypass feature that supposedly bypasses all the digital/DSP for analog stereo signals.
The Onkyo was put up against the well regarded Bryston 4B 300 wpc power amp and a Bryston 2 channel pre-amp. They were driving a pair of expensive floor standing KEF speakers and the source was a high-end Marantz CD player. The person who owns this system is very proud of it and has spent a lot of time getting what he considers to be the best sound possible.
I had the Bryston owner pick the level he wanted to do the comparison at while listening to his system. I then used pink noise to level match the Onkyo to his system while he was out of the room. The Onkyo was running in its "Stereo Direct" analog mode.
I called him back in to listen, he sat down in the sweet spot and I replayed the same CD track he'd used to pick the levels. He immediately started complaining about how bad the Onkyo sounded. He said it sounded thin, compressed, harsh and a few other things. I smiled and turned the Onkyo off and the music kept playing. He'd said all those negative things about his own Bryston gear!
With him red faced, we proceeded to do at least an hour of listening with me swapping cables, or only pretending to, when he requested a switch. He listened to his favorite audiophile CDs. I did lots of swaps and fake swaps and during each would ask him which he thought he was listening to. In the end, his answers were roughly 50% correct which is the same as if he'd been randomly guessing. He even finally admitted, he couldn't tell which was which and WAS only guessing! I took his place and also couldn't hear any difference between the lowly receiver and his prized Bryston gear.
For the analytical among you, I've done some input/output null difference testing as well. This is where you use a wide bandwidth analog (even passive) circuit to subtract the level matched (nulled) input from the output of an amplifier driving real speakers using real music--no sine waves or resistors here. The resulting difference signal is made up of ANY distortion or deviation produced by the amplifier. Some distortion analyzers work this way.
My null tests have shown that even a modest $250 receiver can manage a -60db or better residual signal driving my relatively difficult speakers to fairly loud volumes. Higher quality amps can easily exceed a -70db null. Those are very low levels of residual distortion. Keep in mind this test reveals ANY kind of distortion--audible or not--including THD, phase distortion, IMD, TIM, slew problems, feedback issues, frequency response deviation, etc.
You could argue that you might be able to hear things that are 60db below the signal, but I'm skeptical. It's easy to hear -60db worth of noise during quiet passages of music, but it's downright tough to hear something that's actually 60db below the signal. For those of you who have a volume control calibrated in db, turn it up to a comfortable listening level and then reduce the gain 60db (if you can without shutting the sound off) and see how much is left. It's hard to hear even in a quiet room!
I'm not trying to start a flame war here, but I am suggesting those of you who think high quality amps have magical qualities like "warm", "airy", "detailed", "liquid", etc. might want to do some blind testing. Most of the differences I thought I heard evaporated once I didn't know which amp I was listening to and the others can be attributed to other factors.
It's been my experience that hardcore audiophiles will dismiss the blind tests as flawed and they refuse to participate in these tests or acknowledge the results. Most of the magazines, of course, don't do them as it would be really bad for ad sales across the board. It's hard for any manufacture to sell a $3000 amplifier when the magazine demonstrates it sounds the same as a $300 one. The same goes for salespeople at high-end dealers. The owner of our store told us to keep our mouths shut as "our test was invalid" when we made our blind discovery.
Don't get me wrong here, I'm not saying a $250 amp is all anyone needs. As you spend more money you tend to get more power, higher current capability, bigger power supplies, better construction, longer life, etc. Many of these are especially useful if you have inefficient and/or difficult to drive speakers and like it loud. But listening at the sorts of levels you'd use for critical comparisons, there's usually not enough of a difference to tell quality amps apart in a proper blind test.
I'm also not saying that some amps don't have "euphoric distortion" that some people like (i.e. single ended amps). Some amps are measurably different (i.e. have a rolled off high-end). These amps are relatively easy to pick out in a blind test. I also know the thrill of building your own amp and the extra enjoyment you get from having built it.
Finally, I've found few people that have actually participated in a proper blind amplifier test. Next time you're evaluating an amp, take a few extra minutes to level match it to a "reference" amplifier and have someone else swap cables in a way that you don't know which you're listening to. If you believe amps have their own "sound", you might be VERY surprised at the results!
Grabbing archived data......
Richard Clark has a challenge that will pay you $10k
if you distinguish between the sonics of amplifiers.
You don't lose money if you lose.
Rules found here + some compiled statements by RC:
Note: The challenge is now open to home audio even though it says
car audio. Any amplifier is game, home audio vs. car audio
amp if you wish, $100k home amplifier vs. Yamaha, etc.
Tube vs. SS is game also..... Anything is game as long as it
meets requirements in the rules section.
Talk to RC here, this is his forum;
Other forums debating this issue here;
Many good points there...
-but I still can differentiate between amps from same manufacturer, a generation between.
"Sound-stage", "body", if it's "forward" or "laidback", are usually easy parameters to recognise and separate them.
Wich likes the neary impossible mix of "forward" and "super-analytical".
I went through this phase myself. Eventually, I found that there were certainly amps I could pick out blind. Investigating further, whenever I could distinguish two amps, there was always something like a source Z interaction with the speaker or the level match had drifted. Once the levels were reset or the supposedly crummier amp eq'd to make it sound like the Golden Amp, I couldn't tell them apart. After a year or so spent devastating my own beliefs, I stopped bothering too much about that stuff.
Funny, every time I hear someone say that the Kichimichi 2000 has a much wider soundstage and can place instruments outside the speaker locations which a cheap Pioneer (or whatever) can't, I think, geez, if you blinded me and had a sax player play from one position on my far left, then have him move 2 feet over and play again, I think I'd be able to hear that.
I'm certainly no golden ear. But I often wonder how many golden ears are actually golden ears.
a case of the blind leading the deaf
"I couldn't tell a $300 amp from a $3000 in the store I was working at. Neither could anyone else who worked there."
I don't know whether to feel more sorry for you or the poor fools that bought equipment from said store. I hope no one lost their livelihood faced with such a moral dilemma. In objective experiments in conventional engineering and science disciplines, null results often cause the experimenters to examine the resolution or methodology of the test. I have been down this path in telecom design and troubleshooting too many times to count. I have worked on measurements on telecommunications systems with people several states away working on the same problem. One often get different results could due to the test setups being different in ways that were not noticed or understood at the time. I spent considerable time flying from Texas to California and back in one day, to resolve this issue.
Some of the worst sounding systems I have ever heard were in High End stores. A couple of the best I ever heard were in High End stores also. The cost of the equipment was not the issue either. Room acoustics, AC power quality, the amount of RFI, the cleanliness of the connector contacts, familiarity with the system, the time of day, the listeners experience with live unamplified music........... I can list dozens more factors based on real and measurable variables and not on pseudo science or belief systems. Careful listening, like the appreciation of art and music, is a learned process. A whole new vocabulary describing what listens for has developed in the last twenty 20 years. I'll bet not everyone is familiar with what to listens for or has the reference of live acoustic music. I know nothing about wine tasting but I don't think those who do are deluded or elitist. Some people cannot hear well and some lack the training or desire to develop that skill. That's fine, but don't attack those of us who do.
When the test gives null results, shouldn't one look at the test more carefully; rather than except results based on someone's predetermined expectations. What about changes to a system, made with the expectations of making it sound better, that make it sound worse to the experimenter. How is that an example of hearing what one wants or expects to hear? Are the belief systems of someone expecting no differences any less suspect than one working to examine and develop the reasons that amplifiers sound different? Stating that all decently designed amps sound the same based matching the levels sounds pretty cynical, arrogant, and intellectually lazy to me. Hell, you can match the levels and they won't even measure the same, given adequate resolution measurements. I believe there probably is still some investigation going on the science of how and what we can hear untainted by audiophile delusions. Unlike many dogmatic engineers, many of us motivated by love of the art of audio design, and desire to get further into the music. I think audio designers might even have a lot more experience in the factors that responsible for these differences than reviewers, academics, and salesmen. I have leaned things in serious audio design that have made me a better telecom engineer. I don't think I have had to discard any engineering skills to design audio.
I really fear you have mistaken double deaf testing for double blind testing.
My main reason for posting this was to see what folks here (who should be more technically adept than your average audio forum reader) think of blind tests (and null difference testing).
My secondary reason was to perhaps save some folks some money. If a person with "good ears" can't tell an amp that's full of 5 cent capacitors from one using exotic parts, it stands to reason they should save their money in their DIY projects unless blind testing proves the more expensive parts sound better (and it rarely does).
A friend went down this path with unexpected results. He was building a high-end power amp (someone else's design) and was searching for the very best parts for it. His purpose for building it was to have something better than his mainstream Adcom power amp which he was convinced was holding his system back.
In his quest for the right components, I suggested he use his Adcom as a known reference and then compare his amp against it, using blind testing, as he made various changes to the design and components. You can guess what happened... Right from the start, he couldn't tell his new amp from the Adcom let alone tell the difference between brands of capacitors.
I note that Larry Archibald does his tastings blind.
NO TWO SOUND THE SAME.
And ditto for the rest of your post Fred.
It reflects my own experiences very well.Not everyone has the capability to discern differences but I usually found a majority to agree with what was heard in the long run.
While I find group listening tests rather useless statistically, I always found them useful to me on a personal level.
I found that I could tell the difference between a really bad amplifier and a really good amplifier quickly; put two good ones in and I probably wouldn't. But I don't care, because the real differences aren't in the blind testing.
What about an extended blind test where the testing period were a few weeks or something? That might be more interesting... if you don't know what amplifier you're living with, and you can get used to it in a system you're used to. Then, maybe these so-called imaginary differences might not be so imaginary?
To Fred Dieckmann: A lot has been written on the methodology of blind testing and why it's valid when it's done properly. Very little objective material has been written to discredit it.
Likewise, a lot has been written about how inaccurate non-blind testing is and the psychology behind why we hear differences that aren't really there. It's not unlike the people in drug studies who have side effects and/or results from placebo drugs. They take a sugar pill and it does all sorts of things to them because they EXPECT IT TO. Does that mean manufactures should sell placebo drugs for huge profits because some people claim they work? Or should we trust science instead?
You can criticize high-end audio dealers if you'd like, but many of these tests (such as the one I did with the Onkyo) have been done in people's own homes with music, systems and surroundings they're intimately familiar with--and they still fail.
Audiophiles typically try to attack the blind testing methodology, but they never seem to come up with solid reasons why it's not valid. After all, it DOES expose differences between many things (like a single ended amp versus a class AB one). In fact, when a switchbox is used, it's an extremely accurate indicator of even small differences in sound quality.
A well documented semi-famous blind test was the Sunshine Audio Challenge. A Yamaha integrated amp was used to replace a pair of expensive monoblocks in the owner of Sunshine Audio's home in Florida. He had an extremely high-end system (worth six figures) and regarded himself as very well skilled in hearing all the little nuances that the high-end press loves to write about.
With and without a switchbox (doing cable swapping instead) he couldn't tell the Yamaha from his amps. Nor could his wife. Nor could anyone else present. The methodology in that test was solid and more elaborate than what I've done (at least part of it was double-blind as I recall). The best excuse he could come up with was that he had a couple glasses of wine the night before.
The editors of Stereophile lost a blind challenge to Bob Carver. They couldn't tell a pair of $15,000 monoblock tube amps from a $699 Carver model that Bob lightly tweaked to duplicate the frequency response of the tube amps. They helped define the terms of the test. They were VERY confident they would win. They lost.
If Stereophile could find fault in such a test to save face, don't you think they would have? Instead, they explained it (more or less) by saying the modified Carver amp was "ringer" and way beyond anything else in its class (which of course delighted Bob). The Carver amps have rail-switching class G power supplies which are generally regarded as inferior by audiophiles.
There are several other blind tests documented with similar results. I ask you, if in comparing amplifiers back-to-back in your own system, you can't hear the difference between them when you don't know which you're listening to, how great can any differences be?
|All times are GMT. The time now is 06:07 AM.|
vBulletin Optimisation provided by vB Optimise (Pro) - vBulletin Mods & Addons Copyright © 2015 DragonByte Technologies Ltd.
Copyright ©1999-2015 diyAudio