Blind Listening Tests & Amplifiers

Status
Not open for further replies.
I'll probably be outcast here, but what are everyone's thoughts about blind listening tests of amplifiers?

I've built several of my own amps (some of my own design and some from others) and spent a lot of time listening to high-end gear (which I used to sell) from the likes of Audio Research, PS-Audio and others. I firmly believed there were significant audible differences between amps as I'd heard them with my own ears.

Then I started to hear about some convincing blind tests and finally conducted my own. I was stunned at the results. I couldn't tell a $300 amp from a $3000 in the store I was working at. Neither could anyone else who worked there. It was a major blow to my audio belief system. I'd always thought Julian Hirsch (main reviewer for Stereo Review) was an idiot for arguing all high quality amps, kept within their power limits, sound the same. It turns out he was right!

Since then, I've read Douglas Self's book on amplifier design where he says the same thing--high quality amps with flat frequency response, low noise and low distortion all sound pretty much the same if kept within their power and current capabilities.

I can certainly hear the difference between a class A single ended amp (which measure very poorly) and a conventional amp in a blind test. I can even tell the digital amps I've been involved with (Tripath Class T and various Class D designs) from a class AB amp in a blind test but they also have some measurable problems. But, between high quality low distortion conventional amps that measure well, I flunk the test and so does everyone else I know.

For a recent non-believer, I used an Onkyo SR500 Dolby Digital receiver--purchased reconditioned for $200 (they're $250 - $300 new) against some well regarded separates. It's rated at 65 watts x 2 stereo per the FTC guidelines into 8 ohms. Distortion is 0.08% from 20-20k from 1 watt to 65 watts into 8 ohms. It has a "direct" bypass feature that supposedly bypasses all the digital/DSP for analog stereo signals.

The Onkyo was put up against the well regarded Bryston 4B 300 wpc power amp and a Bryston 2 channel pre-amp. They were driving a pair of expensive floor standing KEF speakers and the source was a high-end Marantz CD player. The person who owns this system is very proud of it and has spent a lot of time getting what he considers to be the best sound possible.

I had the Bryston owner pick the level he wanted to do the comparison at while listening to his system. I then used pink noise to level match the Onkyo to his system while he was out of the room. The Onkyo was running in its "Stereo Direct" analog mode.

I called him back in to listen, he sat down in the sweet spot and I replayed the same CD track he'd used to pick the levels. He immediately started complaining about how bad the Onkyo sounded. He said it sounded thin, compressed, harsh and a few other things. I smiled and turned the Onkyo off and the music kept playing. He'd said all those negative things about his own Bryston gear!

With him red faced, we proceeded to do at least an hour of listening with me swapping cables, or only pretending to, when he requested a switch. He listened to his favorite audiophile CDs. I did lots of swaps and fake swaps and during each would ask him which he thought he was listening to. In the end, his answers were roughly 50% correct which is the same as if he'd been randomly guessing. He even finally admitted, he couldn't tell which was which and WAS only guessing! I took his place and also couldn't hear any difference between the lowly receiver and his prized Bryston gear.

For the analytical among you, I've done some input/output null difference testing as well. This is where you use a wide bandwidth analog (even passive) circuit to subtract the level matched (nulled) input from the output of an amplifier driving real speakers using real music--no sine waves or resistors here. The resulting difference signal is made up of ANY distortion or deviation produced by the amplifier. Some distortion analyzers work this way.

My null tests have shown that even a modest $250 receiver can manage a -60db or better residual signal driving my relatively difficult speakers to fairly loud volumes. Higher quality amps can easily exceed a -70db null. Those are very low levels of residual distortion. Keep in mind this test reveals ANY kind of distortion--audible or not--including THD, phase distortion, IMD, TIM, slew problems, feedback issues, frequency response deviation, etc.

You could argue that you might be able to hear things that are 60db below the signal, but I'm skeptical. It's easy to hear -60db worth of noise during quiet passages of music, but it's downright tough to hear something that's actually 60db below the signal. For those of you who have a volume control calibrated in db, turn it up to a comfortable listening level and then reduce the gain 60db (if you can without shutting the sound off) and see how much is left. It's hard to hear even in a quiet room!

I'm not trying to start a flame war here, but I am suggesting those of you who think high quality amps have magical qualities like "warm", "airy", "detailed", "liquid", etc. might want to do some blind testing. Most of the differences I thought I heard evaporated once I didn't know which amp I was listening to and the others can be attributed to other factors.

It's been my experience that hardcore audiophiles will dismiss the blind tests as flawed and they refuse to participate in these tests or acknowledge the results. Most of the magazines, of course, don't do them as it would be really bad for ad sales across the board. It's hard for any manufacture to sell a $3000 amplifier when the magazine demonstrates it sounds the same as a $300 one. The same goes for salespeople at high-end dealers. The owner of our store told us to keep our mouths shut as "our test was invalid" when we made our blind discovery.

Don't get me wrong here, I'm not saying a $250 amp is all anyone needs. As you spend more money you tend to get more power, higher current capability, bigger power supplies, better construction, longer life, etc. Many of these are especially useful if you have inefficient and/or difficult to drive speakers and like it loud. But listening at the sorts of levels you'd use for critical comparisons, there's usually not enough of a difference to tell quality amps apart in a proper blind test.

I'm also not saying that some amps don't have "euphoric distortion" that some people like (i.e. single ended amps). Some amps are measurably different (i.e. have a rolled off high-end). These amps are relatively easy to pick out in a blind test. I also know the thrill of building your own amp and the extra enjoyment you get from having built it.

Finally, I've found few people that have actually participated in a proper blind amplifier test. Next time you're evaluating an amp, take a few extra minutes to level match it to a "reference" amplifier and have someone else swap cables in a way that you don't know which you're listening to. If you believe amps have their own "sound", you might be VERY surprised at the results!

Comments?
 
Grabbing archived data......

Richard Clark has a challenge that will pay you $10k
if you distinguish between the sonics of amplifiers.
You don't lose money if you lose.
Rules found here + some compiled statements by RC:
http://www.talkaudio.co.uk/vbb/showthread.php?s=&threadid=18815

Note: The challenge is now open to home audio even though it says
car audio. Any amplifier is game, home audio vs. car audio
amp if you wish, $100k home amplifier vs. Yamaha, etc.
Tube vs. SS is game also..... Anything is game as long as it
meets requirements in the rules section.

Talk to RC here, this is his forum;
http://www.carsound.com/ubb/ultimatebb.php?ubb=forum;f=1

Other forums debating this issue here;
http://www.carsound.com/ubb/ultimatebb.php?ubb=get_topic;f=1;t=019632
http://www.decware.com/cgi-bin/yabb/YaBB.cgi?board=General&action=display&num=1036124954
http://www.harmonicdiscord.com/forums/viewtopic.php?t=8749
 
Member
Joined 2002
Paid Member
Many good points there...

-but I still can differentiate between amps from same manufacturer, a generation between.
"Sound-stage", "body", if it's "forward" or "laidback", are usually easy parameters to recognise and separate them.

ArneK

Wich likes the neary impossible mix of "forward" and "super-analytical".
 
I went through this phase myself. Eventually, I found that there were certainly amps I could pick out blind. Investigating further, whenever I could distinguish two amps, there was always something like a source Z interaction with the speaker or the level match had drifted. Once the levels were reset or the supposedly crummier amp eq'd to make it sound like the Golden Amp, I couldn't tell them apart. After a year or so spent devastating my own beliefs, I stopped bothering too much about that stuff.

Funny, every time I hear someone say that the Kichimichi 2000 has a much wider soundstage and can place instruments outside the speaker locations which a cheap Pioneer (or whatever) can't, I think, geez, if you blinded me and had a sax player play from one position on my far left, then have him move 2 feet over and play again, I think I'd be able to hear that.

I'm certainly no golden ear. But I often wonder how many golden ears are actually golden ears.
 
diyAudio Retiree
Joined 2002
a case of the blind leading the deaf

"I couldn't tell a $300 amp from a $3000 in the store I was working at. Neither could anyone else who worked there."

I don't know whether to feel more sorry for you or the poor fools that bought equipment from said store. I hope no one lost their livelihood faced with such a moral dilemma. In objective experiments in conventional engineering and science disciplines, null results often cause the experimenters to examine the resolution or methodology of the test. I have been down this path in telecom design and troubleshooting too many times to count. I have worked on measurements on telecommunications systems with people several states away working on the same problem. One often get different results could due to the test setups being different in ways that were not noticed or understood at the time. I spent considerable time flying from Texas to California and back in one day, to resolve this issue.

Some of the worst sounding systems I have ever heard were in High End stores. A couple of the best I ever heard were in High End stores also. The cost of the equipment was not the issue either. Room acoustics, AC power quality, the amount of RFI, the cleanliness of the connector contacts, familiarity with the system, the time of day, the listeners experience with live unamplified music........... I can list dozens more factors based on real and measurable variables and not on pseudo science or belief systems. Careful listening, like the appreciation of art and music, is a learned process. A whole new vocabulary describing what listens for has developed in the last twenty 20 years. I'll bet not everyone is familiar with what to listens for or has the reference of live acoustic music. I know nothing about wine tasting but I don't think those who do are deluded or elitist. Some people cannot hear well and some lack the training or desire to develop that skill. That's fine, but don't attack those of us who do.

When the test gives null results, shouldn't one look at the test more carefully; rather than except results based on someone's predetermined expectations. What about changes to a system, made with the expectations of making it sound better, that make it sound worse to the experimenter. How is that an example of hearing what one wants or expects to hear? Are the belief systems of someone expecting no differences any less suspect than one working to examine and develop the reasons that amplifiers sound different? Stating that all decently designed amps sound the same based matching the levels sounds pretty cynical, arrogant, and intellectually lazy to me. Hell, you can match the levels and they won't even measure the same, given adequate resolution measurements. I believe there probably is still some investigation going on the science of how and what we can hear untainted by audiophile delusions. Unlike many dogmatic engineers, many of us motivated by love of the art of audio design, and desire to get further into the music. I think audio designers might even have a lot more experience in the factors that responsible for these differences than reviewers, academics, and salesmen. I have leaned things in serious audio design that have made me a better telecom engineer. I don't think I have had to discard any engineering skills to design audio.

I really fear you have mistaken double deaf testing for double blind testing.
 
My main reason for posting this was to see what folks here (who should be more technically adept than your average audio forum reader) think of blind tests (and null difference testing).

My secondary reason was to perhaps save some folks some money. If a person with "good ears" can't tell an amp that's full of 5 cent capacitors from one using exotic parts, it stands to reason they should save their money in their DIY projects unless blind testing proves the more expensive parts sound better (and it rarely does).

A friend went down this path with unexpected results. He was building a high-end power amp (someone else's design) and was searching for the very best parts for it. His purpose for building it was to have something better than his mainstream Adcom power amp which he was convinced was holding his system back.

In his quest for the right components, I suggested he use his Adcom as a known reference and then compare his amp against it, using blind testing, as he made various changes to the design and components. You can guess what happened... Right from the start, he couldn't tell his new amp from the Adcom let alone tell the difference between brands of capacitors.
 
I know nothing about wine tasting but I don't think those who do are deluded or elitist.

No, we're not (well, I'm not, anyway), but we do analytical tasting blind. Just did one today, where we examined the sensory effects of different cork lubricants. Multiple trial, triangle, double blind. People who score significantly run through a second time, with the glass coding rearranged.

I note that Larry Archibald does his tastings blind.
 
diyAudio Senior Member
Joined 2002
NO TWO SOUND THE SAME.

Hi,

I really fear you have mistaken double deaf testing for double blind testing.

Ditto.
And ditto for the rest of your post Fred.

It reflects my own experiences very well.Not everyone has the capability to discern differences but I usually found a majority to agree with what was heard in the long run.

While I find group listening tests rather useless statistically, I always found them useful to me on a personal level.

Cheers,;)
 
I found that I could tell the difference between a really bad amplifier and a really good amplifier quickly; put two good ones in and I probably wouldn't. But I don't care, because the real differences aren't in the blind testing.

What about an extended blind test where the testing period were a few weeks or something? That might be more interesting... if you don't know what amplifier you're living with, and you can get used to it in a system you're used to. Then, maybe these so-called imaginary differences might not be so imaginary?
 
To Fred Dieckmann: A lot has been written on the methodology of blind testing and why it's valid when it's done properly. Very little objective material has been written to discredit it.

Likewise, a lot has been written about how inaccurate non-blind testing is and the psychology behind why we hear differences that aren't really there. It's not unlike the people in drug studies who have side effects and/or results from placebo drugs. They take a sugar pill and it does all sorts of things to them because they EXPECT IT TO. Does that mean manufactures should sell placebo drugs for huge profits because some people claim they work? Or should we trust science instead?

You can criticize high-end audio dealers if you'd like, but many of these tests (such as the one I did with the Onkyo) have been done in people's own homes with music, systems and surroundings they're intimately familiar with--and they still fail.

Audiophiles typically try to attack the blind testing methodology, but they never seem to come up with solid reasons why it's not valid. After all, it DOES expose differences between many things (like a single ended amp versus a class AB one). In fact, when a switchbox is used, it's an extremely accurate indicator of even small differences in sound quality.

A well documented semi-famous blind test was the Sunshine Audio Challenge. A Yamaha integrated amp was used to replace a pair of expensive monoblocks in the owner of Sunshine Audio's home in Florida. He had an extremely high-end system (worth six figures) and regarded himself as very well skilled in hearing all the little nuances that the high-end press loves to write about.

With and without a switchbox (doing cable swapping instead) he couldn't tell the Yamaha from his amps. Nor could his wife. Nor could anyone else present. The methodology in that test was solid and more elaborate than what I've done (at least part of it was double-blind as I recall). The best excuse he could come up with was that he had a couple glasses of wine the night before.

The editors of Stereophile lost a blind challenge to Bob Carver. They couldn't tell a pair of $15,000 monoblock tube amps from a $699 Carver model that Bob lightly tweaked to duplicate the frequency response of the tube amps. They helped define the terms of the test. They were VERY confident they would win. They lost.

If Stereophile could find fault in such a test to save face, don't you think they would have? Instead, they explained it (more or less) by saying the modified Carver amp was "ringer" and way beyond anything else in its class (which of course delighted Bob). The Carver amps have rail-switching class G power supplies which are generally regarded as inferior by audiophiles.

There are several other blind tests documented with similar results. I ask you, if in comparing amplifiers back-to-back in your own system, you can't hear the difference between them when you don't know which you're listening to, how great can any differences be?
 
This is an account of a blind test, which is NOT rigorous, but has convinced me that I'm not just imagining differences between amps:

I've built an AKSA 55, with mild tweaks (Nirvana option + more). Recently I build a gainclone-like kit amp. I've been tuning the gainclone in the past 3 weeks, and it has been swapping places with the AKSA.

My wife sees me connecting / re-connecting amps, but has no interest in the hardware specifics, only in listening. Even then, she doesn't listen out for the audiophile qualities (don't talk to her about PRAT, soundstage, neutrality, ...).

In about 15 iterations, she has correctly identified the AKSA or the gainclone >90% of the time. On earlier occasions, she was able to identify between the AKSA and a Krell about 80% of the time.

Given that she is NOT looking at any power indicators or cabling layout, would this qualify as blind? Yes, levels are not calibrated, and the music varies, but to be able to discern over a relatively short listening period, AND consistently identify the differences in sound indicates pretty substantial variation in sound.
 
nw_avphile said:


The editors of Stereophile lost a blind challenge to Bob Carver. They couldn't tell a pair of $15,000 monoblock tube amps from a $699 Carver model that Bob lightly tweaked to duplicate the frequency response of the tube amps. They helped define the terms of the test. They were VERY confident they would win. They lost.


I bought that Carver amp because of thet challenge. I had it for about two years and sold it, because I couldn't listen to it. I had Zen amp in my system for 7 years and I could still enjoy it.

When test like that is conducted I suspect that the level of excitement is increased and the person tries too hard to choose the proper answer. The better way to carry it out would be when someone switches the amps (without person under test knowing it) but the amp cannot be visible and the person should be under impression that he still listens to his usual amp. If he won't notice that something changed, than I will believe that all the amps sound the same.
 
Yes, levels are not calibrated

Then right there, you've made the test invalid. It's very well established (in blind tests!) that people can detect extremely small level changes (on the order of 0.1 dB). And it's equally well-established that if they believe that the levels are matched, they'll pick the slightly louder amp as sounding "better," and here I have some direct experience.

Doing proper listening tests is tedious, time consuming, and inconvenient for the experimenter (not necessarily for the subjects), so few people bother.
 
"A Nod Is As Good As A Wink To A Blind Bat" - LIfe Of Brian.

Hi Phred and Frank,
I agree 100% with you both, and Sy perhaps something is wrong with your test systems - inadequate speakers etc, etc.

I find it perfectly easy to discern minor sonic differences due to minor tweaks or modifications.
The key is to listen not in a 'N & THD testset' mode.

Let's be honest - just about any modern gear measures pretty damm 'perfect' on a test and measurement bench, however these tests usually do not look for differences in load dependency and mains power supceptability, and this is a key error.

I find the key is to listen in an 'FFT' mode, and to listen for micro 'patterns' in the sound.
This will reveal differing sonics due to differing dynamic behaviour, but takes practice using many types of music and many types of electronics to discern commonalities, and from this then discern fine differences.
Spectral and dynamic behaviours are the keys to listen for, and are the terms that describe the long term musical enjoyment factors.

In my experince just plugging an additional amplifier (not even turned on) into a common mains power board can alter the sound of a system, and this I expect is not a factor heeded in 'blind' testing, and in my view further invalidates conventional blind testing.

When you gain divining or dowsing sense, you are in a much better position to understand what your ears are trying to communicate to you.

Dowsing is about sensing small changes in levels and spectrums in ambient fields, and a similar mode of sensing needs to be applied to discerning differences in audio equipment.
Once you have developed these skills, then categorising audio gear is a piece of cake, and reliably so.

As Phred says not everybody is familiar with live music - acoustic or amplified rock, and indeed this is required as a proper reference.
A live mixing desk contains individual channel tone controls, and despite all inputs running and a cacophony coming out of the FOH speakers, just breathing on a tone control or level fader is clearly audible to experienced ears - indeed this skill is mandatory to achieve pleasant, coherent, musical and dynamic mixes.

The same comment above applies when adjusting the final FOH send graphic eq - changing just one GEQ band by half a dB is clearly discernable to discriminating ears.

To those who belive so solidly in blind testing, I say that the entire test setup and methodology needs to be examined very closely, and this very much includes the skill levels of the listeners.
If you take 6 bricklayers or boilermakers I would fully expect the results to tend toward 'chance' - 50/50.
If however the subjects are very experinced orchestral musicians or conductors, you then have a much more proper and reliable test.

Eric / - You just have to understand your own ears.
 
Peter Daniel said:
When test like that is conducted I suspect that the level of excitement is increased and the person tries too hard to choose the proper answer. The better way to carry it out would be when someone switches the amps (without person under test knowing it) but the amp cannot be visible and the person should be under impression that he still listens to his usual amp. If he won't notice that something changed, than I will believe that all the amps sound the same.

You know, I can't think of anyone of note who has actually, literally claimed that "all amplifiers sound the same."

What I have heard some of note claim is that what causes amplifiers to sound different is no big mystery and rather trivially measureable, such as distortion, frequency response, level mismatch, etc., and that once these differences are narrowed down to below known perceptible limits then the two will be indistinguishable.

But of course once you tweak one amplifier or another in order to minimize its differences from another, you don't exactly have the same amplifier as you did originally.

So where exactly did this "all amplifiers sound the same" come from? It smacks of the same simpleminded rhetoric used by Republicans to characterize Democrats and vice versa.

Also, in "ABX" tests, the listener always knows the identity of A and B. This allows them to swtich from one or the other in order to attempt to discern any differences. Once any differences have been identified, then you can switch to X which will randomly be selected (for each trial, not each switch) to be either A or B.

Once a difference between A and B has been identified, then you can compare A and B to X. so for example if you switch from A to X and you hear no difference, but you hear a difference when switching from B to X, then logically X should be A.

If there is an actual identifiable difference, there should be no trouble in distinguishing it.

Tom Nousaine has set up a number of self-professed golden-eared audiophiles with an ABX comparator in their own home using their own system and allowed to run tests at their own leisure over periods of months or more.

So far, none have been able to statistically discern any differences once basic issues such as distortion, frequency response and level matching have been addressed.

Of course this isn't necessarily what I'd call definitive. But I have always been amused at how "night and day" differences tend to simply vanish once the listener is simply denied the knowledge of which component they're listening to.

They're absolutely sure they hear a difference when switching between A and B. And they're often just as absolutely sure when they compare to X. Yet when the results are examined, no statstically significant correct (or incorrect for that matter) responses seem to manifest.

I would like to think that anything described as "night and day" should be able to be distinguished 100% of the time seeing as there are many "subtle differences" which are easily distinguished by certain individuals 100% of the time in such tests.

se
 
The nulling test...the end of ...(i think it sound better)...

Retourning to the Hafler nulling tests my coments is:

This is my favorite test,that i use for so long...with it you can see all distortions ( invented or yet indiscovered)...it was amazing to do this test in some very expensive comercial amps.. hehehehe!
You can do the test even with music signals...and with a speaker load...in real conditions of operation!!
In a nutshell...if the input is equal to the output the amp desapear from the equation...as it is tottaly transparent...so time to look at the source and speaker!!
By the way...anytime that i improve a amp in the nulling test...i see also a improvement in sound quality...i think is hard to talk against this test..
But i don't beelive that all amps sound the same...as in the nulling test...they also don't show the same residual...

So simple and so true!!

But i wonder who is searching the truth in audio...just a fews!!

Cheers:drink:

Ps: Without a test like this how can we trust in a person saying..that sound good...
for me is the same that a person talk about food...for one is good...for the other with to much salt...for the other without salt at all...:cannotbe:
 
Dear nw_avphile,

To answer your original question: for any assessment which is significantly influenced by psychological/experiential factors the DBRCT is the recognized and accepted "gold standard"..... period!

Anyone wanting to argue with the above statement (on it's own) is embarking upon a personal credibility crisis.

What you are saying is in principle absolutely true, however saying it here is borderline insanity. Those from a scientific background will support you and accept that before we claim to be able to hear "a, b, c, ... x, y, z", we should actually subject ourselves to a rigorous, controlled, blinded environment.

As you suggest, many of the "golden ears" would likely tarnish and excuses would abound.

The industry (as you rightly suggest) has way too much vested interest to ever allow this sort of madness to occur and will claim all sorts of "********" to discredit the process. The process itself remains valid (done properly - which is actually very hard!)

Me ...... well, I built a nice Aleph5 clone, used good quality parts and spent many hours lovingly creating this project. Does this mean it sounds better to me ........ HELL YES!!!

The inescapable truth is that psychological factors do alter our perceptions and appreciation of things in the real world. Our friend who bought the Carver because of the results of the blinded test was happy for a while. But, over time, knowing we are listening to a "modest" product must influence how we perceive it. I do derive more pleasure from my amp because I know I built it well and it is a "quality" product. this knowledge adds to my appreciation and enjoyment.

Would I bet my son's life that I could distinguish between this and another SE amp .......... not in this lifetime :(

Are there companies out there who are taking customers for a ride and profiteering from pseudoscience? ......... I'll leave this to your assessment.

Remember - at the end of the day, enjoyment of the music is the end requirement, and enjoyment is a multidimensional, difficult to measure thing;)

cheers,
mark
 
nw_avphile
to me, it's easier to spot differences than to identify specific sound signatures. The tasks involve two different ways of thinking/listening, quite possibly different sides of the brain are involved. ;)

Since you seem to be into it why don't you try this test: with a bunch of people, play something on one system, then change a component and record differences between the two systems on a piece of paper. Compare notes at the end. I bet you'll find that most everyone will have heard the same difference in sound.
Report back, please.
 
mefinnis said:
Remember - at the end of the day, enjoyment of the music is the end requirement, and enjoyment is a multidimensional, difficult to measure thing;)

Absolutely!

Which is why I ultimately have little interest in blind testing or whether my perceptions are due to physics, psychology or some combination of the two. I'm not so insecure that my ego is threatened at the notion of my being nothing more than a normal human being with all the mortal weaknesses that entails.

When it comes to enjoying music, I'm a "hedonist subjectivist." I don't serve any piece of equipment. I don't serve any blind test. I don't serve any measurement instrument. I don't serve the music. I serve only my own pleasure and satisfaction, regardless of how it may come about.

Well, provided it doesn't violate anyone's rights or liberties. :)

se
 
Status
Not open for further replies.