Blind Listening Tests & Amplifiers

Status
Not open for further replies.
Planet 10... two Statisticians proved beyond any doubt that i had that the only valid result that can be garnered from the ABX test is that 2 DUT are different. If you can't tell a difference the test does not prove or disprove anything.
Actually, this statement is both correct and incorrect (in it's inference). It is also a bit "cake and eat it too" .......... statistics can't prove anything, but they proved this beyond any doubt ??? :bigeyes:

Rob quite rightly states, "Statistical tests don't prove or disprove anything." ..... this is the correct bit

Statistical testing simply gives us the probability, by chance, of observing a particular set of observations. Literature (especially in my field - medicine) abounds with poorly designed studies and mis-used statistics. Mainly because people are trying to "produce" a result consistent with their preconceived ideas ;)

The statement that a negative result "means nothing" is absolutely incorrect, with the proviso that the study was appropriately powered to answer the question asked. We are referring here to beta-error, or the probability of not finding a difference when one truly exists.

As Rob states in nice simple language, if
1. our question was, can "golden ears hear any difference between these devices"
2. we designed the trail well (technically)
3. we used sufficient numbers of golden ears
4. we repeated the study on different occasions and in different locations

This may not "prove" they cannot hear any difference, however, your "learned opinion" would be worth "jack-****" in comparison to anyone with any sort of scientific background.

Negative studies require a process called "power analysis" prior to starting the study to an agreed level of probability for a negative result to be acceptable. That is, we all agree to the ground rules prior to starting.

cheers
mark

PS: I am not a statistician, but it is a necessary evil of my work ..... else I might believe all sorts of nonsense studies and do horrible things to my patients.

PPS: A group of real statisticians went through an entire year of the leading dozen medical journals and over 70% of studies we underpowered to say anything about a negative result, used the wrong statistics, or had designed the study incorrectly. Reassuring ...... isn't it :bawling:
 
analog_sa said:

I was under the impression that nothing made a difference. Why would more output devices make a difference if the amp is not being overdriven? Or a bigger PS? Such excess can possibly be justified only by a headbanger, right?

$8 for a cap? you're surely kidding if you think you can get anything decent for that. Maybe in the seventies?
Well this probably doesn't need a reply, but I never said amps were all the same with respect to power output, low impedance drive capability, etc. These things are (obviously) determined by the size of the power supply, number of output devices, etc. For some people, too much is never enough while others have more modest power/current requirements. My point was this is an area where spending more money clearly gets you measurable and audible gains.

Given the choice of building a 50 watt amplifier with a limited power supply and exotic audiophile parts, or a 100 watt high current amplifier with "regular" parts, I'd choose the 100 watt high current design and would put it up against the 50 watt version with audiophile parts any day in a blind or null test (I'm assuming the same basic circuit design here).

As for capacitors, I was referring to *poly* signal capacitors like you would use to couple the input circuit. In blind and null tests, I've never seen a $0.30 poly film capacitor perform any differencly than an $8.00 "audiophile" one. If you're talking about the main *electrolytic* power supply capacitors, they can indeed cost more than $8 for large ones.
 
By the Way ......

I should clear-up one other small point.

I have not at any point stated person 'X' cannot hear these differences. I have simply stated that DBRCT is the best means of ascertaining if this statement is true ........ it may be, it may not.

The whole point in doing a study well is not to have preconceived ideas as to what the result will be.

Most people arguing here have (I strongly suspect) never placed themselves in a well controlled blinded environment.

mark
 
diyAudio Senior Member
Joined 2002
OH BOY...

Hi,

Given the choice of building a 50 watt amplifier with a limited power supply and exotic audiophile parts, or a 100 watt high current amplifier with "regular" parts, I'd choose the 100 watt high current design and would put it up against the 50 watt version with audiophile parts any day in a blind or null test (I'm assuming the same basic circuit design here).

Well, that will very much depend on the speakers you need to drive with the amp.

I'd drive them with a 50W well designed amp with HQ parts if possible and ample PSU.
Any single part used in an amp is important, even something as silly as a fuseholder.

As for capacitors, I was referring to *poly* signal capacitors like you would use to couple the input circuit. In blind and null tests, I've never seen a $0.30 poly film capacitor perform any differencly than an $8.00 "audiophile" one.

So to you all polyprops are the same sounding? Why do you bother to use polyprops in the first place?
You could save money using polyester there...
Obviously you don't need boutique parts at 25+ $ a piece but I'm sure you wouldn't know where to look for the better non-boutique parts either...mind you, they can even be more expensive than boutique caps.

Does it all make a difference? Yes it does, paying attention to detail and knowing what part is good at what really does pay off.
Than again, that kind of experience does not come overnight.

If you're talking about the main *electrolytic* power supply capacitors, they can indeed cost more than $8 for large ones.

Yes, and there are big sonic differences in electrolytics just the same, not all of them sound alike at all.

No wonder Fred is getting desperate.:bawling:

Cheers,;)
 
nw_avphile said:

On a more technical note, my experience with null testing is much of the error is from phase shift at high frequencies. Is this what you're referring to by "front-to-back time" SY? Amplifiers that have flat response for a few octaves beyond 20khz tend to do better with null testing unless you bandwidth limit the null signal.

Well, you get phase shift anytime you have a rolloff. And the rollof itself will indeed affect the null for a couple of octaves below the corner frequency (and the opposite for an LF rolloff). What I also meant to lump in with that vague term I used was the case where there's a constant delay independent of signal.

The argument of whether an amp that does better on a null test is "performing" better is a semantic issue. It is performing better if the design performance goal is a high degree of null on that test. It is irrelevant (within reason) to the question of audible differences; an amp with 60 dB of null across the audio band might well be audibly transparent and thus indistingishable from an amp with 80 dB of null across the audio band.
 
diyAudio Senior Member
Joined 2002
IN SYNC.

Hi,

Well, you get phase shift anytime you have a rolloff. And the rollof itself will indeed affect the null for a couple of octaves below the corner frequency (and the opposite for an LF rolloff).

Indeed, there is phase lag (running behind) and there is phase lead (running ahead)...

Which is one reason why a CDP, with it's limited bandwith and steep (brickwall like) filtering presents such a big problem...amongst other difficulties.

What I also meant to lump in with that vague term I used was the case where there's a constant delay independent of signal.

Which, in theory at least, should have less of an audible impact since it is delayed in it's entirity.

The argument of whether an amp that does better on a null test is "performing" better is a semantic issue.

There are a number of things the null test does not show, if it was such a perfect test we'd all use it and by now all amps at equal powerrating and passing the null test would be sounding the same as the source, unfortunately that is not the case.

While the test is not a bad one, it sure isn't the only valid one.

Cheers,;)
 
Some numbers!!!

Lets put some perspective in all of this...

As i have told i use this test for many years ago...so lets see some numbers....

With a 2 Volts input in all cases the null is:

Frequencys ----------------------1Khz--------------------10KHz

Sony TAF 3ES-------------------2,5mV--------------------20mV

Pioneer 705 R-------------------2mV---------------------18 mV

harman kardon 680------------12mV--------------------22mV

Onkyo 9511---------------------1mV----------------------5mV

Onkyo 9711 Integra-----------1mV----------------------3mV

Kora designe 30(tubes)-------70mV--------------------70mV

Counterpoint NPS 100---------42mV--------------------400mV

Rotel RB 980 BX-----------------2 mV----------------------6mV

Food for thougt!!;)
 
nw_avphile said:



As for capacitors, I was referring to *poly* signal capacitors like you would use to couple the input circuit. In blind and null tests, I've never seen a $0.30 poly film capacitor perform any differencly than an $8.00 "audiophile" one. If you're talking about the main *electrolytic* power supply capacitors, they can indeed cost more than $8 for large ones.

Are you kidding or you really don't know how much quality "audiophile" caps cost?;)
 

Attachments

  • cap.jpg
    cap.jpg
    89.1 KB · Views: 922
Re: IN SYNC.

fdegrove said:
Indeed, there is phase lag (running behind) and there is phase lead (running ahead)...

Absolute phase doesn't really tell you anything important. Absolute phase includes both frequency-dependent and frequency-independent delays.

Frequency-independent delay is irrelevant. Frequency-independent delay delays all frequencies equally and therefore has no effect on the end result. It's like playing a piece of music now or a few seconds later. It has no effect on the final outcome.

What's of interest is frequency-dependent delay, which has different frequencies arriving at different times.

So what truly matters isn't absolute phase, but group delay. There can be a huge difference from one frequency to another with regard to absolute phase, yet the group delay is virtually zero, with all frequencies arriving at the same time.

In fact, one network may give you much more of a difference in absolute phase compared to another, yet have far better group delay.

So don't get hung up on phase. It's group delay that counts.

se
 
he double binded me with science

"And, FWIW, I don't think folks here are "deluded idiots" for enjoying their hobbies. As I said, it's hard to argue with the concept of spending time and money on something that makes you feel good (for whatever reasons). Further, by your definition, I used to be a "deluded idiot" myself.

But I believe there are also people here with limited budgets looking for the best performance they can get for a given amount of money. Those folks, in my opionion, might be better off spending their limited funds on things that can be demonstrated to make an audible difference like bigger power supplies, more output devices, low distortion circuit designs, etc. Spending $8 each for "audiophile approved" poly capacitors and the like is better left to those who have other priorities.

I guess it's the difference between art and science. If you view DIY amplifiers as art, this thread probably isn't for you. If you view it more as science, there's some good information here."


I went back and read your previous post in particular the first one. I am very puzzled why one who started from the premise that amplifiers sound different and believed that to the extent of building one's own amplifiers is so intent on proving that they don't. Did you get burnt out due to the time and money involved or the lack of success with building your own amps? Was it the frustration of not wanting to spend the large amount of money that you thought was required to have a good system? Did you lose interest in music or resent the time and effort spent on the hobby?

I will be glad to state some of the motivations and rationales for the growing DIY community:

We don't buy into the fallacy that more money spent and following the advice of most reviewers; is getting us closer to enjoying the music. Many expensive and well reviewed products sound bad. Many of the modestly priced and older good sounding pieces can often be made to sound much better with parts upgrades and modifications with a reasonable outlay of cash.

Many professional designers don't design with sonics as the prime objective. Appearance, features, measurements, and what I call yuppie appeal are the dominant factors. Many of there customers are not music lovers but buy equipment for the name recognition and conspicuous consumption status of a product.

New technologies often drive design such as DSP, oversampling, number of bits resolution, and distortion measurements, many of these products are designed by PHDs from other disciplines and are not rooted in audio design. Are listening test even a factor in most of these designs?

Many DIYers and even many High End designers believe that low static distortion numbers are detrimental to the sound of a circuit, because of the topologies and amounts of negative feedback required to achieve those numbers. There many types of distortion measurements and it seems that a new type is discovered every few years. Amplifiers are then designed to minimize this distortion. Yet many of the older simpler design topologies still are still respected for their sonics, even though they have mediocre measurements.

New device technologies are often problematic due to the higher bandwidths causing complex responses in the RF range. This is compound by an increasingly EMI filled environment. The effects of the intermodulation of this RFI products into the audio band are known by instrumentation amplifier designers but ignored by most audio designers. RFI generation by switching and even linear power supplies are audible factors ignored by many audio designers.

New conductive electrolyte capacitors and low temperature coefficient resistor types are being used by the deluded audio tweakers that are unknown many audio designers. The effects of vibration on passive components and the effect on solder types and conductor materials on the sound are being investigated by those outside the mainstream audio industry.

Slowly. many of these factors are being considered by the more open minded engineers inside the traditional audio industry.
The mindset of simplistic measurements and null testing that you are advocating, seem to be geared to a lowest common denominator and budget mentality. That hardly contributes to the advancement of audio engineering. I really wonder what side of this debate on which "science" really resides.

If your idea that "bigger power supplies, more output devices, low distortion circuits" always contribute to an improvement
in an amplifier, I will have to contest those assumptions also. I believe you indicated that you are an engineer and I am very curious in which particular field?

Something's happening and you don't know what it is, do you Mr. Jones.........

Fred Dieckmann
BSEE AND deluded audiophile.
 
diyAudio Senior Member
Joined 2002
DOUBLE BLIND DOUBLE DEAF?

Hi,

Absolute phase doesn't really tell you anything important.

I see, that's why you spun a thread on that ad nauseam?
Oh, and now it's phase? Not polarity?

Absolute phase includes both frequency-dependent and frequency-independent delays.

The proof.

It's like playing a piece of music now or a few seconds later. It has no effect on the final outcome.

Yes, that's just time delay.Not important.

What's of interest is frequency-dependent delay, which has different frequencies arriving at different times.

Yes, and that is either ahead in time or lagging in time...so what's your point?

So what truly matters isn't absolute phase, but group delay.

Again, what's it going to be? Phase or polarity? And if absolute polarity doesn't matter anymore than I'm either death or stone cold dead.

There can be a huge difference from one frequency to another with regard to absolute phase, yet the group delay is virtually zero, with all frequencies arriving at the same time.

Isn't that what we call phase shifts?
And that statement you make doesn't make much sense BTW, I'm only understanding it out of broad, wideband forgiveness...
I know, I know I should have kept my mouth shut.

And it goes on:

In fact, one network may give you much more of a difference in absolute phase compared to another, yet have far better group delay.

Wow...that's new to me.
Imperfect splitters he? Glad you caught up but it seems you still haven't caught all of it.
Just discovered different behaviour in different order filters, or what?

yet have far better group delay.

Not that you would want that.

So don't get hung up on phase. It's group delay that counts.

No, it's frequency dependent delays that you don't want.
And phase accuracy IS major importanto!!!

And no, I don't want to discuss this till kingdom come...

Pheww....:rolleyes:

/Oink.
 
Science takes a stand!

Fred Dieckmann said:
I went back and read your previous post in particular the first one. I am very puzzled why one who started from the premise that amplifiers sound different and believed that to the extent of building one's own amplifiers is so intent on proving that they don't. Did you get burnt out due to the time and money involved or the lack of success with building your own amps?
That's a fair question but you have jumped to the wrong conclusions. I'm also a BSEE by education and profession (well at least part of my profession) so I'm well aware of the technical side of audio design. I also ended up deep in the audiophile world. As I mentioned, I sold high-end audio gear for a while and had the chance to spend a lot of enjoyable time with a lot of expensive gear. The bug had bitten me hard. Not being able to afford $4000 amplifiers, I turned to building my own.

During my high-end audio days I read every issue of The Absolute Sound and Stereophile (and many others). I didn't believe all of it, but I bought into much of it. If someone would have come along and tried to tell me a $250 Onkyo receiver couldn't be distinguished from Bryston separates in a blind listening test, I would have laughed in their face--just like some of you are doing now.

But that was all *before* I started to do some blind tests, more research and more testing. I would argue that I'm not the one who doesn't have an open mind here. I've been where you folks are and have since come to some very different opinions about much of high-end audio. I'm hardly alone, there are many others here and elsewhere who seem to agree with me.

So it's not a matter of being burned out. I still have all the amps I've built, and some of them still sound great. But I now realize they don't sound any better WHEN I DON'T KNOW WHICH AMP I'M LISTENING TO than any number of reasonably priced mainstream commercial amps. They also don't test any better in a null test.

Fred Dieckmann said:
We don't buy into the fallacy that more money spent and following the advice of most reviewers; is getting us closer to enjoying the music.
I don't think you can safely say "We". I don't think your opinions are those of everyone or even the majority on this forum. Clearly, opinions here differ widely. We have at least one member who feels that an $8 poly cap (say an AudioCap, Wima or Solen) is ridiculously cheap. We have several others who state they cannot hear differences between quality amplifiers in blind tests beyond obvious things such as frequency response, etc.
Fred Dieckmann said:
Many of the modestly priced and older good sounding pieces can often be made to sound much better with parts upgrades and modifications with a reasonable outlay of cash.
I'm glad you brought that up. Let's take an example:

Audiophile A spends $250 modifying his CD player, puts it back into his system, sits down with his favorite music playing and is delighted with the results. He genuinely hears all sorts of improvements. The soundstage is deeper, the highs more detailed and less veiled, the bass has more punch and foundation. He's thrilled with his investment.

Audiophile B is a bit more cautious. He's trying to decide if he too wants to spend $250 modifying his indentical player. He borrows Audiophile A's modified player and compares it his own identical, but unmodified player, in a blind listening test. The trouble is, when he doesn't know which one is playing, he can't tell them apart, nor can Audiophile A, nor can anyone else who listens. Should he still spend the $250 for something he can't hear?

Far fetched? Not at all. One of the blind tests I was involved in was a modified Marantz CD player (I forget the model number, but it was the one everyone loved to modify several years back). It had $250 worth of audiophile parts put into it. The power supply, op amps, caps, wiring, etc. were all upgraded to the very best stuff. We loaded the same CDs into both players, ran them into line inputs on the same pre-amp, and switched back and forth. Guess what? Nobody could hear ANY difference on the high-end system we were using when they knew them only as "player 1" and "player 2".
Fred Dieckmann said:
Many professional designers don't design with sonics as the prime objective. Appearance, features, measurements, and what I call yuppie appeal are the dominant factors.
That's true to some degree. But I would argue it really doesn't matter. If someone is comparing two pieces of gear, and they can't hear enough of a difference to know which is which in a blind test, then they should start looking at other reasons to select one over the other.

I already said there's a lot of satisfaction in listening to DIY gear that turns out well. That might be reason alone to build rather than buy, but I would argue you should at least be honest about your reasons.

There's clearly some gear out there that's designed to a price point (or otherwise handicapped) and doesn't sound very good. Like you said, there's also ultra high-end gear that doesn't sound very good either--usually because the designer is so far off in left field he's lost sight of the original goal. 7 watt single ended triodes with 10% distortion fit into that category in my opinion. But the rest of the stuff in the middle, at least when it comes to high-quality amplifiers, sound pretty darn similar if you keep them within their limitations.
Fred Dieckmann said:
The mindset of simplistic measurements and null testing that you are advocating, seem to be geared to a lowest common denominator and budget mentality. That hardly contributes to the advancement of audio engineering. I really wonder what side of this debate on which "science" really resides.
I totally disagree. It's all about focusing on the things that make a REAL difference, not a perceived difference that comes from psychological bias.

I'm all for the advancement of audio engineering. For example, I've lately been working on digital (switching) amplifier designs for a commercial project. There's a lot of room for improvement in that area, and so far, most of the designs I've tested are readily identified as sounding inferior in blind testing and they also do poorly with null testing. But, as Douglas Self points out in his book, it's not hard to make a conventional audio amplifier that's completely transparent. Someday we'll have digital audio amplifiers that are as well (the Crown BCA amps come close).

So far, nobody in this thread has provided any credible objections as to why null testing isn't valid. The closest I've heard is it also identifies inaudible distortions, which is true, but that doesn't invalidate it as a test of amplifier transparency. It's almost as if some of you want to believe there's some other magic "ingredient" to amplifier sound that wouldn't be accounted for by comparing the input to the output in real-world operating conditions. How else can you explain your views? If a tree falls in the woods when nobody is around does it make any noise?

Like I said, it comes down to art and science. For those of you who want amplifiers that "artfully" distort the sound, or prefer to use very costly components because they make *you* think it sounds better, that's fine. Many people prefer art over science. But I also ask that you don't try to discredit the rest of us who prefer more objective proof that one amplifier, or capacitor or whatever, sounds and/or measures better than another one.

Finally, I have a question for everyone...

How many of you think you could tell a $300 mainstream (Asian made) integrated amplifier from high-end separates costing at least ten times as much like those from say Krell or Bryston in a blind test in your own home (your speakers, your music, etc.)?
 
diyAudio Senior Member
Joined 2002
MONEY? NO OBJECT...

Hi,

Geez, I am having a field day here...

My ears were, have been rented for 100$ + a day by high-end importers in my country.
Call them crazy, not me, please.

The bug had bitten me hard. Not being able to afford $4000 amplifiers, I turned to building my own.

Seems to me, you're bitter...you may have a point in that HK is marketing the same product for different markets at different prices, hell nobody said high-end audio had to be expensive...well it has to, but that IS marketing, not audio.

Once you understand that, you may roll your own, or make wise choices...if you're up to it.:goodbad:

Cheers,;)
 
Steve Eddy said:
Tom Nousaine has set up a number of self-professed golden-eared audiophiles with an ABX comparator in their own home using their own system and allowed to run tests at their own leisure over periods of months or more.

So far, none have been able to statistically discern any differences once basic issues such as distortion, frequency response and level matching have been addressed.

Of course this isn't necessarily what I'd call definitive. But I have always been amused at how "night and day" differences tend to simply vanish once the listener is simply denied the knowledge of which component they're listening to.
I think the above was worth repeating. It addresses the issue of listening over a much longer period of time (which is often used to discredit blind tests lasting only a few hours). Those of you who've never done a blind test have never experienced what I and others are talking about here. You have no reason to doubt your perceptions besides a few words from people like me.

Or, some of you say that any kind of tests just don't matter. If you *think* it sounds better, that's all that matters.

What if you were buying a new color printer? A friend showed you print samples from two models with similar speeds and other specifications. You looked hard at the printouts but couldn't see any significant differences. Then your friend tells you that one printer costs ten times more than the other one. Wouldn't it be an easy choice? Or would you argue it was an invalid comparison and buy the more expensive one anyway?

Why is audio any different? If you can't tell two pieces of gear apart when you don't know which is which, shouldn't you start looking for other reasons to choose one over the other instead of trying to discredit the comparison?
 
Good amplifiers

Good amplifier designs are supposed to mean that they are practically transparent - that they add almost nothing to the signal they amplify. If we have several amplifiers that belong to this category and compare them in their linear operating region , by logic they should sound the same ! Since they all add very little the sound is as close to the original as possible - which should be the same on all the amplifiers.

While we do hear differences in amplifiers - sometimes quite huge differences , maybe it is just that we are hearing the differences in what's not transparent about them ! And probably liking some of it . So logically , as they get better and better we should be hearing the same input signal and so hear no differences. So most amps which sound very different must be adding something of their own ( different frequency response, distortion spectra , output impedance etc). Some will be liked and some will not.
Cheers.
 
Status
Not open for further replies.