Blind Listening Tests & Amplifiers

[planet10]

Originally posted by nw_avphile
Hmmm... do you have references on that? I have a feeling we're talking about a different definition of "the same"? I agree a blind test cannot prove two things are the same IN ALL WAYS. It can only prove they're the same in the ways the test is measuring (or judging) them.

If i could find the references buried in my emails i'd have made a web page. As Steve has said, as designed in an ABX test, if you can not distinguish the difference, you have a null result from which nothing can be concluded.

You (planet10) said earlier:

Not me

dave

[fdegrove]

Hi,

For example, in double blind drug studies they compare the drug to a placebo. If the results of the study don't show a statistically significant effectiveness for the drug, it flunks the study.

In French: Oufti.

Please, talk about whatever you're comfortable with and, I hope, know something about.

Pancakes perhaps?

Or will that be a placebo?

P.S. The quote I used isn't necessarily wrong, the idea to relate it to audio is rather far fetched.

Hold on, let me get my aspirine (the most underestimated drug, BTW)

Mama mia,

[nw_avphile]

Originally posted by planet10
If i could find the references buried in my emails i'd have made a web page. As Steve has said, as designed in an ABX test, if you can not distinguish the difference, you have a null result from which nothing can be concluded.

Not me

Well please let us know if you find them. I've looked at a bunch of blind test documentation and have not seen anything that supports that (but I can't say I have anything to disprove it either).

As for the quote, sorry, you're right it was Rob M in post #58.

[planet10]

Originally posted by nw_avphile
Well please let us know if you find them. I've looked at a bunch of blind test documentation and have not seen anything that supports that (but I can't say I have anything to disprove it either).

Don't hold your breath -- i've been looking on & off for a year or so... it is in the archives of the hi-fidelity mail list on Yahoo... the posters were David Klein & Timothy bailey.

dave

[Rob M]

Originally posted by planet10

You will note that i am only refereing to the ABX test. As designed it is unable to conclude that 2 DUT sound the same.

Oh, sorry, I missed that. Sure, if a test is testing the wrong thing, then the results are meaningless. But, that would be true whether you pass the test or you fail it.

[Rob M]

Originally posted by nw_avphile

Why do really large numbers make a difference in this case besides the study gradually becomes more reliable the more samples you have? Do we have any statistical/logic experts here to help out with this part of the discussion?

There's nothing statistical about it. It's just that there are lots of ways to screw up a listening test. The more different tests get done, run by different people at different sites, the greater the chance that you'll get some that weren't screwed up. And the same's true for positive results -- if one test shows an audible difference, then it might be there really is a difference, or it might mean that the levels weren't really matched, or that the listeners figured out the pattern, or any of the other ways that they could guess right without actually hearing anything. More successful tests means less likelihood that the "success" was just a fluke.

[nw_avphile]

Originally posted by fdegrove
In recording just as in hi-fi reproduction, I have one word of advise: keep it simple, keep the signal path as short as possible, don't use feedback to impress others or the datasheets, try to be faithful to the music and yourself.

At least there's something we agree 100% on! I very much agree with the above, but sadly, your advice isn't widely followed in the recording industry these days. As I was saying earlier, the audiophile recordings are often nicely done but the performance itself usually leaves a lot to be desired. There are also just not that many of them even if you did like the obscure performers.

I also agree there are some stunning classical recordings from the late 50's and early 60's. They have lots of tape hiss, but they're otherwise impressive. They're all the more impressive when you think about what the "state of the art" was back then.

As for the rest of your comments, I agree good doesn't have to mean expensive, but I still know there's a big double standard. There's all sorts of stuff in the studio signal path that should make audiophiles make a funny face and cover their ears--especially when you consider it greatly outnumbers the stuff in the playback chain. You yourself have said that just one part--Wondercaps--can ruin the sound of a system. So something doesn't add up here because audiophiles seem perfectly happy with the sound from their systems playing recordings that should be offensive to their ears.

[nw_avphile]

Originally posted by Rob M
There's nothing statistical about it. It's just that there are lots of ways to screw up a listening test. The more different tests get done, run by different people at different sites, the greater the chance that you'll get some that weren't screwed up.

Well I was expecting something like that. I'd guess I'd like to be able to put some numbers to it if that's possible? In other words, if you run one test, it's 60% accurate, if two tests agree it's 65% accurate, if ten different tests agree, it's 90% accurate, etc. Is that possible? I realize the quality of the test is an issue, but keep reading...

I still have to come back to the common sense side of this. If, as you suggest, the levels weren't matched, that would actually bias the test towards HEARING A DIFFERENCE. If someone guessed the pattern that would also bias toward hearing a difference. If someone could see the leads or had other clues, that would also bias the test towards hearing a difference. In other words, all the likely errors I can think of FAVOR HEARING A DIFFERENCE! It's hard for me to come up with things that would actually obscure differences between the amps and cause a "false null"?

I'll outline my procedures, and if someone can tell me how this test would mask the audible differences between the amps, I'd love to hear them. The only credible ones I'm aware of are the time to do the switch if you're cable swapping, and the switching hardware if you're using a switcher.

One can argue the time shouldn't be a big deal as most golden ears claim to have a great "sonic memory". Reviewers frequently compare products they're reviewing to others they haven't heard for several months. We're only talking a few minutes here.

The Onkyo/Bryston test was run as follows (loosely patterned after the Sunshine Challenge):

The test takes place in the listener's own home using their own system and music. We'll call their amp (the Bryston combo) "Amp A".

The listener (or listeners) pick a playback level they're most comfortable with on their system while playing a familiar recording. Tell them you're not going to speak to them once the test starts, and it's their responsibility to ask you when they want a "swap" and that's also when they should tell you what amp they think has been playing. I also ask them, in advance, what percentage of correct votes they should get to indicate they can really tell the difference. Some are so confident they say 100%. The lowest I've heard is 2 out of 3 or 66%. Then ask them to leave the room for a moment.

Leave the gain set where they like it and play a -20db test sine wave on a diagnostic CD (you can buy one for $6 if you don't have one). The frequency isn't terribly critical as we're already assuming the amps have flat response in the audible range. I would suggest 400hz or 1khz.

Use a DMM (you can buy one of those for $25) to read the level at each of the speaker terminals while the sine wave is playing. Write down the voltages.

Connect the new amp or receiver (Amp B--the Onkyo) to the CD player. The most pure way to do this is simply unplug the CD interconnects from the listener's amp/preamp and plug them into the evaluation amp. It's a good idea to temporarily ground the CD player to both amps so that you don't get any nasty hum or other weirdness when you swap cables. You can disconnect the ground once the connections are swapped so there's no chance of it altering the sound. You can also use the preamp outputs if you want to remove the listener's preamp from the evaluation.

Move the speaker leads from Amp A to Amp B. It's best to make the line level connections with the speakers disconnected from both amps. I've had to make adapters if Amp A has really huge speaker wire terminations that don't fit Amp B. The adapters, if anything, would make Amp B easier to pick out.

Play the same test track and use the DMM to set the voltage to be the same as what you wrote down previously for each speaker (adjusting balance if necessary). The two amps are now level matched to a very exact degree. The accuracy of the DMM isn't an issue (unless it has huge drift problems--very unlikely) as you're only concerned about the relative match. You should be able to get the two amps within a few millivolts of each other which is a very tiny fraction of a db at typical listening levels.

Flip a coin to pick Amp A or Amp B to be up first and invite your listener back in and play the music of their choice. Two things are important here. First, the listener should not have any visual clues as to which amplifier is connected either from the listening position or as they enter and leave the room. Most equipment racks allow for this but I've had to do things like stand the DUT up on end behind the rack, set up a bit of audibly transparent camouflage (I bring some pieces of speaker grill cloth), etc.

Second, because this isn't a double blind test, the person running the test needs to not give any clues to the listener. Avoid eye contact, don't talk, ask questions, etc. I usually sit off to the side, out of sight of the listener, when I'm not needed to let them focus on task at hand.

When the person is ready for a "switch", I turn around so they can't see my face and they tell me which amp they thought was playing and I record their answer. They leave the room briefly and I flip the coin again to decide if I swap the leads or only pretend to swap the leads and invite them back in when I'm done. Again, it's obviously important they can't hear or see what you're doing at swap time. If they like (and they usually do) we replay the same music.

When they request the next swap, they again "vote" for which amp was just playing and again I record their answer.

You repeat the above few steps using different music as desired, for as long as the listener desires. If they're scoring really well, I usually terminate the test early as it's obvious they can hear a difference--again, this is biasing the test TOWARDS hearing a difference. If they're scoring about random, in my experience, they usually know they're only guessing and they tend to terminate the test fairly early. It's when they think they hear a difference but are not sure that things drag on.

When the test is over, you tally up the right and wrong answers and compare them to the total number of votes. I then share the results with the person and we discuss them against what they expected to hear. The Onkyo/Bryston guy admitted he was only guessing. He did, however, want to verify that he was indeed listening to both amps--i.e. that I was really swapping sometimes (which is easy enough to verify afterwards).

All that said, for my own personal tests, I built a "toggle" (momentary contact) switcher that simply switches the inputs and speaker leads through very high quality relays and wiring. This allows instant A/B switching with the person necessarily knowing which is A and which is B. Because it's an instant swap, it's more sensitive to differences. Some, of course, argue the extra contacts and wiring degrades the sound and masks any differences, but I've not found that to be the case in doing blind tests comparing the sound with no switcher and with it being in place.

Anyway... there's a DIY procedure for a simple blind home test. Comments anyone?

[mikek]

Originally posted by Steve Eddy


That's all well and good from a purely objective, emotionless point of view.

However, the sole purpose of the amplifier is ultimately to serve the subjective, emotional human beings at the end of the chain. And their single most important criteria is their own subjective satisfaction. Which may be at odds with an amplifier designed by wholly objective criteria.

....must be transparent, neutral, ie not bend signal...if you want an emotional twist, distortion,..etc...added to your music, then introduce it at line-level, with an 'emotion-adder'

Originally posted by Steve Eddy

You seem to be saying that objective criteria is the only valid criteria to use when it comes to designing an amplifier.

if by objective you mean sound, proven, and well founded scientific engineering principals, guilty as charged....

Originally posted by Steve Eddy

If that is what you're saying, then I'm afraid I will have to disagree. The equipment serves us, not the other way around.

se

...indeed the equipment serves us, but we surely must have some unambiguous and transparent method of determining the quality of service?

[Rob M]

Originally posted by nw_avphile

Well I was expecting something like that. I'd guess I'd like to be able to put some numbers to it if that's possible? In other words, if you run one test, it's 60% accurate, if two tests agree it's 65% accurate, if ten different tests agree, it's 90% accurate, etc. Is that possible?

Not really, no. You want to set the test up so that any factors you think might be important are incorporated into the design. That leaves all the stuff you didn't think of (solar flares? seismic activity?) which is going to be pretty hard to assign a number to. And then there's the human factor: actual people will be performing the tests, and they make mistakes. They also cheat, sometimes without even realizing it. So, it's up to you: how many passed listening tests would you need to see to be completely convinced that resistor end cap material audibly changes the sound? And how many would it take before you decided to spend a little more on resistors for your next project? (Personally, I'd say "two" and "one", if they were done by the right people.)

As for the test you outline, I'd say it's got a few problems, but I'll let someone who knows more about psychoacoustics than I do comment on it.