The double blind auditions thread

[VictoriaGuy]

Generally, the listener has switching control in that sort of format. In other test formats (e.g., sorting), the listener also has control of which items are being auditioned.

In several decades of sensory testing experience, I have never run across the concept of reduced test validity with knowledge of when switching takes place. Do you have a cite for that?

No citation, sorry. The fact that you have never run across this concept shows a lot about the state of 'research' in the audio world, IMO.

As I see it, the basic situation in most 'listening tests' is that the listeners are quite sure that they can tell the difference between the units under test. My attitude is: 'Prove it'.
The listener should be presented with random samples from the 'DUT's. Sometimes the same unit will be presented several times in succession, sometimes the unit will change. The listener should identify 'which is which' to prove that he/she can indeed hear a difference.

To eliminate cues (like counting relay clicks) the audio should be silenced during the switching.

Letting the listener control the switching is a move in the wrong direction, IMO.

Most 'tests' seem to be of the 'Now you are listening to unit A' type, which really don't meet even the slackest criteria for validity.

All IMO.

 

[VictoriaGuy]

This isn't entirely true either since when the "switch" occurs one doesn't know whether the DUT is changed or if it remains the same. In other words, the switching should be as random as possible.

John

That's basically what I meant. For most people, 'switch' means 'change to something different'.

If 'switch' means 'change to something different or just pretend to change to something different but really keep things unchanged', then I agree with your point.

 

[SY]

No citation, sorry. The fact that you have never run across this concept shows a lot about the state of 'research' in the audio world, IMO.

It's also unknown in organoleptics, haptics, and optics (I've worked extensively in the first two). So there doesn't seem to be any support for this assertion.

Most 'tests' seem to be of the 'Now you are listening to unit A' type, which really don't meet even the slackest criteria for validity.

The scare quotes are appropriate. Those aren't tests, they're sales efforts.

Sometimes the same unit will be presented several times in succession, sometimes the unit will change. The listener should identify 'which is which' to prove that he/she can indeed hear a difference.

That is only one possible test format. There are many others which are perfectly valid.

 

[VictoriaGuy]

That is only one possible test format. There are many others which are perfectly valid.

What are some of the other ways to test the subject's claim that he/she can identify different audio components? (ie without presenting the same unit in successive trials, which would certainly happen (eventually) with random presentations...)

 

[sofaspud]

Letting the listener control

Maybe this is a better definition of double-blind? Not that neither party knows, but that neither party controls.

 

[SY]

What are some of the other ways to test the subject's claim that he/she can identify different audio components? (ie without presenting the same unit in successive trials, which would certainly happen (eventually) with random presentations...)

I'll plug Linear Audio again- see my article in Volume 2, to be released in about 2 weeks or so. DBT is absolutely necessary, but random presentation is just one possible format for DBT.

One example which I mentioned before was the data compression trials I did.

 

[wavycertified]

This is a post from member millwood. i think he has the point right on and am totally agree with him

that's absolutely true. the study on "placebo effect" is just picking up right now and a lot of interesting things have come out of it. just a few examples:

a) there is a million dollar challenge to anyone who can tell, in a double-blind test, a reference amp from any of your amps, including low-performance car amps. The challenge has been in existence for at least 10 years and no one has been able to take it.

b) in drug testing, 80% of the FDA-approved drugs are LESS effective than sugar pills.

c) in a recent study, almost all hotel maids lost weight AFTER being told that their workload was harder than exercise, against their control group.

d) in another recent study, patients who did not receive acupunture but was told otherwise reported overwhelming improvement from their "fake" acupunture sessions.

e) in a study done in the 1980s, a group of musicians who swore that they could tell tube amps from solid state amps failed a double blind test. The only person who could consistently pick a tube amp from a solid state amp did so by listening to the excessive hum in the tube amp.

f) Bob Carver, who designed Phase Liner, Carver and Sunfire hi-fi amps, challenged the hi-fi establishment in the 1980s by stating that he could mod his amps to sound like any hi-fi amp. the whole experience, called the "Great Hi-Fi challenge" was done in the 1980s. Carver first replicated the sound of ML-2 for Audio Critic (a high-end audio mag back then), and no Audio Critic editors could tell them apart. Carver then replicated the sound of a Conrad-Johnson (then sold for over $20K, in 1985), in less than two days, and without opening up his own amp, for Stereophile (an audio mag). No Stereophile editors could tell the $500 Carver (solid state) amps from $20K Conrad-Johnson (a tube amp).

g) my own experience suggests that 1) in a controlled environment and assuming the amps are working within their own performance envelope, it is next to impossible to tell two reasonably good amps apart; b) it is extremely easy to tell two speakers apart.

Considering that speakers routinely distort 10%, and amps in the less than 1% territory, it is not difficult to understand why that's the case.

and so far, out of millions of "hi-end" cable / interconnect manufacturers who claim that their products will make a "night-and-day" difference for their owners, not a single one of them have agreed to subject their products and themselves to a double-blind test.

That tells you how much faith they have in their own products.

 

[a.wayne]

I made that suggestion and got a non-answer. Res ipsa and all that.

Then publisch the test and its results, so others can verify it.

I did answer ... What ..?

Yes i saw that and I'm no longer involved in the industry so not exposed to or able to perform such test anymore, well apart from my own personal satisfaction,A/B/ testing ..

.

 

[SY]

Yes, but you claimed to have done the tests in the past. So.. write them up in detail with methods, setup, controls, and results.

 

[a.wayne]

Sy that was 15 yrs ago , them der notes and data are long gone .......

 

[SY]

Surely you remember something about the test setup and controls? Level-matching methods? Anything? Bueller? Bueller?

 

[a.wayne]

BUELLER...?

 

[scott17]

What is the concensus on the proper way to perform a double blind ABX comparison?

Any references?

 

[SY]

See the bibliography I referenced earlier.

 

[scott17]

See the bibliography I referenced earlier.

Thanks, I guess I overlooked that, although I'm not sure how.

 

[twest820]

let's discuss any technical aspects of such tests, like the mechanics of organizing one, equipment needed, procedures, etc..

A procedural point: both I and Stik Erik have measured variations in output SPL up to 2dB from A/Bing different amps. We both responded by adjusting our equalization to bring SPL back to target as 2dB is sufficient to alter the tonality of playback. I'm not aware of any studies which have performed comparative measurements of amplifiers and attempted to control for such basic differences. This is unfortunate as, without such end to end measurements, it's difficult to correlate listener preferences with amplifier behavior and use the resulting information to build better sounding amps.

 

[SY]

On the contrary, most DBTs that I referenced earlier used very careful level-matching. It is indeed critical, and at levels well below 2dB!

 

[twest820]

I looked over the links earlier in the thread before posting the above and didn't see data showing SPL variations between amps or mention of equalization to match the frequency responses. Did I miss something? In my experience level matching usually refers to adjusting volumes to match---usually over an unspecified bandwidth with unspecified frequency weighting---which is not quite the same as using EQ patches to match power spectral densities in addition to matching level.

 

[VictoriaGuy]

which is not quite the same as using EQ patches to match power spectral densities in addition to matching level.

Won't the application of such EQ tend to hide any differences that are hypothesized to exist?
Or, to put it differently: Haven't you proven (or assumed) differences in programme material or electronic devices if you need to adjust 'power spectral densities'?

 

[twest820]

In studies controlling for variable involves determining what portion of variability it explains. A way of doing that in the case of linear distortions is to A/B the amplifiers without EQ, A/B them again with EQ to cancel the differences, and see how listener preferences shift. That's not the same as making assumptions.