Where should we focus on if we want to build a good hifi-system

[jeff mai]

Let me guess- you've never done sensory research, have you? These aren't at all requirements of a properly designed test.

No, I haven't done sensory research. I'm sure those aren't all the requirements, but they are requirements. How does this detract from the point I was making?

 

[SY]

No, you misread me- I said that these aren't necessarily requirements. For example, if the diyer is there and doesn't know what X is in an ABX format, it doesn't matter how much people want to please him, it's a fair test. Likewise, the test subjects absolutely can know that they're comparing, say, amp A to amp B as long as they don't know whether "X" is A or B. In fact, in my own tests, I made sure that the subjects believed that they heard a difference between A and B before blinding.

Sorry if my wording was unclear.

 

[keyser]

Let me guess- you've never done sensory research, have you? These aren't at all requirements of a properly designed test.

What are then? How can we properly test equipment?

 

[jeff mai]

No, you misread me- I said that these aren't necessarily requirements. For example, if the diyer is there and doesn't know what X is in an ABX format, it doesn't matter how much people want to please him, it's a fair test. Likewise, the test subjects absolutely can know that they're comparing, say, amp A to amp B as long as they don't know whether "X" is A or B. In fact, in my own tests, I made sure that the subjects believed that they heard a difference between A and B before blinding.

Slipping in the proviso in this final sentence makes it seem peripheral to the argument (your wording makes it seem optional), but without it the entirety is incorrect. You should have given it more emphasis so as not to mislead anyone.

If someone not expecting to hear a difference between A and B knows what A and B are, then they may not hear a real difference due to their preconceptions about A and B. Similarly, if the test subjects know that the DIYer is predicting there will be no difference, they can be affected by this knowledge. Without the proviso you added about ensuring that the subjects thought they heard a difference in sighted tests, the tests are worthless.

If you have negative preconceptions about equipment differences, you are essentially test proof!

 

[SY]

If you don't hear a difference sighted, you won't hear it blind. That's not profound. I also failed to mentioned that the subjects can't be deaf or wearing earplugs. And that the source material be suitable, etc, etc, etc.

The point is that in a proper blind test, it doesn't matter if there's someone that people want to please or someone with a stake in the outcome. It doesn't matter that everyone knows what's being tested. That's OK, in fact, that's preferable. It only matters that no-one in the room can know at any given moment the identity of the unknown except by listening. Science is thus served- you've got a valid subjective test.

The conditions you outlined are not only unnecessary, they're actually inhibiting.

 

[ThorstenL]

Konnichiwa,

No, you misread me- I said that these aren't necessarily requirements. For example, if the diyer is there and doesn't know what X is in an ABX format, it doesn't matter how much people want to please him, it's a fair test.

Actually, I MUST disagree strongly. If the test subject KNOWS what A and B are and has a definite preconceived notion what A & B should behave like the test is not only not "fair", it is completely useless and inconclusive.

I have PERSONALLY used blind AB testing (including a sufficient "X" Factor in cases where I had simply no definite opinion and wanted to find if "A" or "B" came 1) Closest to the reference (usually a straight wire) and 2) which one I preferred disregarless of 1). Such testing I felt would really be served well only if I really did not know what I was listening to.

Also, if you compare the "marginal notes" in some ABX testing (where they are used) you may find that while subjects cannot reliably identify A or B from X, their subjective comments about whatever X was at the time uncannily match the identity of A or B (I found this at a blind test of different 300B Valves which constituted part of my 300B Test for ETM).

Anyway, I personally the subject is a little more complex than you wish it to be, for audio as much as it is for wine tasting.

Sayonara

 

[SY]

If the test subject KNOWS what A and B are and has a definite preconceived notion what A & B should behave like the test is not only not "fair", it is completely useless and inconclusive.

Either they can be distinguished (i.e., X can reliably matched to A or B in the ABX format by at least some subset of listeners) or they can't. There's nothing unfair about that at all.

 

[jeff mai]

Either they can be distinguished (i.e., X can reliably matched to A or B in the ABX format by at least some subset of listeners) or they can't. There's nothing unfair about that at all.

You're sidestepping our concerns here.

Explain the mechanism by which negative preconceptions (excpecting no difference between A and B) are not important in an ABX test. I'm not a sensory test specialist, but your assertions defy logic.

 

[MBK]

Actually it certainly helps if the subjects don't know what they are tested for. That means for instance, that they are supposed to judge the performance of the musician, for instance, or if they are told they have to understand some spoken material, while in fact the experimenter wants to know about playback quality. If they know the test is about playback quality, they might a) freeze up and b) their preconceptions create an additional unknown: positive or negative bias.

 

[analog_sa]

Actually it certainly helps if the subjects don't know what they are tested for. That means for instance, that they are supposed to judge the performance of the musician, for instance, or if they are told they have to understand some spoken material, while in fact the experimenter wants to know about playback quality. If they know the test is about playback quality, they might a) freeze up and b) their preconceptions create an additional unknown: positive or negative bias.

From the perspective of a test subject i would find such a test incredibly difficult.

 

[MBK]

Why? The test subject just focuses consciously on something else, not the actual parameter under test. That leaves him relaxed as to the actual parameter, and all else stays the same (he believes that *something* matters, just that in reality it's something else).

Imagine Milgram would have truthfully told his subjects that he wants to know about obedience to authority, in his famous experiment. It wouldn't have worked. It would have made the subjects conscious of the parameter he was testing them for.

 

[pinkmouse]

To test if some people's negative preconceptions play a part, you can simply see if there is a statistically significant difference in their responses to those of others. It just requires a large enough test sample and a well designed questionaire. Not difficult at all.

 

[Teun]

shall we start a new thread on how to set up the perfect reference test environment, excluding all human perceptions, feelings and character? maybe do it on the space shuttle in zero gravity and in a vacuum? I feel there is only one hard refernce to be made and that is that any reproduced sound should sound the same as an unplugged (fully) session in a perfect sound room. Other then that, there is no way to establish whether we are subjective, objective or just plain ornery

 

[jeff mai]

To test if some people's negative preconceptions play a part, you can simply see if there is a statistically significant difference in their responses to those of others. It just requires a large enough test sample and a well designed questionaire. Not difficult at all.

This presumes they are aware of and forthcoming about their negative preconceptions. Sounds tricky to me.

But still you are talking about large test samples - how does this help the lone DIYer who wants to evaluate changes to his gear?

 

[MBK]

The question of the perfect test only applies if you want to get "universally applicable", scientific results. Doing good tests in this context usually DO take the time, effort and expertise of a PhD thesis.

The question "what should the average DIYer do", as I said above, only has personal preferences as answers. I measure what I can measure, I try out things I can't measure as well, just out of curiosity, and I sometimes seem to prefer modifications, over the long run, which I could not reliably identify in a crude, private, unscientific blind test.

Ultimately, why DIY at all? 1- cost or 2- results? or a third reason, the 3 - enjoyment of the process? For me, all three count, and possibly number three the most.

(actually proper DIY will cost you as much or more than just buying the stuff).

 

[keyser]

what would a scientific audibilitie test look like? Lets cut the observationcrap and go formulating a hypothesis. The null hypothesis, or no-difference hypothesis, is a good type of hypothesis to test.

"There are no audible differences between good amplifiers"

but how do we define 'good amplifiers'?
We measure the amplifiers. Then we have to simply agree which forms and amounts of distortion are allowed in a good amplifier. (for example, we could agree that a good amplifier has no more than 0,5 percent THD. And maybe a frequencie response from 20 to 20000 hertz within 1 dB. etc.)

Next step is to design an experiment. This appears to be the tricky part! There is nothing wrong with using DBT's in any branch of science, so the DBT's are perfectly applicable for testing our hypothesis.
The problem is the test candidate. The candidates' bias and prejudice. If the test subject is convinced there are no audible differences, he will hear them neither sighted nor blind. If the test subject believes there ARE differences he IS useful for the test.

Easiest (and completely scientifically valid) is the ABX test. The test subject is allowed to switch back and forth between component A and B as many times as he'd like, untill he is certain he can distinguish them. Then the subject switches to X, which is either A or B, randomly selected. This sequence has to be repeated many times to eliminate chance. If the subject is able to identify which component X is significantly more than random selection would do, the hypothesis is to be rejected. If not, the hypothesis is accepted. A hypothesis can be disproven, but never proven. This means we can not prove that there are no audible differences between good amplifiers.

 

[SY]

Explain the mechanism by which negative preconceptions (excpecting no difference between A and B) are not important in an ABX test.

Jeff, you're not reading what I've written. Preconceptions, no doubt

I wrote "If you don't hear a difference sighted, you won't hear it blind. That's not profound." Is that clear?

I also wrote "I made sure that the subjects believed that they heard a difference between A and B before blinding." Is that clear?

To recap: One does a blind test to see if, first, the test subjects can distinguish between A and B (as opposed to believe they can), then one can move on to establishing preferences if the results of the first test are positive.

In the case of claims such as "silver plated wire sounds brighter than copper" or "electrolytic coupling caps sound grungy and distorted compared to polypropylene," one uses test subjects who claim to hear that difference under sighted conditions. Under blind conditions, they can either distinguish the two wires/caps or they can't. Nothing complicated about that. They certainly can know (and under this scenario, they should know) that they're comparing copper to silver wire or electrolytics to polypropylenes. That won't bias the test in the slightest; on the contrary, if someone feels that they're attuned to those differences, they know what electrolytics or silver sound like, it gives them a benchmark to listen for.

 

[jeff mai]

So when you said this:

It only matters that no-one in the room can know at any given moment the identity of the unknown except by listening. Science is thus served- you've got a valid subjective test.

The conditions you outlined are not only unnecessary, they're actually inhibiting.

You were assuming this as a necessary condition:

I also wrote "I made sure that the subjects believed that they heard a difference between A and B before blinding." Is that clear?

It looked as though you were denying the importance of this condition - which was, BTW, one of the conditions I outlined. Contrary to your statement above, I guess at least one of the conditions I outlined is necessary.

I'm not denying the validity of scientifically applied ABX testing methods. I'm questioning someone (not you, SY) saying "I did a blind test and I heard no difference" as evidence that all good amplifiers sound the same (or whatever.) The chance that such an uncontrolled test was not influenced by bias is almost nil.

 

[slowmotion]

Hi all

But still you are talking about large test samples - how does this help the lone DIYer who wants to evaluate changes to his gear?

The question "what should the average DIYer do", as I said above, only has personal preferences as answers. I measure what I can measure, I try out things I can't measure as well, just out of curiosity, and I sometimes seem to prefer modifications, over the long run, which I could not reliably identify in a crude, private, unscientific blind test.

In among all this useless verbal masturbation two posts touch upon the real subject at hand. Thanks , Jeff and MBK !

Ultimately, why DIY at all? 1- cost or 2- results? or a third reason, the 3 - enjoyment of the process? For me, all three count, and possibly number three the most.

(actually proper DIY will cost you as much or more than just buying the stuff).

Why diy? Yes, indeed.

Keep in mind that a lot of store bought equipment is rubbish,
all of it small variations of the same mold. If you put your mind to it
you can build diy gear that vastly outperforms comercial offerings.
That means you have to think outside the mold, tho.

And of course the mental part of designing your own gear is very stimulating

cheers

 

[keyser]

I'm not denying the validity of scientifically applied ABX testing methods. I'm questioning someone (not you, SY) saying "I did a blind test and I heard no difference" as evidence that all good amplifiers sound the same (or whatever.) The chance that such an uncontrolled test was not influenced by bias is almost nil.

I think I might know who you are referring to Jeff.
I have not claimed that my the test I did had any scientific validity. It is no cold hard evidense, but an argument based on personal experience.
The first couple of blind tests I did, must have had some validity, as at that time I was still a firm believer in the benefit of high-end audio gear and special cables. In those first tests I thought I heard the difference every time, but it apeared I was wrong almost as many times as I was right. When I started I believed I would be able to hear the differences easily.
I think that those first tests would have been scientifically valid if we would have done more tests on more people like me. ATleast my prejudice could not have negatively effected the outcome of the test.