No, I fear you just outed yourself on bad experimental design. :/
A major point of protocols is to be immune to biases in the first place (as much as possible), because relying on bias stratification is a fools errand. We all come with biases, openly or not! Using the hopefully well understood double-blind ABX test (https://en.wikipedia.org/wiki/ABX_test, not that it's the only way to test) as an example, how would one rig that test? That is assuming tests near the threshold of hearing, at least. I suppose if the differences were so obvious, one could guess selectively choose the wrong answer in a pattern as to make one's "guesses" appear null. If one gets that wrong however, then that points to the null hypothesis. As to the latter, I'm torturing an example.
Pre-screening someone based on biases as opposed to a legitimate inability to be part of the test can easily become a fishing experiment. (and even then, if your protocol is solid, it should cut through that) And, likewise, listener fatigue is a real concern, so trials have to be focused.
Sqaubblers In A Room Full Of Mirrors - LET'S GET REAL!
Squabbling.
Lost in the woods - round and round and round and round.
Stuck in a "roundabout" or "traffic circle".
On purpose.
Turn the wheel , get off, go to a new destination.
Or just STOP.
Advances the dialog ZERO.
All sorts of arguments about entirely meaningless things.
Let's scroll back.
Points that were ignored (perhaps they might advance the understanding or provide avenues to explore - wouldn't want that?):
- the true significance of Walso's Sousa Band test, what does it mean or tell about how we hear, and how does that relate to the actual performance of the signal chain?
- Mooly's test, bear's suggested test(s), aka string of "10" (in my case) identcial opamps to A/B or ABX. What might that show? Who's going to try it? Who wants to help do it? Who WANTS to even know what it might show? Or do you already know?
- bear's comment on LPF (subwoofer xovers) showing audible differences between opamps, who's interested in that?
- the comments on power supply contributions, and the effect of separate supplies for output stages, etc... (just one or two comments on that)
- the idea of a chip without an output stage, for making ur own with class A bias. Well, no need for that, since they're all indistinguishable anyhow? Right.
- the on chip capacitance, "not mos", but what is it? Is it possible that there are some differences between chip fab and discrete, differences between chip resistors and discrete? How much power can a chip resistor handle (for example) and who's measured it's linearity when working in the chip? (stuff like that , stuff that makes no difference)
[btw, how much "stuff that makes no difference" can you cascade before you have "a difference"?? Anyone know?]
there were more... but it seems like the same bunch wants to just argue, argue, argue the same thing over and over and over. Why not DO something that will more or less demonstrate the thesis that opamps DO or DO NOT sound indistinguishable??
I'm talking about HARDWARE that can be both measured and listened to.
With the design expertise here there's no doubt that by collaborating a virtually faultless method and system(s) (might be more than one single test hardware) could be made. I think that between us all we've got enough discretionary $$ to make it happen, and even ship it around the world need be - of course local copies or regional copies would work.
Let's stop the argument phase and move to the TEST phase?
If there's interest PM me or post here, I'll start a new thread if even ONE person wants to do more than talk. Putcher money where ya mouth is, sonny! Step on up - see the giant Egress!!
I rather doubt that anyone is going to take me up on this - you'd rather waste your time pretending to do, pretending to know rather than doing, or knowing, eh?
Who wants to help do it?
Squabbling.
Lost in the woods - round and round and round and round.
Stuck in a "roundabout" or "traffic circle".
On purpose.
Turn the wheel , get off, go to a new destination.
Or just STOP.
Advances the dialog ZERO.
All sorts of arguments about entirely meaningless things.
Let's scroll back.
Points that were ignored (perhaps they might advance the understanding or provide avenues to explore - wouldn't want that?):
- the true significance of Walso's Sousa Band test, what does it mean or tell about how we hear, and how does that relate to the actual performance of the signal chain?
- Mooly's test, bear's suggested test(s), aka string of "10" (in my case) identcial opamps to A/B or ABX. What might that show? Who's going to try it? Who wants to help do it? Who WANTS to even know what it might show? Or do you already know?
- bear's comment on LPF (subwoofer xovers) showing audible differences between opamps, who's interested in that?
- the comments on power supply contributions, and the effect of separate supplies for output stages, etc... (just one or two comments on that)
- the idea of a chip without an output stage, for making ur own with class A bias. Well, no need for that, since they're all indistinguishable anyhow? Right.
- the on chip capacitance, "not mos", but what is it? Is it possible that there are some differences between chip fab and discrete, differences between chip resistors and discrete? How much power can a chip resistor handle (for example) and who's measured it's linearity when working in the chip? (stuff like that , stuff that makes no difference)
[btw, how much "stuff that makes no difference" can you cascade before you have "a difference"?? Anyone know?]
there were more... but it seems like the same bunch wants to just argue, argue, argue the same thing over and over and over. Why not DO something that will more or less demonstrate the thesis that opamps DO or DO NOT sound indistinguishable??
I'm talking about HARDWARE that can be both measured and listened to.
With the design expertise here there's no doubt that by collaborating a virtually faultless method and system(s) (might be more than one single test hardware) could be made. I think that between us all we've got enough discretionary $$ to make it happen, and even ship it around the world need be - of course local copies or regional copies would work.
Let's stop the argument phase and move to the TEST phase?
If there's interest PM me or post here, I'll start a new thread if even ONE person wants to do more than talk. Putcher money where ya mouth is, sonny! Step on up - see the giant Egress!!
I rather doubt that anyone is going to take me up on this - you'd rather waste your time pretending to do, pretending to know rather than doing, or knowing, eh?
Who wants to help do it?
Last edited:
How would one rig that test? By picking people who had a negative bias towards the devices being tested i.e they expect to hear no difference & that's what will most likely happen. You've admitted yourself that you have a bias towards null results add that to the bias of the ABX test itself towards a null result added to your own admission that certain listening tests you have performed in the past were not done with too much care & could have been better (as is often the case when no differences are expected) & what do you think the results will be?
Is this rigging the test? Yes, absolutely.
Somehow you can only see one side of bias control which is not unusual.
Last edited:
Sure, no test is infallible or all-conclusive.
See, Jakob, we can agree at times. 🙂
Unfortunately small, highly-targeted studies is also a big reason that larger, better-controlled studies that stem from those preliminary studies tend towards smaller effects or nulls.
So, no, you haven't actually looked at how to design experiments that are much more immune to bias?
By your own logic, you don't want people who believe there is a difference either, because obviously they'll screw up the test too. The better answer is to make sure you have a broad mix of people. Selection bias is bias, no?
If you live on a different planet where the laws of physics (or maths) are different from here in the Solar System then your attitude make some sense. Even there, however, science would still work in the same way; unless, of course, you were in a non-rationalistic universe where the results of experiments on stuff depend on things other than stuff. I am going to hazard a guess that you actually live here on Earth?mmerrill99 said:
Ditto! 🙂
Prescreening as bad experimental design falls under the "it depends", no? And very much depends on what you're screening for. For example, if you read through Sean Olive's stuff, I'd very definitely want to use well-trained listeners as opposed to anyone off the street. In fact, I'd be much more likely to exclude myself on that characteristic than my biases. 🙂
My comment was more a warning that larger studies tend towards averaging out said selection biases, among so many other biases. Getting large studies in the clinical space is hard enough--good luck getting that level of experimental control in an AES paper, so we do what we can and temper our enthusiasm to both positive and null results. 🙂
For the types of non-academic, non-commercial audio tests we'd expect to see from DiyA members though, pre-screening is not a readily accessible luxury (I realize your own product testing may be different). And in either case, that's in addition to pinning down the actual experimental procedure.
Again, don't start designing experiments until you've tightly defined the question to be answered. That has yet to be done, so worrying about screenings, participants, whatever, is just wasted keystrokes.
And to interject some levity into this discussion about experimental design, one of the more amusing counterpoints to being overly dependent on purely evidence:
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials | The BMJ
(There's a happy mix!)
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials | The BMJ
(There's a happy mix!)
The link to Wiki ABX testing showed that post-screening methods are used "The methods are primarily used to eliminate subjects who cannot make the appropriate discriminations. The application of a post-screening method may clarify the tendencies in a test result. However, bearing in mind the variability of subjects’ sensitivities to different artefacts,caution should be exercised."
For the same reason pre-screening is used. The objective is to eliminate those who "cannot make the appropriate discriminations" i.e who either because of hearing impairments, null bias towards what is being tested or any number of reasons, are unsuitable as participants.
In a lot of cases pre-screening is more useful as it saves the work involved in post-screening. Why test unsuitable subjects?
As Jokob says it does depend on what question is being asked - in this particular case the question under consideration is can different opamps audibly change the output of an audio device.
How about starting with one amp circuit where opamps could be swapped in and out. The circuit should be measured with precision test equipment with each opamp to show that it produces very low distortion in every case.
Then playback recorded cymbals through each opamp and see if anyone can tell a difference.
If so, fine. We have something to investigate further right there.
If not, the circuit could be designed to allow the opamps to amplify their own distortion somewhat, or to provide high gain of an attenuated signal. The idea is to increase measured distortion, until test recordings show audibly detectable differences in the recorded cymbals.
At that point we can infer that most circuits that have some kind of "opamp sound" probably are associated with some minimal distortion level attributable at least in part to the particular circuit topology.
....That's just off the top of my head. Feel free to refine the idea.
Further research could follow, but at least that would be some starting point.
IOW, test a circuit which is nothing like the intended use or the uses being discussed.
I'm sure there's some kind of logic in that, but I can't fathom what that is.
Circle game.
Many around here will have done that, but might have stopped after they figured out there is often no difference, and if there is, there is a rational explanation.
The logic is to provide a simple way to make adjustments of distortion levels in test files in the context of a search protocol to find minimal distortion levels people here in the forum can hear on their systems at home.
It might be good to identify the most sensitive listeners and further study their limits.
"add that to the bias of the ABX test itself towards a null result"
Where does this tendency towards a null result come into play just because the test is an ABX or AB test protocol? That seems to presuppose that this type of testing gives faulty or misleading results. The only time I have seen complaints about ABX testing is here in the audiophile world where some just don't or won't seem to believe that the testing is practical and accurate in its results. Those who insist on peeking seem to be those who are preconditioned to the end results they want to achieve and personal bias is overwhelming the end result.
Where does this tendency towards a null result come into play just because the test is an ABX or AB test protocol? That seems to presuppose that this type of testing gives faulty or misleading results. The only time I have seen complaints about ABX testing is here in the audiophile world where some just don't or won't seem to believe that the testing is practical and accurate in its results. Those who insist on peeking seem to be those who are preconditioned to the end results they want to achieve and personal bias is overwhelming the end result.
- Status
- Not open for further replies.