What does it take to have the results recognized as conclusive?
Has this site ever organized or sanctioned a specific ABX test?
Do these ever satisfy the detractors?
Where does an ABX have to take place to be recognized as valid?
I am proposing to test Machina Dynamica Brilliant Pebbles room treatment devices in such a way that no one could reasonably dispute the results. I need input on how to setup such a test and have it recognized as definitive. TIA for any assistance and/or counsel.
Has this site ever organized or sanctioned a specific ABX test?
Do these ever satisfy the detractors?
Where does an ABX have to take place to be recognized as valid?
I am proposing to test Machina Dynamica Brilliant Pebbles room treatment devices in such a way that no one could reasonably dispute the results. I need input on how to setup such a test and have it recognized as definitive. TIA for any assistance and/or counsel.
There is another issue you need to deal with, the test participants. Record for every participant IN ADVANE whether they:
- believe there will be no effect;
- believe there could be an effect, but or not sure there is one;
- believe there will be an effect.
You don't want only participants that think there is no effect, because if none is found, that will kill you. Probably you don't want ANY of those. If you have only participants that think there is an effect and none is found, that will give you a strong case.
Note: if no effect is found, it does not prove that the effect does not exist. If the effect is found, it proofs that the effect exists.
Jan Didden
- believe there will be no effect;
- believe there could be an effect, but or not sure there is one;
- believe there will be an effect.
You don't want only participants that think there is no effect, because if none is found, that will kill you. Probably you don't want ANY of those. If you have only participants that think there is an effect and none is found, that will give you a strong case.
Note: if no effect is found, it does not prove that the effect does not exist. If the effect is found, it proofs that the effect exists.
Jan Didden
Record every detail about the test protocol.
Make sure that the test subjects believe that they hear a difference before blind protocols are "turned on."
Have an intelligent and qualified skeptic review the test setup and be present during the test.
Make sure that the test is DOUBLE blind to prevent the Clever Hans effect.
Make sure that your test is statistically valid to establish a 0.05 significance (i.e., sufficient trials and/or sufficient listeners). Don't make the mistake of mixing statistics, that is, setting things up for a group, then trying to interpret statistics for one participant.
Make sure you have controls. For example, if you're placing a jar of rocks on a speaker as "A", the place a similar weight of nonmagic rocks on the speaker as "B."
With that stuff in place and a positive result, you'll have something solid and publishable.
Make sure that the test subjects believe that they hear a difference before blind protocols are "turned on."
Have an intelligent and qualified skeptic review the test setup and be present during the test.
Make sure that the test is DOUBLE blind to prevent the Clever Hans effect.
Make sure that your test is statistically valid to establish a 0.05 significance (i.e., sufficient trials and/or sufficient listeners). Don't make the mistake of mixing statistics, that is, setting things up for a group, then trying to interpret statistics for one participant.
Make sure you have controls. For example, if you're placing a jar of rocks on a speaker as "A", the place a similar weight of nonmagic rocks on the speaker as "B."
With that stuff in place and a positive result, you'll have something solid and publishable.
Thanks SY, the Psychology department is where I'll head.
Any suggestions from anyone on what to test for?
Good/Better
Base/Different
Musical/not
or specific effect
larger soundstage
greater resolution
greater dynamics
My main interest is "does it sound better", adding a subjective element to the test.
Any suggestions from anyone on what to test for?
Good/Better
Base/Different
Musical/not
or specific effect
larger soundstage
greater resolution
greater dynamics
My main interest is "does it sound better", adding a subjective element to the test.
I think it would be best to test only for difference, not better/worse, since that is too subjective and so could possibly average out to nothing over a group, even if they all detect a difference.
I agree that college students seem to be a good choice, being generally enthusiastic and still young enough to retain wide bandwidth ears.
Be aware that there are many factors that affect perception that are not immediately obvious, such as atmospheric pressure, temperature, and even the colour of the walls.
I agree that college students seem to be a good choice, being generally enthusiastic and still young enough to retain wide bandwidth ears.
Be aware that there are many factors that affect perception that are not immediately obvious, such as atmospheric pressure, temperature, and even the colour of the walls.
That's really not true on two levels. First, there are a LOT of music geeks/audiophiles on campuses. Second, what you're talking about is the ability to talk geekspeak, not the ability to hear and interpret musical information. Anyone who listens to music can tell that the instrument positions have moved, that reverberation is greater/smaller, that something sounds shriller or duller, etc.
Heck, I don't know any horn players who are audiophiles, but the ones I know are acutely aware of how changes in the mouthpiece or the reed or the horn material change the timbre and projection of their instrument. Guitar players can readily tell if you're playing a Martin versus a Takamine, even if they can't use the words "air" or "soundstage."
Heck, I don't know any horn players who are audiophiles, but the ones I know are acutely aware of how changes in the mouthpiece or the reed or the horn material change the timbre and projection of their instrument. Guitar players can readily tell if you're playing a Martin versus a Takamine, even if they can't use the words "air" or "soundstage."
I have also found that you need listeners that show that they can provide reliable, repeatable impressions, and preferably receive additional coaching. Note that this is not necessarily equivalent with audiophiles. Most people will not provide reliable, repeatable impressions, and as a group, audiophiles are not exempt. In fact, my experience has been that it is a _big_ waste of time to ask most people to serve as listeners. However, a (very) limited number of individuals are quite good, and these you should sign up as part of your regular listening panel.
FWIW, I have not had good results with high-school or college students at all.
FWIW, I have not had good results with high-school or college students at all.
AFAIK ABX can take place anywhere and take as long as the participant wants to in order to get IIRC 18 responses. In order to be meaningful 16 of 18 responses have to be correct to insure a 95% confidence, IOW better than guessing.
Nothing will ever satisfy the detractors, because no matter how much you point out what has been done and how widely accepted it is they CHOOSE to disbelieve.
You would be well advised to post some of your questions on how to conduct the comparisons and how to have it proctored in rec.audio,tech. Arny Krueger who designed the first ABX comparator and Tom Nousaine who as well as being a freind of Arny's has conducted many ABX comparisons. Both of them are regulars there.
Good Luck
[/COLOR]
It's not that the test is not strong enough, it's that any difference that might exist is not strong enough to be audible to the person doing the comparison. For that person it means that there is no difference.
It doesn't apply to anyone else but the person doing the comaprison.
It is however true that certain devices that operate within close enough parameters, have never had anyone who could tell a difference between them. This doesn't mean that it's not possible that someone might be able to tell a difference, only that so far nobody has.
Arthur-itis said:
No, the test is not strong enuff. The only statistically valid result in an ABX test is that 2 DUTs are different. If the test shows no difference, then you cannot conclude anything. An ABX test cannot be used to show that 2 DUTs sound the same, because it is not strong enuff. It cannot even conclude that 2 DUTs are the same when A=B.
dave
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.