Hardly anyone prefer something they consider inferior. Is that a problem?
You can show a false preference with zero difference for a number of reasons. So you determine difference first.
As usual, before doing an experiment (and designing an experiment) a hypothesis/question to be examined shall be stated. Unfortunately it is quite seldom done in these audio tests. Searching for an estimate for the underlying distribution in a population can be the basis for an experiment, but usually in these multidimensional evaluations it is more like "can anyone detect a difference" or also quite often "can this listener detect a difference".
We have discussed these matter in other threads already quite extensively so it can be that I am sometimes writing not detailed enough.
It is not so much about a flaw on any specific test protocol but more of the experiment using a specific test protocol without considering the impact this protocol might have.
We know that any test environment is artifical wrt normal listening and we know that within this artifical environment the impact of the various test protocols differs in addition.
Therefore the meaning of training is twofold in this context, it can be denoting the experience of listeners wrt to the sonic effect under test, but also the familiarization of the listener with the specific conditions of the test.
My "usual analysis" was related to the statistical analysis of tests with a dichotomous outcome like ABX or A/B or Triangle. Usually the analysis is based on the number of correct responses used for doing an exact binomial test under the assumption that the null-hypothesis is true and then to decide if the observed data is more or less compatible with this assumption. The intended level of significance (statistical significance) choosen upfront gives the decision rule for rejecting or non-rejecting the null-hypothesis.
The interesting point of the table excerpt I've posted is, that the number of correct responses is quite different and therefore, if doing the "usual statistical analysis" as described above, rejection of the null-hypothesis depends not only on the detection abilities of the participants but on the test protocol used.
While the null-hypothesis (when using the numbers from this experiment) would always be rejected when using the A/B protocol, it would be so only sometimes for the Triangle and for the ABX.
I'm not sure if we talking past each other about the ABX-protocol. It is listen to "A" then listen to "B" then listen to "X" and finally decide if "X=A" or "X=B"; relistening is allowed but if the participants are not listening consecutively to the three samples it is no longer an ABX - test.
I hope, my paragraph above illustrates where the problem exist; it is the combination of the specific protocol with a certain kind of statistical analysis.
Really detailed documentations are rare, but if existent, iti shows that correct controls are often not applied.
As said above, it of course depends on the hypothesis one wants to examine, but if training under the specific experimental conditions is needed to get correct results which reflect just the audibility is impossible to decide without using appropriate positive controls.
Otherwise risk is high that the test/experimental test conditons are an uncontrolled external variable with unknown impact on the results.
As the excerpt from the tasting experiment shows, the group of participants was well equipped to detect the difference (trained and experienced generally enough for the task of detecting the difference, see the consistency in the replicates) but nevertheless was obviously strongly influenced by the test protocol.
Totally agreed.
I've never seen a price for it, but in this case was talking about the software tool, the so-called "pcabx" afair done by Arny Krueger and downloadable from the website he was running back then. IIRC available alread around ~1999/2000.
Beside doing an ABX without any additional hardware (like in the taste sensory test examples) it was suddenly possible to do it using a PC.
Good news about the Clark ABX; 6- 10 trials can be sufficient if the listener knows what to do and if his detecting abilities are high, 16 trials do not offer an real advantage and replications are always needed.
Totally agreed again.
See my remarks about the different meaning of "training", if not being used to the test conditions it would be more "can any listener detect a difference under the specific experimental conditions" instead of just "can anyone detect a differece" simply based on his listening abilities.
Agreed again,but not just an sound ABX but any good sensory experiments. It needs a clearly stated hypothesis, a really good design of experiments and well execution to achieve correct results.
Could be, but would depend on the hypothesis that should be examined.
Risk of impact due to expectation (being a bit pedantic wrt nomenclature, as Markw4 already pointed out in other threads, the term "expectation bias" denotes usually something different than bias of participants due to their expectations) can be high, but we don't know for sure if the impact was really strong.
Unfortunately humans seem to be always under the impact of lots of different bias effects and most of these are still at work even when used "blinding" in tests.
Last edited:
Gotcha, so a difference test would be a precursor to looking more deeply for a reason, whilst a preference test would be a precursor to.......errrr......something else entirely?
Gotcha, so a difference test would be a precursor to looking more deeply for a reason, whilst a preference test would be a precursor to.......errrr......something else entirely?
No, the difference test would be - when following the test orthodoxy - the precursor for a subsequent preference test and finally finding out what the specific reasons for the preference are.
The preference test allows to shorten the procedure (despite the risk of segmentation that I've mentioned regularly) - as a difference is implicitely given in case of a preference - and to start looking for the reasons for the preference.
Last edited:
It would be interesting if a preference test led to the reason being one of increased distortion, that wouldn't be what I would call "better"
IIRC we've talked about the experiments done by Olson wrt listener preference for extended bandwidth before.
Although overall a clear majority preferred the extended bandwidth reproduction over the low pass filtered version, a surprisingly high percentage (afair ~3x%) of the listeners preferred the filtered version.
Methinks the quite strong intersubject variability was mentioned sometimes before....
Gotcha, so a difference test would be a precursor to looking more deeply for a reason, whilst a preference test would be a precursor to.......errrr......something else entirely?
Whatever. I believe that basically you want to know if there is a statistically significant difference between A and B. I am just pointing out that the question can be phrased in different ways, all leading to the sought answer.
Let us say you have two colors, and you want to know if they are different. You can do an ABX to see if they can reliably identify A and B. Or you can ask 'which one is lighter' to see if they can reliably identify A and B. Many ways to skin etc.
The important point is that it is double blind, ears-only. Or in case of the colors, maybe call it double-deaf, eyes only ;-)
Jan
That's the beastie, thanks, it's living safely in my compoota now.
"The interconnects were then sent to the listener to sort. Unfortunately, the listener decided not to proceed with the test, so this one is still ripe."
Love it!
I suggest you read up on IEEE 802.3 then you will understand it's impossible to apply an audibility test to an ethernet switch or any other networking device.
What is relevant is applying the correct test method to a device, if we learn anything from this thread you would be thinking that audiophiles were connecting a loudspeaker directly to an ethernet switch to conduct an audibility test.
That kinda was my point.
1. Yes, that's what I've been trying to say
2. Oh come on.
The only problem I have with some of the switches (ethernet) would be their power supplies. Most - if not all - have SM power supplies. The quality of the SMPS could (might) cause some noise on the local power lines. Some of the SMPSs for the less expensive ones from China have little or no RF suppression. I can't provide hard data unfortunately.
The only way ethernet data can be corrupted is when an interface, be it electronically or physically compromised, looses connection on the target or source, and then there is no data flow i.e. no signal. As stated so often in this thread by many others, the 803.xx standard says 100% accurate or no data - point, not negotiable. As Indiglo says - READ UP.
- Status
- Not open for further replies.