To add, what is normally done is to define for yourself how sure you want to be that your conclusion is correct.
Let us say you do 20 tests, and you want to be 95% sure that the conclusion you draw from the test is correct.
That would mean that 17 or 18 results out of the 20 have to be right. If you DO get 18 right, you can say: I am 95% sure there is an audible difference.
If you would be happy to be only 80% sure your conclusion is correct, you only need 15 or 16 results right. If then you DO get 15 right, you can say: I am 80% sure there is an audible difference.
Note: I don't know if these numbers are right, I just want to illustrate the concept).
Certainty is almost never provided by Mother Nature
Jan
Let us say you do 20 tests, and you want to be 95% sure that the conclusion you draw from the test is correct.
That would mean that 17 or 18 results out of the 20 have to be right. If you DO get 18 right, you can say: I am 95% sure there is an audible difference.
If you would be happy to be only 80% sure your conclusion is correct, you only need 15 or 16 results right. If then you DO get 15 right, you can say: I am 80% sure there is an audible difference.
Note: I don't know if these numbers are right, I just want to illustrate the concept).
Certainty is almost never provided by Mother Nature
Jan
Last edited:
I believe any extreme difference from 50:50 has significance - consistently wrong is evidence that you are hearing a difference - just that you are (un?)consciously missclassifying it
short listening sessions to prevent fatigue can be done right - you have to commit beforehand to a plan, not throw out any of the listening plan's results after the event to avoid cherry picking
short listening sessions to prevent fatigue can be done right - you have to commit beforehand to a plan, not throw out any of the listening plan's results after the event to avoid cherry picking
Last edited:
Quite true, but equally true one couldn't say the test subject wasn't able to hear a difference a few times.
However, that doesn't necessarily mean one is stuck with no way of figuring out the correct answer.
Please consider than being able to describe the sound of an opamp in a way that correlates with measurements half of the time is not the same as flipping a coin and getting it right half of the time.
For a coin it is usually assumed there are only two possible outcomes, heads or tails.
For describing the sound of an opamp, and for measuring an opamp, there are many more than only two possible descriptions/measurements.
To make the difference more clear, suppose there was a machine that flipped a coin, and the test was to call heads or tails. But, what if someone came along who could only call heads or tails correctly about half of the time, but could also tell you, say for example, the date stamped on the coin about half the time without ever seeing the coin (perhaps by hearing it land or something but we don't know how yet).
If something like that were to happen, it would probably be a mistake on the part of the experimenter to assume that the test subject was guessing at everything. It might suggest a need to perform different tests to better understand exactly what the test subject is and isn't able to do.
Of course, somebody might want to point out that if I make an analogy between coin flipping and naming a date on some of the coins, that is not perfectly analogous to the listening test and it's results. I would have to agree with that. But I didn't bring up the coin flip analogy, I'm just trying to explain how it isn't a very good analogy for what has happened so far in the case of this particular listening test. There are many other ways to try to explain that, and I happened to pick this one. I hope it makes some kind of sense to Jan.
Last edited:
@jcx, Jakob2 has stated there is some research regarding ABX listening tests that he thinks applies to this situation. He said that ABX can work well only if there is significant training and practice with it.
I haven't seen the research he referred to, but if it exists it would seem to make sense to me. At the outset here, I thought ABX should be workable. When I tried it I found myself not as relaxed as think I need to be to do well at listening. I don't have direct control over autonomic nervous system functions, so I can't force myself to relax if some part of my body is not cooperating. Meditation techniques might help, and slowly developing more comfort and confidence with ABX seems like it ought to help too.
So, at this point I still suspect ABX could work, but I it doesn't look like anyone here put in the time and effort that might be needed.
I would agree with what you said about picking wrong and misclassifying, and I would add it would probably be due to unobservable or only indirectly-observable System 1 brain processing.
I haven't seen the research he referred to, but if it exists it would seem to make sense to me. At the outset here, I thought ABX should be workable. When I tried it I found myself not as relaxed as think I need to be to do well at listening. I don't have direct control over autonomic nervous system functions, so I can't force myself to relax if some part of my body is not cooperating. Meditation techniques might help, and slowly developing more comfort and confidence with ABX seems like it ought to help too.
So, at this point I still suspect ABX could work, but I it doesn't look like anyone here put in the time and effort that might be needed.
I would agree with what you said about picking wrong and misclassifying, and I would add it would probably be due to unobservable or only indirectly-observable System 1 brain processing.
Last edited:
Hmmm. I have trouble parsing the point you are trying to make, but that may be me.
I believe that muddling a test by introducing yet another variable (the 'sound' of the opamp), which also is very much arbitrary and undefined, is making the whole exercise somewhat useless.
I did not claim that flipping a coin is or is not a good analogy for what happened so far. I used the coin flip in the context of explaining how to interpret results from a controlled test.
Jan
Last edited:
critical listening is not an easy task, you can try it yourself using this app:
Harman How to Listen: Welcome to How to Listen!
it is hard to achieve higher levels there.
Harman How to Listen: Welcome to How to Listen!
it is hard to achieve higher levels there.
(1) true.Markw4 said:
(2) Most people have an aim in mind when they design a test. If it is a good test it will show whether they are right or not. Not having an aim is a good way to design a bad test. Pano could not have already reached "conclusions", but he hopefully did have an aim.
(3) If the differences are very small then they have very little importance so it doesn't matter if, as you suggest, ABX is not very good at spotting them. The usual whine is that ABX testing makes 'night and day/wife in kitchen' huge differences magically disappear. The whiners need to get their story straight.
(4) see (3)
Maybe I am being dim, but I thought the point of ABX was to see whether someone can reliably tell the difference between A and B unsighted. It is not supposed to be like athletics, where you train to get better. It is not surprising that training can bring a person's actual performance nearer to their imagined performance, but that is surely not the point. If someone can already genuinely tell the difference between A and B unsighted then ABX should confirm this. If they cannot, no matter how fervently they believe that they can and believe that their sighted performance is not being affected by sight, then ABX will show this. It is unsurprising that ABX is criticised for doing exactly what it claims to do. It is unsurprising that a test which determines whether people can actually hear differences they claim to be able to hear should create stress. Stress would be reduced if people approached it with the right frame of mind: seeking truth rather than seeking vindication.Markw4 said:
The result of the ABX test is only meaningful when someone passed the ABX test. If one failed, it could mean 2 things, 1) he could not hear the difference at all, or 2) He could hear the difference, but sometimes not.
As I said, subtle difference can't be heard all the time. You are ignoring the zone between "can" and "can't". Why you are so sure that we human can tell the difference consistently all the time? Human's sense organ is not that consistent in my experience. It is very possible that one can pass the ABX test one day, and the same person fail the same test next day.
This is the wrong idea. Nobody has to pass the ABX test. That is a meaningless concept.
It is not a test of a person. It is a test to find out if there are audible differences between equipment. The result is either an audible difference is heard, or is not heard (with the usual statistical bounderies of course). It is not a pass/fail test at all.
Jan
What plasnu probably meant was that if someone passes an ABX test, you can be fairly sure it is because they did hear differences.
But if someone doesn't pass an ABX test, you can't know if they heard any differences or not. They may have heard a difference some times, but not every time. Or, maybe they never heard any difference and guessed every time.
And, if they score much less than 50% right on an ABX test, as jcx said, they probably aren't guessing, but rather they probably hear something yet somehow their brains are confused and answering backwards.
Last edited:
Yes. For some people that would seem to be the case.
It may also be true that a very few people can hear the small differences more reliably than others, and for the best listeners, it may not seem all that subtle. We just don't know for a fact, I would say, since we haven't really studied all this very well in the past.
With better listening tests, and by testing more people, we should develop a better idea of what is possible for the very best listeners.
The whole concept of 'passing an ABX test' is nonsense. What are the criteria to pass the test?
No! If the heard a difference (much) less than 50%, it only means that there is no reliable detectable difference. Everything else is speculation.
Jan
Last edited:
Jan, not at all. If guessing, the score should be 50% right, not much more or much less.
For a moment, let's consider flipping a coin. If a machine or someone flips a coin, and a test subject can't see the coin, then they should guess right 50% of the time and guess wrong 50% of the time. That's what chance dictates.
But if someone guessed right 90% of the time or guessed wrong 90% of the time, for example, neither one of those things should happen by chance alone or by guessing. It should be just as impossible to guess wrong 90% of the time as to guess right 90% of the time.
If something that should be impossible by guessing happens, then it is very unlikely guessing is the right explanation. It would be mathematically correct to expect there is some causal reason, even if we don't understand what it could be.
For a moment, let's consider flipping a coin. If a machine or someone flips a coin, and a test subject can't see the coin, then they should guess right 50% of the time and guess wrong 50% of the time. That's what chance dictates.
But if someone guessed right 90% of the time or guessed wrong 90% of the time, for example, neither one of those things should happen by chance alone or by guessing. It should be just as impossible to guess wrong 90% of the time as to guess right 90% of the time.
If something that should be impossible by guessing happens, then it is very unlikely guessing is the right explanation. It would be mathematically correct to expect there is some causal reason, even if we don't understand what it could be.
Last edited:
Sure. Two comments: guessing 90% right (or wrong) can happen - it is just not very likely. People do win the lottery. A string of a dozen 'red' at the roulette does happen, once in a century.
The implication is that when somewhen give 90% correct answers in an ABX test, chances are overwhelmingly (but not 100%) that he indeed heard an audible difference.
But even with 90% correct answers we are not absolutely sure that the audible difference is there - just because there is a very slight chance that by guessing you could get 90% correct.
That is why with such a test there is always a statement like: 'there was an audible difference with a 95% certainty', or something like that. The statisticians call it the confidence interval.
Jan
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.