john curl said:
ABX gives different results than informal sighted results. Many people cannot accept these results, so the only avenue open for them is to say ABX doesn't work. Just like that. Sad.
Jan Didden
salas said:
This is just another anecdotal story. Or do have some backup that this actually is true?
For instance, aren't there any audiophiles that select the speaker on the acceptance by their other half? How about the amp that his friend has bought and raves about? You should take to a seasoned marketing guy.
Jan Didden
Any A/B is a test for preference and ABX is a test for difference.
A preference test leads to the same conclusion as a difference test; if a preference is verified by the test result than there must have been a difference.
One could argue that a test for preference is a bit more natural to this field, as we all are used to choose new equipment due to preference decisions.
Both somewhat official recommended listening test procedures (ABC/HR ITU-R BS.1116 and MUSHRA/ITU-R BS.1534) are tests for preference.
Jakob
john curl said:
But you always know which one is playing - A or B. I am sorry but this is equivalent of a sighted test if all you try to conclude is whether there is a difference.
I'm sorry it doesn't and it can't. If all you need to tell from the experiment is if A is different from B, and you always know when you are listening to A (or B), you may just as well have fully sighted test.
Is this really that difficult to see ?
Well, it is not that obvious as you want to make it. In fact, as I've shown above that looked separately, ABX is absolutely equal to your A/B test, with one important distinction - you do not know which amp you are listening to. All you do in ABX is picking the difference between two amps. Just as in A/B.
Hey, look around. That is exactly what industry is doing. Had a look under the hood of a Jeff Rowland amp lately ? Notice how many awards switching designs are getting these days ?
No I have no idea, just as you don't have any idea what kind of electronics design I have done in the last 30 years.
There are the areas of society where improvements are real (cars), and there are areas where improvements are superficial (wristwatches). Market is a complex, living mechanism where perfection can, and does have completely different meaning.
Bratislav, in a A/B test the listener get a choice between two or more stimuli. If these stimuli are randomly allocated to A,B.....Z then it is a blind test.
The listener is forced to rate each stimulus, that can be done by comparison to each other or by comparison to a open reference.
After the number of trials choosen, the results were added and if there is a preference for one of the stimuli than there must have been a difference, otherwise the rating would be distributed if rated by chance.
Jakob
The listener is forced to rate each stimulus, that can be done by comparison to each other or by comparison to a open reference.
After the number of trials choosen, the results were added and if there is a preference for one of the stimuli than there must have been a difference, otherwise the rating would be distributed if rated by chance.
Jakob
Jakob2 said:
OK, if that is the understanding, fine.
Jakob2 said:
Unless this was also a dbt, the preference can be caused by any differences, not just sound.
Jan Didden
Janneman, sorry for not expressing that clearly- i´m talking about blind or double blind tests in this discussion.
Jakob
Jakob
Janneman, what i wrote about A/B was a bit misleading, because in case of two stimuli you could do it as a difference test as well.
By asking the listener after a switch if he votes for same or difference.
In that case you have to include nill tests (switch from A to A or B to B).
It is known (see for example Nousaine in his AES-Convention Paper) that there is a tendency to answer ´different´even if ´same´would have been the correct answer.
This preference for answer ´difference´leads to up to 75% wrong answers for the case `same`.
As there is no way to deal with this fact, it is best to avoid the different/same protocol and therefore any A/B should be a preference test.
Jakob
By asking the listener after a switch if he votes for same or difference.
In that case you have to include nill tests (switch from A to A or B to B).
It is known (see for example Nousaine in his AES-Convention Paper) that there is a tendency to answer ´different´even if ´same´would have been the correct answer.
This preference for answer ´difference´leads to up to 75% wrong answers for the case `same`.
As there is no way to deal with this fact, it is best to avoid the different/same protocol and therefore any A/B should be a preference test.
Jakob
Jakob2 said:
You don't have to rate anything, really. All you need is to detect any difference between two amps in the test.
In A/B test you know amps are different upfront (there is A and there is B). In ABX variant you don't; first you have A/X and then B/X.
I'd say if you can trully detect differences between A and B, you should do just the same with A/X (or B/X), becasue one of these two combinations is going to be the same A/B comparison. Just you won't know which one upfront like with sighted test.
Bratislav, i see your point, but if you decide to do some blind test because humans are imperfect, you have to be aware of these imperfections under blind test conditions. Everything else would be a ´contradictio in ratio`.
So if the specific ABX protocol seems to confuse some listeners, fine, choose another one, that seems to be better in this regard.
And if it seems, that training is needed to reach a high sensitivity level in a blind test- fine train the participants.
In the end the experimentator has to show that his test has reached a sufficient level of sensitivity.
And otherwise every test protocol is acceptable if the test done was objective, reliable and valid.
Jakob
So if the specific ABX protocol seems to confuse some listeners, fine, choose another one, that seems to be better in this regard.
And if it seems, that training is needed to reach a high sensitivity level in a blind test- fine train the participants.
In the end the experimentator has to show that his test has reached a sufficient level of sensitivity.
And otherwise every test protocol is acceptable if the test done was objective, reliable and valid.
Jakob
Btw, Dave Moulton has written some quite interesting articles about a lot of related subjects. Of course his point of view is a bit different due to his profession:
He did an amplifier blind test in 1994:
http://www.moultonlabs.com/more/what_is_the_sound_of_one_amp_clipping/P0/
and a follow up in 1997:
http://www.moultonlabs.com/more/power_amps_revisited/
And this one is about describing a _huge_ difference 🙂 :
http://www.moultonlabs.com/more/yeah_i_think_i_can_hear_it/
Jakob
He did an amplifier blind test in 1994:
http://www.moultonlabs.com/more/what_is_the_sound_of_one_amp_clipping/P0/
and a follow up in 1997:
http://www.moultonlabs.com/more/power_amps_revisited/
And this one is about describing a _huge_ difference 🙂 :
http://www.moultonlabs.com/more/yeah_i_think_i_can_hear_it/
Jakob
Jakob,
Interesting listening, thanks.
I did note that the follow up was fairly informal and anecdotal though. But he seems a very thoughtfull guy that's hard to fool.
Jan
Interesting listening, thanks.
I did note that the follow up was fairly informal and anecdotal though. But he seems a very thoughtfull guy that's hard to fool.
Jan
janneman said:
Yes its obvious that when we talk commercial products we talk social distortion too . But better or not, getting the $$$ golden fascia one that is actually performing worse than the X cheap plain fascia one, does not mean that they perform the same. Getting the wrong one, does not mean that it does not have a distinguishable character. A character that a salesman can even talk us in to thinking is the correct one. I am not talking what is to really get the best for music. I say they do sound different.
''Have your friend flip a coin ten times to determine whether X is going to be A or B for each of ten trials. Then have him/her run the switch for you, playing A and B at your request and then X. Write down your answer before your friend tells you which X is. Your friend should be out of sight behind you, and he/she should be absolutely quiet and still while you are doing this. You will find you become fiercely competitive at this, and absolutely sure you can do it better, and you will start keying off noises, facial expressions, anything you can to get the right answer. This is fatiguing and difficult work. Take a break after each set of ten. Quit after 100 trials, because by that point your brain will have acquired the consistency of overripe zucchini Jell-O.''
Thats from Moulton. Tests do press the brain.
Thats from Moulton. Tests do press the brain.
Anecdotes? Everything that we can know is justifiable with anecdotes. The problem with ABX tests is Type 2 errors, they just go through the roof! Check it out and see for yourself, but until you have tried it, don't push it as the ONLY way to know reality.
Sometimes I’m confused about what is being discussed. I would never say all amplifiers sound the same. The best system I ever listened to was heavily colored to accommodate the tastes of the owner (nothing but RCA LSC’s and Mercury’s). It was a classic 80’s “Absolute Sound” setup, all tube etc. The world is full of people who, when you get right down to it, don’t want the AES idea of accurate sound. I thought the spin on “all amplifiers sound the same” was directed at a different crowd.
As for ABX, I have never seen a “cheat” proof setup. As was said, you look real hard for clues to help you “win”.
As for ABX, I have never seen a “cheat” proof setup. As was said, you look real hard for clues to help you “win”.
Scott, what is, is! IF ABX testing won't give you differences, then you are forced to 'cheat' just to justify your own real experience, since the differences are still there, when the ABX test is removed.
By the way, both you and Bob Cordell went to the AES and the ISSCC. Learn anything that I could use? Anything that anyone into audio design could use? Inquiring minds need to know!😉
By the way, both you and Bob Cordell went to the AES and the ISSCC. Learn anything that I could use? Anything that anyone into audio design could use? Inquiring minds need to know!😉
john curl said:
Not unless you want to make an MP3 player on the head of a pin that draws 50uA from 1.1V that directly feeds an ear implantable MEMS speaker.
🙂
- Status
- Not open for further replies.