janneman said:
Of course it would, as the result of every test is a valid and (maybe) important outcome.
Really questionable is what reason has led to a certain outcome.
If, in your example, the nil hypothesis could not be rejected, then it would be the question why.
There is no chance to answer that question if no controls were included in the test. Of course your answer would (maybe) be that there exists no audible difference, but that would only be based on your own bias, because the test setup has not addressed this question in a scientifically proper way.
If you do not present known audible differences as a control you know nothing about the audibility of a (maybe existing) difference, despite the fact that it most certainly is not "earthshaking" .
My personal bias is, that it´s better to do preference tests (like A-B, ABC/HR or MUSHRA), but i would expect that it is possible to adopt to the somewhat unnormal ABX-Procedure as well, but the learning cure might be more prolonged.
But in the end, that´s just a hypothesis and the proof lies in the control. 🙂
It would be much smarter to give to participants switches so they can switch between A and B, rewind and repeat if needed, then write conclusions. Even visual comparison between 2 pictures standing together requires switching since our memory is not ideal. It is completely stupid to expect somebody to remember sound details to compare in order to find subtle differences.
For scientific purposes A and B may be randomly assigned.
For scientific purposes A and B may be randomly assigned.
This is very similar how i do it. The listeners do not see the instrument tested.
I say "this is A", "this is B". Tell me how long you want to listen to each and when to change.
I say "this is A", "this is B". Tell me how long you want to listen to each and when to change.
Minor recommendation: Don't say it, write it down, then have a third party distribute the instructions.PMA said:
People are amazingly good at reading facial or body language cues, If you have a personal preference many will discern it.
salas said:
On another forum, I mentioned an adjunct to this point, which is:
"There are no experts--there are only people who believe in them."
The statement is taken out of context from that other forum, but it can apply in many cases. Specifically the ones where the less informed commit the act of repeating the action or similar from the person with the greater knowledge in a manner which is not clear headed. Thus, in the psychological sense, awarding a godhead of sorts to the person whom they have taken and utilized the information-from.
The way I do it, is I stand or sit at the place where I do the switching, and I employ as little moment and motion as is possible. The listener cues the changes and they tell me if it is A or B.
Edmond Stuart said:
Harmon (Fred Toole) does all their speaker testing with all the equipment behind a screen on a rotating platform because they found visual bias was so misleading. There used to be an article on their website about it.
I imagine they hide the electronics for listener testing , also.
Jakob2 said:
Ok, then let me modify my test. I wasn't really fair to assume that a BT against a 100$ Sony would have no audible differences. Even I will accept that ther most likely are 😉
So, we use this as a control. Double blind, with whatever other controls you would like. The BT and the 100$ Sony are correctly identified almost all the time, statistically significant.
So, we know that 'the system' is able to show audible differences. Now we replace the 100$ Sony with a 2000$ Sony.
result: correctly identified just above the statistical limit.
Now replace the 2000$ Sony with a 10,000$ AYRE.
Test results: correct identification not different from tossing coins, 50%.
Would that be an acceptable type of test, where you would conclude that this test setup can reveal differences between a BT, a 100$ Sony and a 2000$ Sony, but not between a BT and a 10,000$ AYRE.
Jan Didden
hermanv said:
Indeed. I read of one DB study in medicine where some people received placebo's and the others the real thing. The fact that the person giving out the medicine was aware of who got what, was found to distort the test. Patients, unconciously were picking up the body language, or whatever, of the person distributing the pills and modified their reaction accordingly.
Jan Didden
Floyd is a smart measurements guy, but lord knows he's not a Speaker designer. Not by a long shot. That's always been my take on the subject. He's been in the system a long time, and casts a long shadow. So there's bound to be differing opinions on the situation.
Wavebourn said:
Yet, there are 1000's of posts on this forum where people explain that after changing cap x for cap y they clearly could identify a change in the sound 😉
If what you say is true (and I agree), we can assign all those anecdotal reports to the trash bin.
Jan Didden
janneman said:
During Dr. Richard Bandler's seminar we did one interesting exercise: partner had to imagine one of 3 figures like red triangle, blue circle, yellow square, and say "Done", we had to guess. No reasoning, just guess. Did not matter which channels received the information, reasoning made results worse. Results were up to 90 percents! Surprisingly, one woman who claimed extrasensory perceptions got 50 percent results (expert!) 😀
However, before the exercise we calibrated our sensory perception well and were in good rapport (the whole group of 100+ students)
So, double blind should be assignment of controls, but controls has to be in hands of experts so they can switch, rewind, switch again, any time they want.
Wavebourn and PMA, you have it right! It is the same for me. Double blind and its 'advantages' are possible with a blind A-B test. The assignment of A or B should be arbitrary, but A should be consistent with A, and B with B. There not need be an X. Just A or B. This can still be 'objective' but listener sensitivity and lack of confusion due to the music changing in timbre, will be greatly improved.
Lipshitz and his group could NOT even hear the difference between the worst caps in Walt Jung's junkbox and any other cap in an ABX test. He has already done the test and failed to hear a difference. How is this useful for audio evaluation?
Lipshitz and his group could NOT even hear the difference between the worst caps in Walt Jung's junkbox and any other cap in an ABX test. He has already done the test and failed to hear a difference. How is this useful for audio evaluation?
janneman said:
I'll know what questions to ask the next time someone trumpets the superiority of a Stradivarius. 😉
rdf said:
Heh... I once after concert played Amati cello+contrabas and organ through my PA system, people including the singer Sergey Zadvorny said that it sounded as real including reverberation in the Swiss church where it was recorded. However, right before that everyone herd the difference when Sergey was singing without PA and with PA.
Why?
The answer is simple:
Because during the concert direct comparison was involved, but after the concert they herd recorded music only.
janneman said:
I think this is an unfair comment, it is quite possible to remember certain aspects (like detail, ambiance and focus of soundstage) of a known piece of music on a familiar system for long periods of time. So it is quite possible to hear differences after changing capacitors.
On an unknown system it can be much harder to hear small differences because you have to "quickly" remember what you hear now to compare with another unknown.
André
john curl said:
You're right, it's not very useful.
I recall 30 years or so ago when Lipshitz and Vanderkooy presented their
analysis on the Quad "current dumping" amplifier at AES. At the end, a
member of the audience asked "Well how does it sound?", and they answered
"We didn't listen to it."
At least it was an honest answer, but it implies that measurement is their sole
focus.
Typical subjective equations
">" meaning "prefered to"
A > B
B > C
In the subjective audio world, it often happens that
C > A
">" meaning "prefered to"
A > B
B > C
In the subjective audio world, it often happens that
C > A
janneman said:
OK,
now please explain, or give a weblink for an explanation,
why are the results of this small sample applicable 100% to all people all the time?
how does testing 1000 Jims apply 100 % to Pavel or John?
- Status
- Not open for further replies.