I object to these online ABX tests, yes & for very good reasons. As I said already, all the evidence & research backing it up was laid out in other threads by both Jakob2 & I & like SW, I'm not going to go back & rehash it.
If you have a problem with someone showing the errors in your 'tests' then you should stop the many objections to technical tests posted on this forum - just to mention one current one, I see many objections on the JC thread about the Bybee tests & reported results.
The problem seems to be that listening tests are perceptual tests & not many here seem to know (some really don't care to know) how to conduct these tests in order to achieve some results of value. If pointing out the issues & the research is a problem then just state that openly?
Totally agree with this - pretty much what I have been saying & the only way I can see for removing the great problem of reliance on the end user being studiously rigorous by following a script (PMA would have liked users to test their systems, AFAIK), is to include hidden controls in such online ABX tests. I've suggested one such way to achieve this but I note that the ABX test utilities in Foobar & Lacinato ABX never even considered implementing such hidden controls even though it would have been simple to implement. Such hidden controls would be a far more useful 'test of the test' than the Sine wave test that was done here after the main listening test. It would have allowed the results to be evaluated in light of how many times the differences in the hidden controls were missed during the 'test' i.e how skewed that particular test (end user, playback system, environment, etc) was to false negative results
Last edited:
Yes, the ITU guidelines are exactly that guidelines for how to conduct certain types of blind tests but they have to be understood & translated into the procedures necessary for other types of blind tests, such as ABX tests
Nothing wrong with being pedantic in this, but you missed that it is the _alternative hypothesis_ , which is in this case H1: p > 0.5 .
It is not the criterion like SL = 0.05 .
Fortunately the NHST ritual is slowly but constantly changing;although SL = 0.05 is a useful pointer for something happening it should never be used as a hard threshold criterion for a decision if something is "significant" or "not significant", therefore the mentioning of needed replications.
PMA raised the question in another context and i tried to explain under which conditions it is justified to combine results and when it is not.
As he still thinks it is a matter of "different thinking" i obviously failed miserably in my explanation attempt, but i´ll leave it there.
That it is not a valid approach to combine results from totally different tests/experiments (with zero knowledge about the actual detecting abilities and confounding variables), i´ve already explained at length in several posts in a discussion with DF96.
Btw, already explained then why it is otoh a valid attempt to combine results under the assumption that the null-hypothesis is true (for analysis purposes), with similar "success" 😱
I missed it too when reading Scott´s post the first time, but he is indeed asking for an example of an actual test _done_ (on whatever equipment or effect) where you (and/or me? ) was really satisfied with the choosen protocol and (i assume) the execution.
This seems to be a nice misinterpretation of my posts. I was wondering that it could be possible to combine the results of different members and I explained my strong reservations and disagreement with such combination especially when it was shown that some of the equipment used was unable to reproduce sine tone with distortion low enough. I would appreciate if you would not speak for myself. And it was not me who came here with the sum of results of the participants.
Can you tell original file from tube amp record? - test
Last edited:
Yes
Am I reasonable to assume that same rigor (I guess without hidden controls) applies to sighted listening? I would then also assume most "forum" sighted listening deserves equal scrutiny.
BTW I never said all forum sighted listening was casual, but some by their own admission were not rigorous and lacked any controls.
Sorry PMA, but you explicitely mentioned "thinking differently" and it is not a matter of "thinking differently".
You simply fail to understand the specifics of the statistical analysis in that point. No problem - it is counterintuitive sometimes - and i therefore asked a specific question to help you in understanding it; you refused to answer - for reasons i can´t imagine - and you did so after i repeated my question several times.
Why do you think i insist on it???
Btw, the line:
"And it was not me who came here with the sum of results of the participants."
shows exactly that you still don´t get it, but what can i do?
I only became involved in this thread when the sine wave test was done & the results showed something of interest - a control listening test that showed the flaws with the main blind listening test. It was a chance to understand something about blind testing from an actual test
I already asked you these questions but you dodged answering:
"Is that what this listening test set out to do? What therefore was the point of the listening test? What would have been the outcome if PMA wasn't challenged to post sine wave test signal - would your & others test results have been considered valid?"
Care to consider & answer these questions? Did you ever consider your HDMI receiver was NOT sufficiently discriminating to hear such differences or is it your receiver that's at fault?