This was the specific point I was trying to make a week ago. A very simple switching setup will tell you first whether or not there is a difference. It will also give you a feel for what area of the sound to listen to. From there you can do all the double and quadruple blind testing you like because the participant at least has a bit of a bearing as to what aspect has changed or is different.
To drop someone into a blind test is disorientating at best... Once a listener has their bearings I would guess the accuracy of blind testing would actually improve dramatically with trained or interested participants.
The audible soundscape is a very complex experience and the minutiae mentioned above can be buried in the glare of a multitude of more proment aspects of the sound.
Mike
Mike, that's the point of ABX or tests with a similar structure- the listener is free at any point to switch back and forth "sighted" between A and B. If the listener wants to spend time doing that to get a handle on the differences before beginning his guesses, that's fine.
Sorting tests are a bit tougher unless the listener is given a known pair of references.
Sorting tests are a bit tougher unless the listener is given a known pair of references.
All clear now. So a DBT for the statement "Joshua can hear differences between A and B" is, in your opinion, rigurous and correct (by giving "positive results") only if Joshua can hear differences between A and B.
Ever heard of circular logic?
I've been working as a scientist for about 35 years and I seem to have missed the "rule" about using a variable that is NOT the one being studied as a "positive control." That's a startling assertion.
Equally startling is the idea of post hoc recasting of questions, using statistically insignificant samples, and accepting self-reported results with no controls on scoring. These are fundamental errors of analysis. But hey, if that's "science" to you because you like the result, what can I say?
Uhm, so you are not saying that the DBT procedure itself must give positive results (which in you experience is indeed the case), but that * every single specific DBT * must be proven to be able to give positive results in order to be meaningful. Correct? That's a pretty strange point of view - I think that if you're not going to hear differences between, say, different resistors orientation in a circuit thru a DBT test, maybe it's because resistors orientation doesn't have any effect on circuit's sound - imho it has nothing to do with DBT sensitivity or ability to reveal tiny differences. Unless you're persuaded that merely blinding a test affects the sensitivity of the testers, or establishes some sort of hearing threshold on them - I can't really figure out how that could happen.
L.
No explanation at all- was only speculating on my experience when switching between CD and SACD versions of the same program; it sometimes happened that after discovering in the SACD some details I had never heard before, I noticed I was able to hear the very same details when reverting back to the CD version. Bias power...?
L.
Its sometimes in the mastering when talking CDs and SACDs. Not all SACDs are mastered the same as their CD counterparts (thank God). The target audience for SACDs is not the car stereo or the boom box; it is a bit more driven by sound quality. Of course the format is superior, but that is not all there is to it - mastering differences are important. I've heard many SACDs that are just as crappy as the CD - probably mastered the same. I've heard SACDs that are far better sounding, and it may very well be largely attributable to better mastering. I've also heard exceptional CDs. Many variables in play (so to speak).
Cheers,
Bob
Yes, when we refer to a certain exact test procedure. Once a certain procedure demonstrates the ability to give, even at times, positive results, that procedure is correct, or meaningful. Now, we need to be attentive to the words *exact procedure*.
As I have previously pointed out Joshua any scientifically valid test must contain a falsifiable hypothesis.
I suppose depending upon your disposition , (whether you are a glass half full or glass half empty type), you can look at either falsified or not, as positive, or negative, or visa versa.
rcw
I suppose depending upon your disposition , (whether you are a glass half full or glass half empty type), you can look at either falsified or not, as positive, or negative, or visa versa.
rcw
I am also talking about actual results.
The hypothesis is that something is so, and the result is whether it's true or not.
As already pointed out no scientifically valid test can ever prove the hypothesis true, but can only show that you haven't proven it false, thus true by implication.
rcw
The hypothesis is that something is so, and the result is whether it's true or not.
As already pointed out no scientifically valid test can ever prove the hypothesis true, but can only show that you haven't proven it false, thus true by implication.
rcw
Yes, your concept does not work very well. I found that out over 30 years ago. I prefer A or B, even though I may NOT necessarily what A or B is, or personally care. I test that way, when it is necessary to do so, and have done so for the last 45 years. If I got random results, then I could not go forward and build 'better' designs. I always get random results with ABX testing, with MY designs or the similar designs that have flat frequency response. This level and frequency response requirement is not new to ABX or similar testing. It has been known for more than 50 years, perhaps 100.
I once did a BLIND test at Silver (an HK affiliate) in Nagoya, Japan, with 3 line amp prototypes, all adjusted to be equal level and frequency response.
I did NOT know the specific designs at all, and I just was invited to see if I could hear any difference. The choices were like 1,2, 3. And I could identify them as such, but only as a specific choice. I picked the best, next best, and third best, just from listening to them for a short while. The Japanese were astonished that I could do so, and told me that. I said to them, that this is what I do for a living.
You might say: 'I pulled a Fremer' '-)
I did NOT know the specific designs at all, and I just was invited to see if I could hear any difference. The choices were like 1,2, 3. And I could identify them as such, but only as a specific choice. I picked the best, next best, and third best, just from listening to them for a short while. The Japanese were astonished that I could do so, and told me that. I said to them, that this is what I do for a living.
You might say: 'I pulled a Fremer' '-)
Interesting this apparent need for level matching and frequency response precision to be able to assess sound quality, for many. This is about the last thing that's important to me -- as a good laugh for some, this morning I went through about 2 albums before I realised that I didn't have any significant response below 150Hz or so, a solder joint had come adrift on the powered woofer, the midranges were carrying the whole show at the bottom end.
For me, the sound is either right, or it's not. And if it's not in good shape then altering the volume way up high or down low makes no difference, the characteristics of the distortion are still clear and obvious, irrespective of the level.
Frank
For me, the sound is either right, or it's not. And if it's not in good shape then altering the volume way up high or down low makes no difference, the characteristics of the distortion are still clear and obvious, irrespective of the level.
Frank
_If_ you don´t know what the variable to be studied might be, then you will not know if the "positive control" would be NOT the one.
Your argument seems to be, that it is better to know absolutely nothing about the detection ability under test conditions than to know that the detector reaches a good sensitivity under test conditions wrt to at least one variable.
Please be more specific- what questions were recasted post hoc?
Which samples were statistically insignificant?
I´ve stated clearly what the source was; you otoh were afaik relieing on totally unsolicited anecdotical rumors, or do you have a quality source for the quite new cheap talk about "missing controls on scoring" in the Fremer AES-DBT?
Afair whenever you´ve had questions about sources, did ask for citations, i was posting what was requested or send you pdfs per email.
It is now up to you to return the favour.
I´ve been asking now for months for any data that could back up your assertions.
You were up to now even unable to present some basic calculations to show the insignificance of results.
Yes, it seems that we have a very different point of view wrt to science. I am trying to back up my statements- if any calculations are needed i´d more than happy to provide these, but that need some clear statement from your side which results are "insignificant" wrt to a clearly expressed significance level- if i don´t have any peer reviewed publications i say so, as i´ve done in the Fremer case.
Sorry John for the OT, but it is a bit related to audio quality (perceptive evaluation of it)
- Status
- Not open for further replies.