Really, they are comparing two audio streams MP3 and .wav of the same sound clip. You can't conceive of using 2 AESEBU outputs on a multi-channel sound card and two DAC's as the two streams?
As far as I understand it the subject has to modify one file to specifically match sonically (in their opinion) the reference WAV file (with a known sonic characteristic having been adjusted) along a this specified parameter? They measured the time taken for each trial some of which were MP3 files & some WAV
How exactly do you propose two DACs will be compared in this manner?
Can you detail how this would be done as per my understanding above? Or if my understanding is incorrect, please enlighten me.
Last edited:
These guys openly offer a proposed alternative and seem open to any discussion. If you can find anything in their discussion where they suggest sighted testing might be equally valid please point it out.
And Jakob gave you very specific positive controls to use during blind tests (for DACs which you seem fixated on but it applies to any AB or ABX test) which I seem to remember you summarily dismissed with no discussion.
Did Jakob propose this? I must have missed it - got a link?
Last edited:
This test is probably unorthodox enough to make everyone here unhappy. The DSP is actually used to modify one of the streams according to 4 "mix" tests chosen to match the most common complaints about MP3 vs CD. Maybe Mark could come up with a set for DAC comparisons. Reconfiguring the software and hardware to make the task to take one input data stream and adjust the mix to make the two DAC's sound the same would seem a trivial exercise.
Again you don't read what I say "I don't see how sensitivity to level change has anything to do with two DAC's that can be adjusted to +-.01dB, but use it anyway if you want." is a summary dismissal.
Where did I say Jakob said anything of the sort, I asked if this paper (or any other for that matter) seriously discusses sighted listening for subtle audio impairments.
Let me make one thing clear blind and preferably double blind is an absolute necessity for evidence to be sufficient. This is my opinion, whether blind is sufficient is a different matter. You need to be there when a buyer for a very large wine importer identifies two halves of a magnum as two different wines (DB of course). This is self evident to me people are people and have plenty of frailties.
Last edited:
I would need to know what kind of rivets and what kind of bolts..
And if the tubes were in sockets when hammering the rivets.
yes it is summary dismissal because you completely misunderstand (either intentionally or not) what a positive control actually means despite it being explained to you any number of times.
A positive control does not have to be directly related to the sonic characteristics of what's being tested - it's purely an internal evaluation of the test itself. So a known audible difference (could be any small sonic difference which is known to be audible) is used to do this internal evaluation
I dont know what is so hard to understand about this concept?
I asked "what distinguishes between agenda Vs no agenda?" (& as you know I was specifically asking about why you accuse Jakob of having an agenda but not these guys)
And you answered "These guys openly offer a proposed alternative and seem open to any discussion. If you can find anything in their discussion where they suggest sighted testing might be equally valid please point it out."
I read your answer to mean these guys have no agenda because "These guys openly offer a proposed alternative and seem open to any discussion" - the implication being Jakob did not demonstrate these characteristics
AND these guys have no agenda because "If you can find anything in their discussion where they suggest sighted testing might be equally valid please point it out." Again I read this to imply that Jakob did "suggest sighted testing might be equally valid"
Apologies if I read this wrong.
That seems obvious, how do you know a priory the sonic characteristics of what is being tested. So how do you pick the level of the positive control and why eliminate a subject that fails say a 0.2dB level change test when high frequency IM is being tested for. Lots of things bother you allow me to be bothered by some.
Jakob likes to say how many times I've said things but I know I have said many times (I've lost count) pick the protocol you want, take the stuff home and sit in your comfy chair and spend all the time you want but it must be rigorously blind.
Scott, let me try to explain the basis for positive controls again, briefly as it is tiring & trying as I believe this might be the underlying understanding you are missing
Let me use extracts from the paper you quoted "Current listening test methodology works very well when the comparison between audio systems is easy for the listening test participants. However, when that comparison gets very difficult (when the perceptual difference between systems gets very small), many of these listening test designs break down and fail to produce significant results"
So why should this be the case?
If there's a measured difference which is known to be audible, how can I discern this difference sometimes & not others? Does auditory perception not work the same way every time & therefore I should discern the difference every time? No, is the answer - it doesn't work in this mechanical way that produces the same result every time - it varies. What causes it to vary?
Some text quoted from your paper might give an explanation "The main goal of this approach is to get away from the traditional AB and ABX test methodologies in situations where the differences between audio systems are very small. In these conditions, these tests place too much pressure on the participant, are too short for the participant to recognize differences, sometimes force the participant into having a “preference”, fatigue the participant quickly, and require that the participant remain consistent with their decisions throughout the test. Overall, these tests are difficult to perform and lack focused assessment of specific attributes."
So these authors state that AB & ABX tests are difficult to perform for the highlighted reasons above. Can I ask you if you agree with their assessment of difficulties with these blind test methodologies?
If not then there's no further discussion about this needed
If you do agree with their statement of the " several negative claims were made regarding AB and ABX listening tests in regard to audio systems with very small impairments" then positive controls are one way to evaluate how much these issues have effected the test results. By including hidden controls of known audible differences, a means of evaluating the degree that these issues may have effected the test itself
Let me use extracts from the paper you quoted "Current listening test methodology works very well when the comparison between audio systems is easy for the listening test participants. However, when that comparison gets very difficult (when the perceptual difference between systems gets very small), many of these listening test designs break down and fail to produce significant results"
So why should this be the case?
If there's a measured difference which is known to be audible, how can I discern this difference sometimes & not others? Does auditory perception not work the same way every time & therefore I should discern the difference every time? No, is the answer - it doesn't work in this mechanical way that produces the same result every time - it varies. What causes it to vary?
Some text quoted from your paper might give an explanation "The main goal of this approach is to get away from the traditional AB and ABX test methodologies in situations where the differences between audio systems are very small. In these conditions, these tests place too much pressure on the participant, are too short for the participant to recognize differences, sometimes force the participant into having a “preference”, fatigue the participant quickly, and require that the participant remain consistent with their decisions throughout the test. Overall, these tests are difficult to perform and lack focused assessment of specific attributes."
So these authors state that AB & ABX tests are difficult to perform for the highlighted reasons above. Can I ask you if you agree with their assessment of difficulties with these blind test methodologies?
If not then there's no further discussion about this needed
If you do agree with their statement of the " several negative claims were made regarding AB and ABX listening tests in regard to audio systems with very small impairments" then positive controls are one way to evaluate how much these issues have effected the test results. By including hidden controls of known audible differences, a means of evaluating the degree that these issues may have effected the test itself
Well you pick the level of the positive control in a sensible way. So let's say that the premise being tested is that there is an audibly perceptible difference in how the high frequencies are rendered between one DAC & another. Are you going to use people who have hearing impairments in HF discernment & then declare there is no difference because they have not discerned a difference? (I dislike using freq example because it is so gross but it is just to illustrate the idea). How do you know if they have hearing impairment in HF - one way to do it would be to include a HF tone or tones in some of the trials as controls.
If someone isn't able to retain listening focus for a length of time considered necessary to do such blind tests - are you going to use them & declare their likely null result as of importance?
I asked you before about using measurement instruments that weren't calibrated as the analogy to the above but maybe the better analogy is trusting a technician's reporting of measurements who has what you suspect is sight impairment?
But the point is that you can't test participants for unknowns (such as air between instruments, soundstage depth, naturalness, etc) so all that can be achieved is positive controls that evaluate if the participants were focused enough, not fatigued, not distracted, etc that they actually participated adequately in the test?
You cited a paper & said "The people involved here seem to be making an honest effort to address some of the same AB - ABX issues." But you have no answer to how this could be used for testing two DACs
All Jakob & I have proposed is to include positive controls within existing ABX testing & Jakob stated exactly how this could be done for DAC testing. You & others have accused him of having an agenda.
Please explain what you fiind so wrong with positive controls as suggested?
And please don't reply that my post is "word salad"
Last edited:
I would need to know what kind of rivets and what kind of bolts.
Yes. Also, different brands of hot glue have to be double-blind-listened!
More and more often I see PCB mount caps glued to the chassis in boutique "neatly wired" amps.
I go for the full deoxyribonucleic acid screening & lysergic acid diethylamide testing but then that's just me
Most 'economy' but decent RCA interconnects have what looks like moulded hot melt glue under the shroud enclosing the connections and as strain relief
.Dan.
Last edited:
I already said I have no problem with that discussion in the quoted paper.
Didn't Irwin Pollack prove ABX does not work well for certain sound recognition tasks in the 50's? I keep seeing the same thing here ABX = blind so blind = ABX. Not very good logic. Being a paid subject recognizing the sounds of certain consonants does not come with a lot of personal involvement (ego/financial interest). If you can find some papers that show having a strong vested interest in the outcome of a sighted test does not affect the results I would gladly read them.
How many times have I said I don't care what protocol you use, invent a new one, use any positive controls you want. If you're claiming that a blind test for audio impairment can't be invented, I guess we will just have to go on with the anecdotes.
I have always said I wish folks realized how difficult it is to do these tests for small differences.
What is proposed to be tested... a pair of DACs or what ?.
I well realise daily
Dan..
- Status
- Not open for further replies.