I would contend that the biggest issue with ABX testing is entering into it without having first isolated & identified a particular difference in a particular section of music.
When doing such a test without this is the equivalent of being handed two pictures which seem identical & being asked if there is a difference between them& to spot it i.e. it's a far different proposition for the tester than if they were handed two pictures knowing that there is an actual difference between them.
The first situation quickly leads to the thought that this is likely a waste of my time & demotivates further effort - the second motivates one because it is a challenge to find the known difference.
Now think of doing this with two seemingly identical movies!!
It's standard psychology
I suggested using blind preference testing as the correct approach as it is not cognitively demanding & doesn't have these motivational issues to deal with - just select which one you prefer. If this results in a statistically significant result for one, it tells us a lot - it tells us that there is some difference which we can sense a significant number of times - we might not be able to consciously identify a specific area of music or specific difference but it still says we are sensing a difference
To me this is what these tests are supposed to be about - can we sense a difference between A & B
What has become the clarion call which has been repeated again & again here is the BIG LIE - unless we can "pass the ABX test" then we no other way of difference testing will be countenanced.
Sorry, Nico, if I'm being "too literate" for you - just ask & I'll repost a dumbed down version of this post, just for you 😉
Funny thread. Most people do not even hear a difference between the CD and the vinyl record. How should one hear a difference in DACs? The improvements since the Philips DAC from 1983 are not audible. It does not matter which DAC you use. Maybe a bat will hear a difference? Critical was always only the error correction in CD players, but that does not matter here. Relax!, we have top-hifi since 1983. No reason for any improvement in that area..
Off the wall question, does anyone make a DAC that registers itself as a non audio device that requires bit perfect I/O? I know National Instruments makes data acquisition boxes that are far beyond 24/96 in streaming speed.
Same with potentiometers, no one got hung up about the basic pots used here Doug Self Preamp from Linear Audio #5
This is actually not possible blind, there are no absolutely flawless LP's to use for this. In a more stupid time I actually took an LP where I had two copies and digitized it and removed all ticks and pops (one at a time) that were not common to both from one and then compared it to the CD which as far as I could tell was as close to a simple "dump" of the master tape as possible. I also chose a recording where the tape noise was dominant so the simple cheat of listening to the noise floor was eliminated. I could hear no difference.
Last edited:
Not that I am aware of. Getting bit perfect I/O is presently the responsibility of an application author, and in some cases a driver author. If a DAC did identify itself as something unknown to an OS, it would need specialized application and driver support to run at all. The only benefit that would be accomplished is that accidental misuse or mis-operation could be avoided, but at a cost of compatibility issues in some cases that wouldn't otherwise occur.
That wasn't the only incorrect claim. Don't know where people get some of these things.
This one in particular comes up in the writings of many of the most well known audio critics/magazine pundits. All I can think is that their experience and integrity puts their opinions above suspicion. Avoiding any exercise where they could be wrong keeps it that way.
This is why I included the ability the sneak up on what you are trying to hear. You can start with 8 bit quantization and get 10 out of 10 right. Then 9 bit, then 12 bit..uh oh, suddenly you are getting 8 out of 10, then 13 bit (6/10) then 15 bit (5 out of 5--same as flipping a coin) and then 16 versus 24 bit...forget it.
But if the test is simply "what sounds better" then most will simply gravitate towards that which is slightly louder and has their eq preference. That isn't really demonstrating skill, it's just saying twinkies with extra sugar do better in taste tests than twinkies with less sugar.
If you CAN hear a difference, then ABX isn't cognitively challenging at all. It' only becomes cognitively challenging as your probability of detecting a difference tends towards that of a coin flip.
Yes, what you suggest is good for training - we KNOW there are definite differences between samples & so we test ourselves to see what we can differentiate, trying to establish our thresholds.
But what I'm saying happens in most ABX testing (this one being an example) is that most really don't know if there definitely is a difference so they don't try to find this DEFINITE difference in sighted listening - they blindly 😀 jump into ABX testing expecting small differences will jump out & make it's presence known. Nope, never happens unless differences are so gross that it's obvious & not even worth testing (or they are pretty experienced in perceptual testing)
Well, if there are amplitude differences or tracks are eq'ed differently then the test is flawed, anyway
Ah, if the test is about who is skilled & who isn't - that's a whole different test & why Jon needs to state what he is trying to test - so far he hasn't but the thread title suggests that it's DAcs, not listeners he's trying to test.
Isn't that circular logic - if you can tell a difference therefore the test is not a challenge?
As I said if you haven't trained yourself & identified specific differences prior to doing ABX testing 90% of people will fail. There are few who have enough training in perceptual testing that can use the ABX test itself to train & isolate specific differences.
Here's JJ Johnston's (recognised expertise in perceptual testing i.e MP3 ) summary of such testing: (This slide set from DTS http://www.aes-media.org/sections/pnw/ppt/jj/hashighlevel.ppt)
Slide 17 "Do I have to have controls"
YES
"Well, unless you don’t want to know how good your test is, of course. ☹"
Last edited:
One thing that seems wildly missing from the entire discussion is generalizability. It's being abused.
In short, the more we try to take a specific test (with attendant scrutiny of its actual protocol/etc) and expand its scope, the more likely we are to be utterly and completely wrong. So, there's some discipline that needs applying and clickbait thread titles are not the right thing.
On the opposite side of generalizability, one has to also ask the question "who cares?" when an experiment is so hopelessly contrived as to not apply to anything *but* the test conditions.
In short, the more we try to take a specific test (with attendant scrutiny of its actual protocol/etc) and expand its scope, the more likely we are to be utterly and completely wrong. So, there's some discipline that needs applying and clickbait thread titles are not the right thing.
On the opposite side of generalizability, one has to also ask the question "who cares?" when an experiment is so hopelessly contrived as to not apply to anything *but* the test conditions.
In my view, inserting a control into the test is analogous to a gage study on the human ear. I would suspect that most experimenters would have no problem with a DOE that includes a control. I’ve taken an on-line ABX hearing tests where you start at a easily recognized threshold and continue until you can’t tell the difference (Free Online Audio Tests, Test Tones and Tone Generators).
In the end, I would expect that you would find that what people are able to hear falls within the already well defined distribution. Then you are left with the difficult reconciliation of how is that some people claim to hear well above 20KHz in music or vanishingly small amounts of distortion in a DAC when they can’t hear those frequencies or artifacts on a standard hearing test or ABX test. Hmm...
In the end, I would expect that you would find that what people are able to hear falls within the already well defined distribution. Then you are left with the difficult reconciliation of how is that some people claim to hear well above 20KHz in music or vanishingly small amounts of distortion in a DAC when they can’t hear those frequencies or artifacts on a standard hearing test or ABX test. Hmm...
You are mixing up two situations - one with known differences of decreasing amount that you use for training & the other one that you really don't know where to search for differences - is the difference to be found in the hi-hat, in the bass drum, in a female singer's voice, in a baritone's voice; in the attack ; in the decay; in the soundstage width; in the soundstage depth; where in the music should I listen for this, .......
Not all of these can be focused on (which ABX testing requires) at the same time - each has to be examined - it gets pretty tedious, pretty quickly holding this level of attention for each of these specific elements in order to check if a difference can be consciously heard.
Maybe Jon or some of the participants can tell us exactly how & what they are listening for in their ABX testing? Is it some "generic difference" or something specific & how & where in the music snippets they are listening for specific audible differences?
How did they pick the snippet they are listening to? Pot luck or because it gave some indication that it sounded different - what indication specifically?
Again, blind preference testing is incorrectly dismissed as some sort of corrupt way of listening - it is exactly what is being examined here - is there an audible difference between A & B. This may well be an unconscious effect, not a conscious specific part of the music that can be pointed to. What if we consistently feel fatigue with one device over another (blind, of course). Is this not a difference? But can we spot this in a quick A/B? Probably can't spot it in a A/B preference test either.
Last edited:
I wouldn't have a problem with a preference test if the listener can achieve consistent (statistical) results with their preference.
Exactly - that's what makes it a blind test of any value but many (& we've seen the opinion expressed here) protest that you must FIRST pass a ABX test to establish a difference exists as if that is the only way to establish a difference exists. I wonder why?
Last edited:
^ Yawn. And that's a good way to lose an audience mmerrill -- burn down many strawmen in your lifetime?
ABX is but one of many DBT methodologies. Oftentimes they get conflated. Go forced pairing or triangle if it makes you feel better. How is a blinded preference test (again, ideally double-blinded or computer-based single blinded) not a DBT? It just has a different hypothesis to be tested (A is preferred over B, or vice-versa). Gotta start with what do you want to find out!
ABX is but one of many DBT methodologies. Oftentimes they get conflated. Go forced pairing or triangle if it makes you feel better. How is a blinded preference test (again, ideally double-blinded or computer-based single blinded) not a DBT? It just has a different hypothesis to be tested (A is preferred over B, or vice-versa). Gotta start with what do you want to find out!
Yes but the o/p's & others insistence in ABX as being the only test for differences is what has been discussed. It's also been discussed that the o/p needs to state what hypothesis he is attempting to test.
So, yes it seems we agree - I'm not sure what strawmen you might be talking about?