If you make that, "there was no difference detected," including the word detected in the quote, then I would agree. Otherwise some people think it means "there are no detectable differences between the DUTs," has been proven true for the entire human population.
Other issues we run into with biostatistics are sample sizes and effect sizes, which are expected to be included in published results.
But here's the issue you fail to grasp - ABX testing is designed to eliminate false positives (hearing a difference that doesn't actually exist - this the flaw that sighted listening is cited for): it has nothing in it's design to deal with false negatives (missing a difference that does ACTUALLY exist).
So the outcome of this is that nothing can be concluded from null results - is this null resulting from someone not really trying to hear a difference (due to negative biasing), is it a genuine failure to hear, is it a bad test setup or equipment in which it is not possible to hear such differences, is it the result of a nodding dog hitting random keys or of humans hitting random keys because "life's too short". We simply don't know what gave rise to the null result.
This is fine as long as it is understood & those who have experience with perceptual testing are aware of this. The problem arises when people mistakenly consider this null result as a datapoint & consider the accumulated null results as some sort of 'evidence' of anything - we simply don't know which of the null results are valid attempts, using properly setup ABX tests.
That's why you see everybody who knows anything about blind testing state that controls are needed withing them - to qualify the results in some way & avoid this dilemma.
So, I hope you can see why expecting a null result almost guarantees a null result in ABX testing?
Last edited:
Change this to "So the only result which can be used as the basis of a proof is a non-null result." and I think you may be on the right track
Tony.
It's a first step but the test itself still needs examination as there may be wrong setups used which bias towards one device over another - for instance the order of presentation of devices should be randomised, volume levels should be matched within reason, etc
The whole point is that the ABX test is not designed to eliminate false negatives, it is only designed to minimise false positives.
Therefore null results contain an undetermined number of false negatives & we have no way of determining the level of these false results because there are no controls within the test.
You cannot treat these null results as 'evidence' or 'datapoints' or really of any value. So your statement is incorrect "It can only give some indication about the likelihood of something being true, and hence the likelihood of it being false."
Do you understand this distinction?
Because that is a contradiction. A DAC's job is to connect the dots and recreate the analog signal. I'm sorry if folks disagree but mathematically there is only one answer which can be approached to arbitrary precision with enough time and compute power both of which are limited in an affordable consumer product. It's ironic that the Benchmark DAC3 keeps getting mentioned because that is their design goal.
A "more realistic illusion" simply is a different answer, maybe colored is a strong word but there are DAC's out there that are deliberately designed to be not accurate. Windowing to give a gentle rolloff, no ringing, but allowing some images to fold back below 20k is considered preferable to some. Another was the high order polynomial French curve fit to the dots, again this allows images to leak below 20K. Unfortunately the proponents of these things usually show pictures of waveforms that violate Nyquist and could never have been recorded in the first place.
Last edited:
You mean anti-imaging filters. I would also say none of them because the reality of the situation is that the latency and compute power of a low cost chip set has a finite limit. They keep improving I find it hard to believe the differences are extreme at this point. Then there's the pre-ringing hobby horse. But computing the best fit mathematically exact answer to < 24 bit noise floor is intractable with reasonable latency (I think).
The DAC has no a priori knowledge of the A/D process which opens the door to the possibility that a "wrong" filter in the DAC compensates in some way for the most common A/D's? High speculation here.
I know from an engineering perspective this is the thinking & the disconnect occurs between mathematical precision & physical realisation of this goal. Now we all know that this mathematical precision is not achievable with real world components so ways of measuring this lack of accuracy ("arbitrary precision") are devised in terms of engineering measurements. These use contrived signals to derive measurements - the results of which are usually evaluated with regard to 'thresholds of hearing'. (we have seen here that the auditory perception of complex signals is different to simple tones) - in other words the phrase "arbitrary precision" is dependent on this threshold
So now we have two problems to deal with - the nature of the test signals Vs what we actually listen to. And the evaluation of the resulting measurements with respect to thresholds of audibility is another disconnect.
As has been said before we all know that these measurements do not accurately correlate to auditory perception. So where is the disconnect? Obviously the measurements are not aligned with the end use case which involves auditory perception.
It wouldn't be the first time that engineering has hiccuped between it's measurements & the end use case - some spectacular - bridges that resonate, O-ring seals that fail & others I'm sure - these were also considered to be adequately measured for the role they had to play until they met a circumstance that hadn't been determined/examined by measurement.
It's entirely possible (& is my contention) that this is the situation with DACs - their use case is revealing flaws in areas which have not yet been identified in measurements.
We are using a very sophisticated real-time analyzer in auditory perception which no measurement device is yet capable of emulating
Sure, there are known ways to deviate from the model of 'accuracy' defined by engineering but that is a limited definition of 'accuracy' as I said above & it doesn't mean that DACs which provide a "more realistic illusion" are therefore flawed in some way I would contend the opposite - they are more accurate as far as the auditory perception judge is concerned as they are being perceived as more realistic sounding.
Yes, I can understand the engineering mindset that doesn't consider the disconnect between its measurements & the final use case of the DAC which involves auditory perception, coming to that conclusion but that doesn't make it so (as Jean-Luc would say
)
Last edited:
Last edited:
Yes, I can understand the engineering mindset that doesn't consider the disconnect between its measurements & the final use case of the DAC which involves auditory perception, coming to that conclusion but that doesn't make it so (as Jean-Luc would say
But in both of my examples the DAC creates signals that are not in the original, if this is fair all bets are off and make circuits to create any illusion you want which people do (Carver sonic "hologram", etc.)
Yes but you need to read my words carefully - I said "they are more accurate as far as the auditory perception judge is concerned as they are being perceived as more realistic sounding."
Again, the final arbiter of how realistic the illusion is comes from auditory perception - we have no measurements yet which correlate to this. So again, there must be factors which are important to auditory perception as far as realism of auditory scenes are concerned & if you are saying that your two examples have been consistently perceived as a more realistic illusion (not just an interesting & novel illusion) to auditory perception then I would suggest that they are better meeting some criteria which matches what perception has learned to associate with the behaviour of real instrument/vocal/scene sounds in the real world.
The flaws that you are saying are obvious with these two examples obviously don't override the "better illusion" perception. In the same way that we know certain elements in the sound is perceptually masked, these flaws may become irrelevant when the other factors are more correct (or maybe they are perceptually irrelevant anyway without other factors being present?)
Perception does this all the time - we hear through noise, we hear through room interactions, (both within reason). Perception is adapted to dealing with the real world examples & this is it's template & model from which it has learned what is natural sounding & what is unnatural or slightly off.
Coming at all of this from the perspective of what auditory perception uses to judge sound is the key to unlocking the fixation with accuracy or putting the correct emphasis on accuracy in the correct place & not everywhere within the sound field. For instance we accept ripple in a DAC's passband was mentioned already - what is the typical range of the ripple found in DAC reconstruction filters? Has anybody tested the perceptual effects of amplitude ripple at certain frequencies & not others in complex signals such as music?
Last edited:
Regarding accuracy, many people want equipment that measures well (measures as accurate) and sounds good too.
As a practical matter, it is necessary to measure accuracy in order to design and manufacture equipment.
Of course it is & it should continue to be in the field of design - it has brought us a long way & delivered a good quality of sound. Some say we have gone as far as we can & speakers & room interactions are the only important areas left for improvement - I don't subscribe to that.
What I'm saying is that we need to somehow link where we are to where we want to be which I would contend is design which better recognises the end use - evaluation by auditory perception.
So let's say perceptual research teases out some aspects in the soundfield to which we are much more sensitive than has been considered heretofore & other aspects which we are relatively immune to - should this not direct our design for 'accuracy'?
At the moment we use 'accuracy' in a blunderbuss way & in a relative sense - how much below audible thresholds is this aspect of the measured results. So we are already basing our definition of accuracy on auditory perception thresholds which have their origin in the 40s or earlier.
It's not really consumers ('what people want') that should direct research, discovery & improvement in products - by & large consumers want a good story - specs do not really determine accuracy (in a true sense) - it's just a good story
Last edited:
Some manufacturers only publish specs that portray their product in a favorable light. Many are not equipped to take a full gamut of high quality measurements either. So, to a significant extent its not that there is a huge problem with measurement technology itself, but perhaps more with how it is sometimes used.
There is also price point to consider. Most equipment is designed to sell at a particular price point and to compete only in that domain. Competition is not usually primarily on detailed specifications that consumers don't understand, and probably don't want to learn to understand.
Finally, there is little or no funding to do the kind of research that would be needed to better understand how some very picky people hear, and how to best measure some of it. In the meantime we are left with conducting listening tests along with taking measurements. It's up to each manufacturer to decide how to find listeners and whether or not to train them, and if so, how. Smaller manufacturers may not be able to afford much. So, its very complicated and difficult to design and build a really good product, salable at a good price, and then convince enough people to buy it to justify all the costs.
Last edited:
What does determine accuracy in a true sense? Not the recording, of that you can be certain
I don't agree that it's about what "some very picky people hear" - I'm pretty sure most people recognise a more realistic auditory illusion when they hear it. Doe sit stand up to sloppy ABX testing - not much except gross differences will be found in these tests.
No, we are left with just listening & deciding without 'testing'. Can we be led astray, sure but we can be wrong with many things in life. Can this 'testing' lead us astray? Sure
Commerce is a hole nudder topic
What I mean is that the specs don't really have enough info to state if they are truly accurate - as Mark says they often show the device in the best possible light.
Ok fair enough. I guess the point I'm trying to make is to get to the crux of the issue of where is the information coming from that allows us to form a realistic image perception in our minds, ultimately from the recording but not in any specific way that makes it possible or not possible, it's not that simple an issue, it's largely to do with how we interact with it which must include how the speakers radiate it into the space we are occupying whilst listening
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.