...they will show very personal differences in reactions on stimuli. No need for such subtle measurements. When I react on sounds before I even think that they are reproduced by electronics I know that the system is fair. But when such subconscious reactions don't happen, and I always "remember" that it is the system playing, it is not so good reproduction.
This Saturday my friend come to my house and brought his Yamaha keyboard and Apple notebook to check how well his new software samples grand piano we heard as if real piano playing in my living room. The conclusion was, the software satisfies him. But to my satisfaction I evaluated my home reproduction setup and found it to be quite satisfactory.
Not really. This is one of the most firmly established and powerful tools in sensory research (not just sound, ANY sort of sensory evaluation). And for sound, it works quite well, unless you're selling something that isn't distinguishable without peeking.
Indirect measures like Brad's examples or the MRI stuff from Japan are interesting, but difficult to debug and evaluate. One important point is that, for those sorts of indirect measurements, the matching of results versus stimulus also needs to be done double-blind.
It´s all about scientific reasoning. It is a bit funny to require something like double blind testing for validation reasons, but to refuse on the other hand to validate the double blind test itself.
A controlled listening test relies on a listener as the detector for differences (related to our discussions on these topics), so as an experimenter you ought to know if your detector is doing well -> validate it by using positive controls (aka differences a listener must detect under test conditions) .
Otherwise there is no way to decide whether a difference remains undetected because it is not audible (means no human listener will be able to hear it) or because the listener/detector fails to detect it _under_ the specific test conditions.
You can look at a double blind test as a Bernoulli trial. The one most familiar being a coin toss.
If the coin is "fair" then there is an exact equality in probability in a head or a tail and the distribution quickly converges upon a Gaussian one.
In a double blind test the idea is to eliminate systematic errors because these skew the probability towards one outcome or the other.
If for instance you set a confidence level of five percent then you are saying that your test has up to that much skew towards a particular outcome, and this is convenient because it is the area that is two standard deviations from the mean, and in the vast majority of cases you can get a test that is valid over this range.
Some people then say well your test is invalid because it has an in built uncertainty, but science being basically statistical it always will have, the question is not the presence of uncertainty but how much.
By this method you can know just how uncertain your results are and a ninety five percent chance that a null hypothesis is proven is not bad, a ninety nine percent one is even better but your test is much more difficult to do because removing systematic errors to this extent is very difficult and might well be impossible with realizable sample sizes.
rcw
If the coin is "fair" then there is an exact equality in probability in a head or a tail and the distribution quickly converges upon a Gaussian one.
In a double blind test the idea is to eliminate systematic errors because these skew the probability towards one outcome or the other.
If for instance you set a confidence level of five percent then you are saying that your test has up to that much skew towards a particular outcome, and this is convenient because it is the area that is two standard deviations from the mean, and in the vast majority of cases you can get a test that is valid over this range.
Some people then say well your test is invalid because it has an in built uncertainty, but science being basically statistical it always will have, the question is not the presence of uncertainty but how much.
By this method you can know just how uncertain your results are and a ninety five percent chance that a null hypothesis is proven is not bad, a ninety nine percent one is even better but your test is much more difficult to do because removing systematic errors to this extent is very difficult and might well be impossible with realizable sample sizes.
rcw
Just two remarks; even if the null hypothesis could not be rejected, that does not mean that the null hypothesis was proven, it just means that it could not be rejected.
If an experimenter lowers the probability of falsepositives he raises the probability of falsenegatives (means that the power of the test goes down).
It is possible to balance the two different errors up to a certain degree, which is another benefit of using positive/negative controls.
If an experimenter lowers the probability of falsepositives he raises the probability of falsenegatives (means that the power of the test goes down).
It is possible to balance the two different errors up to a certain degree, which is another benefit of using positive/negative controls.
I'm with you 100% on this. If this kind of listening could (easily enough to be practical) be combined with a blinding, we'd have something. If either is lacking, the whole thing retains a fishy smell to somebody or other.
Thnaks,
Chris
And, how Dean Radin discovered in his early experiments, it is quite difficult task that can be spoiled by effects that have no logical explanation! If you remember, he analyzed data obtained from experiments when generators of true random numbers used by computer to select stimuli, but according to data he analyzed people subconsciously predicted what kind of data will be presented, in advance. Trying to find the source of this error he found that it is not an error, but rather a phenomenon that can't be explained by modern science. His current speculations involve quantum non-localities that entangle true random processes in the Universe with cognition, but it is hard to believe, despite no other explanations fit.
I'm with Robert Neve on this. Passive listening works for me, but I really try not to personally judge my own audio designs or anyone elses, these days. I am getting older and somewhat more tolerant to 'poor sound' , also one could be biased just because they are involved in the creation of the sound.
What I do, is pretty much electronic design, searching for the best topology for a given product. My normal work schedule is 'putting out fires' rather than sitting at the test bench, you know, sourcing parts, getting schematics proofed, etc. I use my hi fi just to listen to music, but not at the very hi fi level that I once maintained more than 20 years ago. The only real reference that I have is a pair of tube driven STAX Pro Lambda electrostatic headphones, that I sometimes use for component evaluation.
My recommendations are based on my experience, and unfortunately, on some 'failed' tests, where I could not measure much, but the general public, (and reviewers) rejected the product. I try to learn from my mistakes, and I would hope that others do not 'fail' either.
What I do, is pretty much electronic design, searching for the best topology for a given product. My normal work schedule is 'putting out fires' rather than sitting at the test bench, you know, sourcing parts, getting schematics proofed, etc. I use my hi fi just to listen to music, but not at the very hi fi level that I once maintained more than 20 years ago. The only real reference that I have is a pair of tube driven STAX Pro Lambda electrostatic headphones, that I sometimes use for component evaluation.
My recommendations are based on my experience, and unfortunately, on some 'failed' tests, where I could not measure much, but the general public, (and reviewers) rejected the product. I try to learn from my mistakes, and I would hope that others do not 'fail' either.
SY said:
All a double blind test will tell you is that the subjects couldn't identify a difference in a blind test. It doesn't mean the differences don't exist or aren't audible.
Recently I heard a person with misophonia (dislike for sounds) complaining about someone mowing their yard far away, while he was inside his home. No one else could hear it. I hear about this all the time, and have experienced it too, of course.
Also read about a nurse who had PTSD could hear medical helicopters coming long before anyone else in the hospital could hear them, even when she'd tell them about it.
The brain turns up the gain on sounds with special meaning or importance, while others are habituated (habituation of perception) to these sounds of no significance.
I don't think the way most people go about double blind testing says much at all about what is and isn't audible.
If you go back to the coin toss analogy a blind test is the same thing as tossing an unfair coin and a double blind test is the closest thing anybody knows to tossing a fair one.
In the former case the test is insensitive to sample size because it has an inbuilt error.
If you look at the y axis of a Gaussian distribution as the true figure then tossing an unfair coin also converges to a Gaussian distribution but the mean is offset to one side.
A fair coin one the other hand has its results randomly distributed around the true mean and thus its accuracy is increased by the number of tosses, i.e. the number of samples you have.
In the end why science works is because it is consensual and not individual.
It is necessary for at least two people to agree that there is a world outside of them that has certain characteristics that are so despite what differences in their subjective of it experience might be. As soon as you do this you have statistics because it is the only way we know of separating the objective from the subjective, and if there was no difference then none of the electronic devices we use would work anyway.
rcw
In the former case the test is insensitive to sample size because it has an inbuilt error.
If you look at the y axis of a Gaussian distribution as the true figure then tossing an unfair coin also converges to a Gaussian distribution but the mean is offset to one side.
A fair coin one the other hand has its results randomly distributed around the true mean and thus its accuracy is increased by the number of tosses, i.e. the number of samples you have.
In the end why science works is because it is consensual and not individual.
It is necessary for at least two people to agree that there is a world outside of them that has certain characteristics that are so despite what differences in their subjective of it experience might be. As soon as you do this you have statistics because it is the only way we know of separating the objective from the subjective, and if there was no difference then none of the electronic devices we use would work anyway.
rcw
Which is exactly what a DBT is supposed to aim to, and the reason why it is a well established - and useful - procedure in drug testing. On the other hand, all an open test will tell you is that subjects are actually able to identify a difference when they know a difference exists - not really useful.
L.
Part of the dilemma is that some people have heard really good audio sound, that's totally convincing, and the others haven't. And the latter therefore believe the former lot are deluding themselves, for various reasons. So the divide remains, those that know what's possible are quite content, in one sense, to be disregarded, they have the quiet satisfaction of knowing the "higher truth"; and the other lot keeps haranguing these miscreants, in their desire to ensure, enable a uniformity of thinking and beliefs.
Me ..., like yesterday, just wind up the system on some reasonably well recorded blues to what some would call deafening, but which are just realistic sound levels, what it would sound like, standing near the muso's, and just smile ...
Frank
Last edited:
Exactly, and after I heard possible quality of reproduction I hear how "different" all setups "from the box in the store" are.
After somebody hears how close to real performance can be sound reproduction that person will never forget it and the difference between "The Real High End" and "Nice sounding set-up" will be obvious. It is like "ear-opening" experience.
Unfortunately I never heard such set-ups that don't beg for some improvements, so the journey is going on. Closer and closer to complete fooling the imagination as if it sounds "here and now", but still not there.
Yes, that's the next "headache" ...
. The better it gets, the more the subtle deficiencies can be heard, the more you can hear into the mix, and the slightest edginess stands out like a sore thumb. But, my happy "discovery", , is that ultimately they can all be knocked over, all these lower and lower level problem areas, until you can't actually imagine it being able to be better ...It really is like tuning a F1 machine, the slightest maladjustment will throw the performance off kilter relatively severely, it can be tiresomely frustrating a lot of the time ...
Frank
Well, I 've certainly had the experience of trying something which does improve one aspect but another worsens, or appears to do so. This is the subjective quality of the sound that I'm talking about, of course. And there are a number of aspects to this ...
One is that frequently the sound is more detailed after changes but apparently also harsher. This I always take as a good sign, this is the proverbial "lifting of the veils", I'm going in the right direction. A smooth but non-informative sound is definitely taking the wrong turn; everything that tells me more and more about what has been captured on the recording is good. This of course means that difficult tracks are a nightmare to listen to for the time, but worry not: this is cutting with coarse strokes to get to the heart of the matter. The last step, so to speak, is to resolve the more subtle issues which are the underlying causes of the unpleasantness of the sound -- this is the sandpapering to continue the analogy -- and then it all falls into place - you get the big sound which is effortless to listen to ...
And another aspect is that when one parameter of sound quality is improved it just allows you to hear "deeper" into the sound: things that were always "wrong", before, are now unmasked, so another round of effort is needed to knock over, smooth out, the next level of less obvious "bumpy" bits ..
Frank
One is that frequently the sound is more detailed after changes but apparently also harsher. This I always take as a good sign, this is the proverbial "lifting of the veils", I'm going in the right direction. A smooth but non-informative sound is definitely taking the wrong turn; everything that tells me more and more about what has been captured on the recording is good. This of course means that difficult tracks are a nightmare to listen to for the time, but worry not: this is cutting with coarse strokes to get to the heart of the matter. The last step, so to speak, is to resolve the more subtle issues which are the underlying causes of the unpleasantness of the sound -- this is the sandpapering to continue the analogy -- and then it all falls into place - you get the big sound which is effortless to listen to ...
And another aspect is that when one parameter of sound quality is improved it just allows you to hear "deeper" into the sound: things that were always "wrong", before, are now unmasked, so another round of effort is needed to knock over, smooth out, the next level of less obvious "bumpy" bits ..
Frank
The whole point is that all of these types of considerations are controlled for by the fact that if the participants dont know what they are listening to, or for, then priori assumptions of these sorts are not relevant because you have to have information to hang them on.
This is why the single blind test that seems beloved of some in diy audio is invalid because one lot of participants know what is being tested and in what order.
If you then only indicate that a switch has taken place, even if it is switched to the same device again and ask someone whether they liked the sound before or after such a nominal switch event, then if there is truly no audible difference the statistical results will be randomly distributed, and in the case of a truly double blind test the skewing effect is only due to the sample size.
All of the subjective baggage that people might bring is as far as possible eliminated, and it is very unlikely that a randomly selected sample will be skewed in any way by such considerations anyway.
This type of test is also applicable to a single individual as well as a randomly selected group, but manufactures are generally not interested in single people but in demographic groups.
rcw
This is why the single blind test that seems beloved of some in diy audio is invalid because one lot of participants know what is being tested and in what order.
If you then only indicate that a switch has taken place, even if it is switched to the same device again and ask someone whether they liked the sound before or after such a nominal switch event, then if there is truly no audible difference the statistical results will be randomly distributed, and in the case of a truly double blind test the skewing effect is only due to the sample size.
All of the subjective baggage that people might bring is as far as possible eliminated, and it is very unlikely that a randomly selected sample will be skewed in any way by such considerations anyway.
This type of test is also applicable to a single individual as well as a randomly selected group, but manufactures are generally not interested in single people but in demographic groups.
rcw
Frank, you should really be more careful about protecting your ears. Live levels can damage hearing of course, but also cause hyperacusis. Audiophiles have a tendancy to gradually turn up the volume over time to get that same level of excitement. This is also an example of habituation. Our tolerances increase but our auditory system can only handle so much. It's not just the sound level but the length of time you listen at that level that causes damage. Better to use restraint than pay later, IMHO.
- Status
- Not open for further replies.