That is a given, but independent from the ABX case.
They all missed a difference during the "official" listening trials.
Afterwards Lipshitz realized that difference in relais sound was detectable, afair Vanderkooy detected an additional difference in the noise floor.
Something very similiar happended in the detmold DSD/PCM listening tests and in the swedish radio tests of audio codecs as well.
As no positive control under ABX conditions was presented we still don´t know if it was "ears-only" or more lower sensitivity under ABX conditions.
A hint might be the fact, that there were audible differences that remain undetected by Tiefenbrunn.
In the quoted sentence you mentioned that a "careful listener.......", but obviously it not only depends on "careful listening" to detect a difference.
And tests without training time for the participants and proper implemented positive/negative controls belong simply not to the category "good test" .
I think Michael Fremer demonstrated that he was capable to detect differences between amplifiers in his double blind test attempt at the AES convention back in ?1989?
Not explicitely but implicitely, otherwise it would not make any sense to mention questionable test results in which listerners failed to detect something.
Of course and there will never be something like a perfect test.
But only _good_ tests lead to useful results.
And most bias effects don´t vanish in a controlled listening test unless the control addresses these effects carefully.
Safeguards in the case of overlooked effects are positive and negative controls.
Double blind ABX tests inhibit hearing ANY subtle differences, except for the few that have been shown to be detectable. Yet, there are many more differences that the ear can detect, especially over a longer period of time.
The proof of this is our listening experience over the decades. IF only the limited criteria of frequency response, reasonably low distortion, level, and absolute polarity were the ONLY subjective differences between amplifiers, or preamplifiers, then EVERYTHING that I have designed in the last 30 years sounds exactly the same. This is because I have taken all of those factors into account for the last 30 years, and more. I told Dr. Lipshitz that in a LTE to TAA back in 1979.
ANY engineer can meet those criteria, so we might as well give up trying to make something 'better'. And you know, many have!
I beg to differ, because I can hear and remember differences in each and every one of the products that I have personally used, and I have a tendency to 'edge out' a great many competing products, without unnecessary advertising and salesmanship. How is this possible? Just about anyone here should be able to make a preamp with some of the cheapest IC's now available that is equal to the Blowtorch. I invite you to do it. Why not?
Of course there might be something wrong with ABX testing, and I think that there is. Then, each and every one of my designs might have a unique signature. That has been my experience.
The proof of this is our listening experience over the decades. IF only the limited criteria of frequency response, reasonably low distortion, level, and absolute polarity were the ONLY subjective differences between amplifiers, or preamplifiers, then EVERYTHING that I have designed in the last 30 years sounds exactly the same. This is because I have taken all of those factors into account for the last 30 years, and more. I told Dr. Lipshitz that in a LTE to TAA back in 1979.
ANY engineer can meet those criteria, so we might as well give up trying to make something 'better'. And you know, many have!
I beg to differ, because I can hear and remember differences in each and every one of the products that I have personally used, and I have a tendency to 'edge out' a great many competing products, without unnecessary advertising and salesmanship. How is this possible? Just about anyone here should be able to make a preamp with some of the cheapest IC's now available that is equal to the Blowtorch. I invite you to do it. Why not?
Of course there might be something wrong with ABX testing, and I think that there is. Then, each and every one of my designs might have a unique signature. That has been my experience.
No, he didn't. Audiophile lore pumped up by Fremer. When you have 5 trials, 100 participants, and someone doesn't turn in his score sheet, what you have is hype, not results.
As for the rest of what you wrote about Tiefenbrun's demonstration that, by ear alone, he could not detect a 16 bit ADC/DAC inserted into an analog feed, it does not accord with the published account of the test nor the recollections of at least one observer who was there.
Circular logic. And patently untrue. But hey, if you can't hear differences, you can't hear differences. Nothing wrong with that, it doesn't diminish the quality of your designs.
Might be, but it was 5 trial dbt, Fremer did turn in his score sheet (John Atkinson did as well), and Fremer correctly identified the amplifier playing 4 times out of 5. So you got Fremer correct 5 times out of 5 on the same/difference question, correct 4 times out of 5 for the question which amplifier (out of 3 possible) was playing. Atkinson and Fremer as a tag team were correct 9 times out of 10 for the same/difference question.
I don´t know if there were 100 participants overall, but it was reported that no other attendee of the AES-convention, who took the test reached a significant result.
Care to calculate the probabilities that Fremer was just guessing and that Fremer and Atkinson as a tag team were just guessing?
Yeah, it was Lipshitz (and not Vanderkooy) who after Tiefenbrunn´s test realized that the noise level could be detectable; both Lipshitz and Vanderkooy knew already in front of the test, that the relay switching noice could be detectable.
Does this really make an important difference to my description wrt to the sensitivity of that test?
Sorry for the inconvenience, but i am not able to see my incorrectness:
Boston Audio Society - ABX Testing article
(written by Stanley P. Lipshitz)
The "Fremer incident" seems to be what i wrote, due to the description of Michael Fremer; afaik no one, not Clark, not Lipshitz expressed that Fremers description of his results was wrong.
Could you please quote the source, which tells other numbers or another story?
I am afraid, but it was not, as the "formal" trials were the ones Tiefenbrunn did.
After that was finished, Lipshitz realized that "at this point i" ....
But, this was not the point, because it confirmed just that a difference, that could have been detected, remained undetected.
As no (other) positive control was tested, we simply don´t know if Tiefenbrunn´s sensitivity was just low under test conditions, or if his sensitivity overall was to low.
And we don´t know if another difference in sound quality, that might have been comparable in effect size to the "hiss problem" or the "realy switching sound problem", remained undetected too.
Last edited:
Thanks for your comment.
In addition to the positive/negative controls, training for the participants is mandatory too.
It is sometimes frustrating that these basic concepts of scientific testing are still not used routinely in audio tests .
@ SY,
sorry, but Tiefenbrunn was challenged and obviously he failed to make the (unofficial) positive controls and the actual task as well. So the normal reaction would have been to let him have some training under test conditions to get used to these and to try again afterwards.
Seems that you have missed the argument.
Please remember to quote the source which corrects Fremers description of the amplifer-DBT; i am very interested, because normally Clark, Nousaine or Krueger commented on every such statement (especially if untrue), but i can´t remember any rebuttal.
DBT
Most everyone has a personal DBT comparison experience to relate, here is the most significant of mine:
In 1988 I was part of a team which demonstrated the first 16-bit high-speed cassette digital bin, running right on the floor of the AES (at the NY Hilton). We duped, loaded and handed out the cassettes of Sheffield source material to interested parties and encouraged them to take the cassettes up to our listening suite and compare them against the source CD. We rented a double suite up in the hotel and had set up a LEDE monitoring system complete with full-wall diffusers, absorption and a UREI monitoring system, all designed and installed with the assistance of Doug Sax. Dan D'Agostino declared our room the best listening environment at the show and brought several clients to it to demonstrate his LEDE techniques.
In addition to allowing the parties to control the source via remote, we also asked if they wanted to participate in a DBT, and had a highly tweaked PC-based DBT switcher with MIL hermetically sealed bifurcated gold contact relays.
The interesting part of the whole experience is the reaction that many of the 200+ world class producers, engineers and musicians showed to the experience. Those who were open minded took the plunge and some indeed described differences. I will never divulge names, but MANY (whom all of you know) started in a very cocky way, then looked very confused part-way through the DBT and claimed they had appointments elsewhere and left in a REAL hurry. This certainly shows the stress that a DBT can cause, and which IMHO will destroy a person's ability to be critical.
All in all, an interesting mish-mash of experiences. Conclusions:
1) A properly made cassette can be vanishingly close to a 16-bit well recorded digital source. Only four or so of the hundreds who took the DBT scored in a statistically significant way. Interestingly enough they were ALL in the cassette industry, particularly Dolby engineer Dennis Staats and our mastering engineer Ellen Threatt repeatedly scored perfectly. This proved that there were indeed audible differences. The test set-up was NOT obscuring the differences.
2) When the effects that these few people were detecting were pointed out to others, the other's scores improved markedly. This shows to me that familarity with the effects being listened for enhances the ability to detect the effect (duh).
3) DBTs are psychologically problematic.
4) Many people in the music industry have enormous egoes (duh), and some do not like exposing themselves to scrutiny, no matter how even the playing field. This makes them rely on their own criteria for claiming sonic superiority instead of agreed-upon criteria which can be mutually discerned and experimented with.
5) Different people can become accustomed to different listening environments (just as with dietary choices) and will not in the short-run like the sound in a different one.
6) My overall conclusion is that the variability of the test conditions when compared to a person's usual listening situation often outweigh the differences which may exist. However if open minds and critical discussions ensue in conjunction with the DBT, many people learn something and gain new insights.
Apologies for the long post, yada yada....my $0.02 worth...
Howie
Howard Hoyt
CE - WXYC-FM
UNC Chapel Hill, NC
www.wxyc.org
1st on the internet
Most everyone has a personal DBT comparison experience to relate, here is the most significant of mine:
In 1988 I was part of a team which demonstrated the first 16-bit high-speed cassette digital bin, running right on the floor of the AES (at the NY Hilton). We duped, loaded and handed out the cassettes of Sheffield source material to interested parties and encouraged them to take the cassettes up to our listening suite and compare them against the source CD. We rented a double suite up in the hotel and had set up a LEDE monitoring system complete with full-wall diffusers, absorption and a UREI monitoring system, all designed and installed with the assistance of Doug Sax. Dan D'Agostino declared our room the best listening environment at the show and brought several clients to it to demonstrate his LEDE techniques.
In addition to allowing the parties to control the source via remote, we also asked if they wanted to participate in a DBT, and had a highly tweaked PC-based DBT switcher with MIL hermetically sealed bifurcated gold contact relays.
The interesting part of the whole experience is the reaction that many of the 200+ world class producers, engineers and musicians showed to the experience. Those who were open minded took the plunge and some indeed described differences. I will never divulge names, but MANY (whom all of you know) started in a very cocky way, then looked very confused part-way through the DBT and claimed they had appointments elsewhere and left in a REAL hurry. This certainly shows the stress that a DBT can cause, and which IMHO will destroy a person's ability to be critical.
All in all, an interesting mish-mash of experiences. Conclusions:
1) A properly made cassette can be vanishingly close to a 16-bit well recorded digital source. Only four or so of the hundreds who took the DBT scored in a statistically significant way. Interestingly enough they were ALL in the cassette industry, particularly Dolby engineer Dennis Staats and our mastering engineer Ellen Threatt repeatedly scored perfectly. This proved that there were indeed audible differences. The test set-up was NOT obscuring the differences.
2) When the effects that these few people were detecting were pointed out to others, the other's scores improved markedly. This shows to me that familarity with the effects being listened for enhances the ability to detect the effect (duh).
3) DBTs are psychologically problematic.
4) Many people in the music industry have enormous egoes (duh), and some do not like exposing themselves to scrutiny, no matter how even the playing field. This makes them rely on their own criteria for claiming sonic superiority instead of agreed-upon criteria which can be mutually discerned and experimented with.
5) Different people can become accustomed to different listening environments (just as with dietary choices) and will not in the short-run like the sound in a different one.
6) My overall conclusion is that the variability of the test conditions when compared to a person's usual listening situation often outweigh the differences which may exist. However if open minds and critical discussions ensue in conjunction with the DBT, many people learn something and gain new insights.
Apologies for the long post, yada yada....my $0.02 worth...
Howie
Howard Hoyt
CE - WXYC-FM
UNC Chapel Hill, NC
www.wxyc.org
1st on the internet
John, it will help to read the BAS writeup. Tiefenbrun, who volunteered to demonstrate the sound-fouling qualities of digital recording, believed that he was able to hear the PCM-F1 insertion when he could see it. As soon as he couldn't, he was unable to distinguish by ear. There were a few sideshows as well, including Tiefenbrun's claim that any other transducers in the room would ruin the sound, where he also scored a null result.
Lipshitz was very gracious about the whole thing and suggested that other testing might show audible differences when the levels were set correctly (eliminating the noise). To date, no one has produced any contradictory data. For all the huffing, puffing, and attempts at obfuscation, contradictory evidence is... zero.
On a different note, I was just sent a copy of a recent AES preprint by Gaskell et al (#8503 from NY Oct 2011) on audibility of opamps. Very interesting work- there are some holes and nits to pick, but they seem to have made a decent effort to back up their claims with data.
Lipshitz was very gracious about the whole thing and suggested that other testing might show audible differences when the levels were set correctly (eliminating the noise). To date, no one has produced any contradictory data. For all the huffing, puffing, and attempts at obfuscation, contradictory evidence is... zero.
On a different note, I was just sent a copy of a recent AES preprint by Gaskell et al (#8503 from NY Oct 2011) on audibility of opamps. Very interesting work- there are some holes and nits to pick, but they seem to have made a decent effort to back up their claims with data.
Good input Howie.
As far as the test of Ivor Tiefenbrun is concerned, he got himself in a bad situation through his overconfidence that he could master ANY test put before him. He failed, AND decades later, his ONE test is brought up by those who don't believe in the differences that both he and I have heard together for the last 35 years. Yes, I use his phono turntable, and have done so since 1974.
SY would just LOVE to get me into a similar test, and then 2 decades later, my 'failure' could be broadcast far and wide, as well. It would serve me right.
As far as the test of Ivor Tiefenbrun is concerned, he got himself in a bad situation through his overconfidence that he could master ANY test put before him. He failed, AND decades later, his ONE test is brought up by those who don't believe in the differences that both he and I have heard together for the last 35 years. Yes, I use his phono turntable, and have done so since 1974.
SY would just LOVE to get me into a similar test, and then 2 decades later, my 'failure' could be broadcast far and wide, as well. It would serve me right.
- Status
- Not open for further replies.