Temporary conclusion: we should focus on pre-conceptions if we want to "build a good hifi system".
It hurts me a bit to throw out all my equipment, including solder/multimeter/woodworking tools.
So far, I don't know how to build anything concrete out of pure psychic effort. Surely a bad preconception of mine. Pleeeeaaase Heelp!
It hurts me a bit to throw out all my equipment, including solder/multimeter/woodworking tools.
So far, I don't know how to build anything concrete out of pure psychic effort. Surely a bad preconception of mine. Pleeeeaaase Heelp!
Konnichiwa,
Yes. And it is clear that IF they genuine believed they heard a difference sighted they would be highly likely to score RANDOM under blind conditions. WHY? Because they EXPECT a difference and will thus hear one EVEN IF NON exists. This is the excact reverse of the condition where the subjects hears no difference in the presence of one as he believes no difference exists.
To recap, blind tests have value ON SUBJECTS where there is a significant emotional value to the outcome ONLY if parameters independent from the actual concious perception can be tested (like it was done by Oohasi et al in their test for the audibility of ultrasonic in music - they measured the actual brain activity and found a correlatable, repeatebale and significant difference in this) OR if the subjects are completely unaware of what ie being tested.
I'll hold that position for another 13 pages of discussion if you like.
Sayonara
SY said:
Yes. And it is clear that IF they genuine believed they heard a difference sighted they would be highly likely to score RANDOM under blind conditions. WHY? Because they EXPECT a difference and will thus hear one EVEN IF NON exists. This is the excact reverse of the condition where the subjects hears no difference in the presence of one as he believes no difference exists.
To recap, blind tests have value ON SUBJECTS where there is a significant emotional value to the outcome ONLY if parameters independent from the actual concious perception can be tested (like it was done by Oohasi et al in their test for the audibility of ultrasonic in music - they measured the actual brain activity and found a correlatable, repeatebale and significant difference in this) OR if the subjects are completely unaware of what ie being tested.
I'll hold that position for another 13 pages of discussion if you like.
Sayonara
Kuei Yang Wang said:
You are wrong here KYW. You are saying these genuine believers hear a difference every time. That means even when they know they might be listening to the same amp. Because that is what this whole thing is all about!
You may be listening to another amp, but it may just as well be the same. They do not EXPECT a difference, because they do not know whether the another amp is connected to the speakers, or its the same one they listened to before...
You were ofcourse speaking to SY, but I personally would find it easier if you'd immediately agree on the above
Konnichiwa,
No, BUT you claimed them as supporting your contention "all gear sounds the same" (I know that this is not your exact position, but it's close enough) and to support your claim that people who spend money because their knowledge and experience disagree with yours are waisting their money as there is no real difference.
In doing so you in effect claimed that your contention was universally applicapble and that is a claim that has the implication "It is scientifically proven beyond reasonable doubt that my contention is true."
And that I have to contend with.
Sayonara
keyser said:
No, BUT you claimed them as supporting your contention "all gear sounds the same" (I know that this is not your exact position, but it's close enough) and to support your claim that people who spend money because their knowledge and experience disagree with yours are waisting their money as there is no real difference.
In doing so you in effect claimed that your contention was universally applicapble and that is a claim that has the implication "It is scientifically proven beyond reasonable doubt that my contention is true."
And that I have to contend with.
Sayonara
Kuei Yang Wang said:
I think you are overstating things a bit. I did not implicate:"It is scientifically proven beyond reasonable doubt that my contention is true."
Yes, I do claim my experience with double blind tests to support my view. You actually did the exact same thing. You have stated somewhere in this thread (I am not gonna search) that you have taken part in DBT's and that you WERE able to hear differences. I don't think you said this NOT to support your contention "all gear sounds differently" (I know that this is not your exact position, but it's close enough)
Nonsequitor. We're talking about testing two different items that people claim sound different. .
I know, for example, that two amps with a level difference of 0.3dB will be distinguishable. I expected this from extensive published work. Why, then, do I score significantly in a blind test? Why do I score significantly in blind tests where there are subtle alterations of frequency response? I know what they are, I know that they're above published thresholds, yet somehow this knowledge doesn't scramble my brain.
And, turning to professional matters, a very famous wine critic set up a blind test of California versus French wines made from the same grapes (he believes that regional differences are unimportant). Although my tasting partner and I firmly believe that there are differences and we have a pretty good handle on the markers, we scored 25 right out of 27, much to the famous critic's distress and surprise. How the heck did we manage to do that? Why didn't our preconceptions cause us to score randomly?
Sy and I seem to be on the same track here.
The way of DBT he explained is an in science commonly used testing method. I'd like to my own post, #123.
Really KYW, I'll quote the part where you are wrong:
This is not true, because the subject on test does not know whether he is listening to the same amp, or another one.
The way of DBT he explained is an in science commonly used testing method. I'd like to my own post, #123.
Really KYW, I'll quote the part where you are wrong:
This is not true, because the subject on test does not know whether he is listening to the same amp, or another one.
Konnichiwa,
I would suggets that the actual differences where sufficiently large that for EXPERIENCED TASTERS (who seem to do little else) the difference overrode the expectation.
Moreover, I notice that Sound Professionals (eg sound engineers score much better in ABX/DB Tests thest than "goldenears", a good example is Tony Faulkner, who seems a rather persistent "lucky coin" in any tests he participates (I could list myself too, but that wouyld be telling).
Sayonara
SY said:
I would suggets that the actual differences where sufficiently large that for EXPERIENCED TASTERS (who seem to do little else) the difference overrode the expectation.
Moreover, I notice that Sound Professionals (eg sound engineers score much better in ABX/DB Tests thest than "goldenears", a good example is Tony Faulkner, who seems a rather persistent "lucky coin" in any tests he participates (I could list myself too, but that wouyld be telling).
Sayonara
Konnichiwa,
I referred to DB Tests in general, not specifically equipment. One example I am well aware of was one "publicity" ABX test in which the Music industry wanted to demonstrate the "inaudibility" of a variety of watermarking schemes. Despite abysimal test setup and conditions TF got a "perfect" score.
And?
Sayonara
keyser said:
I referred to DB Tests in general, not specifically equipment. One example I am well aware of was one "publicity" ABX test in which the Music industry wanted to demonstrate the "inaudibility" of a variety of watermarking schemes. Despite abysimal test setup and conditions TF got a "perfect" score.
keyser said:
And?
Sayonara
To slowmotion: thanks!
To the similar minded (keeping subject in mind):
- fix speakers first. The opposite of the Linn mantra. Without a good transducer, how can you ever evaluate your sources properly? The speakers do the final signal transduction and their type and build heavily influences room interaction. You could say the speaker acts as the penultimate measurement device of your setup (ultimate - and the purpose of it all: your ears).
As a result, with better speakers and better room interaction control, you will have a better tool in hand to judge what came before them. You then might notice quality differences that previously escaped you. Use headphones too.
- train your ears. Buy test CD's or download test files from ABX site etc: once you know what some types of distortion sound at exagerrated levels, you will recognize them more readily at low levels. Experiment by purposely mistuning your system (particularly easy in DIY active crossovers for instance). You will learn what types of errors produce what type of sound signature. For instance, a hump in the 2-3 kHz region can sound like more detail. Exaggerated midbass level will muddy the midrange. Etc.
- do blind tests on occasion for fun. Don't claim universal applicability in the forums. But use the results for your own purposes.
- when doing tests, use a lot of different CD's. Test over a long period of time (not in one sitting I mean - different days, different moods, different people, different CD's, different SPL) rather than in short 10 second bursts switching a/b using a single CD. Listen for how different CD's sound compared to each other. Don't try to figure out whether one single CD sounds "different".
- DIY some simple amps, say chip amps, and hear what differences can result from simple and seemingly unremarkable changes.
- record familiar sounds and play them back on your system. Cheap mics OK. Test various links of your chain.
etc. This is DIYaudio after all
To the similar minded (keeping subject in mind):
- fix speakers first. The opposite of the Linn mantra. Without a good transducer, how can you ever evaluate your sources properly? The speakers do the final signal transduction and their type and build heavily influences room interaction. You could say the speaker acts as the penultimate measurement device of your setup (ultimate - and the purpose of it all: your ears).
As a result, with better speakers and better room interaction control, you will have a better tool in hand to judge what came before them. You then might notice quality differences that previously escaped you. Use headphones too.
- train your ears. Buy test CD's or download test files from ABX site etc: once you know what some types of distortion sound at exagerrated levels, you will recognize them more readily at low levels. Experiment by purposely mistuning your system (particularly easy in DIY active crossovers for instance). You will learn what types of errors produce what type of sound signature. For instance, a hump in the 2-3 kHz region can sound like more detail. Exaggerated midbass level will muddy the midrange. Etc.
- do blind tests on occasion for fun. Don't claim universal applicability in the forums. But use the results for your own purposes.
- when doing tests, use a lot of different CD's. Test over a long period of time (not in one sitting I mean - different days, different moods, different people, different CD's, different SPL) rather than in short 10 second bursts switching a/b using a single CD. Listen for how different CD's sound compared to each other. Don't try to figure out whether one single CD sounds "different".
- DIY some simple amps, say chip amps, and hear what differences can result from simple and seemingly unremarkable changes.
- record familiar sounds and play them back on your system. Cheap mics OK. Test various links of your chain.
etc. This is DIYaudio after all
On a Dutch DIY forum (www.zelfbouwaudio.nl) I have proposed to do a DBT at my place. I will do some further investigation on the protocol of a scientific DBT.
Has anyone of you ever set up of partaken in a DBT? Any advice on how to set it all up would be welcome..
Has anyone of you ever set up of partaken in a DBT? Any advice on how to set it all up would be welcome..
I ran quite a few audio DBTs some years ago- I was a dedicated high-ender pretty ticked off about the assertions made by that Lipshitz clown and wanted to demonstrate where he went wrong. After all, he didn't use speakers like mine or wonderful amps like mine... Anyway, it was an epiphany. I couldn't hear differences that I knew had to be there. Neither could my sharp-eared audiophile girlfriend or any of my audiophile buddies (who afterwards started avoiding me).
I've also (in my professional life) set up controlled sensory tests in haptics and organoleptics; the principles are quite similar other than the bellowing of gored oxen.
In any case, the biggest difficulty with controlled subjective testing in audio is the level-matching requirement. Getting levels matched within 0.1dB and keeping them that way is tedious and difficult but vitally necessary. It takes a precision attenuator and AC voltmeter. And a lot of patience. The problem is that, for tiny level changes, the human ear/brain doesn't recognize the differences as level but rather as subtle differences in clarity (that's probably the origin of a lot of tales of one amp trouncing another in a competition).
If you want to do a specific experiment, I'd be happy to help you design the protocol.
I've also (in my professional life) set up controlled sensory tests in haptics and organoleptics; the principles are quite similar other than the bellowing of gored oxen.
In any case, the biggest difficulty with controlled subjective testing in audio is the level-matching requirement. Getting levels matched within 0.1dB and keeping them that way is tedious and difficult but vitally necessary. It takes a precision attenuator and AC voltmeter. And a lot of patience. The problem is that, for tiny level changes, the human ear/brain doesn't recognize the differences as level but rather as subtle differences in clarity (that's probably the origin of a lot of tales of one amp trouncing another in a competition).
If you want to do a specific experiment, I'd be happy to help you design the protocol.
Konnichiwa,
By defining the purpose of your experiment. In other words, what do you wish to test? Then you should considerdesigning a crosscheck test, in other words you should include a second test that to the participants is identical to the first but uses known audible differences instead of the actual test item in order to establish that your test is in fact sensitive enough to reliably shown differences known to be audible.
There is plenty of material on the net on the subject.
Before proceeding to far I would however recommend a collection of articles and letters compiled by Stereophile on the subject of DB Testing, which might help you in uderstanding the implication and limitations of the methode itself:
The Highs & Lows of Double-Blind Testing
Blind Listening
Also, note some basic material on (non-audio) Double Blind testing:
http://www.answers.com/topic/double-blind
You can then proceed to the site maintained on behalf of those I prefer to call the ABX Mafia:
ABX Double Blind Comparator
Be cautious in assuming their test data presented was derived from well designed and well implemented DB Tests, that is not the case in most if not all of their published tests, plus their tests invariably have very high level of Type 2 Statistical Error possibilty.
Finally a very usefull site to get further information on double blind audio testing and how to set up such tests well, plus a downloadable ABX Comperator for the PC AND downloadable samples to train ones critical hearing is here:
http://www.pcabx.com
Sayonara
keyser said:
By defining the purpose of your experiment. In other words, what do you wish to test? Then you should considerdesigning a crosscheck test, in other words you should include a second test that to the participants is identical to the first but uses known audible differences instead of the actual test item in order to establish that your test is in fact sensitive enough to reliably shown differences known to be audible.
keyser said:
There is plenty of material on the net on the subject.
Before proceeding to far I would however recommend a collection of articles and letters compiled by Stereophile on the subject of DB Testing, which might help you in uderstanding the implication and limitations of the methode itself:
The Highs & Lows of Double-Blind Testing
Blind Listening
Also, note some basic material on (non-audio) Double Blind testing:
http://www.answers.com/topic/double-blind
You can then proceed to the site maintained on behalf of those I prefer to call the ABX Mafia:
ABX Double Blind Comparator
Be cautious in assuming their test data presented was derived from well designed and well implemented DB Tests, that is not the case in most if not all of their published tests, plus their tests invariably have very high level of Type 2 Statistical Error possibilty.
Finally a very usefull site to get further information on double blind audio testing and how to set up such tests well, plus a downloadable ABX Comperator for the PC AND downloadable samples to train ones critical hearing is here:
http://www.pcabx.com
Sayonara
keyser, relying on Stereophile to learn about blind testing is like learning about the history of Israel from the PLO.
Go read the JAES articles by Stanley Lipshitz, Dick Greiner, Floyd Toole, and other real professionals. Look up the NRC (Canada) literature.
When you decide you're ready to try something, I'll be happy to fill in details on how to do it properly.
Go read the JAES articles by Stanley Lipshitz, Dick Greiner, Floyd Toole, and other real professionals. Look up the NRC (Canada) literature.
When you decide you're ready to try something, I'll be happy to fill in details on how to do it properly.
Konnichiwa,
Hmmm, why must you disallow viewpoints that differ from those you disagree with. Even an advocate of ABX testing should recommend the Stereophile material, as it shows clearly many of the potential problems and pitfalls.
Surely you would agree that well researched and founded viewpoints that disagree with one you support should be taken note of? Surely at the very least intellectual honesty demands that one is aware and informed of viewpoints opposing ones own?
How about articles byt professional satisticians, such as Leventahl (his work and points are well covered in the stereophile material, saving the cost of downloading the JAES Article and his AES Conference preprint from the AES)?
May I suggest that Keyser may be better off informing himself, instead of seeking advise from someone who has a definite belief and conviction on the subject? I do not think either you or myself should suggest to him how to set up his experiments, with all due respect. That is why I attempted to provie a range of materials that illustrate technicques, requirements and contain enough material to set up ones own test.
Sayonara
SY said:
Hmmm, why must you disallow viewpoints that differ from those you disagree with. Even an advocate of ABX testing should recommend the Stereophile material, as it shows clearly many of the potential problems and pitfalls.
Surely you would agree that well researched and founded viewpoints that disagree with one you support should be taken note of? Surely at the very least intellectual honesty demands that one is aware and informed of viewpoints opposing ones own?
SY said:
How about articles byt professional satisticians, such as Leventahl (his work and points are well covered in the stereophile material, saving the cost of downloading the JAES Article and his AES Conference preprint from the AES)?
SY said:
May I suggest that Keyser may be better off informing himself, instead of seeking advise from someone who has a definite belief and conviction on the subject? I do not think either you or myself should suggest to him how to set up his experiments, with all due respect. That is why I attempted to provie a range of materials that illustrate technicques, requirements and contain enough material to set up ones own test.
Sayonara
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.