Listening tests
In response to:
To those who belive so solidly in blind testing, I say that the entire test setup and methodology needs to be examined very closely, and this very much includes the skill levels of the listeners.
ITU BS1116-1 addresses these issues. The standard calls for a double blind triple reference with hidden stimulus test. The test subjects must qualify as expert listeners to participate in the test. This is all explained in the ITU document which can be downloaded for free from the ITU website.
-UncleJessie
In response to:
To those who belive so solidly in blind testing, I say that the entire test setup and methodology needs to be examined very closely, and this very much includes the skill levels of the listeners.
ITU BS1116-1 addresses these issues. The standard calls for a double blind triple reference with hidden stimulus test. The test subjects must qualify as expert listeners to participate in the test. This is all explained in the ITU document which can be downloaded for free from the ITU website.
-UncleJessie
i just happened to be reading the A75 article on PassDiy web site
and thought Mr. Pass' comments applied to what is being
discussed in this thread.
The right brain/left brain theory applies here. You use one side
of your brain for objective logical kinds of processing, and the
other for subjective and emotional. When you are enjoying music,
the subjective half is in play, and its process does not relate to
the specs very well.
The same phenomenon explains why A/B comparisons, no matter
how well staged, do not resolve what audiophiles claim to hear.
When you are sitting there trying to hear the differences
between products, the objective side of your brain begins
working and your subjective responses get locked out by the
pressure to make an objective decision.
I think i notice differences in components after i've lived with them
for a while. I just put a new pre-amp in my system, and over the
last few days of listening i've noticed myself becoming more
involved with the music - it just feels better - as well as sounds
better. just my 2 cents worth.
m.
and thought Mr. Pass' comments applied to what is being
discussed in this thread.
The right brain/left brain theory applies here. You use one side
of your brain for objective logical kinds of processing, and the
other for subjective and emotional. When you are enjoying music,
the subjective half is in play, and its process does not relate to
the specs very well.
The same phenomenon explains why A/B comparisons, no matter
how well staged, do not resolve what audiophiles claim to hear.
When you are sitting there trying to hear the differences
between products, the objective side of your brain begins
working and your subjective responses get locked out by the
pressure to make an objective decision.
I think i notice differences in components after i've lived with them
for a while. I just put a new pre-amp in my system, and over the
last few days of listening i've noticed myself becoming more
involved with the music - it just feels better - as well as sounds
better. just my 2 cents worth.
m.
When listening to recorded music I don't try to remember sonic signature of a given equipment or setup, because when differences are small it may become mind boggling indeed. I'd rather prefer to relax and identify my own reaction to the sound I hear. When something is not right, it's usually very obvious, because I can't concentrate on music and sounds exist beside me, without any emotional content on my side. When everything is right, or when the change I just did was significant the music grabs me and gets me involved (at least for awhile) and I'm eager to try another recording to see how it's different this time.
I'm sure many of you went through that and my question is, how this happens, when seemingly in DBRCT you cannot identify the subject consistantly?
I guess I got my answer in previous post
I'm sure many of you went through that and my question is, how this happens, when seemingly in DBRCT you cannot identify the subject consistantly?
I guess I got my answer in previous post
Some really good feedback here! Overall, there appears to be more of you who support blind testing (on some level) than those who do not.
Someone else on another forum said something like: "If buying a $1000 rock and setting it on my amplifier makes it sound $1000 better to my ears, it was $1000 well spent." I can't argue with that logic. Some people buy a performance or luxury car not for the performance or the luxury but because it makes them feel good to own/drive it.
One of you said:
"But I have always been amused at how 'night and day' differences tend to simply vanish once the listener is simply denied the knowledge of which component they're listening to."
The above really sums up my experiences in one sentence. The following is a very good example:
I recorded a friend's audiophile "reference" vinyl album (the one he most often uses for equipment evaluation) onto CD-R when CD-R was still a relatively new thing for audio. I used a studio-grade sound card, my above average turntable/arm/cartridge, and a high quality phono preamp of my own design. I then started the real vinyl playing on his system AND the CD-R playing in his CD player and invited him in to allegedly listen to some new cables I'd brought over and wanted his golden eared opinion on.
This person LOVES vinyl and, pretty much hated CDs. He despised all things digital and had a stellar system with a tricked out Linn LP12 and Ayre electronics driving B&W 604s. He sat down to listen to the new cables and said they were very similar to his own, but he picked out some minor differences. When I revealed he was listening to a CD recording of his vinyl, he didn't believe me. He got up and looked at the input selector on the Ayre preamp and suddenly was angry and defensive!
In reality, he was not only listening to his much hated CD format, but a computer soundcard ADC, a different turntable, different tonearm, different cartridge and different phono section! Now, in fairness, this wasn't a back-to-back test. It had been a few weeks since he'd even listened to that album, but the results were stunning. He not only enjoyed listening to a CD, but one recorded on a computer on someone else's set-up! This was a guy who would spend all day aligning his cartridge and raising and lowering the tonearm until he got the sound just right.
We later did an A/B blind test with the CD vinyl recording level matched to the vinyl. We both passed the blind test with flying colors. The real vinyl sounded plainly better than the CD-R.
But the point of this story, is to show the power of the human mind. When he THOUGHT he was listening to his carefully tweaked turntable, the stunning Ayre phono section, etc. that's exactly what he heard. He only heard tiny differences he attributed to the interconnects I'd brought over. All the familiar ticks and pops were there. He had no reason to believe he was NOT listening to his turntable so he heard what he expected to hear.
Vinyl lovers (including the one above) claim that CDs hurt their ears and have all sorts of gross offensive sonic flaws. They would certainly claim to know one when they heard it. It's not too different than someone who claims they could easily pick out a few hundred dollar mainstream receiver or integrated amp from a quality power amp. It turns out, it's not so easy!
As for "all amps sound the same", I'm not saying that by any means. As I said in my original post, some measure different, and hence sound different. They also differ widely in their power and current (low impedance) capabilities.
Those of you who argue even if they measure the same, they don't perform the same driving real speakers with real music, I suggest you investigate null tests where you can objectively and subjectively evaluate an amplifiers performance doing just that.
By listening to the difference signal, you can get a feel for what sort of distortions the amplifier is producing and by measuring its amplitude and spectral content, you can have a good objective indicator of how it's altering the signal under real-world conditions.
Someone else on another forum said something like: "If buying a $1000 rock and setting it on my amplifier makes it sound $1000 better to my ears, it was $1000 well spent." I can't argue with that logic. Some people buy a performance or luxury car not for the performance or the luxury but because it makes them feel good to own/drive it.
One of you said:
"But I have always been amused at how 'night and day' differences tend to simply vanish once the listener is simply denied the knowledge of which component they're listening to."
The above really sums up my experiences in one sentence. The following is a very good example:
I recorded a friend's audiophile "reference" vinyl album (the one he most often uses for equipment evaluation) onto CD-R when CD-R was still a relatively new thing for audio. I used a studio-grade sound card, my above average turntable/arm/cartridge, and a high quality phono preamp of my own design. I then started the real vinyl playing on his system AND the CD-R playing in his CD player and invited him in to allegedly listen to some new cables I'd brought over and wanted his golden eared opinion on.
This person LOVES vinyl and, pretty much hated CDs. He despised all things digital and had a stellar system with a tricked out Linn LP12 and Ayre electronics driving B&W 604s. He sat down to listen to the new cables and said they were very similar to his own, but he picked out some minor differences. When I revealed he was listening to a CD recording of his vinyl, he didn't believe me. He got up and looked at the input selector on the Ayre preamp and suddenly was angry and defensive!
In reality, he was not only listening to his much hated CD format, but a computer soundcard ADC, a different turntable, different tonearm, different cartridge and different phono section! Now, in fairness, this wasn't a back-to-back test. It had been a few weeks since he'd even listened to that album, but the results were stunning. He not only enjoyed listening to a CD, but one recorded on a computer on someone else's set-up! This was a guy who would spend all day aligning his cartridge and raising and lowering the tonearm until he got the sound just right.
We later did an A/B blind test with the CD vinyl recording level matched to the vinyl. We both passed the blind test with flying colors. The real vinyl sounded plainly better than the CD-R.
But the point of this story, is to show the power of the human mind. When he THOUGHT he was listening to his carefully tweaked turntable, the stunning Ayre phono section, etc. that's exactly what he heard. He only heard tiny differences he attributed to the interconnects I'd brought over. All the familiar ticks and pops were there. He had no reason to believe he was NOT listening to his turntable so he heard what he expected to hear.
Vinyl lovers (including the one above) claim that CDs hurt their ears and have all sorts of gross offensive sonic flaws. They would certainly claim to know one when they heard it. It's not too different than someone who claims they could easily pick out a few hundred dollar mainstream receiver or integrated amp from a quality power amp. It turns out, it's not so easy!
As for "all amps sound the same", I'm not saying that by any means. As I said in my original post, some measure different, and hence sound different. They also differ widely in their power and current (low impedance) capabilities.
Those of you who argue even if they measure the same, they don't perform the same driving real speakers with real music, I suggest you investigate null tests where you can objectively and subjectively evaluate an amplifiers performance doing just that.
By listening to the difference signal, you can get a feel for what sort of distortions the amplifier is producing and by measuring its amplitude and spectral content, you can have a good objective indicator of how it's altering the signal under real-world conditions.
moe29 said:
With all due respect to Mr. Pass, this makes no sense seeing that virtually everyone who tries out a new component, or cable, or tweak, or mod or whatever is engaging in an A/B comparison and audiophiles routinely sit down and try to hear differences and almost invariably they purport that they do hear differences. So this objective side of the brain certainly doesn't seem to be getting in anyone's way.
se
Peter Daniel said:What about the claim that changing passive parts inside the amp changes its sound? It still measures exactly the same (providing same values are used) yet, to some, it sounds totally different. Is there any truth to that?
Well, until you can confidently remove the psychological element, one can't really say with any degree of certainty.
se
Peter Daniel said:
That's pretty much how I approach things. I've never been able to get into that analytical, flyspeck mode of listening and as a consequence have never developed any sort of communications skills to relate "how it sounds."
I simply listen intuitively and go with whatever I end up getting the most subjective pleasure from.
It's like seeing a beautiful woman. You just know it. And when seeing two women, you don't need to examine and compare every pore, mole, nose hair, etc. to know which you find more attractive.
se
Peter Daniel said:What about powering amps from batteries. Any advantage to that, or it's rather also a psychological experience only?
Heheh. Certainly there are technical advantages. Whether they amount to actual audible differences, I've really no idea, nor do I much care. I only know what I like and I never pass off what I like as being anything more than my own personal, subjective preference. That and $5 might get you a cup of coffee these days.
se
moe29 said:
Not to disrespect Nelson Pass at all, but the left/right brain staple of pop psychology has been very much dissproven, see here for a basic article on the subject.
My scientific mind tells me that DB testing is a vital tool in interpreting data with a combined physical and psychological components. Yet I clearly heard differences when building amps with the different GC ICs, with no preconceptions at all, ( I started with LMs weeks before the fashion started). Yet it seems my results were the complete opposite of what Peter Daniels heard. Hmmm
p.s. As a past student of psychology, ( most of which is rubbish), I find it fascinating to observe the group dynamics of this discussion
More on the process itself .....
There are several posts arguing that a DBRCT (that is Double Blind Randomized Conrolled Trial for the few who didn't know) is invalid because you can't use a room full of 'non-golden-ears' and make a valid assessment. "Of course these poor uneducated mugs can't hear a difference ...... they’re Neanderthals!"
This is entirely correct ...... however, does not invalidate the process.
The point is your DBRCT must be constructed properly to answer the question asked. If the measure lacks sensitivity for the factor in question the results become meaningless.
If the question asked is, "can the average man in the street tell the difference between A and B", then the room full of randomly selected people is absolutely correct.
If the question is, "can the average 'golden ear' tell the difference between A and B", then the room should be full of randomly selected 'golden ears'.
If we wish to test forum member X's claim that he/she can easily tell the difference, then the trial is done slightly differently. We use the 1 subject and repeated measures in random order. This might mean conducting the trial over a number of days to remove the "fatigue" element. (plus a lot of other boring bits!)
DBRCT is not a single design entity but a process of approach to answering a question which provides the least bias in the result.
Designing good trials which answer questions accurately with a high degree of certainty is actually a very difficult thing and a science in it's own right.
I return again to the original question, which was (in effect), "is this process a valid way to look at/compare audio" ..... and the answer is that it is the gold standard. We may not like this, however it is a simple truth (within our current paradigm).
cheers
mark
PS: Enjoy the process of design, enjoy the process of making it, take pride in doing it well and most of all enjoy the music at the end.
PPS: For those on a budget, you can save a lot of money avoiding "snake oil" components which will actually make very little (if any) audible difference ....... I think this was one of the main points of the original author and one with which I agree.
There are several posts arguing that a DBRCT (that is Double Blind Randomized Conrolled Trial for the few who didn't know) is invalid because you can't use a room full of 'non-golden-ears' and make a valid assessment. "Of course these poor uneducated mugs can't hear a difference ...... they’re Neanderthals!"
This is entirely correct ...... however, does not invalidate the process.
The point is your DBRCT must be constructed properly to answer the question asked. If the measure lacks sensitivity for the factor in question the results become meaningless.
If the question asked is, "can the average man in the street tell the difference between A and B", then the room full of randomly selected people is absolutely correct.
If the question is, "can the average 'golden ear' tell the difference between A and B", then the room should be full of randomly selected 'golden ears'.
If we wish to test forum member X's claim that he/she can easily tell the difference, then the trial is done slightly differently. We use the 1 subject and repeated measures in random order. This might mean conducting the trial over a number of days to remove the "fatigue" element. (plus a lot of other boring bits!)
DBRCT is not a single design entity but a process of approach to answering a question which provides the least bias in the result.
Designing good trials which answer questions accurately with a high degree of certainty is actually a very difficult thing and a science in it's own right.
I return again to the original question, which was (in effect), "is this process a valid way to look at/compare audio" ..... and the answer is that it is the gold standard. We may not like this, however it is a simple truth (within our current paradigm).
cheers
mark
PS: Enjoy the process of design, enjoy the process of making it, take pride in doing it well and most of all enjoy the music at the end.
PPS: For those on a budget, you can save a lot of money avoiding "snake oil" components which will actually make very little (if any) audible difference ....... I think this was one of the main points of the original author and one with which I agree.
Pissed and Deaf
Sy, I'm not meaning to be be dissing you there.
It is meant as a comment that if you are not hearing subtle differences then it is likely that there is some kind of masking going on.
This masking can be caused by a whole range of causes, ranging from suspect speakers, suspect interconnects, noisey mains power, etc, etc, etc...... .
Your visual system plugged into the same mains source can cause masking too, even if turned off I find.
I note that according to your website, you are into wines in a seemingly big way, and as has been shown to me many times, alcohol is fine for the soul, but typically deleterious to ones hearing.
I get to repair plenty of audio systems that are victims of an owners drunken binge.
Also, it is common knowledge that when doing live sound mixing that ones ears go 'off' if imbibing whilst on the job, and in high end sound reinforcement circles, alcohol is best left alone until after the show is over, and it is not considered 'kosher' to be seen to be under the influence whilst in command of seriously high power systems.
Previous long term exposure to harsh sounding home or PA systems changes hearing response too.
Nowadays there are plenty of 50's something audio operators who have cooked their ears during the late seventies/early eighties on harsh systems, and their hallmark is a live sound with hard and biting mids and high mids.
Educated and youger ears thankfully usually avoid this error, but not always.
I also find plenty of older masterful musicians whose hearing discrimination is quite lost in the midrange.
Eric.
Sy, I'm not meaning to be be dissing you there.
It is meant as a comment that if you are not hearing subtle differences then it is likely that there is some kind of masking going on.
This masking can be caused by a whole range of causes, ranging from suspect speakers, suspect interconnects, noisey mains power, etc, etc, etc...... .
Your visual system plugged into the same mains source can cause masking too, even if turned off I find.
I note that according to your website, you are into wines in a seemingly big way, and as has been shown to me many times, alcohol is fine for the soul, but typically deleterious to ones hearing.
I get to repair plenty of audio systems that are victims of an owners drunken binge.
Also, it is common knowledge that when doing live sound mixing that ones ears go 'off' if imbibing whilst on the job, and in high end sound reinforcement circles, alcohol is best left alone until after the show is over, and it is not considered 'kosher' to be seen to be under the influence whilst in command of seriously high power systems.
Previous long term exposure to harsh sounding home or PA systems changes hearing response too.
Nowadays there are plenty of 50's something audio operators who have cooked their ears during the late seventies/early eighties on harsh systems, and their hallmark is a live sound with hard and biting mids and high mids.
Educated and youger ears thankfully usually avoid this error, but not always.
I also find plenty of older masterful musicians whose hearing discrimination is quite lost in the midrange.
Eric.
TAKING YOUR MUSIC SERIOUSLY.
Hi,
Absolutely.
A dedicated mains spur is a must, anything else hooked up to that should be disconnected from the mains for serious listening.
As Eric says, even when not switched on it will degrade sonics.
Not that I have a quickfix explanation for it though.
Another often overlooked detail is correct mains polarity of each individual apparatus.
Always check for least leakage current, mark the correct polarity and enjoy the difference.
Cheers,
Hi,
Absolutely.
A dedicated mains spur is a must, anything else hooked up to that should be disconnected from the mains for serious listening.
As Eric says, even when not switched on it will degrade sonics.
Not that I have a quickfix explanation for it though.
Another often overlooked detail is correct mains polarity of each individual apparatus.
Always check for least leakage current, mark the correct polarity and enjoy the difference.
Cheers,
- Status
- Not open for further replies.