The goal of almost everyone on this forum is surely to achieve the best "sound quality" for the money and time they are willing to spend. And yet, there is so little consensus as to how "sound quality" can be measured.
Why measure?
In almost every field of scientific endeavour, objective measurement is fundamental to improvement. Undoubtedly a precise and objective measurement system would immensely benefit DIY audio designers and tweakers – the effect of each component change or tweak could be assessed and the best combination.
But we don’t …
Why then is it then that we have (generally) turned our backs on using instrument measurement to support achieving our goal of best “sound quality”?
We at diyaudio.com are not alone here – high end audio generally leans towards listening tests, rather than instrument-based measurement, due to discrepancies between subjective sound quality and measurement results:
• "... Present distortion measurements fail to tally with listening tests ..."
• "... correlation between technical specification and ... subjective sound quality is the most difficult to establish ..."
Why?
Why is this? It seems to me there are four possible reasons as to why instrument-based measurement is not widely used:
• Insufficient Sensitivity – our instruments are not sufficiently sensitive (ie the ear is better than any instrument that can be built)
• Insufficient Measures – we are measuring the wrong things (ie the usual measures of frequency response, THD, IMD, noise and dynamic range are a good start, but are not enough)
• Prefer Distortion – we actually prefer the sound with some distortion – ie using “good distortion” or noise, to mask offensive distortion (eg dithering in digital systems, low-order odd harmonic distortion of tube amplifiers, a la Jean Higara, etc)
• Fooling Ourselves – the emperor has no clothes – ie we prefer to fool ourselves that our equipment sounds better, regardless of what an objective test would show
My views
My personal views on these possibilities are:
• Insufficient Sensitivity – unlikely, give the range and capabilities of modern hardware and software instrumentation
• Insufficient Measures – highly likely, since the usual measure to not cover the dynamic characteristics of music – see Memory Distortion Philosophies for an example; of course current instruments may need new capabilities to perform additional tests
• Prefer Distortion – possibly, to some extent; to resolve this, we need to be able to distinguish between the desired “good distortion” and the unwanted “bad distortion”
• Fooling Ourselves – regrettably, in some cases, yes
And so …
As amplifier designers and builders, our job is made easier, since our focus is the electrical domain rather than the acoustic, the results of our efforts must appear at the loudspeaker terminals – measurable as a simple two-dimensional (time-voltage) variable. Our friends in the loudspeaker forum have a far more complex task to perform accurate measurements in the acoustic domain.
With a PC, a good quality soundcard and suitable analysis software, everyone on this forum could perform a wide range of measures on their equipment at reasonable cost – the hardware is up to it, but the software needs more work to perform some additional tests.
Let’s embrace the challenge and find ways to measure what we hear.
Any comments on these thoughts?
Why measure?
In almost every field of scientific endeavour, objective measurement is fundamental to improvement. Undoubtedly a precise and objective measurement system would immensely benefit DIY audio designers and tweakers – the effect of each component change or tweak could be assessed and the best combination.
But we don’t …
Why then is it then that we have (generally) turned our backs on using instrument measurement to support achieving our goal of best “sound quality”?
We are not alone …
We at diyaudio.com are not alone here – high end audio generally leans towards listening tests, rather than instrument-based measurement, due to discrepancies between subjective sound quality and measurement results:
• "... Present distortion measurements fail to tally with listening tests ..."
• "... correlation between technical specification and ... subjective sound quality is the most difficult to establish ..."
Why?
Why is this? It seems to me there are four possible reasons as to why instrument-based measurement is not widely used:
• Insufficient Sensitivity – our instruments are not sufficiently sensitive (ie the ear is better than any instrument that can be built)
• Insufficient Measures – we are measuring the wrong things (ie the usual measures of frequency response, THD, IMD, noise and dynamic range are a good start, but are not enough)
• Prefer Distortion – we actually prefer the sound with some distortion – ie using “good distortion” or noise, to mask offensive distortion (eg dithering in digital systems, low-order odd harmonic distortion of tube amplifiers, a la Jean Higara, etc)
• Fooling Ourselves – the emperor has no clothes – ie we prefer to fool ourselves that our equipment sounds better, regardless of what an objective test would show
My views
My personal views on these possibilities are:
• Insufficient Sensitivity – unlikely, give the range and capabilities of modern hardware and software instrumentation
• Insufficient Measures – highly likely, since the usual measure to not cover the dynamic characteristics of music – see Memory Distortion Philosophies for an example; of course current instruments may need new capabilities to perform additional tests
• Prefer Distortion – possibly, to some extent; to resolve this, we need to be able to distinguish between the desired “good distortion” and the unwanted “bad distortion”
• Fooling Ourselves – regrettably, in some cases, yes
And so …
As amplifier designers and builders, our job is made easier, since our focus is the electrical domain rather than the acoustic, the results of our efforts must appear at the loudspeaker terminals – measurable as a simple two-dimensional (time-voltage) variable. Our friends in the loudspeaker forum have a far more complex task to perform accurate measurements in the acoustic domain.
With a PC, a good quality soundcard and suitable analysis software, everyone on this forum could perform a wide range of measures on their equipment at reasonable cost – the hardware is up to it, but the software needs more work to perform some additional tests.
Let’s embrace the challenge and find ways to measure what we hear.
Any comments on these thoughts?
I d'ont think that it is possible to place a number on sound quality. And if their where one manufacturer would already be using this magical figure to rate their stuff.
And even if we define a way to mesure and get a final verdict their will always be people who will tell that it's not better than something who was rated lower.
And how would will be rating solid state amp versus valve amp.
And even if we define a way to mesure and get a final verdict their will always be people who will tell that it's not better than something who was rated lower.
And how would will be rating solid state amp versus valve amp.
A major problem is that our hearing mechanism, mechanically
and neurologically is radically different from our measurement
apparatus, and so it can only give objective, but not
necessarily subjectively truthful information.
The task of reconciling the descrepancy is beyond me. I just
alter, measure, listen, and think. Over and over.
pass/ - works with what he's got
and neurologically is radically different from our measurement
apparatus, and so it can only give objective, but not
necessarily subjectively truthful information.
The task of reconciling the descrepancy is beyond me. I just
alter, measure, listen, and think. Over and over.
pass/ - works with what he's got
JBL - whist your signature says "---Nothing is impossible---", I assume from your reply that excludes measuring sound quality 
- however if absolute quality cannot be measured, one should at least be able to do relative measurements, ie: along the lines:
Option A sounds better then Option B and the measured effect is ...
Nelson, thank you for your reply - could you kindly clarify for us:-
in your (vast) experience, with professional test gear can one always measure the changes you hear or can one make changes that are audible, but have no measureable effect?
- however if absolute quality cannot be measured, one should at least be able to do relative measurements, ie: along the lines:Option A sounds better then Option B and the measured effect is ...
Nelson, thank you for your reply - could you kindly clarify for us:-
in your (vast) experience, with professional test gear can one always measure the changes you hear or can one make changes that are audible, but have no measureable effect?
- If changes are always measureable, then why the difficulty of reconciling perceptions with measurement - just too many variables?
- If not always measureable, then what's wrong with our measurement systems and can they be improved?
[/list=a]
As difficult, almost, as telling what Quality the MUSIC has
MUSIC is a matter of taste.
Some think some badly tuned Punk Guitars are the ultimate quality.
Others think that, worlds most famous violinist
playing Paganini, is the best.
And also is what we think is Good
depending on our situation.
When you are in sorrow, you might need a sentimental song,
to help you cry your sadness away.
When you are in partymod, you might want some Boney M. disco
to really get you going.
There is a music for every occation.
And for every single person.
"Little music every day,
keeps your doctor away." /halo 2003
MUSIC is a matter of taste.
Some think some badly tuned Punk Guitars are the ultimate quality.
Others think that, worlds most famous violinist
playing Paganini, is the best.
And also is what we think is Good
depending on our situation.
When you are in sorrow, you might need a sentimental song,
to help you cry your sadness away.
When you are in partymod, you might want some Boney M. disco
to really get you going.
There is a music for every occation.
And for every single person.
"Little music every day,
keeps your doctor away."
/halo 2003Attachments
And I am not going to contredict the fact that nothing is impossible.
But sound quality is a direct relation to how much we like it. So if I where(for example) to design a system to measure sound quality it will probably be working in my case but not for another.
And for comparing amps(or anything) between them the same problem arise.
Up to now nothing as been able to beat a good listening test.
But sound quality is a direct relation to how much we like it. So if I where(for example) to design a system to measure sound quality it will probably be working in my case but not for another.
And for comparing amps(or anything) between them the same problem arise.
Up to now nothing as been able to beat a good listening test.
It seems that the music sound quality is in most cases measured by own faculties of perception. Once when I was driving my car through the bottom of the sunset-colored sky, I was by chance caught by a beautiful and emotional music strolling out of the car FM radio. Later, I tried the exactly same music on my home audio system. But, the emotion had gone. I added one bottle of wine for some help. Nonetheless, I could not get it back at all.
The measurement could be possible only by the ears of whom have lived long long years with various music, and Yeah… the audio system should be developed by the persons having such experienced ears.
JH
The measurement could be possible only by the ears of whom have lived long long years with various music, and Yeah… the audio system should be developed by the persons having such experienced ears.
JH
Most of the responses above focus on music 
quality. And most would agree that is difficult (OK, impossible) to define, let alone measure.
But we are concerned here with presentation of a recorded performance (ie the source signal), the question - let's accept each person's choice the music as a given. The question is what does our equipment do to that recording that changes the perceived sound quality and can we measure those change effectively?
We need to consider three possible design approaches:
quality. And most would agree that is difficult (OK, impossible) to define, let alone measure. But we are concerned here with presentation of a recorded performance (ie the source signal), the question - let's accept each person's choice the music as a given. The question is what does our equipment do to that recording that changes the perceived sound quality and can we measure those change effectively? We need to consider three possible design approaches:- Replication – designing to present a signal to the loudspeaker that is as close the original source as possible
- Enhancement – presenting a signal to the loudspeaker, that is different to the original signal, so that, in conjunction with other factors (eg loudspeaker characterists, room acoustics, recording deficiencies, etc), the perceived sound quality is improved - obvious examples being: equalisation, reverberation and other “surround sound” enhancements, etc
- Masking – aiming for the “good distortion” to mask the unwanted distortion, as noted above
[/list=a]
From a measurement point of view, approach "a" is clearly easiest, since our goal is to minimise the difference between the output signal and the original source. Approach "b" is relatively easy if we know precisely what enhancements we intend to add (as opposed to other, unintended alterations to the signal). Approach "c" is generally simpler, however it may not be so easy to distinguish between good and bad distortion.
Perhaps this is where the difficulty lies. If approach "a" is our goal, then reconciling perceived and measured sound quality is purely a measurement problem. With approachs "b" and "c" - the problem may lie in determining what changes to the signal one will improve perceived sound quality?
What approach do you follow, my friends?
Dr. No uses an HP
"Underneath the mango tree...."
Michael Lonsdale had one of the best Bond lines: "You foil my attempts at arranging an amusing death for you, Mr. Bond."
I'd think that a distortion measurement apparatus and spectrum analysis is most useful in tracking modifications to the design on paper, the yielded gain, bandwidth, and whether the wee beastie is stable.
Whether it actually sounds any good requires much more discriminating equipment, like a sip of VSOP or a wee bit of single malt, and hours in front of the speakers!
"Underneath the mango tree...."
Michael Lonsdale had one of the best Bond lines: "You foil my attempts at arranging an amusing death for you, Mr. Bond."
I'd think that a distortion measurement apparatus and spectrum analysis is most useful in tracking modifications to the design on paper, the yielded gain, bandwidth, and whether the wee beastie is stable.
Whether it actually sounds any good requires much more discriminating equipment, like a sip of VSOP or a wee bit of single malt, and hours in front of the speakers!
Nicwix said:
Add to that Presentation, which puts a nice frame on the
experience, and you have a good list. For example, last night
when I saw the Stardust Cowboys live, I was sitting next to
the fiddle player, and even though the (violin) was amplified
by cheesy equipment, it was great. This is perhaps why
home theater guys have a totally alien aesthetic regarding
good sound.
But let's face it, the whole subject is a big can of worms.
pass/ - got tired of Jet Society, now listening to Comfort
Zone
"In almost every field of scientific endeavour, objective measurement is fundamental to improvement. "
Quite right.
I mean, it is impossible to argue with. No one will argue that using listening tests as a measure of improvement is wrong. So why should using intermediate measures be wrong?
IMO the reason intermediate measures are not more deterministic is that they are too simplistic. I do not believe that equipment is not adequate - there are plenty of instruments that can measure things that we cannot hear; we are only flesh and blood. Even a modest digital storage scope will show some difference between a amps input and output signal. I think the issue is that we, the amateurs, don't always have the time, money or understanding to figure out what measures are most important. There is also a tendency to keep things simple by measuring the things that the off-the-shelf equipment is designed to measure. Very little commercially available test equipment was designed for audio measurement.
I believe some high-end companies do pursue objective measurements. They don't divulge them. Hifi magazines have tried hard to find good measures and have dabbled with things like dynamic spectrum analyzer plots and so on, but to be honest the people who work for the magazines are no more experienced in T&M methods than us amateurs.
I think this is an area of opportunity.
Quite right.
I mean, it is impossible to argue with. No one will argue that using listening tests as a measure of improvement is wrong. So why should using intermediate measures be wrong?
IMO the reason intermediate measures are not more deterministic is that they are too simplistic. I do not believe that equipment is not adequate - there are plenty of instruments that can measure things that we cannot hear; we are only flesh and blood. Even a modest digital storage scope will show some difference between a amps input and output signal. I think the issue is that we, the amateurs, don't always have the time, money or understanding to figure out what measures are most important. There is also a tendency to keep things simple by measuring the things that the off-the-shelf equipment is designed to measure. Very little commercially available test equipment was designed for audio measurement.
I believe some high-end companies do pursue objective measurements. They don't divulge them. Hifi magazines have tried hard to find good measures and have dabbled with things like dynamic spectrum analyzer plots and so on, but to be honest the people who work for the magazines are no more experienced in T&M methods than us amateurs.
I think this is an area of opportunity.
I would say the ideal approach should embrace YET transcend
the things you propose.
If measure then measure. Please proceed.
But don't forget that as humans we have at least the 3 aspects,
Physical, Intellectual, Emotional.
The heart of great music is emotional.
How should we measure this? And besides it's only the mind
that wants to, the body and the emotion want nothing to do
with it.
the things you propose.
If measure then measure. Please proceed.
But don't forget that as humans we have at least the 3 aspects,
Physical, Intellectual, Emotional.
The heart of great music is emotional.
How should we measure this? And besides it's only the mind
that wants to, the body and the emotion want nothing to do
with it.
Ultimate Comfort Zone
"The goal of almost everyone on this forum is surely to achieve the best "sound quality" for the money and time they are willing to spend. And yet, there is so little consensus as to how "sound quality" can be measured."
Yes, and we need to further define what we, individually or collectively regard as 'quality sound'.
I have long since come to the conclusion that infinite presicion is not really required in an audio system.
By this I mean that factors like absoloute frequency response flatness, and zero THD etc are not nearly so critical as many would believe, and indeed the colourations wrought can increase enjoyment of the music.
The exception to my ear is that IMD must be low for pleasant reproduction.
Fun Factor -
I had the opportunity to 'tweak' the instruments and PA of a live show on the weekend.
The 'tweak' is a product that I am developing currently, that when used correctly immediately transforms any live or replay system that I have tried it on.
The 'tweaking' went in stages between brackets, the first change being treating the two vocal mics, and the two guitar signal leads.
At the next break, the guys sat with my GF and myself, and gave me some initial feedback on what they heard.
During this break I treated the single 240V AC lead supplying instruments and PA, and the effect was immediately apparent to all those listening during this break on pre-recorded filler music. (Rolling stones CD).
After the next bracket, changes were to treat the kick mic, and further treat the lead guitar amplifier.
During the last set the guitarist was exploring and reaching sounds that he has not had before, and the whole time immersing himself into his instrument and GRINNING from ear to ear.
The final result of this informal experiment session was positive, positive, positive from the band, GF and me - stage sound was the best I've heard it, the PA was the best I've heard it, and the band played the best I've heard them yet.
The venue is a sandstone stage with a tarpailin roof in the outdoor beergarden of a localish pub on top of a hill out in the middle of nowhere with bush views for ten miles, a crowd of about 10, and so only natural sounds to compliment the music.
Nelson mention sitting next to the fiddle player - my GF and I were seated 20m across the beergarden under a shady tree - 39*C day.
For the last set, I laid back in my chair with my head against the paper bark tree, a beer in my hand, enjoyed live music like I have not done so before, and listened and relaxed, and grinned from ear to ear too !.
Along the lines of what Nelson says about his 'Stardust Cowboys' show -
My wonderful GF, cold beer, natural surrounds, and world class musos playing cooking hot blues on the nicest sounding live system that I have yet heard - I think that musical enjoyment better than this is not possible.
The downside to this weekends experiment is that I now know the sound of nicer sounding microphones and instruments, and every recording that I now listen to sounds rotten.
Anyway there are enjoyment factors in audio replay systems that currently can not be explained by standard audio measurement methods, and these are the ones that make that that difference between just nice sound and really enjoyable sound.
There is more going on in an audio system than standard electronic theory explains, and it is not due to any psychoacoustic expectation factor.
Eric / - 'Tone Stone' - you heard it here first.
"The goal of almost everyone on this forum is surely to achieve the best "sound quality" for the money and time they are willing to spend. And yet, there is so little consensus as to how "sound quality" can be measured."
Yes, and we need to further define what we, individually or collectively regard as 'quality sound'.
I have long since come to the conclusion that infinite presicion is not really required in an audio system.
By this I mean that factors like absoloute frequency response flatness, and zero THD etc are not nearly so critical as many would believe, and indeed the colourations wrought can increase enjoyment of the music.
The exception to my ear is that IMD must be low for pleasant reproduction.
Fun Factor -
I had the opportunity to 'tweak' the instruments and PA of a live show on the weekend.
The 'tweak' is a product that I am developing currently, that when used correctly immediately transforms any live or replay system that I have tried it on.
The 'tweaking' went in stages between brackets, the first change being treating the two vocal mics, and the two guitar signal leads.
At the next break, the guys sat with my GF and myself, and gave me some initial feedback on what they heard.
During this break I treated the single 240V AC lead supplying instruments and PA, and the effect was immediately apparent to all those listening during this break on pre-recorded filler music. (Rolling stones CD).
After the next bracket, changes were to treat the kick mic, and further treat the lead guitar amplifier.
During the last set the guitarist was exploring and reaching sounds that he has not had before, and the whole time immersing himself into his instrument and GRINNING from ear to ear.
The final result of this informal experiment session was positive, positive, positive from the band, GF and me - stage sound was the best I've heard it, the PA was the best I've heard it, and the band played the best I've heard them yet.
The venue is a sandstone stage with a tarpailin roof in the outdoor beergarden of a localish pub on top of a hill out in the middle of nowhere with bush views for ten miles, a crowd of about 10, and so only natural sounds to compliment the music.
Nelson mention sitting next to the fiddle player - my GF and I were seated 20m across the beergarden under a shady tree - 39*C day.
For the last set, I laid back in my chair with my head against the paper bark tree, a beer in my hand, enjoyed live music like I have not done so before, and listened and relaxed, and grinned from ear to ear too !.
Along the lines of what Nelson says about his 'Stardust Cowboys' show -
My wonderful GF, cold beer, natural surrounds, and world class musos playing cooking hot blues on the nicest sounding live system that I have yet heard - I think that musical enjoyment better than this is not possible.
The downside to this weekends experiment is that I now know the sound of nicer sounding microphones and instruments, and every recording that I now listen to sounds rotten.
Anyway there are enjoyment factors in audio replay systems that currently can not be explained by standard audio measurement methods, and these are the ones that make that that difference between just nice sound and really enjoyable sound.
There is more going on in an audio system than standard electronic theory explains, and it is not due to any psychoacoustic expectation factor.
Eric / - 'Tone Stone' - you heard it here first.
gtrmaker,
Emotion is the result of an individuals brain processing pressure waves that impinge upon their eardrums. It is not necessary or pertinent to measure emotions.
Measuring how accurately the pressure waves are reproduced between the original instrument and the listeners ear is what it is all about. And this is a matter of mechanics and electronics and nothing else.
Unless, of course, you are the sort of person who believes in homeopathic medicine (except in a psychological sense) in which case your method of reasoning is anathema to mine. ;-)
BAM
Emotion is the result of an individuals brain processing pressure waves that impinge upon their eardrums. It is not necessary or pertinent to measure emotions.
Measuring how accurately the pressure waves are reproduced between the original instrument and the listeners ear is what it is all about. And this is a matter of mechanics and electronics and nothing else.
Unless, of course, you are the sort of person who believes in homeopathic medicine (except in a psychological sense) in which case your method of reasoning is anathema to mine. ;-)
BAM
JBL said:
I think you hit the nail on the head. Our problem is that we don't know how to define "sound quality". Therefore, we don't know how to measure it. It is also the reason that each of has his own standards of what we find "good sound quality".
The problem for the designer is to built an amp that everyone agrees gives "good sound quality" which is as a result from the above impossible.
That is IMHO why there are so many different topologies and technical solutions. That is also, IMHO, the reason why there is so much humbug and snake oil in audio.
To summarize: if you can define it, I can measure it. Guaranteed.
Jan Didden
Let's make a parallel with paint (as art). Did you think about Mr Pass as a great painter like Picasso? When you analyze all the paints from some painter in the time, you see a noticeable difference in how the colors are made and how hes describe the environment. This difference is explained by the physiological changes in the eye’s retina. So quality for a particular painter has changed. What about the great hears of Mr Pass?? Let’s say every other variables are fixed, the sound quality of his amplifier will be modified depending on the period he designed it.
Have fun
Let’s say every other variables are fixed, the sound quality of his amplifier will be modified depending on the period he designed it.Have fun
How difficult can it be ?
Imagine yourself following the replication criteria of “quality”, described by NicWix, to determine the worth of a component (e.g. an amplifier). The amplifier you are going to test/measure is in a black box and you cannot determine what is inside as all you can see on the outside is an input socket and an output socket (it can have a blue LED as well if you like
).
What sort of measurements would you make ?
THD ?
TIM ?
Frequency sweep ?
Impulse response ?
Voltage gain ?
Power output into a number of differing loads ?
Etc, etc, etc
After many measurements you conclude that this indeed a measurably good amplifier. As a final test you plug in a CD/phono source, some speakers and put some of you favourite music on. You quickly realise there is something seriously wrong – this amplifier “bleeps” out any expletives in the source material – this is a family amplifier
The bottom line is, unless you have some (accurate) a-priori knowledge about the system under test, you do not really know what to measure. This is important as you cannot measure everything as this would need an infinitely long period test signal. In addition, the more complex the system under test (the expletive removal system would be a seriously complex example), the more difficult this will be to determine.
BTW, I don't trust my ears too much either ...
Dave -
suit at the ready ...
Imagine yourself following the replication criteria of “quality”, described by NicWix, to determine the worth of a component (e.g. an amplifier). The amplifier you are going to test/measure is in a black box and you cannot determine what is inside as all you can see on the outside is an input socket and an output socket (it can have a blue LED as well if you like
). What sort of measurements would you make ?
THD ?
TIM ?
Frequency sweep ?
Impulse response ?
Voltage gain ?
Power output into a number of differing loads ?
Etc, etc, etc
After many measurements you conclude that this indeed a measurably good amplifier. As a final test you plug in a CD/phono source, some speakers and put some of you favourite music on. You quickly realise there is something seriously wrong – this amplifier “bleeps” out any expletives in the source material – this is a family amplifier
The bottom line is, unless you have some (accurate) a-priori knowledge about the system under test, you do not really know what to measure. This is important as you cannot measure everything as this would need an infinitely long period test signal. In addition, the more complex the system under test (the expletive removal system would be a seriously complex example), the more difficult this will be to determine.
BTW, I don't trust my ears too much either ...
Dave -
suit at the ready ...
François said:Let's make a parallel with paint (as art). Did you think about Mr Pass as a great painter like Picasso? When you analyze all the paints from some painter in the time, you see a noticeable difference in how the colors are made and how hes describe the environment. This difference is explained by the physiological changes in the eye’s retina. So quality for a particular painter has changed. What about the great hears of Mr Pass??
Have fun
Yes Francois! And I think there is a great parallel between paintings and amps. You cannot say: OK, take a frame of such and such dimensions, paint with a contrast of such ratio, etc, etc and you will make a great painting. Why? because you cannot define what "a great painting " is any more than you can define "good sound quality".
Jan Didden
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.