Measuring sound quality - can it be done ???

[Nicwix]

The goal of almost everyone on this forum is surely to achieve the best "sound quality" for the money and time they are willing to spend. And yet, there is so little consensus as to how "sound quality" can be measured.

Why measure?

In almost every field of scientific endeavour, objective measurement is fundamental to improvement. Undoubtedly a precise and objective measurement system would immensely benefit DIY audio designers and tweakers – the effect of each component change or tweak could be assessed and the best combination.

But we don’t …

Why then is it then that we have (generally) turned our backs on using instrument measurement to support achieving our goal of best “sound quality”?

Some quotes from this forum:
"Like Prohibition, chasing low distortion figures was an experiment that didn't work. Let's learn from the mistake and go on..."
"... they tend not to have enough feedback for very low distortion. But then we don't necessarily judge an amp on that basis, do we?"
"Reducing the feedback will increase the measured distortion and reduce damping factor. Whether it will sound better or not I will leave to others..."
"... of course, distortion measurement does not tell the whole story, as we all know ..."

We are not alone …

We at diyaudio.com are not alone here – high end audio generally leans towards listening tests, rather than instrument-based measurement, due to discrepancies between subjective sound quality and measurement results:
• "... Present distortion measurements fail to tally with listening tests ..."
• "... correlation between technical specification and ... subjective sound quality is the most difficult to establish ..."

Why?

Why is this? It seems to me there are four possible reasons as to why instrument-based measurement is not widely used:
• Insufficient Sensitivity – our instruments are not sufficiently sensitive (ie the ear is better than any instrument that can be built)
• Insufficient Measures – we are measuring the wrong things (ie the usual measures of frequency response, THD, IMD, noise and dynamic range are a good start, but are not enough)
• Prefer Distortion – we actually prefer the sound with some distortion – ie using “good distortion” or noise, to mask offensive distortion (eg dithering in digital systems, low-order odd harmonic distortion of tube amplifiers, a la Jean Higara, etc)
• Fooling Ourselves – the emperor has no clothes – ie we prefer to fool ourselves that our equipment sounds better, regardless of what an objective test would show

My views

My personal views on these possibilities are:
• Insufficient Sensitivity – unlikely, give the range and capabilities of modern hardware and software instrumentation
• Insufficient Measures – highly likely, since the usual measure to not cover the dynamic characteristics of music – see Memory Distortion Philosophies for an example; of course current instruments may need new capabilities to perform additional tests
• Prefer Distortion – possibly, to some extent; to resolve this, we need to be able to distinguish between the desired “good distortion” and the unwanted “bad distortion”
• Fooling Ourselves – regrettably, in some cases, yes

And so …

As amplifier designers and builders, our job is made easier, since our focus is the electrical domain rather than the acoustic, the results of our efforts must appear at the loudspeaker terminals – measurable as a simple two-dimensional (time-voltage) variable. Our friends in the loudspeaker forum have a far more complex task to perform accurate measurements in the acoustic domain.

With a PC, a good quality soundcard and suitable analysis software, everyone on this forum could perform a wide range of measures on their equipment at reasonable cost – the hardware is up to it, but the software needs more work to perform some additional tests.

Let’s embrace the challenge and find ways to measure what we hear.

Any comments on these thoughts?

[JBL]

I d'ont think that it is possible to place a number on sound quality. And if their where one manufacturer would already be using this magical figure to rate their stuff.

And even if we define a way to mesure and get a final verdict their will always be people who will tell that it's not better than something who was rated lower.

And how would will be rating solid state amp versus valve amp.

[Nelson Pass]

A major problem is that our hearing mechanism, mechanically
and neurologically is radically different from our measurement
apparatus, and so it can only give objective, but not
necessarily subjectively truthful information.

The task of reconciling the descrepancy is beyond me. I just
alter, measure, listen, and think. Over and over.

pass/ - works with what he's got

[Nicwix]

JBL - whist your signature says "---Nothing is impossible---", I assume from your reply that excludes measuring sound quality - however if absolute quality cannot be measured, one should at least be able to do relative measurements, ie: along the lines:
Option A sounds better then Option B and the measured effect is ...

Nelson, thank you for your reply - could you kindly clarify for us:-
in your (vast) experience, with professional test gear can one always measure the changes you hear or can one make changes that are audible, but have no measureable effect?[list=a][*]If changes are always measureable, then why the difficulty of reconciling perceptions with measurement - just too many variables?[*]If not always measureable, then what's wrong with our measurement systems and can they be improved?[/list=a]

[halojoy]

MUSIC is a matter of taste.
Some think some badly tuned Punk Guitars are the ultimate quality.
Others think that, worlds most famous violinist
playing Paganini, is the best.

And also is what we think is Good
depending on our situation.
When you are in sorrow, you might need a sentimental song,
to help you cry your sadness away.
When you are in partymod, you might want some Boney M. disco
to really get you going.

There is a music for every occation.
And for every single person.

"Little music every day,
keeps your doctor away." /halo 2003

[Nelson Pass]

Originally posted by Nicwix
can one always measure the changes you hear or can one make changes that are audible, but have no measureable effect?

You can usually measure the changes, but not always.


pass/ - has two copies of Dr. No

[JBL]

And I am not going to contredict the fact that nothing is impossible.

But sound quality is a direct relation to how much we like it. So if I where(for example) to design a system to measure sound quality it will probably be working in my case but not for another.

And for comparing amps(or anything) between them the same problem arise.

Up to now nothing as been able to beat a good listening test.

[jh6you]

It seems that the music sound quality is in most cases measured by own faculties of perception. Once when I was driving my car through the bottom of the sunset-colored sky, I was by chance caught by a beautiful and emotional music strolling out of the car FM radio. Later, I tried the exactly same music on my home audio system. But, the emotion had gone. I added one bottle of wine for some help. Nonetheless, I could not get it back at all.

The measurement could be possible only by the ears of whom have lived long long years with various music, and Yeah… the audio system should be developed by the persons having such experienced ears.

JH

[Nicwix]

Most of the responses above focus on music quality. And most would agree that is difficult (OK, impossible) to define, let alone measure.

But we are concerned here with presentation of a recorded performance (ie the source signal), the question - let's accept each person's choice the music as a given. The question is what does our equipment do to that recording that changes the perceived sound quality and can we measure those change effectively?

We need to consider three possible design approaches:[list=a][*]Replication – designing to present a signal to the loudspeaker that is as close the original source as possible
[*]Enhancement – presenting a signal to the loudspeaker, that is different to the original signal, so that, in conjunction with other factors (eg loudspeaker characterists, room acoustics, recording deficiencies, etc), the perceived sound quality is improved - obvious examples being: equalisation, reverberation and other “surround sound” enhancements, etc
[*]Masking – aiming for the “good distortion” to mask the unwanted distortion, as noted above[/list=a]
From a measurement point of view, approach "a" is clearly easiest, since our goal is to minimise the difference between the output signal and the original source. Approach "b" is relatively easy if we know precisely what enhancements we intend to add (as opposed to other, unintended alterations to the signal). Approach "c" is generally simpler, however it may not be so easy to distinguish between good and bad distortion.

Perhaps this is where the difficulty lies. If approach "a" is our goal, then reconciling perceived and measured sound quality is purely a measurement problem. With approachs "b" and "c" - the problem may lie in determining what changes to the signal one will improve perceived sound quality?

What approach do you follow, my friends?

[Neutron Bob]

"Underneath the mango tree...."

Michael Lonsdale had one of the best Bond lines: "You foil my attempts at arranging an amusing death for you, Mr. Bond."

I'd think that a distortion measurement apparatus and spectrum analysis is most useful in tracking modifications to the design on paper, the yielded gain, bandwidth, and whether the wee beastie is stable.

Whether it actually sounds any good requires much more discriminating equipment, like a sip of VSOP or a wee bit of single malt, and hours in front of the speakers!