This doesn't mean that it is the only or even the best method however.
Were designers to carry out robust repeatable testing of their designs including end user testing and sensory response evaluation, their "craft" would improve and could be scientifically validated.
Unfortunately, a truism of "handcrafted" design is that it is generally carried out by crafty hands. The crafty tend not to respect evidence too much.
Nope, the answer is simple - the enginers can't be bothered measuring the finer points of the behaviours, but the ears certainly can, and they can't help noticing when things are wrong. It's all about laziness - no need to bother, it's only audio ...
Frank, and fas42: It's not about "hiding" behind science, it's about getting reproducible results. Yes ,you can use your ears to evaluate changes, BUT you need to do blind testing, or your results really don't mean anything useful. You need to document what you're doing, and test without knowing which of two states you're listening to.
I strongly suspect that once you begin doing blind testing, you will find that most of your "differences" magically go away. Others have found that very thing.
Consider how drug companies test drugs. The patient doesn't know if he's getting one medication or another, and the results aren't tainted by placebo effect. Same for audio.
Until you do such tests, your results are your opinion only, and aren't to be trusted by anybody else.
I strongly suspect that once you begin doing blind testing, you will find that most of your "differences" magically go away. Others have found that very thing.
Consider how drug companies test drugs. The patient doesn't know if he's getting one medication or another, and the results aren't tainted by placebo effect. Same for audio.
Until you do such tests, your results are your opinion only, and aren't to be trusted by anybody else.
With minor changes the same could be said for the "listeners", viz:
the listeners can't be bothered accurately and in an untainted way measuring the behaviours they perceive, even though they certainly could. It's all about arrogance - no need to bother, if I perceive it, it must be there...
Just for clarity - I am all for sensory testing as a method for assessing the quality of an item. I just don't accept that a single user (or even a group) self-administering a test where they are fully involved in the process of setting up the test and invested in its outcome is able to make a valid judgement. This also puts aside the issue of the known shortcomings in our ability to remember sounds accurately enough to identify minute (and sometimes even gross) changes that are even a matter of seconds apart.
Bottom line - its not listening tests that I am against - its poorly carried out tests that do not recognise (let alone allow for) the known issues in testing.
Most assuredly, fas are the initials of my name; 42, apart from the pop culture thing, is a street address that has figured strongly in my past.Hi,
It's all about hiding behind that comfort zone called science.
BTW, is your first name really Frank?
Ciao,
I agree that proper measuring would be ideal, but it is most certainly difficult to do, especially in the areas I deal with. I would literally have to unsolder, and resolder, or get someone else to do that, between each iteration of say, a blind test. So, I have as a natural process evolved a technique of using difficult recordings, which immediately make it obvious whether I'm genuinely altering or not - 'good' recordings are completely useless, because one's mind can easily compensate for subtle variations - a 'poor' recording has nowhere to hide ...
I agree that proper measuring would be ideal, but it is most certainly difficult to do, especially in the areas I deal with. I would literally have to unsolder, and resolder, or get someone else to do that, between each iteration of say, a blind test. So, I have as a natural process evolved a technique of using difficult recordings, which immediately make it obvious whether I'm genuinely altering or not - 'good' recordings are completely useless, because one's mind can easily compensate for subtle variations - a 'poor' recording has nowhere to hide ...
Actually, its not difficult to do. You just either don't know how to do it, or have chosen not to. You could, for example, make high resolution copies of the before and after output signals, and then ABX test using those recordings. If you can reliably identify the samples, your change has probably made the difference.
All you have to do now is identify which is subjectively "better". For that, you will need a group, since what is "better" to you is not necessarily better for me. Our hearing will be measurably different as a starting point for the reasons for that.
Have you ever had the technique validated in any way by anyone other than yourself? Because unless you have taken that step and the validation confirms the effectiveness and reliability of the technique, you are still no more immune from all of the effects of self administered sensory testing than you were before - possibly you are in a worse position since you firmly beleive the technique (which you are invested in having invented it) is reliable...
This is not a point for debate - all sensory testing research and science supports my concerns regardless of whether the subject is food, music or pain perceptions.
To bring us back to the ORIGINAL topic - this validity question is at the heart of all snake oil. Without validated data based on relevant and reliable testing, any claim is just that - a claim. If more people understood that, the various snake oil sales folks in whichever discipline you care to look at would have a harder job.
Fortunately for them Mr Barnum was right (even if it wasn't him that said it...)
That's an enormous challenge in itself - it's an obvious way of approaching the issue, and if it were relatively straightforward to do, in the sense of being a reliable way of resolving a lot of these audio questions, then it would have been done a long time ago ...
Only in the sense that the final 'product' does the job properly - it passes "the wife in the kitchen" test,
. But seriously, all those I have demonstrated in person the concept to 'get it'. I say, listen to the quality of these cymbals at this precise point, try to ignore all the other sounds happening; now, listen again with this changed; right, now go back to the previous state, etc ...IOW, you're training yourself to listen to the precise qualities of an individual sound in the mix - never as to whether it's "better" or not, except in the sense of "how does it compare to what your memories of such a sound made live are?". Though, the "betterness" can be accurately monitored if the sound exists in isolation at one point in the clip, and at another point is heavily overlaid with other sound elements - wind up the volume, a defective system will mulch the sound qualities once the mix turns busy, a clean reproduction will maintain that sound element with full integrity at all times.
If the consumer has full capability of trying something, and returning it if it doesn't do the job, where's the problem? All medicines are in one sense 'snake oil' - it may do you good as a consumer, or it may not, for a myriad of reasons - this is not Black and White ...
Last edited:
No, its not an enormous challenge. Its ridiculously easy and needs only very rudimentary equipment. Its straightforward to do, easy to set up protocols for, repeatable and best yet - the files can be stored, shared distributed etc for verification by others. With slightly more effort, you can actually "null" one file with the other to identify specific positions where the waveforms show there ARE differences (if any).
It has been done long ago and even recently on this forum. Unfortunately, it involves honesty, measurement and testing and the golden eared ones don't approve because they are not allowed to peek. Nonetheless, its there.
Only in the sense that the final 'product' does the job properly - it passes "the wife in the kitchen" test,
IOW, you're training yourself to listen to the precise qualities of an individual sound in the mix - never as to whether it's "better" or not, except in the sense of "how does it compare to what your memories of such a sound made live are?". Though, the "betterness" can be accurately monitored if the sound exists in isolation at one point in the clip, and at another point is heavily overlaid with other sound elements - wind up the volume, a defective system will mulch the sound qualities once the mix turns busy, a clean reproduction will maintain that sound element with full integrity at all times.
So, not.
They are relying on the builder's/tweaker's expertise and, having invested in the item, are just as unreliable as they are. "Fuel saving" devices for cars and other such nonsense are classic examples of the genre (as are owner reviews and opinions of cars themselves). In essence you are invoking the "if I do no harm" rule to justify potentially doing nothing with any real effect.
No, licensed remedies have been through a testing regime that identifies what they do and under what circumstances they can be expected to do what they claim. They are the antithesis of snake oil in that they clearly identify when they DON'T work, and don't claim to work on everything from hair-loss to dropsy to warts on the end of your willie.
Last edited:
Immediately forgetting little issues like going through a whole new set of circuitry of the A/D and D/A to make it happen. And, that the loudspeaker is taken out of the equation. Just saying that they're 'transparent', or not relevant, would be a brilliant example of scientific self-deception, to put the shoe on the other foot ...
This "nulling" nonsense is laughable - constantly repeated as if somewhere out there is magical software to do. Sorry, it ain't - having struggled for many days trying to get DiffMaker to give meaningful results on a trivially distorted file I gave up - again, no-one is bothering to put the effort into it.
As kgrlee points out, if it is done precisely in the way that suits the listener then it may work. Otherwise, it's just another version of 'snake oil', to suit the scientists,
.That's an argument?
If the user perceives a beneficial effect, then everyone is happy - if he doesn't, then no skin off his nose. A staggering amount of life in other areas would love to have at least that going for them ...
So you can prove, 'scientifically', that a medicine will have a positive benefit for a patient, I think you should get a patent on it,
, ... like a hell of a lot of life, it's often trial and error, maybe it will, maybe it won't ..BTW, there are major ripples of disquiet in the pharmaceutical industry - 'proven' medicines are suddenly being shown not to work, in proper trials - they are now technically a placebo - the money tree is starting to shrivel up ...
Immediately forgetting little issues like going through a whole new set of circuitry of the A/D and D/A to make it happen. And, that the loudspeaker is taken out of the equation. Just saying that they're 'transparent', or not relevant, would be a brilliant example of scientific self-deception, to put the shoe on the other foot ...
Which of itself makes the unproven assumption that the a/d and d/a circuit makes an audible difference.
And yes, the speaker is taken out since it only reacts to the waveform it is given, its irrelevant (unless the changes are to the speaker). It can still be attached and operating during the record phase, and so it is not absent as far as recording the waveforms is concerned. And of course you can use it to play back, although the use of headphones during the ABX would be preferable
So no, not self deception - its all considered and allowed for if not mentioned in my necessarily brief description.
Others seem capable. It can be done.
As kgrlee points out, if it is done precisely in the way that suits the listener then it may work. Otherwise, it's just another version of 'snake oil', to suit the scientists,
Not sure what this refers to.
No, its an observation.
Indeed. However, when you are making assertions of fact, as against just saying meh, I think this might have happened, then you should be able to back them with evidence. If not, its blowing smoke.
So you can prove, 'scientifically', that a medicine will have a positive benefit for a patient, I think you should get a patent on it,
No, and not what I said. What I pointed out is that licensed remedies are required to undergo testing and publish data on their efficacy (among other stuff). None would claim they work in every situation and for all people. It is not, however, trial and error. Its about understanding the data and applying it rationally.
BTW, there are major ripples of disquiet in the pharmaceutical industry - 'proven' medicines are suddenly being shown not to work, in proper trials - they are now technically a placebo - the money tree is starting to shrivel up ...
Yes, there are frauds and deceptions in medicine as in all areas of life. They do not invalidate the points I've made. The placebo one is interesting since it is directly related to the mechanism I suspect is at play in your work. Good to see you recognise the possibility, even if in a different field.
A lot of tweaking sadly leads to nowhere good, maybe different but not actually better. It is a shame we can't rely absolutely on our hearing or memory. Real science (measurements and DBT) is real hard work, tweaking is easy, and every now and then you may actually get an improvement.
The secret is to know when the system sounds as good as it is possible for it to sound, given its components and environment, then stop and enjoy it for what it is. Too much repeated listening and tweaking to improve say the shimmer of cymbals or such, often leads to unbalanced sound.
The secret is to know when the system sounds as good as it is possible for it to sound, given its components and environment, then stop and enjoy it for what it is. Too much repeated listening and tweaking to improve say the shimmer of cymbals or such, often leads to unbalanced sound.
Methinks the shoe is on the other foot - it's up to whoever to prove that the a/d and d/a makes no difference. Or do you just assume that a piece of test equipment is always right, just because it has the name of an established manufacturer on it?
If, and a very large if, extraordinary efforts are made to be rigorous then it it possible that something meaningful will be extracted. Of course, this will never be done ... so the doubters can rest easy,
.Links, please ...
The member kgrlee, who is very strong on correctly running DBTs, has posted his POV many times ...
That what might have happened?
The trial and error is whether it works for a particular individual. The reality of medicine is that the doctor is given the hard sell by drug companies, and he says, OK, I'll give it a go, maybe it could be useful for this patient.
These are not frauds, these are fully recognised medicines, that have been used for years, that are well known even by the man on the street.Yes, there are frauds and deceptions in medicine as in all areas of life. They do not invalidate the points I've made. The placebo one is interesting since it is directly related to the mechanism I suspect is at play in your work. Good to see you recognise the possibility, even if in a different field.
Consistency is the real arbiter in any field - if the same results come in over and over again, from a number of the different angles, then that establishes a structure of thinking, which you project your next action from. I've had a good run with this one, 30 years and still ticking over nicely ...
Last edited:
First off, I would like to state that I do not place myself in the subjectivist camp. This being said, I feel that a lot of the die hard objectivists on this forum have overstepped and take things way too far around here. At least it appears that way to me.
My reason for stating this:
1 - Can we hear things that can't be measured with current technology? : I would say the chances of this are VERY slim and the answer has got to be a tentative NO
2 - In practice, do we (does anybody) take sufficient measurements to account for all the things our ears can discern? : My opinion here is that this never happens either.
----------------------
Yes it can be disheartening to see snake oil peddlers running obvious scams that can't possibly be explained except for the placebo effect. I am not defending this in the slightest.
I would just like some of the engineer types to take a step back and ponder all of the possible limitations to any of the tests that are done and realize that the numbers are only valid within the parameters of the test, and are nearly always only a small part of the big picture.
The point I am trying to make is that some of you go one step too far and jump to conclusions that have a definite possibility of not being true. I don't want to get into specific subjects where I feel this might be happening as it will detract from the main point I am trying to make.
My reason for stating this:
1 - Can we hear things that can't be measured with current technology? : I would say the chances of this are VERY slim and the answer has got to be a tentative NO
2 - In practice, do we (does anybody) take sufficient measurements to account for all the things our ears can discern? : My opinion here is that this never happens either.
----------------------
Yes it can be disheartening to see snake oil peddlers running obvious scams that can't possibly be explained except for the placebo effect. I am not defending this in the slightest.
I would just like some of the engineer types to take a step back and ponder all of the possible limitations to any of the tests that are done and realize that the numbers are only valid within the parameters of the test, and are nearly always only a small part of the big picture.
The point I am trying to make is that some of you go one step too far and jump to conclusions that have a definite possibility of not being true. I don't want to get into specific subjects where I feel this might be happening as it will detract from the main point I am trying to make.
its been mentioned before - drug companies, clinicians are encountering problems in drug studies because placebo effect is so strong
its recommended to use the newest antidepressant - before they are proven ineffective
believers can experience different subjective sensations due to their beliefs, expectations about their expensive, "storied" equipment - works for drugs and sugar pills: The placebo effect: A new study underscores its remarkable power - The Globe and Mail
its recommended to use the newest antidepressant - before they are proven ineffective
believers can experience different subjective sensations due to their beliefs, expectations about their expensive, "storied" equipment - works for drugs and sugar pills: The placebo effect: A new study underscores its remarkable power - The Globe and Mail
You missed the salient point that I made - the impact has to be audible. There are plenty of articles on exactly this - the ability of even trained listeners to identify differences arising from a (well engineered) A/D and/or D/A process. Broadly, the consensus is no, given a reasonable sample rate etc etc, the process is not able to be reliably detected.
So no. I'm not assuming anything about equipment (named or otherwise). I'm relying in the current state of knowledge in the field. If anyone want to run counter to that knowledge, they should provide evidence to back their claims.
Anything claimed to have changed/improved/altered in the perceived sound following a tweak that has little or no engineering basis to explain it
Thats true, but its a process informed by the prior published research, not on the guesswork implied by the phrase "trial and error"
Perhaps I used fraud too loosely. I meant based on faulty or fabricated research
Consistency is the real arbiter in any field - if the same results come in over and over again, from a number of the different angles, then that establishes a structure of thinking, which you project your next action from. I've had a good run with this one, 30 years and still ticking over nicely ...
Its easy to be consistent when subjective assessments and measurements are made by the same person effecting the work! The best that can be said under those conditions is the result consistently please the individual.
Of course, that is often the driving force - in which case there is no real problem.
Well, you're in luck! Everything I've done leads me to be certain that the quality of sound reproduction can be lifted, at any point, well beyond what you have at that moment. There have been moments when atrociously unpromising material has opened up to reveal great depths of value in what was captured, and just normal recordings are truly wonderful, in every way.
If you get unbalanced sound, then you have made a wrong move. Backtrack to the last optimum, and cautiously feel your way forward again - unfortunately, and there is no way around this I've found so far, when a system is running at a very high standard then the slightest wrong deviation can kneecap the sound badly - you'll panic, and think, "What have I done wrong?!". But there is no real problem, the system is just letting you know, very strongly, that your last choice was not the best one. I can say this with complete certainty, because it's happened to me many, many times - retrace your steps, and have another good think about the next move ...
And there are plenty of people who will strongly disagree with that. If some scientists select a sample of motor vehicles for sale, and test their capabilities, do we now know what any vehicle is, or is not, capable of? How about, we run that A/D and/or D/A process through on some sample, copy of a copy style, 3 times, 5 times, 10 times, 100 times - when do we get to the point that the deafest person on the planet can pick the difference? The thing being, there will always be degradation, so then we are playing a sensitivity game, who can pick it up ...
Most people who fiddle with audio, that is of decent quality, to any degree know that doing anything, almost anything, changes the sound - it's just accepted that that is the way of it. The amplifier, and digital source threads here on this forum are riddled with subjective assessments that altering something altered the sound - are they all deluding themselves?
Most times a good engineering basis can be found, quite easily - but there are some devious interactions out there, I've scratched my head far too much over these ones, for one lifetime,
.If it were only so! ... I've experienced extreme frustration over the years, because I have a SQ issue, and I don't know to effect a long lasting solution - all sorts of workarounds in procedures of using the system are used to counter these, while trying to find a real solution. And the sigh of relief when I get resolution ...
.My big advantage is that I always knew, know, exactly what I'm after - I'm never running blind, the goalposts are rock solid visible, audible, always.
Last edited:
- Status
- Not open for further replies.