I would like to see the limitations of such a test and whether it could help in designing a non human test system.
Would you let the listeners choose the source material or inform them what they are testing?
Would you give the listeners multiple answers to choose from and the possibility to explain what they are hearing? It seems a limited use of their expertise limiting their answers.
If you don't know what you're trying to determine with an experiment, questions about the design and structure are meaningless.
another thing some don't "get" about science experiments, especially human subject experiments is that even with a good experimental protocol there are problems in execution that can only be reduced with critical practice - for both the experimenters and subjects
all of us are fallible, make more errors in to us novel circumstances
it can be fair to criticize the "stress" of the testing procedure as possibly affecting the subject's ability - the way to reduce it is with familiarization, practice sessions
the experimenter's need for practice may be less recognized from the school book picture of "the scientific method"
in a undergrad Microbiology project lab - for 3rd year students, expected to have a couple of experimental lab courses as prerequisite - the grad students TAs told us that our 1st try with transducing phage would likely fail - only one of ~20 teams succeeded in executing the well documented, thoroughly "debugged" protocol - while being coached to write it out in our lab notebooks, check off each step...
"amateur" Psychoacoustics experiments are likely to have this problem given the difficulty of assembling/scheduling any useful number of volunteer test subjects, their lack of time commitment to trial runs to "debug" the experimental procedure, to build the experimenter's experience to accurately execute
then internal positive controls have to be included to check the quality of the experiment - again lengthening the subjects time commitment, stretched even further by having multiple time limited sessions with each subject to avoid working into fatigue
as a practical matter you could have your prospective subjects work through the Harmon "Golden Ears" listening test to increase their comfort with Blind, ABX testing, perhaps also serve as useful audio discrimination training
Audio Musings by Sean Olive: Harman's "How to Listen" - A New Computer-based Listener Training Program
http://harmanhowtolisten.blogspot.com/2011/01/welcome-to-how-to-listen.html
I don't have academic or professional psychoacoustic test design, experimentation experience even though I have been a subject a couple of times
I expect the experimenters need to do trial runs, with exit interviews on the subjects experience of the directions, testing - at 1st with maybe your buddies - but trial runs with "naïve subjects" that have the same experience as your full test's subject pool
and compare your test protocol resolution/accuracy against published psychoacoustic thresholds on known signals/modifications in the positive controls you should include
http://www.delta.dk/imported/senselab/AES125_Tutorial_T4_Perceptual_Audio_Evaluation_Tutorial.pdf may help with Auio Perceptual experiment design - though the book does seem to spend more time on standards organizations publications than the "science"
all that is a lot of work - why I mostly reason about "audibility" from actual professional scientist's published Psychoacoustic work and EE Signal Theory, measurements
all of us are fallible, make more errors in to us novel circumstances
it can be fair to criticize the "stress" of the testing procedure as possibly affecting the subject's ability - the way to reduce it is with familiarization, practice sessions
the experimenter's need for practice may be less recognized from the school book picture of "the scientific method"
in a undergrad Microbiology project lab - for 3rd year students, expected to have a couple of experimental lab courses as prerequisite - the grad students TAs told us that our 1st try with transducing phage would likely fail - only one of ~20 teams succeeded in executing the well documented, thoroughly "debugged" protocol - while being coached to write it out in our lab notebooks, check off each step...
"amateur" Psychoacoustics experiments are likely to have this problem given the difficulty of assembling/scheduling any useful number of volunteer test subjects, their lack of time commitment to trial runs to "debug" the experimental procedure, to build the experimenter's experience to accurately execute
then internal positive controls have to be included to check the quality of the experiment - again lengthening the subjects time commitment, stretched even further by having multiple time limited sessions with each subject to avoid working into fatigue
as a practical matter you could have your prospective subjects work through the Harmon "Golden Ears" listening test to increase their comfort with Blind, ABX testing, perhaps also serve as useful audio discrimination training
Audio Musings by Sean Olive: Harman's "How to Listen" - A New Computer-based Listener Training Program
http://harmanhowtolisten.blogspot.com/2011/01/welcome-to-how-to-listen.html
I don't have academic or professional psychoacoustic test design, experimentation experience even though I have been a subject a couple of times
I expect the experimenters need to do trial runs, with exit interviews on the subjects experience of the directions, testing - at 1st with maybe your buddies - but trial runs with "naïve subjects" that have the same experience as your full test's subject pool
and compare your test protocol resolution/accuracy against published psychoacoustic thresholds on known signals/modifications in the positive controls you should include
http://www.delta.dk/imported/senselab/AES125_Tutorial_T4_Perceptual_Audio_Evaluation_Tutorial.pdf may help with Auio Perceptual experiment design - though the book does seem to spend more time on standards organizations publications than the "science"
all that is a lot of work - why I mostly reason about "audibility" from actual professional scientist's published Psychoacoustic work and EE Signal Theory, measurements
Last edited:
#6885
Whom: We have already trained the listeners beyond the capabilities of the intended user group. I don't know how I can change that.
How: We have a test setup, a piece of equipment to test and a question to ask. Is the test rig suggested by Tattoo not adequate to do the tests?
Whom: We have already trained the listeners beyond the capabilities of the intended user group. I don't know how I can change that.
How: We have a test setup, a piece of equipment to test and a question to ask. Is the test rig suggested by Tattoo not adequate to do the tests?
Last edited:
"Can I hear the difference between A and B in my system at matched levels with a source of XX and a load of YY?"
Large scale hedonic testing presupposes that the differences are discernible to all participants. You have to establish that first before designing the hedonic portion.
Large scale hedonic testing presupposes that the differences are discernible to all participants. You have to establish that first before designing the hedonic portion.
As jcx points out, that the experiment is 100% correctly executed is crucial, and as SY says, the right question has to be asked. For me, that would be that the DUT is transparent, that the presence of it in the chain is undetectable, that I could not be confident that I knew when it was in operation. And since this is in the discussion loop as well, what I might then do would be to select a recording that is notoriously "bad" for sibilance, where it is particularly irritating normally - I would do the AB and just listen to the quality of that sibilance - is one version of that sound less objectionable than the other, does the "flavour" of the sibilance alter between A and B ... that's the approach I use.
Last edited:
I probably expected the piece of equipment that is to be tested to have a gain of 1.
Both A and B use one source. I don't know how the source is going to react to the different inputs of A and B.
Both A and B use one source. I don't know how the source is going to react to the different inputs of A and B.
Luuvve it!! .. as someone who was driven "insane" by the software world, I can only roll my eyes ... 😉
And this is exactly where things can get nasty - maybe the presence of B, operating, is affecting the performance of the source, in itself - that influence could be very indirect, but it may turn out that that is actually the cause of a variation in perceived quality. Simplistic thinking about what is going on may make life easier, but the rule should be "No assumptions, ever ... !!"
That's part of the initial debugging process- getting rid of clicks, pops, or any other sort of noise which might give an unintended cue, making sure everything is stable into the loads at each end, and that your switching system is doing what you expect it to do. And getting rid of ground loops.
You'll find out quickly that the requisite level-matching (and you may want to take the trouble to have the two boxes matched in gain but a master volume control accessible to the listener) is very tedious to accomplish, but it's vital- human ears are VERY sensitive to small changes in level. Frequency response as well. This is going to take you some time to get correct and stable, so be patient during the process. Ditches must be dug.
Are the signal powers roughly the same in and out (e.g., line level buffer or A/D->D/A chain) or is this a power amplifier with a load?
My suggestion is that if you feel level changes are influencing results is to deliberately build more overt changes into the experiment; have A at normal, and +1, or even 2, and -1, etc dB as samples; and same for B. Partly, people will then learn how to hear "past" those aspects, and those who are constantly tricked by such can be weeded out, so to speak ...
Last edited:
Again, you're confusing the usual "does this sound better than that?" frame of mind with a "can you detect a problem here, compared with that?" attitude. I'm coming from the same angle as those people whose job it is to detect compression codec artifacts - when someone picks that a certain sound is not well converted, the compression is not "transparent", then I would be fairly certain that he could still hear it occurring if you then dropped the sound level by 3dB, or raised it by the same amount, for that matter.
The analogy is pretty strong - I listen to sound as if it's going through a compression algorithm, and that translator has flaws in it, it's not "perfect", it introduces little glitches depending upon the nature of the sound. So all I'm interested in is whether I can detect those glitches or not - nothing else matters, if the glitches are there then the sound is "faulty", and something has to be fixed. No glitches == optimised sound, and I can relax ...
The analogy is pretty strong - I listen to sound as if it's going through a compression algorithm, and that translator has flaws in it, it's not "perfect", it introduces little glitches depending upon the nature of the sound. So all I'm interested in is whether I can detect those glitches or not - nothing else matters, if the glitches are there then the sound is "faulty", and something has to be fixed. No glitches == optimised sound, and I can relax ...
If we are allowed to preselect/train the listeners. I don't see why we cannot also preselect/configure the listening method to ensure no errors in the results.
- Status
- Not open for further replies.