Go Back   Home > Forums > >
Home Forums Rules Articles diyAudio Store Blogs Gallery Wiki Register Donations FAQ Calendar Search Today's Posts Mark Forums Read

The Lounge A place to talk about almost anything but politics and religion.

John Curl's Blowtorch preamplifier part III
John Curl's Blowtorch preamplifier part III
Please consider donating to help us continue to serve you.

Ads on/off / Custom Title / More PMs / More album space / Advanced printing & mass image saving
Reply
 
Thread Tools Search this Thread
Old 21st November 2018, 01:06 PM   #10011
Jakob2 is offline Jakob2  Germany
diyAudio Member
 
Join Date: Jun 2004
Location: germany
Quote:
Originally Posted by PMA View Post
<snip>

Now, to your test with black boxes. I have read your description carefully. Please tell me, you, who say that 16 attempts in ABX is not enough and you consider it statistically unimportant, how valid is your 5/5 result in a semi-sighted, semi-blind test as you have described.
First of all it wasnt "semi-blind" or "semi-sighted" it was an A/B paired comparison preference test "double blind" .Neither did i know about the marking of the variants when handing them over, nor did the listener know which of the two DUTs they "had to prefer".
And during my own test we followed the same procedure....

Addressing your question for the statistical part unfortunately needs a short recap of some basics. (for simplicity i neglect all philosophical differences between NHST, Fisherian experiments and the Neyman/Pearson approach)

The null hypothesis is stated before (random guessing, H0: p = 0.5), so two different errors can be made when analysing the results of a listening experiments:

The null hypothesis could be rejected although it is true (so-called alpha error)
and
the null hypothesis could _not_ be rejected although it is _false_ (so-called beta error)

The guard against alpha errors is to choose a significance level that is sufficiently low; the usual SL = 0.05 means that in the long run the alpha error will be <= 5%.
So this required SL decides which minimum number of samples is needed.

Therefore 5 samples are sufficient (in a one-sided test) as the probability to get 5 hits in 5 trials is 0.032, which is below 0.05

The guard against beta errors is the statistical power (1- beta), so if you want to keep the beta error as low as the alpha error, you need a statistical power of (1-0.05) = 0.95.

The usual minimum required statistical power is 0.8, which means to accept a beta error of 0.2 , that is already 4 times higher as the alpha error.

To calculate the statisticial power you have to assume something about the difference under test conditions.
The calculations ive shown before therefore assumed p2 = 0.6 (instead of the p = 0.5 as assumed under the null hypothesis) .

I hope that helps to understand why the requirements for low alpha and low beta errors are different.

And i hope it shows why training under the specific test conditions helps, because it might (if a difference is detectable) the proportion of correct answers and therefore lowers the number of samples required.

Last edited by Jakob2; 21st November 2018 at 01:16 PM.
  Reply With Quote
Old 21st November 2018, 01:31 PM   #10012
mmerrill99 is offline mmerrill99
diyAudio Member
 
Join Date: Aug 2014
Quote:
Originally Posted by DF96 View Post
So you accept that what you claimed has not not been said 'on here' has been said 'on here'? Even if it had not been said before (which I doubt) it has now been said by you, and you were 'on here' when you said it.
Wow, is this not a perfect example of pedantry? What are you trying to prove with this?


Quote:
Hhoyt has given you some evidence.
I asked him for the details of this - do you have the details?


Quote:
My bias is that I would take any reasonable unsighted test over any sighted test. There: my bias is out in the open for all to see. It seems to me that this is a common-sense position to take, given all the false positives which can easily be introduced into sighted tests.
I know what your bias is - you will accept any test results even if all are false negatives, as long as it supports your worldview. Your failure to accept that ABX testing, in particular, is skewed towards false negatives is a perfect example of your bia sin this matter. If you understood statistics you would see this but you don't need to be able to understand statistics to see that Jakob2's evidence & the use of internal controls is a crucially necessary factor in ABX testing

Quote:
My bias is that a genuinely obvious difference will be heard in an ABX test; not necessarily by everyone, and not necessarily every time, but often enough (as shown by the statistics). Whether some other blind test will be better or worse at doing this is a separate issue. I make no claims about how 'scientific' ABX is; I merely believe that all sighted tests are 'unscientific' and some are merely marketing.
Yes, that your bias is based on your beliefs is very obvious. And, indeed ABX tests, as seen on audio forums, are performance art & pseudo-science
  Reply With Quote
Old 21st November 2018, 01:37 PM   #10013
scottjoplin is offline scottjoplin  Wales
diyAudio Member
 
scottjoplin's Avatar
 
Join Date: Dec 2015
Location: Penrhyndeudraeth
Quote:
Originally Posted by mmerrill99 View Post
I know what your bias is - you will accept any test results even if all are false negatives, as long as it supports your worldview. Your failure to accept that ABX testing, in particular, is skewed towards false negatives is a perfect example of your bia sin this matter. If you understood statistics you would see this but you don't need to be able to understand statistics to see that Jakob2's evidence & the use of internal controls is a crucially necessary factor in ABX testing
Excellent execution of the social justice warrior's 5 Step protocol, I would have expected no less, congratulations, full marks
  Reply With Quote
Old 21st November 2018, 01:38 PM   #10014
PMA is offline PMA  Europe
diyAudio Member
 
PMA's Avatar
 
Join Date: Apr 2002
Location: Prague
Quote:
Originally Posted by gpapag View Post
Ive already said that the amp Pavel used does not change the dynamic range of the signal.

I have asked myself a relevant question though on this point.
When I am to arrange such a comparative test: In case one of the DUTs changes the dynamic range appreciably (>1-2dB), either due to increased noise floor or due to smooth compression of the high level peaks or both, what instrument reading should I use for to level equalize the two files (target is equal perceived level of loudness)?
Peak hold, VU peak, average RMS power?

George
I think you hit the point, George - the amp should not change the dynamics of the recording and also should not audibly increase the noise level of the recording.

And you are asking a good question - unfortunately I am not able to give you similarly good answer .
  Reply With Quote
Old 21st November 2018, 01:58 PM   #10015
mmerrill99 is offline mmerrill99
diyAudio Member
 
Join Date: Aug 2014
Quote:
Originally Posted by DF96 View Post
...

My bias is that I would take any reasonable unsighted test over any sighted test. There: my bias is out in the open for all to see. It seems to me that this is a common-sense position to take, given all the false positives which can easily be introduced into sighted tests..
So how do you judge what is "any reasonable unsighted test"?
  Reply With Quote
Old 21st November 2018, 02:10 PM   #10016
DF96 is offline DF96  England
diyAudio Member
 
Join Date: May 2007
Quote:
Originally Posted by scottjoplin
The marketing men will never give up trying to persuade you otherwise, it's their job
You are surely not trying to tell me that money could corrupt a technical discussion? After all, we have been assured that people can overcome their bias in sighted tests. Someone who can overcome sight/knowledge bias and simply report what their ears hear should surely have enough self-control to overcome any financial bias too?

Quote:
Originally Posted by mmerrill99
Wow, is this not a perfect example of pedantry? What are you trying to prove with this?
I am merely demonstrating that that which was alleged to be 'not said' 'on here' has now been said 'on here' by you at least. I strongly suspect it has been said in the past too, either by you or others with a similar view to you. You were talking as though this forum has a uniform view; it does not. On the basis that if something is not impossible then it is inevitable (if you wait long enough) then it is highly likely that almost every possible view on audio has been expressed by someone on this site at some point in the recent past. Hence it seems rather silly to claim that a particular view has not been said, as though that sums up everyone here.

Quote:
I know what your bias is - you will accept any test results even if all are false negatives, as long as it supports your worldview.
I am not aware that I have 'accepted' any test results, although I may have given some more credence than others. I genuinely don't know what would satisfy you, apart from an agreement to accept everything you assert on all subjects.

Everyone has a worldview; everyone tends to reject anything which conflicts with that worldview, so I am happy to admit that I am a fairly normal person. Those who think that they are different from this are merely better at fooling themselves.

Quote:
Yes, that your bias is based on your beliefs is very obvious.
Everyone's bias is based on their beliefs. You should be pleased that in my case this is obvious. Obvious bias is easier to deal with.

Quote:
And, indeed ABX tests, as seen on audio forums, are performance art & pseudo-science
And sighted tests are not?

I will show another piece of bias: I suspect that your position is at least partly motivated by a commercial interest in audio. What form that takes I don't know, but my experience is that dogged persistance (and faith in sighted tests) is often a sign of commercial interest. My experience is also that people with a commercial interest are often reluctant to say so, until one day it spills out when they use commercial success bragging in lieu of reasoned argument.

Quote:
So how do you judge what is "any reasonable unsighted test"?
I only added the "reasonable" adjective to avoid being accused of accepting any old junk as evidence.

Last edited by DF96; 21st November 2018 at 02:11 PM. Reason: extend
  Reply With Quote
Old 21st November 2018, 02:25 PM   #10017
mmerrill99 is offline mmerrill99
diyAudio Member
 
Join Date: Aug 2014
@DF96, your pedantry is getting tedious for everybody, I suspect so I'l keep it short?

It's not a 'competition', just a debate on various flaws in listening tests

So you proffer no definition of what is a "reasonable" unsighted test to you - you simply "added the "reasonable" adjective to avoid being accused of accepting any old junk as evidence." So tell us how you know the unsighted test isn't just comprising of false negative results or are you not concerned about that, only wishing to minimize false positives?
  Reply With Quote
Old 21st November 2018, 02:44 PM   #10018
scottjoplin is offline scottjoplin  Wales
diyAudio Member
 
scottjoplin's Avatar
 
Join Date: Dec 2015
Location: Penrhyndeudraeth
Oh, I don't know, most of his posts make me smile
  Reply With Quote
Old 21st November 2018, 02:46 PM   #10019
TNT is offline TNT  Sweden
diyAudio Member
 
Join Date: Apr 2003
Location: Sweden
mmerrill99, your are in the audio business - right? Did I recall correct that you have a website?

//
__________________
More distortion to the people!
  Reply With Quote
Old 21st November 2018, 02:47 PM   #10020
scottjoplin is offline scottjoplin  Wales
diyAudio Member
 
scottjoplin's Avatar
 
Join Date: Dec 2015
Location: Penrhyndeudraeth
Bet he won't answer
  Reply With Quote

Reply


John Curl's Blowtorch preamplifier part IIIHide this!Advertise here!
Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


New To Site? Need Help?

All times are GMT. The time now is 08:21 PM.


Search Engine Optimisation provided by DragonByte SEO (Pro) - vBulletin Mods & Addons Copyright © 2019 DragonByte Technologies Ltd.
Resources saved on this page: MySQL 14.29%
vBulletin Optimisation provided by vB Optimise (Pro) - vBulletin Mods & Addons Copyright © 2019 DragonByte Technologies Ltd.
Copyright ©1999-2019 diyAudio
Wiki