Listening Test Part II. Can you tell which......

[johnego]

Have you listened yet, what do you think? There are some strange noises around 58s on the left channel

I used default Windows 8 app (with headset icon). I can only open one file at one time. Took too long to change from one file to the other. I couldn't hear the noise you mentioned. Your sound system must be very good!

But i think one file is fatiguing, should be the tube loop, if level etc has been taken care of.

 

[scottjoplin]

Could you not download the zip file straight from the dropbox page without using dropbox? I've only listened so far on my chromebook and Grado headphones, I will try again tomorrow and listen through my stereo

 

[Pano]

The problem with a control like the sine wave test is that is it limited to 2 or 3 frequencies - the fundamental and a couple of harmonics. In the case of a pure versus distorted sine wave, it may only be testing the FR of the listening gear.

For example: I could not hear the H2 (2 kHz) at first. When I checked the FR of my headphones the reason appeared. The headphones had a 2K response almost 10dB below that of 1K, making the H2 very hard to hear. Once the headphone FR was flattened I picked out the distorted sine every time, no matter the DAC or amp. Another listener used Grado headphones that have a measured peak of nearly 10dB right at 2K. Much easier to now hear H2 with that peak. If I had EQ'd my Fostex headphones (or some others) to the Grado response then picking the distorted 1K sine would have been much easier. A difference in response of almost 20dB, right were the difference lies.

Does that prove the audibility of the test distortion by one set of equipment or ears over another? Not really. Because in this test the FR difference happened to lie right where the 2nd harmonic was. Even with the flatter EQ that allowed me to hear the 2K harmonic, I still could not reliably pick out the distortion on music. The distorted music did not have a difference only at 2 kHz, it would be broad spectrum. That overall distortion remained inaudible for me, despite the new audibility of the sine's H2.

The distorted 1 kHz sine wave is a useful control only for testing if the electronics can reproduce the first few harmonics of 1K. It tells us nothing about other frequencies or harmonics.

What the test does tell us, however, is whether or not we can hear the added distortion on our equipment and with our ears. It demonstrated that very well, but does not tell us why we can or can not hear it.

 

[turk 182]

2 scotches for Pano!

 

[mmerrill99]

The problem with a control like the sine wave test is that is it limited to 2 or 3 frequencies - the fundamental and a couple of harmonics. In the case of a pure versus distorted sine wave, it may only be testing the FR of the listening gear.

For example: I could not hear the H2 (2 kHz) at first. When I checked the FR of my headphones the reason appeared. The headphones had a 2K response almost 10dB below that of 1K, making the H2 very hard to hear. Once the headphone FR was flattened I picked out the distorted sine every time, no matter the DAC or amp. Another listener used Grado headphones that have a measured peak of nearly 10dB right at 2K. Much easier to now hear H2 with that peak. If I had EQ'd my Fostex headphones (or some others) to the Grado response then picking the distorted 1K sine would have been much easier. A difference in response of almost 20dB, right were the difference lies.

Does that prove the audibility of the test distortion by one set of equipment or ears over another? Not really. Because in this test the FR difference happened to lie right where the 2nd harmonic was. Even with the flatter EQ that allowed me to hear the 2K harmonic, I still could not reliably pick out the distortion on music. The distorted music did not have a difference only at 2 kHz, it would be broad spectrum. That overall distortion remained inaudible for me, despite the new audibility of the sine's H2.

The distorted 1 kHz sine wave is a useful control only for testing if the electronics can reproduce the first few harmonics of 1K. It tells us nothing about other frequencies or harmonics.

What the test does tell us, however, is whether or not we can hear the added distortion on our equipment and with our ears. It demonstrated that very well, but does not tell us why we can or can not hear it.

Yes, it's a crude control but if there's a problem with the test equipment correctly reproducing the first few harmonics of 1KHz (or the listener being similarly insensitive in this area), would you consider the equipment or the listener suitable for wider frequency, multi-dimensional testing of Tube Vs SS amplifier?

I also gave another example of a control that adjusted the volume in the A sample & introducing A Vs A(modified)in some random trials during ABX testing.

This is what I would attempt to implement if I had written & had control over the source of the ABX utility.

There may be better controls that can be used - these were just my initial thoughts but the ITU recommendations for most blind testing involves the equipment & room specification & participants pre-tested using controls before and/or after the test - this can result in eliminating some results from being included because they are considered invalid or in some participants from being excluded from the test itself

 

[johnego]

This is what I would attempt to implement if I had written & had control over the source of the ABX utility.

<snip>

this can result in eliminating some results from being included because they are considered invalid or in some participants from being excluded from the test itself

Why the ABX itself cannot be considered as sufficient control? To avoid false negative? If nobody pass the ABX then there is no sample space and no 'conclusion' should be made. If people still make wrong conclusions, its their problem, some people will make wrong conclusion no matter what.

If you design a new product and you want to find out if it is any better than a certain reference, first you want to gather expert listeners. You make a test scenario to find these people. And you want positive result so you also make sure the test system is good enough to avoid false negative...

 

[mmerrill99]

Why the ABX itself cannot be considered as sufficient control?

There is no control within ABX itself so I don;t know what you mean, "sufficient control"

To avoid false negative? If nobody pass the ABX then there is no sample space and no 'conclusion' should be made. If people still make wrong conclusions, its their problem, some people will make wrong conclusion no matter what.

Sure, some people will make wrong conclusions no matter what but the accepted recommendations on blind testing are that controls should always be used so why should this ABX test be any different?

How many time have you seen people being challenged to do an ABX test to prove they can hear what they claim? Do you think that a null result is considered a shrug-the-shoulders, tells us noting, to the person issuing the challenge? How many people are corrected on this forum (& other fora) when they use ABX results as "evidence" that something is not audible?

So, yes, I know that is supposed to be the idea of the null hypothesis but many people don't have a grounding in statistics or mathematics & don;t understand what this means. I have seldom seen them corrected when they make wrong conclusions about null results.

Yet when this is pointed out & the nuances/issues & problems with perceptual testing outlined there is a lot of push back.

I have no problem for both types of forum reports of listening to be treated as of equal value but that is not the way it is & we all know it. We all accept the sighted listening has no control over false positives but few will admit that forum style blind testing has no control over false negatives - so when someone posts a sighted listening impression, a false positive is often queried for this result whereas a false negative is seldom queried for blind test.

If you design a new product and you want to find out if it is any better than a certain reference, first you want to gather expert listeners. You make a test scenario to find these people. And you want positive result so you also make sure the test system is good enough to avoid false negative...

Sure but we are talking about forum run blind testing - ABX testing, in the main

 

[Pano]

Please start your own thread on the problems of ABX testing and stop thread crapping in these test threads. Point to your ABX thread here if you must, but please stop taking over threads with your personal agenda.

 

[BYRTT]

...As always, please:

1) tell which file you prefer by listening,
2) try to guess which one is the original rip
3) as an option you might add an ABX report, appreciated but not necessary condition.

Again, it is an original rip vs. tube amp loop record of the same file. Opening part of the concert was chosen, with very high dynamics and both silent and loud passages.

My post 34 was using active diy speaker, this opinion and ABX was using HD650 head phones:

1) Screw
2) Screw
3) ...:

foo_abx 2.0.5 report
foobar2000 v1.4.1
2018-12-16 06:50:37

File A: nail.wav
SHA1: b67b06219b6dfa12c87c4a1cfd95ca1d489362f6
File B: screw.wav
SHA1: e012634713c70a44fc096404a65f4fdc3b397f5a

Output:
WASAPI (push) : JRiver Media Center 23 (JRiver Media Center 23), 24-bit
Crossfading: NO

06:50:37 : Test started.
06:54:55 : 00/01
06:56:47 : 01/02
06:57:51 : 01/03
07:00:51 : 02/04
07:04:00 : 03/05
07:07:16 : 04/06
07:10:34 : 04/07
07:12:43 : 05/08
07:12:43 : Test finished.

----------
Total: 5/8
Probability that you were guessing: 36.3%

-- signature --
dc80cf25fd8b98ce5bd6cba1e88b5d8a8f2b2922

 

[PMA]

I did three runs. Lol, you tell me I feel one has a softer feel and attack to it, a bit less glassy but as always I can't get Foobar to agree

(I've got the full version playing now )

Thanks for trying the files and posting the screenshots, Karl. I will try as well, but I am not very optimistic what I would get.

This time, distortion at peak amplitudes is up to 1%. However, 2dB decay at 20kHz was eliminated and frequency response is flat now. So we have almost "pure" harmonic distortion with profile already shown in previous tests.

 

[PMA]

BYRTT, thank you for the new ABX. It still looks like guessing, do you think you could concentrate on 16 trials test?

 

[BYRTT]

Welcome PMA, your amp seems too good for my skills and replay systems especially into that ABX test and agree result looks like guessing, good thing about head phones for me is it drains bit less energy than using speaker, i give it a run using 16 trial test tonight using head phones and see were it gets me with the guessing : )

 

[Mooly]

Mooly i was wondering the Abx results you posted are they in order? if so seems you where better able to differentiate fresh out of the gate, or mere coincidence?

Yes, they were in order. I'll try again this morning.

Could you not download the zip file straight from the dropbox page without using dropbox? I've only listened so far on my chromebook and Grado headphones, I will try again tomorrow and listen through my stereo

You can do a direct from the Dropbox page without installing Dropbox. Just wait a few seconds for the options to appear at the top right of page.

 

[Mooly]

Another three.

 

[PMA]

The headphones had a 2K response almost 10dB below that of 1K, making the H2 very hard to hear. Once the headphone FR was flattened I picked out the distorted sine every time, no matter the DAC or amp. .

I have prepared a set of new test sine tones.

http://pmacura.cz/testsinetones.zip

There are tones with frequency of 300Hz, 1000Hz, 3000Hz. Originals and distorted of 0.9% THD, decreasing harmonic profile. Tones have 18s duration and -3dBFS amplitude. The distorted tones were recorded through the tube amplifier under test, so they reflect peak distortion under real conditions.

 

[Mooly]

I notice the edited poll isn't allowing multiple choices and there doesn't seem anyway to bring those options back...…

Let me see what I can do.

Hopefully this should be OK now. I'm going to vote

 

[PMA]

Another three.

You are getting better, 16/24 makes only 7.5% probability of guessing .

 

[Mooly]

Success

Would the person who was only able to vote once for which file they preferred please pm me and I can add your other choice for which you think was original to the poll

 

[Mooly]

You are getting better, 16/24 makes only 7.5% probability of guessing .

That makes me feel better Perhaps I should quit now while I'm ahead

 

[PMA]

Welcome PMA, your amp seems too good for my skills and replay systems especially into that ABX test and agree result looks like guessing, good thing about head phones for me is it drains bit less energy than using speaker, i give it a run using 16 trial test tonight using head phones and see were it gets me with the guessing : )

Anyway thanks for your efforts and for answering the 2 questions. Let's wait again until the poll is closed, for the disclosure.