tan,
I'm not sure if we will do any more. Keith who is getting the Marchland has determined that the DSP units don't work in his system and the remaining 4 of us DIY guys - I'd say we all agree that the differences between the units were not big enough for us to want to give it more time. All of us are experimenting with waveguides (or planning to) and I think at this point we'll be more interested in that side of things.
Trevor,
It isn't quite so black and white. It's easy to think that from a distance. I wrote a blog post recently about a major flaw with many blind tests:
Red Spade Audio: The biggest problem with blind testing
Please keep in mind that in general I'm in favour of blind testing. The problem is that normally in a blind test, you rely on audio memory. This is a bigger limitation than listener bias in many cases.
If everyone came away from the tests with their opinions completely unchanged, then you might have a point. I was not overly concerned with listener bias - I know the guys in the test and their biases reasonably well.
Keith expected to easily hear differences and he did. We even tested him blind and it was obvious to all that he could in fact hear and describe what he heard. Turns out he has some good listening skills. Once he hears something he zeroes in on that and blocks out everything else.
Murphy owned the modded DCX and felt in his system it was clearly superior, but admitted in the sighted tests that he felt they sounded the same. His bias didn't stop him from coming away with a very different opinion.
Gainphile - if I had to guess I'd say that he wasn't expecting to hear much if anything and his impressions seemed to back that up. If anything his bias may be preventing him from hearing very subtle things - the more you believe, the more you are likely to listen for small differences.
Antripodean - I'd say our opinions here are probably the closest, both of us expecting that differences will be subtle if present at all.
I did hear some differences, but they were the kind that would normally run under the radar. If I have to listen very critically, then it's subtle and I'm not that interested.
Which of these biases makes the test invalid?
If we had a stronger subjectivist contingent full of strong "cable believers" then I'd be wanting to make the test blind and more rigorous. However, even in that case, the instant switching is very revealing. I've seen people change their bias rather than hang onto them when instant switching is done.
Would you have considered it valid had we done a blind test but still had audio memory to content with? I would argue that this invalidates such a test to a greater degree if you are dealing with subtle differences. Put simply, we can't recall sounds with sufficient precision to compare a sound we hear presently to one we remember, even if there is only a one second pause. It will only work for easy to pick differences.
We did at times test blind, but in general it was technically a sighted test. Keep in mind that unless you are mentally tracking which is switched at the time, most of the time you don't actually know which ones is being played.
Due to the revealing nature of the switching method, I consider blind or not to be a minor factor. You are listening for any difference at the point of switching, so we switched frequently, sometimes 5 seconds apart.
I'd encourage people to try their own tests and make the effort to set up instant switching. If you do I think you will conclude as I have that it makes the comparison much easier and the need to test it blind becomes moot.
The second test was conclusive about one thing - the differences were not huge. They were at that level where some claim to hear a difference while others don't. That is enough conclusion for me. It tells me that my time is better focused elsewhere.
The tests weren't set up to be "statistically significant." They were informal in nature and intended to satisfy the curiousity of the participants. Afterwards I decided to share the results because I realise it's of great interest to many other diyers who mostly haven't had the luxury of comparing active crossovers in this way.
I'm not sure if we will do any more. Keith who is getting the Marchland has determined that the DSP units don't work in his system and the remaining 4 of us DIY guys - I'd say we all agree that the differences between the units were not big enough for us to want to give it more time. All of us are experimenting with waveguides (or planning to) and I think at this point we'll be more interested in that side of things.
Trevor,
It isn't quite so black and white. It's easy to think that from a distance. I wrote a blog post recently about a major flaw with many blind tests:
Red Spade Audio: The biggest problem with blind testing
Please keep in mind that in general I'm in favour of blind testing. The problem is that normally in a blind test, you rely on audio memory. This is a bigger limitation than listener bias in many cases.
If everyone came away from the tests with their opinions completely unchanged, then you might have a point. I was not overly concerned with listener bias - I know the guys in the test and their biases reasonably well.
Keith expected to easily hear differences and he did. We even tested him blind and it was obvious to all that he could in fact hear and describe what he heard. Turns out he has some good listening skills. Once he hears something he zeroes in on that and blocks out everything else.
Murphy owned the modded DCX and felt in his system it was clearly superior, but admitted in the sighted tests that he felt they sounded the same. His bias didn't stop him from coming away with a very different opinion.
Gainphile - if I had to guess I'd say that he wasn't expecting to hear much if anything and his impressions seemed to back that up. If anything his bias may be preventing him from hearing very subtle things - the more you believe, the more you are likely to listen for small differences.
Antripodean - I'd say our opinions here are probably the closest, both of us expecting that differences will be subtle if present at all.
I did hear some differences, but they were the kind that would normally run under the radar. If I have to listen very critically, then it's subtle and I'm not that interested.
Which of these biases makes the test invalid?
If we had a stronger subjectivist contingent full of strong "cable believers" then I'd be wanting to make the test blind and more rigorous. However, even in that case, the instant switching is very revealing. I've seen people change their bias rather than hang onto them when instant switching is done.
Would you have considered it valid had we done a blind test but still had audio memory to content with? I would argue that this invalidates such a test to a greater degree if you are dealing with subtle differences. Put simply, we can't recall sounds with sufficient precision to compare a sound we hear presently to one we remember, even if there is only a one second pause. It will only work for easy to pick differences.
We did at times test blind, but in general it was technically a sighted test. Keep in mind that unless you are mentally tracking which is switched at the time, most of the time you don't actually know which ones is being played.
Due to the revealing nature of the switching method, I consider blind or not to be a minor factor. You are listening for any difference at the point of switching, so we switched frequently, sometimes 5 seconds apart.
I'd encourage people to try their own tests and make the effort to set up instant switching. If you do I think you will conclude as I have that it makes the comparison much easier and the need to test it blind becomes moot.
The second test was conclusive about one thing - the differences were not huge. They were at that level where some claim to hear a difference while others don't. That is enough conclusion for me. It tells me that my time is better focused elsewhere.
The tests weren't set up to be "statistically significant." They were informal in nature and intended to satisfy the curiousity of the participants. Afterwards I decided to share the results because I realise it's of great interest to many other diyers who mostly haven't had the luxury of comparing active crossovers in this way.
qusp, I have an RME FF400..... what makes the digital attenuation in the RME cards better than that in (eg.) miniDSP?
Also, can you give us the gist of the 8 channel ackodac?
eg. input connectivity, price, balanced outs? etc.
I assume it's the akd18 listed on the ackodac site, but a search for that term yields nothing.
Thanks!
So, audio memory is a problem
May i ask if this is indeed the case, what is the point of improving one's system? Do you constantly click an A/B switch when listening for pleasure? Or you simply enjoy the warm feeling that this has been proven in an A/B comparison to be a better system? A mystery to me. Yet, another example that speaker guys listen and hear differently to amplifier/dac guys. Btw, your switching box looks like a prop from a horror movie to me. It could well be everything sounds the same through it.
It's not the biases, it's some issues with the test setup that make the validity far lower than it needed to be. There's a frequency response difference (and possibly some level matching) with the stock DCX that needed to be tracked down and fixed before the comparison.
hello...
About our memory and the evaluation of the very subtle differences, I think Paul Spencer is absolutely right: nothing more to expect.
All the DCX users who play with the remote must have noticed it: switching between various settings from the listening spot can reveal for exemple big differences in the space restitution. But, if I have just to cross the room for calling the other setting on the DCX, time to come back , I miss all. Vanished.
To come back in the objective world, if any DEQX owner is around here, it would be very nice to show us the step of any kind of boomer-medium, just to see if, when using the 300 dB, the response is symmetric and shows the famous pre ring/post ring. It's supposed to be the big trade off of the FIR, but maybe the designers have done something ?
About our memory and the evaluation of the very subtle differences, I think Paul Spencer is absolutely right: nothing more to expect.
All the DCX users who play with the remote must have noticed it: switching between various settings from the listening spot can reveal for exemple big differences in the space restitution. But, if I have just to cross the room for calling the other setting on the DCX, time to come back , I miss all. Vanished.
To come back in the objective world, if any DEQX owner is around here, it would be very nice to show us the step of any kind of boomer-medium, just to see if, when using the 300 dB, the response is symmetric and shows the famous pre ring/post ring. It's supposed to be the big trade off of the FIR, but maybe the designers have done something ?
because the way i'm talking about doing it, the attenuation is done with the RME mixer in 64bit float before its sent to the dac, rather than in the midst of DAC/ADC/DAC. I also have owned the FF400 in the past and still have a modded 9632 that I use for AES and BNC into the ackodac. the MCU based volume runs pretty much the same, but internal to the dac, until the volume is limited a decent amount, the DNR is completely unaffected and after that still I cannot pick flaws in it, plus its integrated which I love. either way results in completely transparent adjustment to these ears.
as far as the ackodac, I would rather not derail the thread MUCH further with that and i'm not privy to all the details, but suffice to say its basically an 8 channel ackodac, take all the connectivity options of the 2 channel ES9012 based ackodac (high rez async USB, i2s and multiple aes/spdif) and add 8 channel output along with 8 channel MCU based volume and to some degree filter control (which will also have wireless ipad control pretty shortly) you can arrange the 8 channels however you want. good chance given the ability of the 2 channel version to do so, that you can put 2 together for 16 channel, but dont quote me on that.
no idea on price, or timeline sorry
hope that helps, but I suggest emailing him for more detail, or PMing me if you have any Q's relating to the 2 channel, but i'd rather not continue OT here.
I'm starting to think that maybe I "wanted" to notice a huge difference in my system between stock and modded DCX,, perhaps some bias??
But after 2 gtg's, in 2 different systems, with instant switching, level matching, blind and not blind,,, it is clear that my Modded DCX sounds pretty much the same as stock DCX, Mini DSP, DEQX.. I suppose the only soultion for me is to actually do an instant switching test in MY system between stock and modded DCX..
I know one thing for sure, that is I thoroughly enjoyed the DIY part of constructing and installing the mods
Cheers..
I find that a brief pause less than a second is long enough to mean that audio memory is required. There were times when a switch was made during a pause and it was long enough that I was having to remember. Hence, I ignored those switches.
There were times we didn't get a clean switch over and so sometimes we had to ignore obvious differences this made.
I believe I gained some insight into difficulties getting results with blind tests. Had I been blind tested in the usual way, I would have failed along with the others. The reason is simple - most of the time I heard no difference. Let's say 80% of the time I perceived no difference at all, then for the remaining 20% of the time where I felt I heard a difference, I might still get it wrong. That means I end up with a result no better than random chance.
But let's say we modify the test. I only have to give an answer when I feel that I can actually hear something. So now we test the 20% of the time I claim I hear a difference. If I get it right 70% of the time, then we have some kind of a result. But then these numbers basically say that I'm splitting hairs over trying to hear something small. I lose interest at this point.
Murphy,
Audio memory is a tricky thing! You think you hear something, then you hear option B and wonder "did I really?" On and on it goes. That's why I was obsessed with instant switching.
I suspect if you repeated the test at home you'd get a similar result.
It would have been better had we precisely matched both response and level, however I was conscious of time and aiming to get through quite a lot, as well as not making the process too tedious for others. I was not quite so rigorous as some would like to see because we had a simple goal and people should keep this in mind before declaring that our tests were invalid. Our goal was to listen to the different units and to see if we could hear a difference and if we could to get a sense for how big those differences were. We made an attempt to match level and response as much as we could within the restraints of the day.
Had we matched both response and level very precisely, then the differences would have been even harder to pick and perhaps they would have vanished. To me it does not matter and the lack of this level of rigour doesn't make the comparisons completely invalid because in the end my goal was reached. I was satisfied that they were close enough in sound to not warrant further comparisons.
If we said that they sounded very different, then the lack of precise matching would be more of an issue because it could completely undermine the conclusion and make it the complete opposite.
So people should keep in mind the goal was not to publish an AES paper or satisfy the requirements of a refereed journal. This was an informal comparison between a small group of DIY guys with a bit of a social event mixed in. Insistance that we meet certain criteria to satisfy others can be a little irritating. In truth, the comparisons we've done are far more valid than the methods most will use to form an opinion on these units. Typically someone will integrate it into their system and compare the final result to other references based on memory only. There are so many uncontrolled variables that it's hard to make a comparison of something as subtle as a line level active crossover.
If anyone feels that these results are not valid enough, perhaps they should organise their own comparison and they will have the option to do it blind ABX style. I'd certainly be interested to read about the result.
And I'm always interested to add another comparison to my blind test results page:
Red Spade Audio: Blind Test Results
At the moment I only have two tests that revealed a difference in there.
There were times we didn't get a clean switch over and so sometimes we had to ignore obvious differences this made.
I believe I gained some insight into difficulties getting results with blind tests. Had I been blind tested in the usual way, I would have failed along with the others. The reason is simple - most of the time I heard no difference. Let's say 80% of the time I perceived no difference at all, then for the remaining 20% of the time where I felt I heard a difference, I might still get it wrong. That means I end up with a result no better than random chance.
But let's say we modify the test. I only have to give an answer when I feel that I can actually hear something. So now we test the 20% of the time I claim I hear a difference. If I get it right 70% of the time, then we have some kind of a result. But then these numbers basically say that I'm splitting hairs over trying to hear something small. I lose interest at this point.
Murphy,
Audio memory is a tricky thing! You think you hear something, then you hear option B and wonder "did I really?" On and on it goes. That's why I was obsessed with instant switching.
I suspect if you repeated the test at home you'd get a similar result.
It would have been better had we precisely matched both response and level, however I was conscious of time and aiming to get through quite a lot, as well as not making the process too tedious for others. I was not quite so rigorous as some would like to see because we had a simple goal and people should keep this in mind before declaring that our tests were invalid. Our goal was to listen to the different units and to see if we could hear a difference and if we could to get a sense for how big those differences were. We made an attempt to match level and response as much as we could within the restraints of the day.
Had we matched both response and level very precisely, then the differences would have been even harder to pick and perhaps they would have vanished. To me it does not matter and the lack of this level of rigour doesn't make the comparisons completely invalid because in the end my goal was reached. I was satisfied that they were close enough in sound to not warrant further comparisons.
If we said that they sounded very different, then the lack of precise matching would be more of an issue because it could completely undermine the conclusion and make it the complete opposite.
So people should keep in mind the goal was not to publish an AES paper or satisfy the requirements of a refereed journal. This was an informal comparison between a small group of DIY guys with a bit of a social event mixed in. Insistance that we meet certain criteria to satisfy others can be a little irritating. In truth, the comparisons we've done are far more valid than the methods most will use to form an opinion on these units. Typically someone will integrate it into their system and compare the final result to other references based on memory only. There are so many uncontrolled variables that it's hard to make a comparison of something as subtle as a line level active crossover.
If anyone feels that these results are not valid enough, perhaps they should organise their own comparison and they will have the option to do it blind ABX style. I'd certainly be interested to read about the result.
And I'm always interested to add another comparison to my blind test results page:
Red Spade Audio: Blind Test Results
At the moment I only have two tests that revealed a difference in there.
The fact that it wasn't a proper double blind test with statistical inferences to draw a proper valid conclusion. As well, the tester which is yourself shouldn't be part of the test so your memory issue gets thrown out of the window
Why do people always make excuses for not conducting these tests properly ? Don't you think it would add much more credibility to your aim of comparing equipment ?
regards
Trevor
Trevor,
Have you set up and conducted a double blind ABX test yourself? One with instant switching and rigorously set up with a statistically significant sample and careful statistical analysis?
It wasn't our intention to set up such a test. This was an informal comparison where we made some attempt to reduce down uncontrolled variables. However, to do blind ABX and to do it right is a serious undertaking. I'm not making excuses - you are simply insisting on a level of "credibility" with others that was not part of our goal. To put it bluntly, we did it for our own interest, not yours. Later Keith decided to post it elsewhere and I decided to share it abroad as I knew many would be interested.
Have you set up and conducted a double blind ABX test yourself? One with instant switching and rigorously set up with a statistically significant sample and careful statistical analysis?
It wasn't our intention to set up such a test. This was an informal comparison where we made some attempt to reduce down uncontrolled variables. However, to do blind ABX and to do it right is a serious undertaking. I'm not making excuses - you are simply insisting on a level of "credibility" with others that was not part of our goal. To put it bluntly, we did it for our own interest, not yours. Later Keith decided to post it elsewhere and I decided to share it abroad as I knew many would be interested.
- Status
- This old topic is closed. If you want to reopen this topic, contact a moderator using the "Report Post" button.