ABX, MP3 and other stories

[Evan Robinson]

This might be a poor choice for my first post on these boards.

John Hope: Have you ever done an ABX comparison of a CD to a mp3 encoded with a high quality codec such as LAME? If you were to rip a CD and encode it at a setting that produced mp3s with a VBR of around 200kbps and run an ABX test, I doubt you could identify the mp3 even on a high end system. A guy I know ran this test on a $100,000+ rig with two guys that do studio work and four others that own a hi-fi store and I don't believe any of them passed. Well, I should say none passed at this quality setting or higher.

 

[John Hope]

Compression schemes

Hi erobinson

Welcome to the forum and to this thread.

I would observe that the compression used in the current systems intended for consumer audio such as DAB and the audio of DTV uses technology that is outdated. I stand corrected, but I think it's MPEG1. The highest bit rate used on DAB is 192kbps (BBC3 classical) and most channels use 128kpbs.

Virtually everybody I've come across can discern the quality impairment on these compression systems, even on relatively modest hifi equipment, especially when playing 'busy' musical pieces with a lot of percussive high frequency content. The coders simply run out of available bits to code. Something noisy from the 70's, like Bob Seger's Katmandu is an excellent test.

The thrust of this thread is on the disappointing quality of recording of some CDs. So we're already developing a good idea of the hearing abilities of some recording engineers! These guys are certainly not Ear Gods. There's a school of thought that believes that persons with long-term visual impairment often make the most critical audio listeners, because their sense of hearing is fine tuned to compensate for their visual deficiencies.

I'm not familiar with this LAME codec you mention and so couldn't comment on how subjectively transparent it might be, except to say it has an unfortunate name! Perhaps you have some links to sites offering details of this system ?

Best Wishes

John Hope

 

[Evan Robinson]

The LAME codec does have an amusing name. It is an open-source project, and the acronym stands for "Lame Ain't an Mp3 Encoder." Of course, the first line on their website states that LAME (uppercase) is an LGPL mp3 encoder. It is about six years old now and what I find most people ripping their CD's at high quality use. Here is a link to their webpage: http://lame.sourceforge.net/

You're definitely right about 128kpbs recordings. Of course, this depends on the original source as you mentioned. I can't stand when someone has obviously ripped their CD's to 128. They will put a burned CD into my car stereo system and I often have to ask them "what bitrate is this recording?"

Thanks for the welcome by the way, and sorry for steering this thread a little off track. I'm a young guy (20) and don't have a lot of CDs (around 70), so I don't have a lot of material to compare recording qualities against. I do have an embarrasing number of mp3's, but how can you be sure it is the mp3 or original recording that causes the problem?

As I get closer to the topic of this thread, I have begun to notice this "loudness" battle going on in the recording industry. Mostly on rock and hip-hop (probably not very popular in this forum). Some CDs it is just so obvious they have overly used compressors, making the CD lose all of its dynamics. And without dynamics, you are losing one of the most important things to quality music! I may notice this more than the average CD buyer because I've played in a symphonic band for 10 years. I can't count the number of times the director would ask for more dynamics, especially from the horn section! Of course, the average buyer, listening on his stock radio will think a piece that sounds "loud" also sounds better. But, it will leave the ones that care with a look of distaste on their faces.

One last thing. How can such great artists like the Red Hot Chili Peppers (subjective of course) produce such poorly recorded CDs? I guess they just don't care enough about the fans that have quality equipment to make that extra effort.

Evan

 

[Bas Horneman]

'The Essential Billy Joel" Columbia

Bought Billy Joel's greatest hits for 1 euro (Vinyl) sounds allright!

Bought Lois Armstrong's greatest hits for 1 euro..sounded brilliant...lifelike....probably recorded with tube preamps!

I somehow stopped buying cd's..on a subconscious level...(only realized this recently since I own a turntable (good one)) I have not been re-discovering vinyl...I am discovering it..since I grew up in the CD era.

John Hope: Have you ever done an ABX comparison of a CD to a mp3 encoded with a high quality codec such as LAME? If you were to rip a CD and encode it at a setting that produced mp3s with a VBR of around 200kbps and run an ABX test, I doubt you could identify the mp3 even on a high end system. A guy I know ran this test on a $100,000+ rig with two guys that do studio work and four others that own a hi-fi store and I don't believe any of them passed. Well, I should say none passed at this quality setting or higher.

interesting but what can we conlude from it? ....plus .....price says nothing about the sound quality...nor does your occupation.

But I must say that music recorded in this Millenium have been noticeably better.

. Which recordings are you refering to?

 

[SimontY]

If you were to rip a CD and encode it at a setting that produced mp3s with a VBR of around 200kbps and run an ABX test, I doubt you could identify the mp3 even on a high end system. A guy I know ran this test on a $100,000+ rig with two guys that do studio work and four others that own a hi-fi store and I don't believe any of them passed. Well, I should say none passed at this quality setting or higher.

That's very interesting to hear.

Once my house mates (when I was still a student ) played me a track taken from CD and put into 320kb MP3 and the same thing encoded as full quality OGG. I could tell the difference, and I preferred the OGG.

Maybe when these pros were listening on the ultra-expensive system their expectation was too great and ruined the test. I was under no pressure and expected to hear no difference.

I have tested using throwaway quality PC speakers for differences between 128kb and top quality VBR MP3s and scored well enough to conclude there was a real difference. I'd imagine higher bit rates could be differentiated using a hi-fi, in a relaxed environment.

I think MP3 sound is not off-topic on this thread, as some of the common failings of MP3-sound seem to afflict CDs - notably some that feature copy-protection.

Listening to a disapointing CD and then looking on the back to find a 'copy-protected' warning is sort of the opposite to going "wow, this Joni Mitchell sounds amazing" later to find the tiny HDCD logo there... hehe. If only they were all HDCDs!!!!

 

[Evan Robinson]

interesting but what can we conlude from it? ....plus .....price says nothing about the sound quality...nor does your occupation.

What can be concluded? Well, if many sets of trained ears could not identify the mp3 at the quality setting it was encoded at then it can be concluded that it is unlikely other people could tell the difference either, and thus a mp3 of that bitrate is satisfactory to encode since you will never know if you are listening to the original recording or not.

The insult at my "occupation" as an engineering/premed student was uncalled for. I don't need a degree in an audio field to understand an ABX test and how it is the only truly scientific method.

Once my house mates (when I was still a student ) played me a track taken from CD and put into 320kb MP3 and the same thing encoded as full quality OGG. I could tell the difference, and I preferred the OGG.

But this also was not an ABX test, and how can we not say psychology played a role?

Maybe when these pros were listening on the ultra-expensive system their expectation was too great and ruined the test. I was under no pressure and expected to hear no difference.

They were doing it for fun, not an article. They are all friends and wanted to see if they could tell a difference. I'm not sure what pressure they would have been under. Even if they were, they were under no more pressure than they would be in a non-ABX test, and ABX like I said before is the only valid method.

I hope I have not come to a forum where ABX testing is not considered valid. If so, what other way do you guys suggest?

 

[Bas Horneman]

The insult at my "occupation" as an engineering/premed student was uncalled for. I don't need a degree in an audio field to understand an ABX test and how it is the only truly scientific method.

I did not mean YOUR occupation..but ONE'S occupation. Working in a Studio or hifi shop does not automatically mean you have good hearing. The proof is an incredible amount of rubbish studio's and hifi shops sell and produce.

I therefore don't find it a valid test WHATSOEVER...a dutch computer magazine did the same type of test...with a similar crowd..and got the same results...TO ME that does not mean that one cannot hear the difference...I am convinced that in a GOOD system...almost everyone could hear it. But I'm not attacking you.....I just find the 100 000$ system and people's job (hifi store/studio people) completely irrelevant. However for the people involved in that test..it was completely true.


If you knew me...you would know that I NEVER intentionally insult people...EVER. And for the brief period you felt insulted by my less than perfect English I profusely apologize.

 

[SimontY]

I hope I have not come to a forum where ABX testing is not considered valid. If so, what other way do you guys suggest?

Please see the many, many threads on ABX/blind testing.

The way I test is to gauge enjoyment over a long period with all music. This is what tends to work for me. But it aint scientific, and it's slooow progress sometimes. I would imagine a combination of methods should produce the most insight when auditioning equipment.

Anyway, this has nothing to do with bad CDs. Music listening is all about enjoyment, or lack thereof.

 

[SY]

And as a quick note from someone who does sensory testing for a living, ABX is NOT the only valid method. It's just one tool (a damn good tool!), but unfortunately, it's become stuffing for many straw men.

 

[SY]

BTW, to answer your question about other methods, there are many, many, many of them.

For example, A/B preferences. A subject is given a pair of stimuli and chooses a preference. This is repeated with the stimuli being randomly shuffled in order. If the subject consistently picks one as preferred, that's a significant result.

Here's another: paired stimuli with "same" or "different" randomly determined. If the subject consistently can detect same or different, the result is significant.

 

[Evan Robinson]

BTW, to answer your question about other methods, there are many, many, many of them.

For example, A/B preferences. A subject is given a pair of stimuli and chooses a preference. This is repeated with the stimuli being randomly shuffled in order. If the subject consistently picks one as preferred, that's a significant result.

Here's another: paired stimuli with "same" or "different" randomly determined. If the subject consistently can detect same or different, the result is significant.

Although it was of course not obvious at all, these other tests you mention were kind of implied in my post. I assume both these methods are blind?

 

[Evan Robinson]

I did not mean YOUR occupation..but ONE'S occupation. Working in a Studio or hifi shop does not automatically mean you have good hearing. The proof is an incridble amount of rubbish studio's and hifi shops sell and produce.

I don't believe in Golden Ears. And, therefore their occupation is relevant to the test, as the defining difference between two people with healthy hearing is their ability to analyze the music. Although two people with equal ears hear the same music, a sound engineer is more likely to notice subtelties that the average person would not. I'm sorry I misinterpreted your meaning with regards to occupation.

I therefore don't find it a valid test WHATSOEVER...a dutch computer magazine did the same type of test...with a similar crowd..and got the same results...TO ME that does not mean that one cannot hear the difference...I am convinced that in a GOOD system...almost everyone could hear it. But I'm not attacking you.....I just find the 100 000$ system and people's job (hifi store/studio people) completely irrelevant. However for the people involved in that test..it was completely true.

Why are you convinced of this? I mean, I understand that by paying for an audiophile system you would hope to be able to hear the difference between a 200kbps VBR mp3 vs. the original. It might be disappointing that one cannot tell a difference, but it does not make it untrue.

You are contradicting yourself by saying on "a GOOD system almost everyone could hear it" and then saying "I just find the $100,000 system (...) completely irrelevant."

I really am curious what has you so convinced. Do you believe in Golden Ears?

If you knew me...you would know that I NEVER intentionally insult people...EVER. And for the brief period you felt insulted by my less than perfect English I profusely apologize.

Once again, I apologize as well, and I forgot to take notice of your country. This is the first board I have been on with such an international membership.

Please see the many, many threads on ABX/blind testing.

I'll have a look for them. Are there any ones that stand out with a particularly good debate you can link me to?

The way I test is to gauge enjoyment over a long period with all music. This is what tends to work for me. But it aint scientific, and it's slooow progress sometimes. I would imagine a combination of methods should produce the most insight when auditioning equipment.

The bolded portion is my point. How can you say with any confidence that your method is valid evidence to hearing a difference? I guess being an engineering student and looking to go into medicine make me a pretty scientific person, and I put a lot of confidence in the scientific method.

 

[SY]

Although it was of course not obvious at all, these other tests you mention were kind of implied in my post. I assume both these methods are blind?

Any valid and useful sensory test is blind or in some other way properly controlled. I don't mean to be pedantic, but ABX is a very specific protocol which is distinct from the two examples I gave.

 

[leadbelly]

Why does even a simple thread about crappy CD's have to devolve into another polemic on the evils on non-science-the-way-I-define-it? This place is slowly rotting away as a place for new ideas. I say that purely out of sadness.

 

[Bas Horneman]

You are contradicting yourself by saying on "a GOOD system almost everyone could hear it" and then saying "I just find the $100,000 system (...) completely irrelevant."

No I don't think so ..you can buy a KRELL or a AUDIO NOTE ONGAKU for that...neither perse mean it is a good system...

But then again a good system is person independent?

a Good system is perhaps not the right word...I think there are systems where such a test (a test as you described) can be much more apparent that the other system...I think such a system would be say a single driver loudspeaker (no crossover) with a single ended direct heated triode amplifier...in such a system the sound is much more sensitive to any particular change..for instance capacitor...whereas a pushpull amp with feedback and a complex crossover in the loudspeaker..you can change resistor A and replace it with B and not hear a thing...does that mean resistors don't have their own sound?...changing a cable will probably not be audible either in such a system because microdetails tend to masked by millions of resistors/transistors or get lost somewhere in the crossover point (pp) the crossover or the feedback loop.

That is why I am so convinced...I think ABX test have concluded that different types of wire don't influence sound...does that mean that it true? To some yes.... to me no. And I doubt if I could hear the difference between cables in a proper ABX test...but I am very convinced that I heard a difference between my DIY cat5 or Bastanis silver coated speaker cable. ABX would probably expose me as a liar.

 

[Evan Robinson]

Any valid and useful sensory test is blind or in some other way properly controlled. I don't mean to be pedantic, but ABX is a very specific protocol which is distinct from the two examples I gave.

I agree. All I meant was, by ABX I basically meant blind testing under scientific (controlled) conditions. I am aware ABX is a very specific format. Thanks for posting other examples.

Why does even a simple thread about crappy CD's have to devolve into another polemic on the evils on non-science-the-way-I-define-it? This place is slowly rotting away as a place for new ideas. I say that purely out of sadness.

So, anytime there is a debate in a thread it has devolved? This has been all very civil. That's fine if you can place faith in incredibly unscientific methods, but I cannot do that logically. I mean, who am I to tell someone not to spend their own money on $500 cables that don't sound any different than $20 cables if it makes them happy?

 

[Bas Horneman]

So, anytime there is a debate in a thread it has devolved?

Yeah leadbelly I don't see the problem....;-) it's all about exchanging ideas...I've never really gotten into the ABX discussions...cause I don't know enough about them...so let me

There aren't too many really new ideas anyway in audio...like someone's sig probably says..it's all about the music anyway...and if an old idea works to bring the music closer to me ....i'll take it as well....in fact I'll embrace it.

 

[leadbelly]

Knock yourself out then. I had considered your discussion threadjacking but I seem to be in the minority.

 

[Bas Horneman]

I had considered your discussion threadjacking but I seem to be in the minority.

No it is threadjacking...you're right.

 

[Evan Robinson]

No I don't think so ..you can buy a KRELL or a AUDIO NOTE ONGAKU for that...neither perse mean it is a good system...

But then again a good system is person independent?

a Good system is perhaps not the right word...I think there are systems where such a test (a test as you described) can be much more apparent that the other system...I think such a system would be say a single driver loudspeaker (no crossover) with a single ended direct heated triode amplifier...in such a system the sound is much more sensitive to any particular change..for instance capacitor...whereas a pushpull amp with feedback and a complex crossover in the loudspeaker..you can change resistor A and replace it with B and not hear a thing...does that mean resistors don't have their own sound?...changing a cable will probably not be audible either in such a system because microdetails tend to masked by millions of resistors/transistors or get lost somewhere in the crossover point (pp) the crossover or the feedback loop.

That is why I am so convinced...I think ABX test have concluded that different types of wire don't influence sound...does that mean that it true? To some yes.... to me no. And I doubt if I could hear the difference between cables in a proper ABX test...but I am very convinced that I heard a difference between my DIY cat5 or Bastanis silver coated speaker cable. ABX would probably expose me as a liar.

Ok, you are correct in saying that if a person for instance cannot identify the difference between two cables this does not prove there to be no audible difference. It does give significant evidence that this person cannot hear the difference. And, if a hundred people complete the test and all fail and all are devoid of hearing problems, then that is strong evidence that almost no one can hear the difference.

The only way to definitively prove (that I can think of) that audible differences can be heard between components is to first test the person's hearing by an audiologist. This will identify any defiencies in their hearing compared to a normal person. Then, you analyze the A/B signals and compare against the results of the hearing test and determing if any audible changes exist. Of course, if someone says they hear a difference, even if they fail an A/B style test, I cannot prove they are lying. How unfortunate. Anyway, all that matters is your level of enjoyment.