MP3 vs. CD: The Beer Test

Unless you’re a dog, a whale, or a computer, you’re not going to be able to tell the difference between a good MP3 and CD audio.

Think you’re an audiophile? I’ll bet you beers you can’t tell the difference between 256 Kbps VBR MP3s and CD audio in a blind test.

Track Selection

12 tracks have been selected from commercially sold, mass-produced audio CDs (i.e. “silver” discs – no CD-R, no online purchases, etc.). The primary criteria for track selection are length of track (2-4 minutes), availability (I have to own them), and pure subjectivity.

Selected tracks are:

Song Artist Album Genre
Cats & Dogs Howard Shore The Truth About Cats & Dogs (Soundtrack) Jazz
The Four Seasons: II. Largo Antonio Vivaldi Classical Christmas Favorites Holiday
Goldberg Variations Johann Sebastian Bach Tune Your Brain with Bach: Heal Classical
I Wonder As I Wander The Night Heron Consort A Celtic Celebration Holiday
I’m On Fire Bruce Springsteen Born In The U.S.A. Rock
No! They Might Be Giants No! Alternative, Children’s Music
The Nutcracker Suite – Dance of the Sugar Plum Fairy Peter Ilyich Tchaikovsky Classical Christmas Favorites Holiday
She’s Always A Woman Billy Joel The Stranger Rock
Surfin’ U.S.A. The Beach Boys Endless Summer Rock
Violin Concerto No. 2, K. 211, III Itzhak Perlman,Violine; Wiener Philharmoniker, James Lavine Mozart: Violin Concertos No. 2 & 4 Classical
We Will Rock You Queen ESPN Presents Stadium Anthems Rock
Zombie Zoo Tom Petty Full Moon Fever Rock

Order

Songs are ordered alphabetically by song title.

CD to MP3

MP3 tracks were imported to iTunes with these settings:

  • stereo bit rate = 256 Kbps Variable Bit Rate (VBR)
  • Quality = Highest
  • Sample Rate = 48 kHz
  • Channels = Audio
  • Stereo Mode = Normal
  • Smart Encoding Adjustments = Off
  • Filter Frequencies Below 10 Hz = On

CD tracks were imported to iTunes using the Apple Lossless Encoder.

Randomness

To determine which track comes first in a pairing, I rolled a six-sided die. If odd, the MP3 track comes first. If even, the CD track comes first.

Sound Level

I created a playlist with the 24 tracks and sound-leveled all tracks with iVolume v3.1.4.

CD Burning

A CD-R was burned with my iMac using iTunes. The CD drive is 24x read/write, Matshita DVD-R UJ-875, firmware revision DB09.

Listening

Using any stereo components desired, subject may listen to tracks on audio CD for for as long and as often as desired switching tracks at will, making notes, etc.

Subject must work alone and not seek opinions from others.

To avoid the possibility of finding differences in the original and compressed files, subject must agree not to rip any tracks from the CD or in a any way use a computer to analyze the CD.

Subject must use his/her ears only. No visual cues (including, but not limited to, graphic equalizers) may be used.

Identification

For each pair of tracks representing the same original CD track, the subject must identify whether the A track or the B track is the original.

Scoring

Based on pure chance, the odds for getting a specific number of tracks identified correctly are:

N Percentage
0 0.024%
1 0.293%
2 0.611%
3 5.371%
4 12.085%
5 19.336%
6 22.559%
7 19.336%
8 12.085%
9 5.371%
10 0.611%
11 0.293%
12 0.024%

What the results mean:

Number Correct Conclusion
5, 6, or 7 No better than random chance. Sorry, you’re not an audiophile.*
3, 4, 8, or 9 Better than chance, but not significant enough to make claims about the ability to distinguish between original and compressed files without further testing.*
0, 1, or 2 Statistically significant. However, since the subject identified most tracks incorrectly, there will be much mocking.*
10 or 11 Statistically significant. Subject can reliably distinguish between original and encoded files, may claim such, and may spend time re-ripping entire CD collection at a higher bitrate. Subject will be treated to beers at a mutually agreed upon establishment.
12 Accolades of auditory excellence. Subject may also have a cheeseburger.

*Plus you owe me beers.

Credits

In May 2005, I bragged about my ability to distinguish between 128 Kbps MP3s and CD audio. My MIT friend Dave Kaffine then challenged me with this test. His test was double-blind, used an HP-48SX for ordering the tracks, and use Perl scripts for the randomness. My test is not double-blind, used alphabetical order for the tracks, and used a six-sided die for randomness, so it’s admittedly less rigorous. But I’m no Dave, nobody is. When I took Dave’s test, I got 7 of 12 tracks correct, which is statistically insignificant. However, I maintain that I was certain (and could tell within five seconds) on at least a couple of tracks. I concluded that if I could tell the difference on one track, then I must re-rip my entire CD collection from 128 Kbps CBR to 256 Kbps VBR, which I did. So I declared moral victory but lost the bet.

I think I’ll be getting free beers from anyone that takes me up on this challenge.

Related Posts

17 Replies to “MP3 vs. CD: The Beer Test”

  1. “Subject must work along and not seek opinions from others.”

    Do you mean “along” or “alone”?

    That latter seems more likely. But maybe this is just part of the test?

    Did I pass?

  2. One major problem people have brought up is that you normalized the audio files, and people seem to feel that majorly shifts things. I don’t fully agree, but you might want to not normalize the files.

  3. Greetings David,

    Except that the sound-leveling that I did does not involve re-encoding the files. I used iVolume, which adds a value to the iTunNORM tag, making use of the built-in “Sound Check” feature of iTunes:

    http://www.mani.de/ivolume/help.html

    I could have easily skipped this test and had no impact on the results. I just like all my tracks to be sound-leveled, and when you’re mixing genres, the default is usually a mix of too loud and too soft.

    Regards,
    Erik

  4. So, where are the results of the test? Would be nice if you published a histogram showing the numer of people and the score they obtained. Also, could you upload the sequence of songs to a website (include only maybe 20 seconds of each to avoid copyright problems) so that we can all take part? Thanks

  5. Greetings BFN,

    Good one. It wouldn’t be a blind test if I published the answers, now would it? Plus I can’t upload the songs – that would be copyright infringement. Not good form for an IP attorney. So if you really want to lose some beers, send me your USPS address, I’ll lend you the CD, you send me you answers, then we can figure out when/where you buy me beers.

    Alternately, you can follow my methodology with a trusted friend and pay the beers to him/her. Nobody has passed the test yet, but no dogs or whales have taken the bait.

    Regards,
    Erik

  6. Well, I think that if you just upload 20 seconds it would be all right and everyone would understand it’s for just research purposes. But fair enough if you don’t wanna do it. I’m in the UK so don’t see the point of sending the cd, besides, the idea would be for many people to participate.
    How many people have taken the test so far? Can you publish the results?
    Cheers!

  7. I would be willing to give this a shot, but I am also in the UK so sending a CD seems a bit much. If you do upload some stuff, drop me an email and I’ll give it a go then. Here in the UK, anything recording over 50 years old is out of copyright, so that might be a possibility.

    One point to consider: with the lossless file, there’s no point in increasing the sample rate to 48 kHz if you’re encoding a CD. A CD has only got a sample rate of 44.1 kHz. In actual fact, recording a 44.1 kHz signal at 48 kHz may actually make the resulting audio file sound worse than it would have done at 44.1 kHz, depending on how iTunes deals with the conversion.

Leave a Reply

Your email address will not be published. Required fields are marked *