How to deal with music-file disk corruption?

Backup. That's what we're recommended to do to manage disk file corruption. That's what I did. But disk corruption doesn't come with a warning, when it occurs. So when it happened to me, although I didn't know that it had, I backed-up and over-wrote the good copies with the corrupted ones. I imagine some of you might have done the same? :worried:


Before I post what I have done to combat disk corruption, please tell me your ideas? How do you keep your music files safe?
 
Sounds great, but I wonder if it protects against corruption of individual files? As far as I know, file corruption can happen at any time (although it doesn't happen too often, thankfully), and may have nothing to do with failure of the whole disk drive. It sounds like SMART deals mainly with whole-disk failure modes.



I don't use a NAS drive. My music files - 1.2 TByte of FLAC and MP3 files - are stored (and backed-up) on external USB drives. I suffered corruption on the master drive, and that was that. Once I'd over-written the good copies while backing up, there was no way back. I still don't know how many files were damaged, and which ones they were. I find them when foobar2000 complains the file is corrupt while playing.


So I took the best steps I could think of to protect my music files. Now, I use the Microsoft file checksum integrity verifier (downloaded free from here). Before I back up, I check the master files for corruption using the Microsoft program. Then, when I know that no further corruption has occurred, I back up.



Now I think my music files are as safe as I can make them. Any suggestions for improvements?


P.S. I can upload the files I wrote to do this, if anyone would like copies? They're simple Windows batch files, and you would need to edit them to meet your own needs. But you're welcome if you want them?
 
Music files are basically read only right? I mean we are not talking about the files used in a session, but simple stereo audio for playback?

If so, import the file, and record the file hash (More or less any hashing algorithm will do, even MD5), then when you write the backup you hash the backup file and compare the hashes, if they match then you know to a very high level of confidence that both files match what was originally written, if not then you need to restore from an earlier backup that did match.

RAID is about availability NOT backup, backup should be offline and should be rotated, so that a failure while writing the backup does not leave you with both your working disks and their backups corrupt, I still commend LTO tape as a backup medium.

The problem with a RAID array is that while it protects you from a disk failure, it does little against the far more common HUMAN screwup, you need backup as well.

Regards, Dan.
 
Administrator
Joined 2004
Paid Member
I store my music library on 2 external USB drives, just like PatternChaser. A main library and a backup. Recently the main started to suffer file corruption from bad sectors. Fortunately I had been slack in making backups, so had not overwritten the old files with the new corrupt files before discovering the problem.

I can't remember exactly what I did to sort out the bad files (too many tests and attempts to count) but somehow did. Thanks for this thread and its warning. Putting in place a reliable system for correct backup is a top priority. :up:
 
Business data changes constantly and requires incremental and full backups. That is not the case with music files, which I only change when adding or correcting metadata (Artist name, release year, genre, etc.). I only delete music files in rare cases when I acquire an enhanced version of the same album, like an anniversary edition.

Instead of making full backups I keep mirror data sets which I synchronize with the main one. All my music is encoded in FLAC format, which I chose for many reasons, one of them being the embedded MD5 checksum. Using the FLAC encoder itself, I can verify the condition of all existing files.
 
grimberg: "Instead of making full backups I keep mirror data sets which I synchronize with the main one."

...until the main one becomes corrupt?


grimberg: "All my music is encoded in FLAC format, which I chose for many reasons, one of them being the embedded MD5 checksum. Using the FLAC encoder itself, I can verify the condition of all existing files."

Ah, I didn't know that! Thanks! I'll look into that, to see if it makes things any easier than my current scheme.

Thanks again.
 
Raid serves its purpose - if your drive goes wrong, just put in a new one and resynchronize. No need to pull backups and copy from them. Nowhere here was it said that raid would substitute backups.

Raid in ZFS allows on-the-fly correction of hidden file corruption. Without raid only detection is available.

Which filesystem detects file corruption apart of those few making hashes of blocks?

BTW the original poster complained about hidden file corruption overwriting his proper backups. Even if he had two, three, ...N rotating copies, eventually without proper detection he would have overwritten all of them with the corrupted file.

If I were to build a secure and reliable solution for home storage, I would use ZFS + Raid-Z with single drive redundancy and backup incrementally to offline ZFS drive/array with zfs send/receive. Very fast backup, safe-guarded operation, with history of backups under full control. If I ever needed to use the backup (the unlikely two drives failure), just zfs send the backup array to the new array and be set.

At work we rotate two offline arrays, likely superfluous at home, but easy to accomplish.
 
...until the main one becomes corrupt?

Correct. When that happens, I have the backup data sets to rebuild a new main.
I use Western Digital hard disks and run their maintenance software Data Lifeguard Diagnostic once a month. That software is available for free from their web site. I also run the FLAC encoder with option -t monthly to verify the integrity of the files. I finished encoding all my CDs in 2009, use the server daily and have never lost music files. I am now in my third main data set. The first rebuild was due to failure in the main and only one album was affected and recovered from the backup. The second rebuild was caused by running out of space in the main one.
It is important to understand that once the first scan error is detected in a hard disk, its days are numbered and you should replace it. Google did an internal survey on the subject and published their findings back in 2007.

https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf
 
Member
Joined 2002
Paid Member
"Modern filesystems detect file corruption well." - davidsrsb

No, I'm afraid they don't. That's rather the problem. We discover corruption when we come to play the affected file, not when the corruption happens.

They do.

Loss of data is 100% the fault of the user, but we always blame the computer.

Proper backup solutions have been available forever. We choose between effort required and convenience.
 
Member
Joined 2002
Paid Member
Every OS I have used has some form of disk checking. You can see the bad blocks and you determine your course of action. Corporate solutions can monitor the health of your whole system.

In my experience corruption is a rare occurrence usually a result of ignoring warnings. Failing disk drives and lack of disk space are usually know in advance.

If your data is of extreme importance then your backup strategy has to take this into account. Most people are happy with a daily backup.