That has not been my experience. In the case of my main music library there were no official warnings, just bad sectors that caused file corruption. The only sign of trouble was that the USB drive had been slow to connect a couple of times, so I set about to find out what was wrong. That's when the bad sectors and files were discovered. Fortunately I had not over-written the backup files.
I suppose I'm lucky, in 25 years of using hard drives this is the first that has failed me at home. And it's been rare at work.
What I would like to know (and I suppose the OP as well) is how to regularly - or better yet automatically - scan for file errors on the main library to avoid corrupting the backup.
I think we all know disk drives don't last forever. So we should be doing check disks regularly and checking the error logs, trying to anticipate when the number of bad blocks is significant, but we don't. 
BTW: I always thought it was the OS that handles these issues not the application.
Here's my strategy. I use 3 USB drives. All with identical music files. One is on my ripping PC, one is on my LMS and the other is floating. I usually stick the other one of my pCP clients.
So I rip new CDs on my ripping PC. I sync from PC to my LMS. I use WinSCP which gives my a list of files that it is going to copy. This lists only the new files. I then sync from my ripping PC to USB drive on pCP. Now I have 3 identical copies.
I disconnect USB drives from ripping PC and pCP. I should store them with a neighbor, but I don't.
A few years goes by, LMS starts acting up. Slow to boot, USB drive in read only mode, some tracks won't play. What do I do? Ignore it of course until LMS completely stops working. I then buy a new drive and sync from one of the other two.
It seems simple and logical to me but I know no one will agree.
regards
Greg
BTW: I always thought it was the OS that handles these issues not the application.
Here's my strategy. I use 3 USB drives. All with identical music files. One is on my ripping PC, one is on my LMS and the other is floating. I usually stick the other one of my pCP clients.
So I rip new CDs on my ripping PC. I sync from PC to my LMS. I use WinSCP which gives my a list of files that it is going to copy. This lists only the new files. I then sync from my ripping PC to USB drive on pCP. Now I have 3 identical copies.
I disconnect USB drives from ripping PC and pCP. I should store them with a neighbor, but I don't.
A few years goes by, LMS starts acting up. Slow to boot, USB drive in read only mode, some tracks won't play. What do I do? Ignore it of course until LMS completely stops working. I then buy a new drive and sync from one of the other two.
It seems simple and logical to me but I know no one will agree.
regards
Greg
I have a small NAS based on the HP Gen 8 microserver with 8G of ECC unbuffered RAM and 4 x 2TB NAS drives in mirror.
Every 28 days, FreeNAS is scheduled to do a scrub o check for corrupted files, but I haven't had one in over five years. It sits under the desk and quietly hums away while being accessed by Daphile a few rooms away.
Every 28 days, FreeNAS is scheduled to do a scrub o check for corrupted files, but I haven't had one in over five years. It sits under the desk and quietly hums away while being accessed by Daphile a few rooms away.
How does Your bank protect it's customer data, account balances etc from data corruption ? I believe they use raid systems with backups and even on different datacenters around the world. … Fire, storms and hearth-quakes happen.
The problem is... in order to check our multi-terabyte music collections, our computer systems are too slow. A simple copy of 1TB of music (150000 files) to my QNAP Nas on a 1Gb network needs a whole night unless we invest in a IBM Z mainframe with 64 processors and 8TB of RAIM (not a typo) memory...
The problem is... in order to check our multi-terabyte music collections, our computer systems are too slow. A simple copy of 1TB of music (150000 files) to my QNAP Nas on a 1Gb network needs a whole night unless we invest in a IBM Z mainframe with 64 processors and 8TB of RAIM (not a typo) memory...
The bank datacenters I've seen have the SAN on the first row then all the servers in the rows behind.
I think they do this because the SANs look so nice... really big, shinny black box with blue lights.
I am amazed that hackers can get into these systems because the effort required to do even legitimate work is enormous.
I think they do this because the SANs look so nice... really big, shinny black box with blue lights.
I am amazed that hackers can get into these systems because the effort required to do even legitimate work is enormous.
Yes, but the OS utility only checks for information consistency in the used portion of the disk. It does not do anything about the disk's unused area. Manufacturer provided utilities scan the entire surface of the disk, whether or not it is in use, and reports any error it encounters.
I do it once a month. On the 27th in the evening I start the Full Scan feature of the Data Lifeguard Diagnostic utility, which takes a few hours to complete. In the morning of the 28th, if no error was reported, I generate and start the script to run flac.exe -t on each and every music file. Late in the afternoon I have the result.
That's a type of strategy I could work with. Only move (sync) new files to the extra drives and don't do backups. Any files with modified metadata like tags, would be considered new - and this would sync.
The trouble has come with backups or manual copies from drive to drive.
Thanks to those mentioning disc utilities, I'll have to give them a try.
@Tromperie
Oh! I have had 2 disks fail in a RAID configuration just days apart. Only just finished rebuilding first disk before the second one failed. 100% up time, backup not required. Felt like I dodged a bullet.
I have also had a RAID disk controller failure taking out both disks. Lucky it was pre-production. I don't know if that is still possible on modern systems?
Oh! I have had 2 disks fail in a RAID configuration just days apart. Only just finished rebuilding first disk before the second one failed. 100% up time, backup not required. Felt like I dodged a bullet.
I have also had a RAID disk controller failure taking out both disks. Lucky it was pre-production. I don't know if that is still possible on modern systems?
Last edited:
"In my experience corruption is a rare occurrence usually a result of ignoring warnings. Failing disk drives and lack of disk space are usually know in advance." - Greg Erskine
OK, Windows has chkdsk, but it isn't normally run regularly or automatically. File corruption can occur when the disk is in the first stages of complete failure, in which case it will fail eventually, and needs replacement. But this isn't the problem I'm aiming at. Corruption can also be caused by bad sectors, improper shutdown, malware, corruption, physical damage, etc. And when it occurs, there are no automatic warnings generated.
Do I have this wrong? Do you know better? If so, please help. Please tell me how I can avoid random file corruption on my external USB drives (i.e. corruption that is not simply the first sign of drive failure, as mine was not).
OK, Windows has chkdsk, but it isn't normally run regularly or automatically. File corruption can occur when the disk is in the first stages of complete failure, in which case it will fail eventually, and needs replacement. But this isn't the problem I'm aiming at. Corruption can also be caused by bad sectors, improper shutdown, malware, corruption, physical damage, etc. And when it occurs, there are no automatic warnings generated.
Do I have this wrong? Do you know better? If so, please help. Please tell me how I can avoid random file corruption on my external USB drives (i.e. corruption that is not simply the first sign of drive failure, as mine was not).
My server runs Ubuntu. I use Unison to synchronise my library to a USB disk, which I store elsewhere.
Any additions are easily replicated.
If an old file has changed (never happened to me), Unison would find this by checksum change and offer to replicate the change. It does not do this operation automatically, so I would notice the unexpected action request and investigate.
It is possible to automate far more, but music libraries are very static.
I am dubious about RAID after a bad experience with both drives in an array failing at the same time, I suspect a software bug trashed both drives MBRs
Any additions are easily replicated.
If an old file has changed (never happened to me), Unison would find this by checksum change and offer to replicate the change. It does not do this operation automatically, so I would notice the unexpected action request and investigate.
It is possible to automate far more, but music libraries are very static.
I am dubious about RAID after a bad experience with both drives in an array failing at the same time, I suspect a software bug trashed both drives MBRs
The answer to your question is that I don't know, which is part of the problem. I *think* the corruption may have occurred while backing up from one drive to another. I don't know. I know there were no system messages to tell me that something had gone wrong. I found out when my music player started rejecting files as broken. [They are broken, it's not the music player that's the problem.] I love to play my whole collection on shuffle play, so I find the broken files when I try to play them.
The bottom line is that file corruption on disk can occur. When it does, there's nothing to tell you that it's happened. And it can't be avoided. [Happily, it's not too common.] If you can't find a way to verify that your master files are OK, you will over-write your good copies with corrupted ones, as I did.
Backup isn't enough. Only verify-followed-by-backup will keep your files as safe as they can be.
Backup isn't enough. Only verify-followed-by-backup will keep your files as safe as they can be.