Backups.

[sn00p]

It happened again....

This time a vital machine at work went down - this happened a couple of years ago with another machine, so I built a backup system using rsync that created daily backups of important data. Both ends of the backup system would email if there was a problem i.e client was unable to backup to the server or that the server had not seen machine x backup for y number of days.

Brilliant, it worked fantastic.

That is until you put a human into the equation and they happen to just ignore the emails saying that "machine x has not backed up for 230 days".

Woke up in the middle of the night in a panic because there was critical data on that machine.

Managed to get somebody to put the drive into another machine this morning and *luckily* I was able to copy out the important data from the drive. The drive is buggered, there's a physical fault on it, but fortunately not on the important data. Bizarrely enough, the rsync backup scripts do a sync locally before doing it across the network and the backup script had been happily backing up data locally! There were a couple of years worths of daily backups, I grabbed those as well just incase the data was buggered, far easier to recover from data thats a couple of days/weeks old than 12 months.

Moral of the story, always assume that humans will cover up the big red flashing light with something rather than act on it!

Backups people, double check!

 

[Silent_Scone]

At home all my data is backed up to two external 3TB drives. Using SSDs you really have to be extra cautious. At work all important data I use is on a RAID5 NAS, and uploaded offsite every evening. You really don't have to go mental to insure your data is safe, as long as it is backed up. People are more careless than they are brainless

 

[Munson]

I'm not in an IT environment, but to me, backups are as important as breathing.

I use Time Machine on an external USB drive for first defence. Plus simple duplicated data on at least two other drives, plus disk images on at least one external drive and of course on my NAS in RAID5. I also backup super critical files to Giganews' Dump Truck.

People who don't backup data are retards.

 

[Longy]

What type of data was it? can it not be stored on the server/cloud share? Can't trust any end user to do anything.

 

[sn00p]

Data is backed up on a daily basis to the backup server, the data is backed up to a hot swap drive, hot swap drives are rotated every week and taken off sites, this means at worse the data is one week old should everything on sites be destroyed.

As it's all based on rsync over ssh, off site could be done just as easily.

However, the failure here was ignoring an email which said that machine x had not been backed up for 230 odd days.

We also have a machine which backs up to time machine, we then use my scripts to backup the time machine image incrementally (using hard links), this means I can pick out how the time machine looked on a specific date. Always handy when time machine corrupts itself.

Still can't believe that the email was just ignored, think I might put some flashing defcon lights in the office!

 

[iain187]

Direct the email to a distribution list and copy in the most annoying mitherer you can think of. Alternatively copy in yourself...

 

[SamP]

I love CDP. Backups taken every 6 hours and weekly snapshots archived offsite and kept for a month. Screw loosing data, it's just not worth it.