Backup, Backup, Backup!

 

As a Zen master once said, "Ahh...to go forward, one must first back-up."

 

Anybody who has worked with technology knows the importance of backing up your data. But there is nothing like the loss of data to remind you just how important it is to not only have a backup, but to have a 2nd copy of the backup at a 2nd location.

 

With out going any further, in case you don't have time to read what happened...just be sure:

1. That you backup right now...anything that you don't want to loose. Storage prices are at an all time low and ever decreasing. Just the other day, I bought an 8 gb thumb drive for $30 and it has backup software built into it. An external 250gb drive is about $60 which is pretty much enough to back up 100,000 photos and then some.

2. Back it up a second time to ANOTHER external drive and take that to another location.

3. Routinely rotate the backups.

 

This isn't my first time to loose data...but I am sure going to try to make sure this is the last time. The irony is that I have followed a pretty good back up plan up until now...it just wasn't good enough.

 

Here is my story that I hope you will all learn from.

 

The incident:

Part 1: Electrical Challenges at the WB. All year long, 2007-08, we had power challenges...with one half of the two legs of 220 volts going out every few days. It was really frustrating as anything with a motor would have be unplugged to prevent it from burning up since there was just enough power going to some circuits to trick equipment into thinking it had electricity...but not enough to run the device. So, refrigerators, AC units, water fountains, etc. had to be unplugged. Lots of equipment failed all over the school during the period of the power outages. Fortunately, I had UPS units on all of the critical computers. The problem is that UPS units only run so long, especially the older ones. So, if the electricity stays off long enough, and the battery dies, then the computer is going to die a hard death. Now, today's UPS units are pretty clever. If you set them up right, they are supposed to power down the system before the battery dies. But, that is IF you set it up right and it works right. The reason why the loss of electricity is so hard on a computer is not just with the power supply...but specifically the hard drives. If the computer dies apparently...there is a significant risk of the drive crashing permanently. Point of this...I knew I needed longer run time UPS as a result of all of the power outages. This was my warning...and I ignored it. "I knew then, yet did nothing" to paraphrase a line from one of my favorite movies.

 

Part 2: Construction Zone and Electrical Work at the WB. While the electrical challenges were finally worked out with one of the massive generators being replaced, little did I know they construction crew erecting the new building would be playing with the electricity while we were gone. Actually, I wasn't really gone for the summer...so I left the computers plugged. I came back after being gone a few days to find that the electricity had been off. I didn't realize the extent of that just yet...but soon came to realize that the server was choking on it's error log...a drive had died!!!! And, two other computers were dead!!

 

Part 3: I get careless with one of the more current backups. Some of the more important data was backed up to a 4gb SanDisk Cruzer...that SOMEHOW...I lost. I had been going back and forth to school to work on the data recover efforts of the server drive. I thought I left the drive on my desk...and left for the weekend. When I came back on Tuesday, the drive was gone. Unfortunately, if it fell out of my pocket, then it could be anywhere from the beach where I went on Thursday to any number of places I visited on Friday.

 

What was lost: Pretty much EVERYTHING!!! When I say everything, I may be exaggerating a little. Very few of my pictures were lost, this time. But that is because the drive that failed wasn't my photo drive. The server had a total of 5 drives (well, actually, 6 if you count the two that make one virtual drive array)...an internal 40gb boot drive, an internal 160gb data drive for the network user shares (this is the one that is two drives to make one large drive), an external 80gb drive for the clipart, an external 250gb drive for photo storage, and an external 750gb drive for video storage (I teach video production). Only one drive failed on that one computer, but the amount of data loss is staggering. My last full back up of my primary teaching folder (tests, worksheets, PowerPoint presentations, etc) was 2.5 years ago. WHAT??? 2.5 years ago???? Well, I wasn't actually THAT bad in my backup...backup data existed on a completely different computer and a thumb drive. In fact, the stuff I was currently working on was backed up at the end of the school year, when I backed up other items, to the thumb drive.

 

In addition to loosing my own personally created documents (and modifications of documents older than 2.5 years), I lost an incredible amount of data and resources that was generated by students spanning back over 5 years. I have been in the process of converting all of the classroom examples, magazine cutouts, samples from students, everything, to a digital format. I had collected the best work from the students, and even had students do voluntary and involuntary (those serving detention) scanning and taking photos of samples, equipment, etc. Many of the samples that were scanned were tossed in the trash can after they were scanned. Fortunately, I am a pack rat...so, I actually do have a lot of the hard copies when saving the hard copy didn't really take much space (good news for those serving detention time this year). I don't know why I didn't have this particular folder backed up...but I didn't. Some of the data exists in PowerPoint presentations...other data exists on the internet from a webshare that was not lost.

 

Attempts at Recovery: Pretty much all in vain. Due to the way the data was stored on the server, the failed drive prevented any recovery from the good drive. Specifically, for those who want to know, the server was set to Raid 0, with two drives forming one virtual drive of 160gb. This is good for data throughput as the drives controller board is able to split traffic to the two drives simultaneously. But this is HORRIBLE for data security with ZERO redundancy and protection from loss of data. I did try some software that specializes in the recovery of data from a failed disk array...but it works mainly if the drive physically works. Since one of the drives physically does not work, recovering BOTH strips of data has proved to be impossible. The only thing I really was able to recover was the hierarchy of folders and actual file names of the majority of the drive. The recovery software LOOKED like it was going to recover almost everything. But, after spending some 6 hours recovering what I thought were intact files, turned out to be corrupted...and striped files...such that on the pictures that would actually open...they looked like a zebra with large bands of missing data running through the picture. But, thanks to the software, right away, I was able to recover the organization of things and have already been able to start replacing the some of the more important documents.

        

 

Update: 3/11/2009

A very fortunate turn of events. The Grandfather of a current student works in data recover in Houston. I sent him the drive in October on a whim that he might be able to recover some of the data. He was optimistic, but I wasn't. Well, he was successful in recovering about 80%. I can't express enough gratitude for the countless human hours recovered. Keep in mind, I didn't just lose my work of 2.5 years of new and modified files, but also students work, AND students EXTRA work for me in the afternoon scanning many files that were then tossed in the trash can. These files represented many years of student examples and archives of project examples which serve an invaluable resource in the study of work.

 

Unfortunately, data loss happens. Just today, 3/17/09, I have managed to get a nightly backup working critical data. The only problem with my system even still is that the nightly back up and the source drive (the data being backed up) are in the same room. The following screen shot is from a MAJOR website hosting service provider that is disclosing that people backup may actually have been lost.

 

 

And as Ed Ladd of D-Cubed Media Conversions, linked over to me: http://www.informationweek.com/news/storage/systems/showArticle.jhtml?articleID=212902034&cid=ref-true and said...use Seagate at your own peril, even if you think you have a good plan, plan that you primary and your back up drive fail.