April 1, 2015 - Moved in new furniture for my then ~6 month child. You can imagine what might have been stored on my drives. This required that I find new locations for the stuff already in its place. This necessitated that I move a bunch of electronics into temporary locations in the basement until I got things straightened out and figured out where their permanent homes would be. Not a big deal in my mind as I was planning to host a LAN party that weekend and we always setup in the basement. I go ahead and setup computers for the LAN party. I reiterate, every last computer I owned (quite a few at the time) was in the basement.
April 3, 2015 - I get a phone call at ~9:00am from my wife. Long story short, the basement has 12cm of standing water and the breaker tripped. Apparently, between the time I left for work and when she called, we had 10cm if rain drop in a span of only 20 minutes, an electrical event tripped the breaker preventing primary sump pump operations for both wells, and the backup pumps were incapable of handling the volume of incoming water and went offline when the batteries shorted. Putting aside the long drawn out and still ongoing drama with the insurance company, every last computer was damaged by the flood
. Even my gaming laptop was oddly and unfortunately located in it's bag on the ground. A buddy of mine grabbed his trumpet an played Taps over the computer graveyard.
From here I go into full recovery mode. I first had to rebuild a machine to do recovery on. I'll thank a good friend of mine who decided that that was an excellent time to upgrade the HDDs on his NAS to 6TB HGST drives and lent them to me prior to installation for recovery purposes. I started with my custom FreeNAS box that was built into a Thermaltake Xaser III full tower chassis. This older style chassis has the PSU in the top rear instead of the bottom rear. While unfortunate in the sense that there was no way the motherboard wasn't going to get soaked, it was fortunate that the PSU was free and clear meaning less electrical damage to downstream components given there was no capacitor explosions like in the other PSUs. Many of my HDDs were fully submerged, though. I removed the circuit and cleaned them with a non-ionizing solution. I noted that some had parts of the PCB peeling up and solder bulbs that had popped. I was able to reflow the solder on some of these, but the PCB peel was a lost cause.
I started the recovery process using DD with one of the submerged drives. It actually went rather smoothly recovering everything that drive had to offer. Unfortunately, the next drive was not so lucky. I started getting block errors almost immediately, but they were sparse, sporadic, and inconsistent. Changing the block size for DD allowed me to recover more of the data, but went painfully slow. I decided to use the larger block size, output error block locations to calculate new start points, and run at these start points a smaller block size through the error region. Then it was a "simple" matter of merging the data into a single cohesive image
. After some frustration in getting the image merged, I mounted the single drive as a zfs volume and it worked
. That said, while I recovered most of the data on that volume, I didn't get everything. Unfortunately, the mirror drive was a PCB peeler. Then I had the thought, I can replace the PCB with the PCB from the drive I just recovered from. Just swapping the PCB didn't work as I had hoped, but after a little soldering, I was able to get it into a workable state. Predictably, this volume also had block errors, so I repeated the procedure above. When I mounted the new image back into the ZFS mirror configuration it was originally setup in, ZFS began to automagically cross compare and recover the data. So far 100% success.
Then I went for a volume in which neither drive was submerged. One drive wasn't accessible at all and nothing I did changed that. I found discoloration at the power connector and power circuit solder point that indicated a surge event had taken place. Best guess is the surge suppressor (located on the floor) failed at its job once submerged. Who would have thought it. The other drive had more error blocks than the submerged drives by two orders of magnitude. This is when I dropped DD (for that drive) and found DDRescue. Quite a nice recovery method (see documentation), but burnt my bacon that I wasted my time reinventing a small part of that wheel. Nonetheless, between DDRescue and the ZFS file duplication I had active on that volume (lucky) I recovered all but 16.3Kb of data. Two unimportant log files and a thumbnail IIRC. DDRescue was my new best friend for the rest of the NAS recovery effort. Between that and ZFS I lost a grand total of ... 16.3Kb of data. Given that I had the vast majority of my data back up to the NAS, my less successful efforts recovering NTFS and XFS volumes weren't an issue. I did recover a few things though, so I suppose it was worth the effort.
Now for the fun. I took my recovered volumes out of my new file server and put in volumes with the partially recovered data. I then gave my friend back his drives which I promptly put to use. Then I proclaimed victory to my wife and asked her to take a look at network shares to see my good work. She started browsing around and found some good files, but then started finding corrupted files. I told her I must have gotten the wrong copy. I said I'd call my friend while she checked the rest of it out. I came back a minute later and told her that he had already done a secure wipe of the drives. She was hot
. She then immediately went to the baby pictures which were full of artifacts and cutoff pictures. I decided I'd better own up to the deed before she started crying. When I put the real drives back in, everything was cool