[hate5six-labs] Boolean algebra & data recovery

Last night two hard drives in my storage system failed but I didn’t lose any data. I got some messages from people asking how data recovery works at a bit level within a RAID system. hate5six runs* on a RAID-Z3 which can tolerate 3 drive failures. Here’s a quick lesson on how it works. Shout to my high school computer science teacher Mr Callinan for teaching me Boolean algebra back in 2000.

The two dying drives had uncorrectable sectors so I took them out and popped two empty 4TB drives in. The rebuild (“resilver”) process has taken over 20 hours, but it’s finally done and the pool is healthy again. The resilvering is more or less the process of solving a more complex series of algebraic equations as described in the video.

* For storing my source files & editing only. I have two FreeNAS RAID-Z3 pools: one containing everything I filmed from 2008-2018, and a second pool for everything I’ve filmed since then. The first pool is running 14 4TB drives, the second has 12 4TB drives, totaling 70TB. I’m probably gonna add another 30TB soon. I installed a 10 Gigabit ethernet LAN so that I can edit directly off the storage array and not have to worry about keeping files on my desktop/laptop.

RAID is NOT a backup. It only provides you redundancy in the case of drive failures. For backup I have a second RAID-6 array running on a Synology Diskstation 1812+ that I normally keep offsite in another location. When it’s running I have a cronjob that uses rsync to mirror data from the FreeNAS box onto the Diskstation, aka a carbon copy/true backup.

All of this ends up being more cost effective than hosting on the cloud on top of the peace of mind of having custody over my data….on top of being able to quickly retrieve and edit files over the 10gbe LAN.