TR Forums

I had a Crucial MX100 bitrot at me a while back, and I have no desire to repeat the experience. Btrfs-raid1 sounds like just the ticket to complement my backup scheme (what with checksumming allowing it to correct errors, not merely detect), but it also sounds kinda overkill for my needs, so I thought I'd ask the gerbils what you all would do.

I should probably note that I'm not planning on buying new hardware immediately. If I decide on a plan that involves buying stuff, it'll probably happen sometime in the next few months.

The current breakdown of my data is ~6.5 GB for the OS and programs (Arch), ~7.5 GB for the data that I actually care about, and ~150 GB for games and other unimportant data, all on a 512 GB Sandisk X300s (not my choice of drive, long story ). The OS/programs shouldn't grow too much, I could see the data I actually care about getting as big as ~50 GB in the not-too-distant future, and games expand indefinitely but I'm fine with running cleanup on them aggressively if I'm low on space. Speed isn't a huge deal so long as it's not BX200 levels of bad, though I wouldn't object to some NVMe goodness if I could find an excuse for it. It's secondary to reliability, though. NVMe and reliability aren't fundamentally opposed, because I've got an mITX board with the M.2 slot on the back and the case gives decent ventilation to it.

The plan that feels overkill is an NVMe ext4 OS drive (256 GB?) and btrfs-raid1 2.5" data drives (128 GB?) mounted directly at my core data folder. It would feel weird having a bigger OS drive than my data drives, but that plan would get NVMe speed to most stuff, integrity where it's needed, and wouldn't involve getting too spendy on duplicated space. Going to 512 GB for the OS drive would be nice, but really all 256 GB means is deleting games I haven't played in a while a bit more aggressively, which I don't mind much.

An obvious modification of this is to skip the OS drive and just run the whole system from a btrfs-raid1 pair, but that's slower, it looks like a dragon or two might be hiding around that boot process, and because of the extra duplicated space it's unlikely to be cheaper (unless I just get another X300s, but if I'm going to be messing with storage I'd rather get MLC flash).

Theoretically, it should be possible to run a btrfs-raid1-like anti-bitrot scheme by duplicating data across two partitions on one drive. It looks like btrfs might even be able to do this. This sounds hacky, though. It's obviously not as good for integrity as separate drives, and it sounds like it might be slow, but I'm still trying to figure out just how good or bad it might be.

Scaling back further would lead to some variety of snapshot system. This would help, but it seems like there are a lot of errors it would have a tough time catching (especially for frequently changing files). At some point it's not much different from normal backups, but without the external media.

Speaking of that, theoretically backups could have some auto-diff functionality modulo modification times. This is probably even standard in some places, it's just not a thing I've looked into. I like being able to handle my backups with the most basic of tools, and I wouldn't consider such a thing a replacement for the kind of in-place anti-bitrot system I'm asking about, but if there's suitably universal tooling that does this I'd be interested.

Thoughts?

synthel2 wrote:

Thoughts?

Ultimately, the question is what are you trying to protect? Because if it is just that 7.5 GB of data, and that data doesn't change frequently, you might want to just tar that up, md5sum it, and then I don't know, put it on a bunch of flash drives/ old harddrives, and leave them in different places: work, family, friends. I'd cloud storage it somehow too, 7.5GB will easily fit into a free google drive and plenty of other places too I'd imagine.

Like you said, pretty much everything else sounds like overkill if it is just that data you are concerned about. Even spending a lot of money on harddrives using any FS scheme won't match the protection of what I described above, because what if the computer gets burnt/flooded/stolen/zapped/knocked-over-while-running-by-pet?

I use crashplan, on an ECC computer with a 6 volume ZFS raidz2 plus two different RAID1 btrfs filesystems as well, and even then, the really important stuff is *STILL* dispersed in the fashion I described above.

My next home server build will be using btrfs or ZFS...

JBI wrote:

My next home server build will be using btrfs or ZFS...

As a forewarning, btrfs RAID5/6 support isn't "experimental", it's broken. !Do Not Use!, there is a bug ("fatally flawed" with the very ominous implication that someone should, you know, do "a full scrap and rewrite") as it intermittently fails ("in average every three tests I got 1 bug..." ) to recompute parity correctly on scrubs/rebuilds.

https://www.mail-archive.com/linux-btrf ... 55161.html

I use it for RAID1, and have already run into a serious bug: One of the drives starting failing (ancient 7200.10s), and then it died completely. As in, it was no longer detected by anything, it was just gone.

This is not handled well by btrfs (or the marvell controller, but I only have suspicions and it's irrelevant to this discussion). For one thing, it kept logging errors like crazy instead of recognizing that the device was kaput. For another, the actual bug, you can get into a catch-22 where you can only mount the remaining disk as degraded,ro but then cannot add a new replacement disk because the array is read-only.

http://www.spinics.net/lists/linux-btrfs/msg53437.html

I didn't lose any data, and maybe if you had a hot-swap setup you could get around this, but the mailing list is again ominous ("We need this issue be fixed for the real production usage." ).

You effectively lose the ability to re-use the array, you have to get the data off and recreate it. In my case, well, I didn't have any use for the remaining healthy 7200.10 disk anyway, so I did what I described to Synthel2, which is put a new btrfs single volume on it and write a bunch of backup data on it and then put it on a shelf at work. I've got a bunch of disks that are either obsolete or questionable, and that's my standard practice.

Then I bought new two new harddrives and remade the btrfs array. I'm a glutton for punishment I guess.

Glorious wrote:

Ultimately, the question is what are you trying to protect? Because if it is just that 7.5 GB of data, and that data doesn't change frequently, you might want to just tar that up, md5sum it, and then I don't know, put it on a bunch of flash drives/ old harddrives, and leave them in different places: work, family, friends. I'd cloud storage it somehow too, 7.5GB will easily fit into a free google drive and plenty of other places too I'd imagine.

For now it is just the 7.5 GB, and spamming backups everywhere they'll fit is pretty much what I do already. The main weakness is that that data has a decent rate of change, so while the checksums let me ensure that a backup I'm restoring is the same as the backup I saved, they don't protect against some data getting corrupted while on my computer and then being backed up that way. If that happens, then whenever I discover the problem (which may not be for quite a while), I just have to go fishing through old backups until I (hopefully) find an unaffected copy, which is an annoying and insufficiently foolproof scheme. Also, if my core data expands to 40 or 50 GB, it'll be tougher to keep lots of old versions around.

Glorious wrote:

I use it for RAID1, and have already run into a serious bug: One of the drives starting failing (ancient 7200.10s), and then it died completely. As in, it was no longer detected by anything, it was just gone.

This is not handled well by btrfs (or the marvell controller, but I only have suspicions and it's irrelevant to this discussion). For one thing, it kept logging errors like crazy instead of recognizing that the device was kaput. For another, the actual bug, you can get into a catch-22 where you can only mount the remaining disk as degraded,ro but then cannot add a new replacement disk because the array is read-only.

The btrfs replace instructions I see here look like they avoid that problem, and even if they don't, having to juggle some data in the event of a proper drive failure is something I'm okay with. I'm much more interested in getting rid of potential for silent failure than I am in maximizing uptime.

Can't you compile in whatever you want pretty easily with Arch? If so, no reason to pick btrfs over zfs, the latter is far far more mature.
Though even the more prebuilt-oriented distros are making it less and less painful.

If you're really worried about the data, having it on multiple computers in different locations is the only real backup though.

I chose to use btrfs on my current system, and came to regret it over using ext4 or xfs.

Here's one heck of a big reason to stay off btrfs when it comes to raid5 or 6: http://www.phoronix.com/scan.php?page=news_item&px=Btrfs-RAID-56-Is-Bad

synthtel2 wrote:

I had a Crucial MX100 bitrot at me a while back, and I have no desire to repeat the experience. Btrfs-raid1 sounds like just the ticket to complement my backup scheme (what with checksumming allowing it to correct errors, not merely detect), but it also sounds kinda overkill for my needs, so I thought I'd ask the gerbils what you all would do.

Thoughts?

RAID isn't backup - it's real-time redundancy, and for consumers it's most useful for your OS drive to improve your odds at "up-time".  Just get a good (good rep. MLC or SLC) couple of small SSD's for your OS and put them in RAID (..and I"m assuming Linux here so probably ZFS).  If I remember correctly, Samsung's 850 Pro has cell-size similar to SLC - and it has a very good rep.. so 2 x 128 gb should be more than sufficient:

http://www.accessories2016.com/index.ph ... rrency=USD


For the program/data side I personally went with Snapraid and 5 of these (two parity disks and three data, though you'd probably only need 2 data):
https://www.amazon.com/dp/B00DCM8ZB4/?tag=pcpapi-20

synthel2 wrote:

The btrfs replace instructions I see here look like they avoid that problem

They don't.

You can't delete a device that no longer exists, you can't replace a disk on read-only filesystem, and you can't mount it as anything other than -o degraded,ro

The only way to avoid it is to ensure you do not touch the volume the first time it's mounted as degraded. And I'm being very literal in regards to 'touch'.

Because, once you do, it will never let you mount it as degraded,rw ever again. This is extremely counter-intuitive behavior, and it's something the maintainers acknowledge as a bug.

The wiki doesn't deal with this at all, but it's all there in the mailing list I linked to.

synthel2 wrote:

and even if they don't, having to juggle some data in the event of a proper drive failure is something I'm okay with.

Right, I'm just making sure everyone knows. The issues with btrfs aren't hypothetical or far-fetched, they've happened to me, on a very recent kernel.

That said, I'm still using it, but when people say "oh yeah btrfs, don't worry about that experimental bit, etc..." take it with a grain of salt.

boing wrote:

Here's one heck of a big reason to stay off btrfs when it comes to raid5 or 6:

Yes, I discussed that and posted the mailing list discussion to which Phoronix is undoubtedly referring...

Bauxite wrote:

Can't you compile in whatever you want pretty easily with Arch? If so, no reason to pick btrfs over zfs, the latter is far far more mature.
Though even the more prebuilt-oriented distros are making it less and less painful.

I would consider ZFS, but it is a bit more trouble to set up, and the feature I'm looking for from btrfs is a thing that it provides just fine.

Bauxite wrote:

If you're really worried about the data, having it on multiple computers in different locations is the only real backup though.

CScottG wrote:

RAID isn't backup - it's real-time redundancy, and for consumers it's most useful for your OS drive to improve your odds at "up-time". Just get a good (good rep. MLC or SLC) couple of small SSD's for your OS and put them in RAID (..and I"m assuming Linux here so probably ZFS). [...]

I do have good backups, and no I am not confusing this with backups. I also am not any more concerned with uptime than the average gerbil. There is a pretty specific issue I am trying to solve here.

The scenario I'm trying to prevent is one where a bit here or there flips, I don't notice immediately, I go on with my backup scheme as normal, and then years later I notice that there's a problem and it's been propagated into all my backups.

Normal RAID helps with this, because it at least lets me know when there's a problem, but if I'm going to bother with hardware for that I might as well set it up to be self-healing at the FS level.

If anyone has a good solution for this that doesn't involve buying more hardware, substantially slowing down disk access, or excessive manual effort, I'm all ears.

http://www.snapraid.it/compare

You can also do what I do as far as the OS is concerned - run from system memory "live", with ecc memory.

Ah, sorry, I missed the snapRAID ref. I think I still prefer btrfs, but that does look cool.

I plan to be doing a mobo/CPU/RAM upgrade early next year and the top priority in doing so is to get ECC. Once I do that, I'll probably abuse ramdisks quite a bit.

You're describing the perfect use case for ZFS. Bitrot / silent errors are exactly what it was built to detect (and fix).

You can do it on a single drive (copies=2) as well, which kills half your space, but you can do it with one drive. Scrub monthly (which doubles as a refresh of the data if the drive controller is stupid and doesn't do it on its own) to correct latent errors and ensure data doesn't rot without you knowing it. ZFS mirrors are stupid easy as well, and you'll gain the benefits of compression in the mix as well (if your data is compressible).

Yes, I have a hard on for ZFS. I have petabytes of it deployed and I've learned to very much trust it (both the true ZFS tree and ZFS on Linux).

Aha, copies=2 looks like just the ticket! That looked like it would be more complex on btrfs.

My big concern with ZFS is that it looks like it involves either running a downgraded kernel or doing a lot more poking around with the kernel than I otherwise have reason to. I don't especially want to have extra kernel maintenance to do, and I don't like the idea of a downgraded kernel because Arch's rolling-release deal expects all packages to be at latest as of the most recent system update. I don't know exactly what kind of breakage can happen if that isn't followed, and I don't really want to find out.

synthtel2 wrote:

Aha, copies=2 looks like just the ticket! That looked like it would be more complex on btrfs.

My big concern with ZFS is that it looks like it involves either running a downgraded kernel or doing a lot more poking around with the kernel than I otherwise have reason to. I don't especially want to have extra kernel maintenance to do, and I don't like the idea of a downgraded kernel because Arch's rolling-release deal expects all packages to be at latest as of the most recent system update. I don't know exactly what kind of breakage can happen if that isn't followed, and I don't really want to find out.

What kernel are you running that it's so bleeding edge you can't build ZoL? 4.6 is supported as of today in the most recent stable release...

EDIT: You can run the FUSE version of ZoL if you're on a weird kernel that isn't supported yet. It's not *fast*, but it's reasonably quick.

I don't know, I was mainly looking at this. I don't do kernel hackery and have no desire to have anything interesting about my system in that area.

Even if the current state of things is pretty good, I'm uncomfortable with relying on a non-Arch source for anything kernel-level every time I do an update. It's probably not a big deal, but it isn't a part of my system I have a particularly good understanding of, and if something were to go wrong I'd have a much tougher time fixing it than other parts of the system. I would just go learn about it until I feel more confident, but my list of things like that to learn about is full of more important things right now.

FUSE ZFS would bypass all that. I like that idea a lot, thanks!

Waco wrote:

I have petabytes of it deployed and I've learned to very much trust it (both the true ZFS tree and ZFS on Linux).

My experience of using both leads me to recommend the same. ZFS hasn't given me issues even when I initially did things I knew were stupid, like building a pool by device (/dev/sd*) instead of disk (/dev/disk/by-id).

Pro-tip, don't do that, go with by-id.

But that's the beauty of ZFS, because when I did something that I thought would cause me significant issues, I managed to work that out with minimal fuss. Whereas, with btrfs, I did something I never expected to cause any problems that literally made a very simple setup unusable in apparent perpetuity with no solution. :/

So, long story short, I echo your statement. With Ubuntu it's incredibly easy too, as they (legal mutterings aside) included it natively in 16.04.

synthel2 wrote:

FUSE ZFS would bypass all that. I like that idea a lot, thanks!

Yeah I think Waco's got you covered here, and the performance hit just really isn't going to affect you much from what your use-case sounds like.

Damn, you guys have got me thinking about a new home server build (and ZFS) even though the existing one is completely adequate for my needs.

I was looking at LSI SAS/SATA cards yesterday, and thinking, "damn, I should've just bought one of these instead of futzing around with those mediocre onboard ports and random aftermarket PCIe cards when I set up that server". I even had it (and the required SAS/SATA breakout cables) in my Amazon cart, and backed out at the last minute on the checkout page.

Now I want to go back and buy that card, put Ubuntu Server 16.04 + ZFS on the hardware from my just-retired FX-8320 desktop, and start migrating all over again.

How hard is it to grow existing ZFS pools by adding disks, and what are the downsides to building up an array gradually? I'd prefer not to have to buy a bunch of new disk drives. The existing server has multiple MD RAID-1 arrays in it. Could I create a ZFS pool just large enough to contain one of the existing arrays (e.g. using the existing storage drives from the retired FX-8320 box...), move one array's worth of content, then take the drives for that RAID array out of the existing server and add them to the ZFS server prior to migrating the next array? That way I could migrate everything one RAID array at a time, and re-use all of the existing drives.

Glorious wrote:

ZFS hasn't given me issues even when I initially did things I knew were stupid, like building a pool by device (/dev/sd*) instead of disk (/dev/disk/by-id).

Pro-tip, don't do that, go with by-id.

It's very easy to fix that - just import with "-d /dev/disk/by-id". I tend to use the short names to build pools, then export / import with the by-id labels. It'll forever use the by-id labels until you tell it not to.

just brew it! wrote:

How hard is it to grow existing ZFS pools by adding disks, and what are the downsides to building up an array gradually? I'd prefer not to have to buy a bunch of new disk drives. The existing server has multiple MD RAID-1 arrays in it. Could I create a ZFS pool just large enough to contain one of the existing arrays (e.g. using the existing storage drives from the retired FX-8320 box...), move one array's worth of content, then take the drives for that RAID array out of the existing server and add them to the ZFS server prior to migrating the next array? That way I could migrate everything one RAID array at a time, and re-use all of the existing drives.

Growing arrays in ZFS isn't trivial, or straightforward.

You have to add VDEVs at a time - entire virtual devices (think RAID arrays). So if you're doing mirrors, it's not bad, but balancing can be a problem. If you're building RAIDZ1/2/3s, you have to grow in entire array increments. E.g., if you have 10 drives, your only options for growing are to split it into 5 mirrors, or some set of RAIDZ arrays (two 4+1, 3+2, etc).

You can grow arrays without changing VDEVs by swapping out smaller disks for larger disks one-by-one, but in general, growing zpools is hard to do without spare hardware and/or all the drives up front.

Because I didn't feel like doing it the hard way in terms of swapping things in an out all the time, I went with a bunch of mirrors for my server. It let me grow the array in 2 TB chunks, then I just rebalanced it by hand once all the data was placed and all the disks were in the array.


EDIT: By the way, the cheap LSI controllers are Ebay are wonderful for around $60. I have three of them, and aside from the initial pain of putting them in IT mode and updating firmware, they're rock solid. Ensure you have at least some airflow over them though, they run hot. I cheated and replaced the heatsinks with nice tall ones from Digikey for a few dollars each.

Hmm... OK, maybe I'll put the ZFS project off for a bit then.

I will also make sure I position the LSI controller near the side panel vent and install an intake fan there, if I decide to get one.

One thing to keep in mind is that a pool is only as resilient as the weakest link. If you have multiple VDEVs, one failing *will* take out the entire array. I've seen people build a nice RAIDZ3 with a handful of drives, then make the whole thing scary by trying to add a single drive to the pool. That's effectively a RAIDZ3 as a "drive" in a RAID 0 with another single drive. That last drive failing will take all your data with it.

The best tactic I can suggest is to read a lot before you go into building anything.

Waco wrote:

EDIT: By the way, the cheap LSI controllers are Ebay are wonderful for around $60. I have three of them, and aside from the initial pain of putting them in IT mode and updating firmware, they're rock solid. Ensure you have at least some airflow over them though, they run hot. I cheated and replaced the heatsinks with nice tall ones from Digikey for a few dollars each.

I'd have to recommend the same at this point, as the marvell controller I got is implicated in a system hang. It happened in the midst of that drive failing; btrfs was logging errors like nuts, smartd was going crazy, and I was doing a batch thing in the middle of the night that involved moving a lot of files from the failing array to another array...

...on the same controller.

In the middle of that, my system froze. And it turns out there is a long standing issue with that marvell chip in which high load + SMART traffic = freak out and drop all disks.

Not system hangs, but hey, where's there is smoke there is likely fire.

If I did it again, I'd go the ebay + LSI route. I'm still using the marvell controller though, with two fingers crossed.

EDIT:

Waco wrote:

It's very easy to fix that - just import with "-d /dev/disk/by-id". I tend to use the short names to build pools, then export / import with the by-id labels. It'll forever use the by-id labels until you tell it not to.

Yeah, that part is painless. What isn't is when you're still importing by dev and building the server, adding drives etc.., and accidentally unplug one of the disks in the ZFS array without realizing it at first, causing a confusing and conflicting re-enumeration.

I wasn't in any serious trouble because I was still messing around and nothing on there was serious yet, but I despite thinking I might be in some trouble it was painless and I didn't lose or damage the filesystem at all.