Personal computing discussed

Moderators: morphine, Steel

 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Fun with Linux software RAID

Wed Jul 18, 2018 10:31 am

I'm in the process of moving from a home file server with multiple RAID-1 volumes to one with a single large RAID-6 volume. Both are running Ubuntu Linux (14.04 LTS for the old one, 16.04 LTS for the new), with Linux software RAID. A few observations:

1. Ubuntu 18.04 LTS RAID seems to have issues (drives intermittently dropping out of arrays after reboots). After spending most of an evening trying to figure out why this was happening (with little progress), I backed off to 16.04 LTS, which does not seem to have this issue.

2. In retrospect, I could've probably saved myself a lot of time by pulling one drive of each of the old RAID-1 arrays (allowing the arrays to continue in degraded mode), moving the drives to the new server, and copying the data off. This would've avoided the need to copy all of the data over the network, and would've sped things up considerably.

3. Software RAID-6 performance is pretty impressive for streaming reads. Nearly 800 MB/sec on a 5-stripe array using 5xxx RPM "NAS" type disks.

4. Praise the Lord for the --re-add option. Saves your butt if you accidentally pull the wrong drive tray (assuming you notice immediately, and the array isn't being written to at the time). Without the ability to (in effect) say "oops, I really didn't mean to yank that drive", I would've needed to do let it do a full array rebuild.

5. Now that most of the data has been successfully transfered, a couple of the drives from the old server are being re-purposed as members of the new server's RAID-6 array to expand its size. For the drives being repurposed, I made the mistake of not wiping the first few MB (partition table and RAID superblock) before inserting them into the new server. RAID subsystem on the new server immediately attempted to assemble a pair of degraded RAID-1 arrays when it saw the drives. Then I compounded the mistake by attempting to wipe the partition tables and RAID superblocks while the RAID subsystem apparently still thought it was managing those drives (not sure why it was doing that, I had issued STOP commands on them). This sent the drives into some sort of stupid limbo state where I couldn't get the RAID subsystem to let go of them, and couldn't get it to add them to the RAID-6 array either because it thought they were still "busy". The kernel also seemed to be pissed off that I had wiped the partition table out from under it. After poking at this a bit I finally gave up and rebooted the box; everything returned to normal.

6. Reshaping a RAID-6 array to add more drives and expand the capacity is really slow; estimated completion time is over a day to add a pair of 3 TB drives to an existing 9 TB array. This is not really surprising since all of the data on the disks must be reshuffled. The array is still "usable" during the reshape, but performance is absolutely abysmal (hence the scare quotes). Yes there are ways to throttle the reshape to leave more performance available, but then you're potentially looking at multiple days for the operation to complete; I'll just let it run full speed, since I'm not planning on using it much anyway over the next couple of days.
Nostalgia isn't what it used to be.
 
SuperSpy
Gold subscriber
Minister of Gerbil Affairs
Posts: 2332
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Fun with Linux software RAID

Wed Jul 18, 2018 10:48 am

just brew it! wrote:
I'm in the process of moving from a home file server with multiple RAID-1 volumes to one with a single large RAID-6 volume. Both are running Ubuntu Linux (14.04 LTS for the old one, 16.04 LTS for the new), with Linux software RAID. A few observations:
5. Now that most of the data has been successfully transfered, a couple of the drives from the old server are being re-purposed as members of the new server's RAID-6 array to expand its size. For the drives being repurposed, I made the mistake of not wiping the first few MB (partition table and RAID superblock) before inserting them into the new server. RAID subsystem on the new server immediately attempted to assemble a pair of degraded RAID-1 arrays when it saw the drives. Then I compounded the mistake by attempting to wipe the partition tables and RAID superblocks while the RAID subsystem apparently still thought it was managing those drives (not sure why it was doing that, I had issued STOP commands on them). This sent the drives into some sort of stupid limbo state where I couldn't get the RAID subsystem to let go of them, and couldn't get it to add them to the RAID-6 array either because it thought they were still "busy". The kernel also seemed to be pissed off that I had wiped the partition table out from under it. After poking at this a bit I finally gave up and rebooted the box; everything returned to normal.


I have been here so many times it's not funny. I hate this "feature". The worst part with Debian based distros is you can't seem to escape it when inside the installer, as the first thing it does is scan for any md-marked partitions and attempt to 'help' you by making them into mdX-devices. I basically resort to starting with a bare gParted live CD to dd away the first few MB of the disk (I was never sure of where the superblocks lived, so I overcompensated), then rebooting into the Debian installer.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
chuckula
Gold subscriber
Gerbil Jedi
Posts: 1867
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Fun with Linux software RAID

Wed Jul 18, 2018 11:05 am

SuperSpy wrote:
just brew it! wrote:
I'm in the process of moving from a home file server with multiple RAID-1 volumes to one with a single large RAID-6 volume. Both are running Ubuntu Linux (14.04 LTS for the old one, 16.04 LTS for the new), with Linux software RAID. A few observations:
5. Now that most of the data has been successfully transfered, a couple of the drives from the old server are being re-purposed as members of the new server's RAID-6 array to expand its size. For the drives being repurposed, I made the mistake of not wiping the first few MB (partition table and RAID superblock) before inserting them into the new server. RAID subsystem on the new server immediately attempted to assemble a pair of degraded RAID-1 arrays when it saw the drives. Then I compounded the mistake by attempting to wipe the partition tables and RAID superblocks while the RAID subsystem apparently still thought it was managing those drives (not sure why it was doing that, I had issued STOP commands on them). This sent the drives into some sort of stupid limbo state where I couldn't get the RAID subsystem to let go of them, and couldn't get it to add them to the RAID-6 array either because it thought they were still "busy". The kernel also seemed to be pissed off that I had wiped the partition table out from under it. After poking at this a bit I finally gave up and rebooted the box; everything returned to normal.


I have been here so many times it's not funny. I hate this "feature". The worst part with Debian based distros is you can't seem to escape it when inside the installer, as the first thing it does is scan for any md-marked partitions and attempt to 'help' you by making them into mdX-devices. I basically resort to starting with a bare gParted live CD to dd away the first few MB of the disk (I was never sure of where the superblocks lived, so I overcompensated), then rebooting into the Debian installer.


That's kind of frightening actually. I don't mind auto-detection of a USB stick but I don't want auto-merge in a RAID array (especially when it goes wonky).

Is there a service you can turn off to avoid that issue? I used a simple mdadm setup for a RAID1 system and I remember that you had to go through the manual setup process to add drives. I hope that throwing another random drive into the system wouldn't make it automatically try to change the array (hope).
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-1080 sold and back to hipster IGP!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
Waco
Gold subscriber
Minister of Gerbil Affairs
Posts: 2494
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Fun with Linux software RAID

Wed Jul 18, 2018 11:37 am

I hate mdraid with a passion (mostly because of issue #5 you noted, rebooting is simply NOT an option sometimes). If you're going software RAID and you don't need a particular FS over it...use ZFS.
Z170A Gaming Pro Carbon | 6700K @ 4.4 | 16 GB | GTX Titan Xm | Seasonic Gold 850 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | Cosmos II | Samsung 4K 40" | 2048 + 240 + LSI 9207-8i (128x8) SSDs
 
titan
Grand Gerbil Poohbah
Posts: 3371
Joined: Mon Feb 18, 2002 7:00 pm
Location: Great Smoky Mountains
Contact:

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:23 pm

I was going to ask how you achieved the RAID (dmraid, or something else).

On my server I user Btrfs RAID1. Some day, when I have money, I'll get 6 more disks to stuff in the case.

Or at least one more.
The best things in life are free.
http://www.gentoo.org
Guy 1: Surely, you will fold with me.
Guy 2: Alright, but don't call me Shirley.
 
SuperSpy
Gold subscriber
Minister of Gerbil Affairs
Posts: 2332
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:42 pm

chuckula wrote:
That's kind of frightening actually. I don't mind auto-detection of a USB stick but I don't want auto-merge in a RAID array (especially when it goes wonky).

Is there a service you can turn off to avoid that issue? I used a simple mdadm setup for a RAID1 system and I remember that you had to go through the manual setup process to add drives. I hope that throwing another random drive into the system wouldn't make it automatically try to change the array (hope).


Well, it doesn't just randomly glue drives together, but if say, you wanted to do a reinstall of a machine with a md RAID array, or you were re-using drives that used to be part of an array, it would reassemble them and create a mdX device for them even in your intention was to destroy the array in the installer. So you end up in this catch-22 where you basically have to use an outside tool to destroy the md magic data in order to use them in a different RAID/partition layout.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:44 pm

chuckula wrote:
That's kind of frightening actually. I don't mind auto-detection of a USB stick but I don't want auto-merge in a RAID array (especially when it goes wonky).

Is there a service you can turn off to avoid that issue? I used a simple mdadm setup for a RAID1 system and I remember that you had to go through the manual setup process to add drives. I hope that throwing another random drive into the system wouldn't make it automatically try to change the array (hope).

Just to be clear, it doesn't affect existing arrays. It's overly aggressive at going "hey, it looks like you've inserted part of another array here, let me try to assemble that for you". Kind of like MS Office Clippy for RAID. :lol:

(And then it isn't always obvious how to reverse what it just did.)

Waco wrote:
I hate mdraid with a passion (mostly because of issue #5 you noted, rebooting is simply NOT an option sometimes). If you're going software RAID and you don't need a particular FS over it...use ZFS.

titan wrote:
I was going to ask how you achieved the RAID (dmraid, or something else).

On my server I user Btrfs RAID1. Some day, when I have money, I'll get 6 more disks to stuff in the case.

Or at least one more.

I thought about switching to ZFS, but mdraid is the "devil I know" at this point. I get the impression there's a fairly steep learning curve for ZFS. Maybe for the next upgrade...
Nostalgia isn't what it used to be.
 
Waco
Gold subscriber
Minister of Gerbil Affairs
Posts: 2494
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:48 pm

The learning curve isn't anywhere near as high as many make it out to be IMO. Build array (after reading or asking questions). Ignore until failure occurs or you want to rebuild. :P
Z170A Gaming Pro Carbon | 6700K @ 4.4 | 16 GB | GTX Titan Xm | Seasonic Gold 850 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | Cosmos II | Samsung 4K 40" | 2048 + 240 + LSI 9207-8i (128x8) SSDs
 
Waco
Gold subscriber
Minister of Gerbil Affairs
Posts: 2494
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:48 pm

Sigh, double post. Posting from my phone since construction crews decided to chop the neighborhood cable feed...
Last edited by Waco on Wed Jul 18, 2018 12:57 pm, edited 1 time in total.
Z170A Gaming Pro Carbon | 6700K @ 4.4 | 16 GB | GTX Titan Xm | Seasonic Gold 850 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | Cosmos II | Samsung 4K 40" | 2048 + 240 + LSI 9207-8i (128x8) SSDs
 
chuckula
Gold subscriber
Gerbil Jedi
Posts: 1867
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Fun with Linux software RAID

Wed Jul 18, 2018 12:49 pm

Waco wrote:
The learning curve isn't anywhere near as high as many make it out to be IMO. Build array (after reading or asking questions). Ignore until failure occurs or you want to rebuild. :P


I see you have enabled RAID1 support on your posts too!
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-1080 sold and back to hipster IGP!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
SuperSpy
Gold subscriber
Minister of Gerbil Affairs
Posts: 2332
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Fun with Linux software RAID

Wed Jul 18, 2018 1:25 pm

Now we just need a phpBB plugin to automatically combine posts that start with the same 4 characters into one post since they clearly belong together. :roll:
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
DragonDaddyBear
Gerbil Elite
Posts: 772
Joined: Fri Jan 30, 2009 8:01 am

Re: Fun with Linux software RAID

Wed Jul 18, 2018 2:11 pm

Waco wrote:
The learning curve isn't anywhere near as high as many make it out to be IMO. Build array (after reading or asking questions). Ignore until failure occurs or you want to rebuild. :P


I'm no expert but I've used both LVM+MD+EXT4 and ZFS and when you start doing volumes with a RAID ZFS becomes a lot less complicated IMO. It's blasphemy to some Linux people to mix an FS, Volume Manager, and disk manager but I'm OK with it. You also get the benefit of checksums at the block level. It's not as fast but I imagine it's good enough for JBI's use case.
 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Fun with Linux software RAID

Thu Jul 19, 2018 12:08 pm

Yeah, I don't really need fast since my network is still plain ol' Gb Ethernet.

The array reshape and file system resize to add the 2 repurposed disks from the old server went without incident. 14 TB ought to hold me for a while!

We also had a brief power outage this morning. The big array came back up without incident in spite of the unclean shutdown. Need to get a new battery for the UPS so I can put this thing on reliable power, and set up a script to do a clean shutdown when the power fails.

Just want to tidy up some of the cable routing, and then I will move it from my workbench to its permanent location in the data center corner of the crawlspace.

Will probably shove an additional drive in as a hot spare at some point (there are 2 unused bays).
Nostalgia isn't what it used to be.
 
captaintrav
Gerbil First Class
Posts: 162
Joined: Thu Dec 12, 2013 12:51 pm
Location: Saskatchewan, Canada

Re: Fun with Linux software RAID

Thu Jul 19, 2018 1:21 pm

The new installer in 18.04 is garbage. I had no success getting a system to boot after creating a simple 2 disk RAID 1 mdraid array and booting from it. Reverted back to server 16.04 and it's fine.
 
Bauxite
Gerbil Elite
Posts: 768
Joined: Sat Jan 28, 2006 12:10 pm
Location: electrolytic redox smelting plant

Get off the "windows" of unix filesystems

Thu Jul 19, 2018 1:43 pm

You're already using ubuntu which already decided license-armchair-lawyering be damned and put it in out-of-the-box:

Learn ZFS, never look back.
2018: at 120 Zen cores and counting, so pretty much done with intel on the desktop.
E5 2696v4 22c44t 2.2~3.7Ghz - The last great gleam of the pre-nerf HEDT era.
E5 1680v2 8c16t 4.5Ghz - "Yes Virginia, there were unlocked xeons" /weep for them.
 
Waco
Gold subscriber
Minister of Gerbil Affairs
Posts: 2494
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Get off the "windows" of unix filesystems

Thu Jul 19, 2018 1:51 pm

Bauxite wrote:
You're already using ubuntu which already decided license-armchair-lawyering be damned and put it in out-of-the-box:

Learn ZFS, never look back.

*as long as you're not booting from it.

That said, nobody should be booting from their data arrays!
Z170A Gaming Pro Carbon | 6700K @ 4.4 | 16 GB | GTX Titan Xm | Seasonic Gold 850 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | Cosmos II | Samsung 4K 40" | 2048 + 240 + LSI 9207-8i (128x8) SSDs
 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Fun with Linux software RAID

Thu Jul 19, 2018 2:40 pm

captaintrav wrote:
The new installer in 18.04 is garbage. I had no success getting a system to boot after creating a simple 2 disk RAID 1 mdraid array and booting from it. Reverted back to server 16.04 and it's fine.

18.04 LTS installed fine for me, but on first boot the console was getting set to an invalid video mode. Had to futz with the GRUB configuration to straighten that out. The bigger issue for me was that it kept breaking the RAID array on shutdown.

So yeah... 18.04 seems like it is not quite ready for prime time... 16.04 LTS seems stable (so far).

Edit: And booting from a software RAID array has always been a little dicey. Back in the mid-aughts it was a PITA, but was improving. Then came GPT and UEFI, and it all went to hell again. You can make it work, but it's not for the faint of heart.
Nostalgia isn't what it used to be.
 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Fun with Linux software RAID

Sat Jul 21, 2018 8:50 pm

Latest chapter: Was copying the final volume over from the old server, and it was going S-L-O-W. Determined that the problem was one of the new drives in the new server's RAID array. For whatever reason, it was running at a fraction of the speed of the other drives, and holding back the performance of the entire array. My first thought was "infant mortality... this drive is getting RMAed". But just to rule out a problem with the port or backplane slot, I swapped the drive with the one in the adjacent bay. Both drives had normal throughput. Swapped 'em back. Still normal. WTF.

Only thing I can figure is, the hot swap backplane the drive was plugged into had been sitting in the spares stash for a couple of years. Dust or corrosion on the SATA connector? Swapping the drives back and forth could've scraped that off, I suppose.

Oh well, I guess I'll just need to keep an eye on it. One of the hazards of "ghetto server" builds.
Nostalgia isn't what it used to be.
 
Waco
Gold subscriber
Minister of Gerbil Affairs
Posts: 2494
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Fun with Linux software RAID

Sat Jul 21, 2018 11:18 pm

Do any of the SMART attributes look funny? Is the firmware on the drive up to date? I've seen drives do some awfully funky things over the years and sometimes a power cycle of the drive would resolve any issues.
Z170A Gaming Pro Carbon | 6700K @ 4.4 | 16 GB | GTX Titan Xm | Seasonic Gold 850 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | Cosmos II | Samsung 4K 40" | 2048 + 240 + LSI 9207-8i (128x8) SSDs
 
just brew it!
Gold subscriber
Administrator
Topic Author
Posts: 51637
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Fun with Linux software RAID

Sun Jul 22, 2018 7:29 am

Waco wrote:
Do any of the SMART attributes look funny? Is the firmware on the drive up to date? I've seen drives do some awfully funky things over the years and sometimes a power cycle of the drive would resolve any issues.

Yeah, I checked the SMART attributes, and they looked normal. Haven't checked whether the firmware version is the most current, but The firmware is the latest according to Seagate's site, and it is the same version as the three other drives of the same make/model. Moving it to another bay would've power cycled it, of course; that's likely what cleared whatever it was unhappy about.
Nostalgia isn't what it used to be.

Who is online

Users browsing this forum: No registered users and 1 guest