Raid for dumbies

All things storage here: hard drives, DVD RW drives, little wicker baskets.

Moderators: morphine, Steel

Raid for dumbies

Postposted on Wed Oct 09, 2013 3:08 pm

i currently have a 240 gb SSD for my boot drive and programs. i am running win7 64 pro. i use two 2tb WD caviar black hard drives in a raid 1 for storage. i store music, movies and my photos from my nikon d600 on the raid and then backup the music and photo's to carbonite. my raid is starting to fill up and as it does i am starting to think about some options and issues. the movies i could care less about, the music is backed up on my laptop but the pictures are a priority. i am slightly concerned that the two WD drives have been spinning for about 3 years non-stop and could die of old age. i am thinking i could upgrade to two 3tb constellation drives (i just saw them on newegg for $170 each).

my issue or hesitation is the speed of my raid. when i take pictures for work i tend to copy them to my laptop which has a 256gb ssd drive. i tend to take 100-300 pictures at a time and shoot raw so each pic is about 24mb. i use nikon view to sort and nikon capture or photoshop to process. when i review the folder on my laptop it populates pretty quickly and is easy to work from. when i create a folder on my raid it takes substantially longer for everything to work. sometimes it seams like the mechanical drives red line for about 30 seconds and almost lock up before i can do anything. i also hear other people quoting insane read/write speeds on mirrored raids. is there anything i can do to speed up my raid without spending more than a few hundred dollars that will not introduce a complexity beyond my ability. i.e. is there a pci raid card that is $150, faster than a mb raid and can be setup in windows? other thoughts or ideas?

i know raid is not a backup but i am a cautious person (read paranoid/anal/ocd). every time i have lost a file it was to mechanical breakdown and not user error.

i would count myself as a skilled hobbyist. i have no trouble building a raid in disk management but beyond that you may need to dumb it down for me.
i7-930, GA-X58A-UD3R, Agility 2 240GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
hiro_pro
Gerbil
 
Posts: 62
Joined: Wed Feb 16, 2011 10:20 am
Location: DC

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 3:26 pm

Have you figured out approximately how many MB/sec you're getting on the existing RAID-1? At the end of the day, unless you're running software RAID-5 or -6 (which can be limited by CPU speed), you're probably being bottlenecked by the transfer rate of the hard drives themselves, so a $150 RAID card probably won't help much.

Getting 4 drives and doing RAID-10 might get you a nice little boost though...
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 3:55 pm

A few years ago I was in largely the same situation. I decided to purchase one more drive of the existing two I had and do a RAID 5 (using the integrated controller on my MB, nothing fancey). That doubled my space (went from 640GB to ~1.2 TB) for the cost of a single drive, and still gave me redundancy. It also doubled the read speed (don't expect writes to be faster, but reads were noticeably faster than before).

The trick was I had an external 1 TB laying around. I copied the ~500 gigs to that, rebuilt the array, then copied it back over. If your motherboard is capable and you want to take the plunge it would benefit you to use the builtin controller to build the raid instead of using the Windows tools. But either way it should be an improvement :)
LaChupacabra
Gerbil First Class
Gold subscriber
 
 
Posts: 136
Joined: Tue Dec 30, 2008 10:59 pm

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 4:05 pm

I would hesitate to do a software (or motherboard) RAID-5 in his case, since it sounds like he frequently writes large batches of data to the RAID array. While faster CPUs have mitigated the issue quite a bit, RAID-5 without an intelligent RAID controller will still result in some performance loss.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 4:24 pm

That is completely true. I am biased against RAID 10 for separate reasons (uses half your disks and still can result in data loss if you lose the wrong 2 drives). The parity calculations in a RAID 5 with 3 disks are incredibly minimal, and instead of purchasing 4 new drives at 170 bucks a piece to get his storage requirements, RAID 5 will result in increased storage space, increased read speeds (so, viewing the folders should go down from ~30 seconds to ~15), and cost less than a quarter of what a brand new RAID 10 array will. That's 90% of the goals with 20% of the cost. You have to admit, that's a pretty big win.
LaChupacabra
Gerbil First Class
Gold subscriber
 
 
Posts: 136
Joined: Tue Dec 30, 2008 10:59 pm

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 4:47 pm

just brew it! wrote:I would hesitate to do a software (or motherboard) RAID-5 in his case, since it sounds like he frequently writes large batches of data to the RAID array. While faster CPUs have mitigated the issue quite a bit, RAID-5 without an intelligent RAID controller will still result in some performance loss.

He's writing large blocks though - I'd bet the performance would be more than reasonable with that kind of workload.
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 4:54 pm

just brew it! wrote:Have you figured out approximately how many MB/sec you're getting on the existing RAID-1? At the end of the day, unless you're running software RAID-5 or -6 (which can be limited by CPU speed), you're probably being bottlenecked by the transfer rate of the hard drives themselves, so a $150 RAID card probably won't help much.

Getting 4 drives and doing RAID-10 might get you a nice little boost though...


i know i am seeing 120 mbs write speeds. i am not sure what the bottleneck is with the reads when i open folders with more than 1gb. i dont have an obvious way to measure those reads. i also notice when i open my video library there is a huge lag while it rebuilds the thumbs. i would have thought that once it cashed itself you wouldn't see those kind of delays? i dont really care as much about the movies though. i just pick one and hit play. it is more annoying when trying to cull down 250 NEF's and edit the best 30.
i7-930, GA-X58A-UD3R, Agility 2 240GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
hiro_pro
Gerbil
 
Posts: 62
Joined: Wed Feb 16, 2011 10:20 am
Location: DC

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 4:59 pm

i guess the other question is how raid-able is a WD black drive? i have read reports that the black drives will fail in anything more complicated than a mirror (i.e. stripe) or should i go to the constellation drives and not look back? and it sounds like a 3 disk raid 5 with 3 constellation drives is safe enough and easy enough in windows 7?
i7-930, GA-X58A-UD3R, Agility 2 240GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
hiro_pro
Gerbil
 
Posts: 62
Joined: Wed Feb 16, 2011 10:20 am
Location: DC

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 6:00 pm

Unless you need to have uptime on your system. I would try to avoid RAID solutions as much as possible. RAID 1 is about as far as you want to go if you are trying to implement a contingent plan on top of your back-up schemes. Example, I forgot or the program failed to run the schedule back-up and HDD had a hardware failure. RAID 0 is only good for scratch disks. Going beyond RAID 1 isn't worth the cost, potential headaches and overhead if you aren't looking for uptime.

Performance-wise HDD-based RAIDs no longer make much sense for vast majority of desktop users. SSDs are faster for a fraction of cost and overhead. SSD-based RAIDs are just overkill and they only make sense if you run applications that take advantage of the I/O bandwidth (mostly in the domain of professional world).
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC-12800, EVGA 660Ti, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
Krogoth
Maximum Gerbil
Silver subscriber
 
 
Posts: 4410
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime

Re: Raid for dumbies

Postposted on Wed Oct 09, 2013 6:39 pm

How much are your "work" images and how much are your other stuff that you could not care less (movies and music)? I would say buy a separate drive, move those off to the new drive, and stay with your current setup. You probably want to do serious work on the SSDs anyway. This new drive can be the cheap "greens" if you want.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24422
Joined: Mon May 24, 2004 2:19 am

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 12:24 am

hiro_pro wrote:i know i am seeing 120 mbs write speeds.

Thats what I see from my single WD greens. You could add more drives to your RAID 1 array to improve performance but from what I read that is up to the controller and not all controllers offer that performance increase.

The more immediate concern is the 30 second delay/lockup. The only thing I can think of is that you have set your drives to sleep after inactivity and the delay you are experiencing is the array waking up. The delay should only happen with the initial activity. The array should be immediately responsive after it has woken up. If the delay/lockup is happening with every copy it might be best just to buy a big drive (or two to backup your backups) and retire your array.

I've got photos archived onto a 3TB green on a Windows 8 rig. I just did a test with a folder that contain 2700 10 MP NEF images. After all the thumbnails were generated for the first time they appear instantly when I go back to that folder.

Don't work off of your backup/archive drive. I would buy another SSD to use as a work in progress drive. Once you are done editing move them over to your long term storage solution.
Last edited by End User on Thu Oct 10, 2013 9:48 am, edited 1 time in total.
i7-3770K@4.7 | H100 | P8Z77-V PREMIUM | 16GB | 2 GTX 770 4GB SLI | M500 960GB | EVO 840 250GB | AX850 | Obsidian 550D | R.A.T. 9 | U2713H | U2711
End User
Gerbil Elite
Gold subscriber
 
 
Posts: 951
Joined: Fri Apr 16, 2004 6:47 pm
Location: Toronto, Canada

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 6:26 am

I played with soft-RAID10 for a while, using Spinpoints and Greens. RAID5 is a definite no-no without dedicated hardware though.

I use Steam Mover to swap 200GB of game installs from the SSD to mechanical on a fairly regular basis. I ended up going with 3 old 1TB drives in RAID0 for throughput - I was getting sequential writes of about 250MB/s - and then doing a nightly robocopy /MIR to clone the RAID0 onto a 3TB single drive for some measure of redundancy. It has the added benefit that your redundant data copy is not RAIDed and you can read it without finding a RAID controller of the same type.

If I can get 250MB/s out of three old 90MB/s drives I'm pretty sure 500MB/s is achievable with a handful of faster drives.
<insert large, flashing, epileptic-fit-inducing signature (based on the latest internet-meme) here>
Chrispy_
Gerbil Jedi
Gold subscriber
 
 
Posts: 1881
Joined: Fri Apr 09, 2004 3:49 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 8:31 am

Flying Fox wrote:How much are your "work" images and how much are your other stuff that you could not care less (movies and music)? I would say buy a separate drive, move those off to the new drive, and stay with your current setup. You probably want to do serious work on the SSDs anyway. This new drive can be the cheap "greens" if you want.


great point. about half is pictures. music is barely 10% and the rest is movies/tv. i could dump the videos on a separate drive and if i lose them so what i could just _____ ____ ____ _________.__ (edited per rule 1).
i7-930, GA-X58A-UD3R, Agility 2 240GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
hiro_pro
Gerbil
 
Posts: 62
Joined: Wed Feb 16, 2011 10:20 am
Location: DC

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 8:38 am

End User wrote:
hiro_pro wrote:i know i am seeing 120 mbs write speeds.

Thats what see from my single WD greens. You could add more drives to your RAID 1 array to improve performance but from what I read that is up to the controller and not all controllers offer that performance increase.

The more immediate concern is the 30 second delay/lockup. The only thing I can think of is that you have set your drives to sleep after inactivity and the delay you are experiencing is the array waking up. The delay should only happen with the initial activity. The array should be immediately responsive after it has woken up. If the delay/lockup is happening with every copy it might be best just to buy a big drive (or two to backup your backups) and retire your array.

I've got photos archived onto a 3TB green on a Windows 8 rig. I just did a test with a folder that contain 2700 10 MP NEF images. After all the thumbnails were generated for the first time they appear instantly when I go back to that folder.

Don't work off of your backup/archive drive. I would buy another SSD to use as a work in progress drive. Once you are done editing move them over to your long term storage solution.


maybe i need to go back and look at NEF's from some older cameras. i know the drives are choking on folders with larger files. i should say the machine is not locking up, it is just the operation on that drive is grinding to a painful slowdown. i have my drive meter up so i can see that one drive max out and with old age i have learned to just wait til it finishes before going to work.

i like your idea of re-introducing a scratch disk or working disk. i could get a new ssd (i just need the excuse for a newer bigger faster what ever) and clone my boot drive to it and then format the old ssd boot disk and use it for active jobs.
i7-930, GA-X58A-UD3R, Agility 2 240GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
hiro_pro
Gerbil
 
Posts: 62
Joined: Wed Feb 16, 2011 10:20 am
Location: DC

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 9:05 am

I second RAID10. Nothing else gives you the write performance you are looking for.

Use a real RAID Controller:
http://www.newegg.com/Product/Product.a ... 6816118129
http://www.newegg.com/Product/Product.a ... 6816118106 - w/ Cache - Enable Write Cache, buy a good UPS (I like Eaton Equipment).

Setup Email Alerting for any Disk Issues. Continue Backup to Carbonite, RAID does not equal Backup.
prb123
Gerbil
 
Posts: 14
Joined: Wed Apr 25, 2007 10:04 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 9:55 am

If all you want is sustained read/write performance a software RAID 5 (on the integrated Intel) with the RAM cache enabled can perform surprisingly good as long as you don't get into random write scenarios. A UPS is a must as is a backup solution (which I hope you have anyway!).
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 10:34 am

hiro_pro wrote:maybe i need to go back and look at NEF's from some older cameras. i know the drives are choking on folders with larger files.

I did the same test with a folder that contained 1500 20MB RAW images from my RX100. After the initial icon previews were created they were displayed instantly once I reopened the folder.
i7-3770K@4.7 | H100 | P8Z77-V PREMIUM | 16GB | 2 GTX 770 4GB SLI | M500 960GB | EVO 840 250GB | AX850 | Obsidian 550D | R.A.T. 9 | U2713H | U2711
End User
Gerbil Elite
Gold subscriber
 
 
Posts: 951
Joined: Fri Apr 16, 2004 6:47 pm
Location: Toronto, Canada

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 11:33 am

I wonder if it's set to not cache them? Isn't that a per-folder setting in Vista/7/8?
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 12:54 pm

Waco wrote:I wonder if it's set to not cache them? Isn't that a per-folder setting in Vista/7/8?

I made no changes to such an option. The settings I am using are the default.
i7-3770K@4.7 | H100 | P8Z77-V PREMIUM | 16GB | 2 GTX 770 4GB SLI | M500 960GB | EVO 840 250GB | AX850 | Obsidian 550D | R.A.T. 9 | U2713H | U2711
End User
Gerbil Elite
Gold subscriber
 
 
Posts: 951
Joined: Fri Apr 16, 2004 6:47 pm
Location: Toronto, Canada

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 12:59 pm

prb123 wrote:http://www.newegg.com/Product/Product.a ... 6816118106 - w/ Cache - Enable Write Cache, buy a good UPS (I like Eaton Equipment).


Buy the battery backup unit for it. Some cards won't reach their full potential without it.
http://www.newegg.com/Product/Product.a ... 6816118163

hiro_pro wrote:i guess the other question is how raid-able is a WD black drive?


You probably shouldn't if your concerned about your data. The WD Blacks have consumer firmware that isn't optimized for RAID arrays, and it will cause them to be dropped from the array quicker then if they had RAID firmware. Don't get me wrong, it's doable. It's just not the greatest idea when you need data security.
Flatland_Spider
Gerbil Elite
 
Posts: 852
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 1:46 pm

Flatland_Spider wrote:You probably shouldn't if your concerned about your data. The WD Blacks have consumer firmware that isn't optimized for RAID arrays, and it will cause them to be dropped from the array quicker then if they had RAID firmware. Don't get me wrong, it's doable. It's just not the greatest idea when you need data security.

I keep hearing this but I can tell you that we use consumer drives in very large RAID 5 and 6 arrays and they very rarely drop out from the horrors people attribute to them.
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 1:53 pm

Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.
cphite
Gerbil Elite
 
Posts: 557
Joined: Thu Apr 29, 2010 9:28 am

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 1:55 pm

cphite wrote:Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.

If there are regular backups and a little downtime from a bad drive is acceptable...RAID 0 all the way.
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 2:51 pm

Waco wrote:
cphite wrote:Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.

If there are regular backups and a little downtime from a bad drive is acceptable...RAID 0 all the way.


Meh.

I can see it being nice having one large space instead of two; but the performance gains are minimal and if you lose one drive you lose the whole array; so it's actually slightly less reliable than non-RAID.

For me, RAID 0 just isn't worth it unless you absolutely need a space that's bigger than you can get from a single physical drive.
cphite
Gerbil Elite
 
Posts: 557
Joined: Thu Apr 29, 2010 9:28 am

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 2:56 pm

You also get large jumps in sustained read/write performance. If backups are in place and a temporary loss of the array isn't that much of a pain...I don't see any reason not to run a striped setup for this workload. It's damn near ideal.

I'm fairly sure everyone understand that it's not a redundant array but the premise was that it doesn't matter.
Z68XP-UD4 | 2700K @ 4.4 GHz | 16 GB | 770 | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 1 TB WD Black w/ SRT
Waco
Gerbil Elite
 
Posts: 746
Joined: Tue Jan 20, 2009 4:14 pm

Re: Raid for dumbies

Postposted on Thu Oct 10, 2013 3:30 pm

Waco wrote:I keep hearing this but I can tell you that we use consumer drives in very large RAID 5 and 6 arrays and they very rarely drop out from the horrors people attribute to them.


Like I said, it's doable, but it's not the best hardware to use when the idea is to remove as much risk as possible. There are ways to run commodity hardware in a production environment, it just requires data to replicated across several servers in near-realtime.
Flatland_Spider
Gerbil Elite
 
Posts: 852
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539


Return to Storage

Who is online

Users browsing this forum: No registered users and 5 guests