Personal computing discussed

Moderators: renee, morphine, Steel

 
hiro_pro
Gerbil
Topic Author
Posts: 68
Joined: Wed Feb 16, 2011 10:20 am
Location: DC
Contact:

Raid for dumbies

Wed Oct 09, 2013 3:08 pm

i currently have a 240 gb SSD for my boot drive and programs. i am running win7 64 pro. i use two 2tb WD caviar black hard drives in a raid 1 for storage. i store music, movies and my photos from my nikon d600 on the raid and then backup the music and photo's to carbonite. my raid is starting to fill up and as it does i am starting to think about some options and issues. the movies i could care less about, the music is backed up on my laptop but the pictures are a priority. i am slightly concerned that the two WD drives have been spinning for about 3 years non-stop and could die of old age. i am thinking i could upgrade to two 3tb constellation drives (i just saw them on newegg for $170 each).

my issue or hesitation is the speed of my raid. when i take pictures for work i tend to copy them to my laptop which has a 256gb ssd drive. i tend to take 100-300 pictures at a time and shoot raw so each pic is about 24mb. i use nikon view to sort and nikon capture or photoshop to process. when i review the folder on my laptop it populates pretty quickly and is easy to work from. when i create a folder on my raid it takes substantially longer for everything to work. sometimes it seams like the mechanical drives red line for about 30 seconds and almost lock up before i can do anything. i also hear other people quoting insane read/write speeds on mirrored raids. is there anything i can do to speed up my raid without spending more than a few hundred dollars that will not introduce a complexity beyond my ability. i.e. is there a pci raid card that is $150, faster than a mb raid and can be setup in windows? other thoughts or ideas?

i know raid is not a backup but i am a cautious person (read paranoid/anal/ocd). every time i have lost a file it was to mechanical breakdown and not user error.

i would count myself as a skilled hobbyist. i have no trouble building a raid in disk management but beyond that you may need to dumb it down for me.
i7-930, GA-X58A-UD3R, Crucial MX100 512GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
Nikon D600, Tamron 24-70 & 70-200 2.4 VC, Nikon 50 1.4, 85 1.8, Sigma 35 1.4
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Raid for dumbies

Wed Oct 09, 2013 3:26 pm

Have you figured out approximately how many MB/sec you're getting on the existing RAID-1? At the end of the day, unless you're running software RAID-5 or -6 (which can be limited by CPU speed), you're probably being bottlenecked by the transfer rate of the hard drives themselves, so a $150 RAID card probably won't help much.

Getting 4 drives and doing RAID-10 might get you a nice little boost though...
Nostalgia isn't what it used to be.
 
LaChupacabra
Gerbil First Class
Posts: 145
Joined: Tue Dec 30, 2008 10:59 pm

Re: Raid for dumbies

Wed Oct 09, 2013 3:55 pm

A few years ago I was in largely the same situation. I decided to purchase one more drive of the existing two I had and do a RAID 5 (using the integrated controller on my MB, nothing fancey). That doubled my space (went from 640GB to ~1.2 TB) for the cost of a single drive, and still gave me redundancy. It also doubled the read speed (don't expect writes to be faster, but reads were noticeably faster than before).

The trick was I had an external 1 TB laying around. I copied the ~500 gigs to that, rebuilt the array, then copied it back over. If your motherboard is capable and you want to take the plunge it would benefit you to use the builtin controller to build the raid instead of using the Windows tools. But either way it should be an improvement :)
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Raid for dumbies

Wed Oct 09, 2013 4:05 pm

I would hesitate to do a software (or motherboard) RAID-5 in his case, since it sounds like he frequently writes large batches of data to the RAID array. While faster CPUs have mitigated the issue quite a bit, RAID-5 without an intelligent RAID controller will still result in some performance loss.
Nostalgia isn't what it used to be.
 
LaChupacabra
Gerbil First Class
Posts: 145
Joined: Tue Dec 30, 2008 10:59 pm

Re: Raid for dumbies

Wed Oct 09, 2013 4:24 pm

That is completely true. I am biased against RAID 10 for separate reasons (uses half your disks and still can result in data loss if you lose the wrong 2 drives). The parity calculations in a RAID 5 with 3 disks are incredibly minimal, and instead of purchasing 4 new drives at 170 bucks a piece to get his storage requirements, RAID 5 will result in increased storage space, increased read speeds (so, viewing the folders should go down from ~30 seconds to ~15), and cost less than a quarter of what a brand new RAID 10 array will. That's 90% of the goals with 20% of the cost. You have to admit, that's a pretty big win.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Wed Oct 09, 2013 4:47 pm

just brew it! wrote:
I would hesitate to do a software (or motherboard) RAID-5 in his case, since it sounds like he frequently writes large batches of data to the RAID array. While faster CPUs have mitigated the issue quite a bit, RAID-5 without an intelligent RAID controller will still result in some performance loss.

He's writing large blocks though - I'd bet the performance would be more than reasonable with that kind of workload.
Victory requires no explanation. Defeat allows none.
 
hiro_pro
Gerbil
Topic Author
Posts: 68
Joined: Wed Feb 16, 2011 10:20 am
Location: DC
Contact:

Re: Raid for dumbies

Wed Oct 09, 2013 4:54 pm

just brew it! wrote:
Have you figured out approximately how many MB/sec you're getting on the existing RAID-1? At the end of the day, unless you're running software RAID-5 or -6 (which can be limited by CPU speed), you're probably being bottlenecked by the transfer rate of the hard drives themselves, so a $150 RAID card probably won't help much.

Getting 4 drives and doing RAID-10 might get you a nice little boost though...


i know i am seeing 120 mbs write speeds. i am not sure what the bottleneck is with the reads when i open folders with more than 1gb. i dont have an obvious way to measure those reads. i also notice when i open my video library there is a huge lag while it rebuilds the thumbs. i would have thought that once it cashed itself you wouldn't see those kind of delays? i dont really care as much about the movies though. i just pick one and hit play. it is more annoying when trying to cull down 250 NEF's and edit the best 30.
i7-930, GA-X58A-UD3R, Crucial MX100 512GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
Nikon D600, Tamron 24-70 & 70-200 2.4 VC, Nikon 50 1.4, 85 1.8, Sigma 35 1.4
 
hiro_pro
Gerbil
Topic Author
Posts: 68
Joined: Wed Feb 16, 2011 10:20 am
Location: DC
Contact:

Re: Raid for dumbies

Wed Oct 09, 2013 4:59 pm

i guess the other question is how raid-able is a WD black drive? i have read reports that the black drives will fail in anything more complicated than a mirror (i.e. stripe) or should i go to the constellation drives and not look back? and it sounds like a 3 disk raid 5 with 3 constellation drives is safe enough and easy enough in windows 7?
i7-930, GA-X58A-UD3R, Crucial MX100 512GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
Nikon D600, Tamron 24-70 & 70-200 2.4 VC, Nikon 50 1.4, 85 1.8, Sigma 35 1.4
 
Krogoth
Emperor Gerbilius I
Posts: 6049
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime
Contact:

Re: Raid for dumbies

Wed Oct 09, 2013 6:00 pm

Unless you need to have uptime on your system. I would try to avoid RAID solutions as much as possible. RAID 1 is about as far as you want to go if you are trying to implement a contingent plan on top of your back-up schemes. Example, I forgot or the program failed to run the schedule back-up and HDD had a hardware failure. RAID 0 is only good for scratch disks. Going beyond RAID 1 isn't worth the cost, potential headaches and overhead if you aren't looking for uptime.

Performance-wise HDD-based RAIDs no longer make much sense for vast majority of desktop users. SSDs are faster for a fraction of cost and overhead. SSD-based RAIDs are just overkill and they only make sense if you run applications that take advantage of the I/O bandwidth (mostly in the domain of professional world).
Gigabyte X670 AORUS-ELITE AX, Raphael 7950X, 2x16GiB of G.Skill TRIDENT DDR5-5600, Sapphire RX 6900XT, Seasonic GX-850 and Fractal Define 7 (W)
Ivy Bridge 3570K, 2x4GiB of G.Skill RIPSAW DDR3-1600, Gigabyte Z77X-UD3H, Corsair CX-750M V2, and PC-7B
 
Flying Fox
Gerbil God
Posts: 25690
Joined: Mon May 24, 2004 2:19 am
Contact:

Re: Raid for dumbies

Wed Oct 09, 2013 6:39 pm

How much are your "work" images and how much are your other stuff that you could not care less (movies and music)? I would say buy a separate drive, move those off to the new drive, and stay with your current setup. You probably want to do serious work on the SSDs anyway. This new drive can be the cheap "greens" if you want.
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
 
End User
Minister of Gerbil Affairs
Posts: 2977
Joined: Fri Apr 16, 2004 6:47 pm
Location: Upper Canada

Re: Raid for dumbies

Thu Oct 10, 2013 12:24 am

hiro_pro wrote:
i know i am seeing 120 mbs write speeds.

Thats what I see from my single WD greens. You could add more drives to your RAID 1 array to improve performance but from what I read that is up to the controller and not all controllers offer that performance increase.

The more immediate concern is the 30 second delay/lockup. The only thing I can think of is that you have set your drives to sleep after inactivity and the delay you are experiencing is the array waking up. The delay should only happen with the initial activity. The array should be immediately responsive after it has woken up. If the delay/lockup is happening with every copy it might be best just to buy a big drive (or two to backup your backups) and retire your array.

I've got photos archived onto a 3TB green on a Windows 8 rig. I just did a test with a folder that contain 2700 10 MP NEF images. After all the thumbnails were generated for the first time they appear instantly when I go back to that folder.

Don't work off of your backup/archive drive. I would buy another SSD to use as a work in progress drive. Once you are done editing move them over to your long term storage solution.
Last edited by End User on Thu Oct 10, 2013 9:48 am, edited 1 time in total.
 
Chrispy_
Maximum Gerbil
Posts: 4670
Joined: Fri Apr 09, 2004 3:49 pm
Location: Europe, most frequently London.

Re: Raid for dumbies

Thu Oct 10, 2013 6:26 am

I played with soft-RAID10 for a while, using Spinpoints and Greens. RAID5 is a definite no-no without dedicated hardware though.

I use Steam Mover to swap 200GB of game installs from the SSD to mechanical on a fairly regular basis. I ended up going with 3 old 1TB drives in RAID0 for throughput - I was getting sequential writes of about 250MB/s - and then doing a nightly robocopy /MIR to clone the RAID0 onto a 3TB single drive for some measure of redundancy. It has the added benefit that your redundant data copy is not RAIDed and you can read it without finding a RAID controller of the same type.

If I can get 250MB/s out of three old 90MB/s drives I'm pretty sure 500MB/s is achievable with a handful of faster drives.
Congratulations, you've noticed that this year's signature is based on outdated internet memes; CLICK HERE NOW to experience this unforgettable phenomenon. This sentence is just filler and as irrelevant as my signature.
 
hiro_pro
Gerbil
Topic Author
Posts: 68
Joined: Wed Feb 16, 2011 10:20 am
Location: DC
Contact:

Re: Raid for dumbies

Thu Oct 10, 2013 8:31 am

Flying Fox wrote:
How much are your "work" images and how much are your other stuff that you could not care less (movies and music)? I would say buy a separate drive, move those off to the new drive, and stay with your current setup. You probably want to do serious work on the SSDs anyway. This new drive can be the cheap "greens" if you want.


great point. about half is pictures. music is barely 10% and the rest is movies/tv. i could dump the videos on a separate drive and if i lose them so what i could just _____ ____ ____ _________.__ (edited per rule 1).
i7-930, GA-X58A-UD3R, Crucial MX100 512GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
Nikon D600, Tamron 24-70 & 70-200 2.4 VC, Nikon 50 1.4, 85 1.8, Sigma 35 1.4
 
hiro_pro
Gerbil
Topic Author
Posts: 68
Joined: Wed Feb 16, 2011 10:20 am
Location: DC
Contact:

Re: Raid for dumbies

Thu Oct 10, 2013 8:38 am

End User wrote:
hiro_pro wrote:
i know i am seeing 120 mbs write speeds.

Thats what see from my single WD greens. You could add more drives to your RAID 1 array to improve performance but from what I read that is up to the controller and not all controllers offer that performance increase.

The more immediate concern is the 30 second delay/lockup. The only thing I can think of is that you have set your drives to sleep after inactivity and the delay you are experiencing is the array waking up. The delay should only happen with the initial activity. The array should be immediately responsive after it has woken up. If the delay/lockup is happening with every copy it might be best just to buy a big drive (or two to backup your backups) and retire your array.

I've got photos archived onto a 3TB green on a Windows 8 rig. I just did a test with a folder that contain 2700 10 MP NEF images. After all the thumbnails were generated for the first time they appear instantly when I go back to that folder.

Don't work off of your backup/archive drive. I would buy another SSD to use as a work in progress drive. Once you are done editing move them over to your long term storage solution.


maybe i need to go back and look at NEF's from some older cameras. i know the drives are choking on folders with larger files. i should say the machine is not locking up, it is just the operation on that drive is grinding to a painful slowdown. i have my drive meter up so i can see that one drive max out and with old age i have learned to just wait til it finishes before going to work.

i like your idea of re-introducing a scratch disk or working disk. i could get a new ssd (i just need the excuse for a newer bigger faster what ever) and clone my boot drive to it and then format the old ssd boot disk and use it for active jobs.
i7-930, GA-X58A-UD3R, Crucial MX100 512GB, 2 TB WD Black *2 mirror, ATI 5850 1 GB, 12GB (3*4GB)
Nikon D600, Tamron 24-70 & 70-200 2.4 VC, Nikon 50 1.4, 85 1.8, Sigma 35 1.4
 
prb123
Gerbil
Posts: 43
Joined: Wed Apr 25, 2007 10:04 pm

Re: Raid for dumbies

Thu Oct 10, 2013 9:05 am

I second RAID10. Nothing else gives you the write performance you are looking for.

Use a real RAID Controller:
http://www.newegg.com/Product/Product.a ... 6816118129
http://www.newegg.com/Product/Product.a ... 6816118106 - w/ Cache - Enable Write Cache, buy a good UPS (I like Eaton Equipment).

Setup Email Alerting for any Disk Issues. Continue Backup to Carbonite, RAID does not equal Backup.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Thu Oct 10, 2013 9:55 am

If all you want is sustained read/write performance a software RAID 5 (on the integrated Intel) with the RAM cache enabled can perform surprisingly good as long as you don't get into random write scenarios. A UPS is a must as is a backup solution (which I hope you have anyway!).
Victory requires no explanation. Defeat allows none.
 
End User
Minister of Gerbil Affairs
Posts: 2977
Joined: Fri Apr 16, 2004 6:47 pm
Location: Upper Canada

Re: Raid for dumbies

Thu Oct 10, 2013 10:34 am

hiro_pro wrote:
maybe i need to go back and look at NEF's from some older cameras. i know the drives are choking on folders with larger files.

I did the same test with a folder that contained 1500 20MB RAW images from my RX100. After the initial icon previews were created they were displayed instantly once I reopened the folder.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Thu Oct 10, 2013 11:33 am

I wonder if it's set to not cache them? Isn't that a per-folder setting in Vista/7/8?
Victory requires no explanation. Defeat allows none.
 
End User
Minister of Gerbil Affairs
Posts: 2977
Joined: Fri Apr 16, 2004 6:47 pm
Location: Upper Canada

Re: Raid for dumbies

Thu Oct 10, 2013 12:54 pm

Waco wrote:
I wonder if it's set to not cache them? Isn't that a per-folder setting in Vista/7/8?

I made no changes to such an option. The settings I am using are the default.
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Raid for dumbies

Thu Oct 10, 2013 12:59 pm

prb123 wrote:
http://www.newegg.com/Product/Product.a ... 6816118106 - w/ Cache - Enable Write Cache, buy a good UPS (I like Eaton Equipment).


Buy the battery backup unit for it. Some cards won't reach their full potential without it.
http://www.newegg.com/Product/Product.a ... 6816118163

hiro_pro wrote:
i guess the other question is how raid-able is a WD black drive?


You probably shouldn't if your concerned about your data. The WD Blacks have consumer firmware that isn't optimized for RAID arrays, and it will cause them to be dropped from the array quicker then if they had RAID firmware. Don't get me wrong, it's doable. It's just not the greatest idea when you need data security.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Thu Oct 10, 2013 1:46 pm

Flatland_Spider wrote:
You probably shouldn't if your concerned about your data. The WD Blacks have consumer firmware that isn't optimized for RAID arrays, and it will cause them to be dropped from the array quicker then if they had RAID firmware. Don't get me wrong, it's doable. It's just not the greatest idea when you need data security.

I keep hearing this but I can tell you that we use consumer drives in very large RAID 5 and 6 arrays and they very rarely drop out from the horrors people attribute to them.
Victory requires no explanation. Defeat allows none.
 
cphite
Graphmaster Gerbil
Posts: 1202
Joined: Thu Apr 29, 2010 9:28 am

Re: Raid for dumbies

Thu Oct 10, 2013 1:53 pm

Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Thu Oct 10, 2013 1:55 pm

cphite wrote:
Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.

If there are regular backups and a little downtime from a bad drive is acceptable...RAID 0 all the way.
Victory requires no explanation. Defeat allows none.
 
cphite
Graphmaster Gerbil
Posts: 1202
Joined: Thu Apr 29, 2010 9:28 am

Re: Raid for dumbies

Thu Oct 10, 2013 2:51 pm

Waco wrote:
cphite wrote:
Frankly, I would drop RAID altogether... RAID is all about up-time. You've got backups going to Carbonite - if something should happen to your local drive, it's an inconvenience, yes; but it's not like you're doing business on that machine and losing money while it's down.

Honestly, unless you're just really keen on using RAID because you find it interesting or whatever, it just seems like an added level of complexity that really doesn't buy you much. As long as you're doing regular backups, which it sounds like you are.

If there are regular backups and a little downtime from a bad drive is acceptable...RAID 0 all the way.


Meh.

I can see it being nice having one large space instead of two; but the performance gains are minimal and if you lose one drive you lose the whole array; so it's actually slightly less reliable than non-RAID.

For me, RAID 0 just isn't worth it unless you absolutely need a space that's bigger than you can get from a single physical drive.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Raid for dumbies

Thu Oct 10, 2013 2:56 pm

You also get large jumps in sustained read/write performance. If backups are in place and a temporary loss of the array isn't that much of a pain...I don't see any reason not to run a striped setup for this workload. It's damn near ideal.

I'm fairly sure everyone understand that it's not a redundant array but the premise was that it doesn't matter.
Victory requires no explanation. Defeat allows none.
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Raid for dumbies

Thu Oct 10, 2013 3:30 pm

Waco wrote:
I keep hearing this but I can tell you that we use consumer drives in very large RAID 5 and 6 arrays and they very rarely drop out from the horrors people attribute to them.


Like I said, it's doable, but it's not the best hardware to use when the idea is to remove as much risk as possible. There are ways to run commodity hardware in a production environment, it just requires data to replicated across several servers in near-realtime.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On