Is RAID 5/6 dead due to large drive capacities?

All things storage here: hard drives, DVD RW drives, little wicker baskets.

Moderators: morphine, Steel

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 3:46 pm

Have you considered FlexRaid? Seems to have a lot of cool features like not losing the whole array if two or more drives die.

Wish TR would review it.
Igor_Kavinski
Gerbil XP
 
Posts: 338
Joined: Fri Dec 22, 2006 2:34 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 5:11 pm

Obviously you can't go much beyond the probabilities in terms of calculating failure - but beyond that - has anyone ever dealt with a 14+ TB RAID 5 array and had it successfully rebuild consistently?

The math would suggest that a failure to rebuild would be seen very quickly even in very few rebuild trials.

This is tempting me to build a large array just to try to map this out experimentally rather than mathematically. Granted, it'd be with a single set of hardware, but if the array consistently fails to rebuild over a certain size at a certain rate I'm sure it'd be interesting (at least for consumer drives).
Z68XP-UD4 | 2700K @ 4.7 GHz | 16 GB | GTX 780 SLI | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 480 GB Extreme Pro
Waco
Gerbil Elite
 
Posts: 816
Joined: Tue Jan 20, 2009 4:14 pm

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 7:06 pm

I was going to post that if you are interested in stuff like that you might as well go with ZFS as it is free. While reading the wiki page of ZFS though I ran across an interesting note:

Silent data corruption

The worst type of errors are those that go unnoticed, and are not even detected by the disk firmware or the host operating system. This is known as "silent corruption". A real life study of 1.5 million HDDs in the NetApp database found that on average 1 in 90 SATA drives will have silent corruption which is not caught by hardware RAID verification process; for a RAID-5 system that works out to one undetected error for every 67 TB of data read.[27][28] However, there are many other external error sources other than the disk itself. For instance, the disk cable might be slightly loose, the power supply might be flaky,[29] external vibrations such as a loud sound,[30] the Fibre Channel switch might be faulty,[31] cosmic radiation and many other types of soft errors, etc. In 39,000 storage systems that were analyzed, firmware bugs accounted for 5–10% of storage failures.[32] All in all, the error rates as observed by a CERN study on silent corruption, are far higher than one in every 1016 bits.[33] Webshop Amazon.com confirms these high data corruption rates.[34]


http://en.wikipedia.org/wiki/ZFS

Note 28 links off here: http://queue.acm.org/detail.cfm?id=1866298 which goes in to more detail.
Tachyonic Karma: Future decisions traveling backwards in time to smite you now.
Convert
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3134
Joined: Fri Nov 14, 2003 6:47 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 7:14 pm

And again I would warn that statistical measurements are inherently fuzzy and not hard limits. After all, that's why they call it statistics.

The problem with this thread is that some read the MTBF as a hard-coded point in time after which the drive will 100% and irretrievably fail. That's not how statistics work, and MBTF is inherently statistical.
Life is hard; but it's harder if you're stupid. Big Al.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20641
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 8:09 pm

I think the fundamental issue here is that except for striping to get performance, RAID is all about trying to beat the percentages. Unless the possibility is 0% or 100% for any given event you will find people to which is has "never" happened and "always" happened.

To answer the original question, I personally believe the answer is "yes" for RAID 5. I don't know of any enterprise level storage vendor that doens't use RAID6 instead of RAID5 these days. Much lower odds at very little extra cost for a large array. This includes for storage systems with two and three terabyte drives. I also know of no enterprise level storage vendor that sells RAID1 or RAID10 arrays.

At the consumer level, RAID5 was never really an option due to cost and now that cost has been coming down, almost all controllers I've seen that support RAID5 also support RAID6. Since most people (TR gerbils are not most people) a not really ever going to build more than a four drive array, there is fundamentally no difference in disk overhead between RAID 10 and RAID6.

The only realy point of question in my mind is for arrays between four and eight drives. Four and below, use mirroring. Eight and above use RAID6 unless you are trying for crazy performance and even then, the RAID6 is likely not going to be the limiting factor in performance. In between those two points I suppose it depends on how much you want to pay for performance, or how much you want to pay for space.

So yes, IMHO, RAID5 is dead but not due to drive size but due to drive cost and RAID6.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1737
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 8:40 pm

Captain Ned wrote:The problem with this thread is that some read the MTBF as a hard-coded point in time after which the drive will 100% and irretrievably fail. That's not how statistics work, and MBTF is inherently statistical.

I hope I haven't given the impression that on an exact number of reads, writes, or time a drive will die. The URE rate and MTBF rate is merely a guidance. However, at a certain point you are bound to have one or more within a given time period IF you have enough drives. It becomes a certainty after a while. Most of my advice is from an enterprise viewpoint where URE rates and MTBF's actually have an impact. We have more than a few storage appliances which house about one hundred drives in each rack (essentially 2 JBODS connected to a SAS backplane + 1U server w/ R1 or R10 on the OS drives). On average we see maybe one or two drives die a year. At this level R5 would just be nonsensical.
Core i7 920 @stock - 6GB OCZ Mem - Adaptec 5805 - 2 x Intel X25-M in RAID1 - 5 x Western Digital RE4 WD1003FBYX 1TB in RAID 6 - Nvidia GTX 460
kc77
Gerbil Team Leader
 
Posts: 242
Joined: Sat Jul 02, 2005 2:25 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Wed Mar 20, 2013 9:46 pm

SecretSquirrel wrote:Since most people (TR gerbils are not most people) a not really ever going to build more than a four drive array, there is fundamentally no difference in disk overhead between RAID 10 and RAID6.


Thanks for your input in this thread SS. I think the point you made earlier about performance is pretty important when it comes to our personal uses too. If you can saturate a gigabit link with the array then it should fit most needs even for a gerbil.
Tachyonic Karma: Future decisions traveling backwards in time to smite you now.
Convert
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3134
Joined: Fri Nov 14, 2003 6:47 am

theory vs practice

Postposted on Thu Mar 21, 2013 12:05 am

Ryu Connor wrote:What can be said about the probabilities that exist that hasn't been said already?

What more can the math of odds really tell us? They give a % value that varies depending on a number of factors, but quantify that into something more than just a dice roll. The percentages are fascinating theory, but averages manifest in life in rather unpredictable ways. You could argue that due to the way human perceive life and remember events, that all odds regardless of their actual % distill into a simple dichotomy: did happen, didn't happen.


I can say this about probabilities. If you have a lot of data, there is a

100%

chance you will have failures. Plan for them.

Considering Joe Blow can afford quantities of data only dreamed of in the past and its only going to get cheaper, the 'ethusiast' should start learning about dealing with stuff now rather than later.

Some stuff from the ZFS crowd (not the only option, but what I went with) old wisdom is just that, old.

1 in 10^16 bits is a pure academic whiteboard pipe dream. The real world is many orders of magnitude worse. (yeah, ripped from the wiki page, which has the sources, theres way more in various mailing lists/forums/etc)


Silent data corruption

The worst type of errors are those that go unnoticed, and are not even detected by the disk firmware or the host operating system. This is known as "silent corruption". A real life study of 1.5 million HDDs in the NetApp database found that on average 1 in 90 SATA drives will have silent corruption which is not caught by hardware RAID verification process; for a RAID-5 system that works out to one undetected error for every 67 TB of data read.[27][28] However, there are many other external error sources other than the disk itself. For instance, the disk cable might be slightly loose, the power supply might be flaky,[29] external vibrations such as a loud sound,[30] the Fibre Channel switch might be faulty,[31] cosmic radiation and many other types of soft errors, etc. In 39,000 storage systems that were analyzed, firmware bugs accounted for 5–10% of storage failures.[32] All in all, the error rates as observed by a CERN study on silent corruption, are far higher than one in every 1016 bits.[33] Webshop Amazon.com confirms these high data corruption rates.[34]


If you have a decent sized array/NAS that is actively used it easily does way way more r/w than that every year, hell I've downloaded more than 67TB some months. (yay fios)


edit: I fail, someone beat me to it
blah blah blah signature blah blah blah
Bauxite
Gerbil Elite
 
Posts: 619
Joined: Sat Jan 28, 2006 12:10 pm
Location: electrolytic redox smelting plant

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 3:51 pm

Ok, I have no idea who uses Areca RAID controllers in business. Maybe for custom build stuff along with the LSI MegaRAID controllers? Most of the big vendors I've dealt with, Dell and HP, have their own solutions, the PERC and the SmartArray lines. That being said, here's what I have found. I will post the results/raw numbers/graphs when I get a chance to send the spreadsheet I've been working on to my home address so I can upload it to my geoblocked-from-work web space so y'all can download it.

SAS and SSD
HP DL380 G5 458565-001, P56 5/2/2011 firmware
2 x Xeon E5430 2.66 GHz
32 GB RAM
P400i 512 MB BBWC, 7.24 firmware
146 GB 15K 6G DP SAS 512547-S21
256 GB Samsung 840 Pro SATA
Set as RAID 1+0 with 2 drives, 256K stripe, for RAID test.
Set as 2 RAID 0 arrays with 1 drive each, 256K stripe, Windows Dynamic Disks, mirrored, for WIN test.
Set as 1 RAID 0 array with 1 drive, 256K stripe, for single drive test.

SATA
Dell PE2900ii, 2.7.0 firmware
2 x Xeon E5345 2.33 GHz
12 GB RAM
PERC 5/i 256 MB BBWC, 5.2.2-0072 firmware
500 GB 7.2K Western Digital
Set as RAID 1 array with 2 drives, 128K stripe, for RAID test.
Set as 2 drives on Intel 5000X chipset's SATA controller, Windows Dynamic Disks, mirrored, for WIN test.
Set as 1 drive on Intel 500X chipset's SATA controller for single drive test.
Set as RAID 0 array with 1 drive, 128K stripe, for single drive test.
For 15K SAS drives, same settings as the RAID tests for this system.

Testing Procedures:
SAS and SSD followed TR iometer testing procedures. Outlined here: viewtopic.php?f=5&t=86757
SATA followed TR testing procedures with the exception that a 100 GB test file was used instead of the whole disk.
OS was Win2K8 R2 Enterprise, fully-patched at the time of testing.

Observations:

The RAID mirror using SATA drives plateaued when the queue depth hit 4, in one instance when it hit 8. The software mirror continued to scale as the queue depth rose. The SATA drives closely tracked each other in performance up until the RAID mirrors plateaued. In all cases with the SATA drives, a mirrored drive configuration provided significant performance over a single drive. The single drive RAID configuration did not scale at all while the single drive SATA configuration did. This may be a quirk of the PERC 5/i controller.

The RAID mirror using SAS drives closely tracked the software mirror configuration. In 3 out of 4 cases, the RAID mirror appeared to be pulling away in performance as queue depth hit 32. Higher queue depths were not tested so it is unknown if the trend would have continued to a queue depth of 256. In general, the software mirror performed slightly better for most of the tests--though when we're talking a max of 50 IOps difference when the scores are going up to 800, it's not a huge difference. Mirror configurations produced significant performance advantages over a single drive, the largest being a score of twice the single drive in the Web Server test pattern at a queue depth of 32.

The RAID mirror using SSD drives tracked similarly to the software mirror in 3 out of 4 tests. The RAID typically performed better that the software mirror in these tests, sometimes up to 800 IOps with the scores reaching a high of 7000 to 15000. Of note, at queue depth of 32, the software and RAID mirror scores reached parity. The File Server test pattern was odd with the RAID mirror performing almost twice as much at queue depth of 4 as the software mirror, 8000 to 4500 IOps. RAID and software then reached parity at a queue depth of 8 and closely tracked up to a queue depth of 32. Single drive testing was not done on SSD configurations.

Further testing using the same HP 15K DP 6G SAS drives in the PERC 5/i system showed how much of a performance difference proper enterprise drives make compared to regular desktop drives. The PERC 5/i was competitive with the SmartArray P400i and performed far better than when it used the 7.2K SATA desktop drives. A SAS/SATA interposer may help increase the SATA drive performance but that testing is beyond the scope of this set of tests. Plus, I don't think many businesses use interposers, they just buy SAS instead of SATA.

At the highest queue depth tested, the P400i begins to pull away in performance versus the PERC 5/i. This may be a consequence of the P400i's 512 MB of cache versus the PERC 5/i's 256 MB of cache.

Conclusions:

The PERC 5/i seems to suck at higher queue depths with a mirror configuration. The drives attached to it were no screamers, but the software mirroring continued to scale at higher queue depths whereas the RAID did not. The P400i continued to scale at higher queue depths. Further testing will be done with the same 15k SAS drives used in the P400i to see if the PERC 5/i is just being held back by slow drives.

Hardware-based RAID can either be better or worse than software-based RAID, depending on the controller and drives. Often, the differences are not significant.

Mirrored drive configurations provided significant performance gains over single drive configurations.

Enterprise-grade drives, especially 15K dual-ported 6G SAS drives, easily outperform desktop-grade SATA drives. Though this should be obvious, it's interesting to see the large performance gap in the real world.

Raw Data

http://screenshots.rq3.com/monk/mirror.xlsx

Made in Excel 2010 but it works in LibreOffice's Calc too, though the graph labels didn't quite make it. I didn't split off the read versus write performance but the data's all there for anyone who wants to graph that stuff.

For the graphs, red and green are single drives. Green is a single drive on a RAID controller versus the red which was on the SATA controller for that system. The SAS system didn't offer a raw SATA link to test a non-RAID single drive. Orange is RAID mirror, blue is Windows-based mirror.
Last edited by Scrotos on Tue Mar 26, 2013 6:44 am, edited 5 times in total.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 4:42 pm

Ryu, at the risk of a ban or getting the thread locked, I will address you directly. I'm going to address you directly not as a moderator but as someone contributing to the thread. I was looking for real-life experience. You had some decent discussion with RAID 10/01 and probabilties, but the rest of it seemed to be a generic "RAID is not a backup" mantra and flat out dismissing UREs. Look, the articles out there at least try to back up their assertions.

You're assuming 10^16, most drive ratings are 10^15 which does change the equation. I mean, take a look at the specs you dug up plus WD Black ones:

http://www.wdc.com/wdproducts/library/S ... 771386.pdf
http://www.wdc.com/wdproducts/library/S ... 701276.pdf

Most rates I've seen are 1 in X, not 10 in X. You shift the 10 over and it's <1 in 10^15 which, heck, I'd still round up to 1. The Black is 1 in 10^14. But that's kind of academic because typically you wouldn't use 3.5" drives in modern storage arrays, you'd use 2.5" drives. And vendors of real SAS stuff don't typically quote their URE/NRE/whatever error rates. Case in point:

http://h18004.www1.hp.com/products/quic ... 44_div.PDF

Most 3.5" drives seem to be 1 in 10^15. The 2.5" Seagate Cheetahs, the 15K, are 1 in 10^16. However, I have no idea what generation of rebranded Seagate I'm getting from HP or Dell when I get drives from them for their servers. For those under service contracts who get new drives from these vendors, you know what I mean. I have a failed HP DG146ABAB4 / 431954-003 which is a Seagate ST9146802SS which I can see is 1 in 10^16 only AFTER I found the Seagate model: http://www.seagate.com/staticfiles/supp ... 84760d.pdf (section 5, page 23 of the PDF)

That doesn't help with planning if you can only find this info out AFTER you have purchased the product/solution.

If you're really concerned about this kind of stuff, I would think you would take a worst-case scenario and plan for that. You seem to be leaning in the non-paranoid everything-is-fine direction about storage arrays. Is your plan to just go to tape whenever an array fails? Can you handle that kind of downtime while a slow restore happens? If you do disk-to-disk backups, how is the other "disk" protected? Do you just run a bunch of redundant arrays in a SAN? If you're not concerned about this, why are you posting in the thread?

I don't know why you locked the thread. I took a break from it for the last week because I was tired of the pedantic nitpicking about the performance of mirrors. I put my money where my mouth was to at least give another, modern data point.

This thread actually is of interest to several posters in both the hobby and business sense. Locking it because... I don't even know why. People weren't just flat out agreeing with you? Please, just post your experiences and move on. I'd love to know what kind of RAIDs you've dealt with and the storage capacities, some real-life stuff instead of empty platitudes that you should have a backup plan and each RAID scenario is different for each application. That's not what this is about! And that's not even a "you have to have a bigger e-peen than me before I listen to you" thing, I just want to know if you have experience in what I'm asking about!

It's a real, specific question, about whether or not people with large capacity RAID arrays are seeing actual rebuild issues as was predicted by tech pundits. If you have actual experience, tell. Otherwise, please let us discuss and argue and postulate. This isn't R&P, I'm not on a pro-RAID 5/6 or against-RAID 5/6 kick, I just want to learn.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 6:59 pm

Scrotos wrote:Ryu


Wrex.

Some of this post is a good question some of it is stuff that should have been in a PM.

You're assuming 10^16


I did?

This is an example of how conversations go down hill. I used a singular example and an enterprise class drive example at that. That I didn't mention other examples does not constitute an assumption or even an intentional fallacy of omission. It was an intent to present an example of the probability metrics of a drive intended for RAID use. As discussed such metrics are fraught with variables depending on the hypothetical scenario. It is impossible to model all the variable you will face in real life in a forum post.

This is the crux of why I said this thread was reaching the end of discussion. My concern was that all that remained to be said from here forth is arguments over dice rolls and sniping over the fact that someone wasn't quite detailed enough. Not to mention possible battle lines forming over this pedantic and presumptive interaction.

If you're really concerned about this kind of stuff, I would think you would take a worst-case scenario and plan for that. You seem to be leaning in the non-paranoid everything-is-fine direction about storage arrays. Is your plan to just go to tape whenever an array fails? Can you handle that kind of downtime while a slow restore happens? If you do disk-to-disk backups, how is the other "disk" protected? Do you just run a bunch of redundant arrays in a SAN? If you're not concerned about this, why are you posting in the thread?


I'm not concerned with it? I said that?

I recall mentioning that these metrics should be factored into risk assessments. Such metrics would also matter for 9's of uptime calculations. Wouldn't hurt to have a general idea of the MTTR values for both a rebuild or restore. Training and guidelines set in your incident response policy would also be good.

Of course none of those things really apply to the average TR person. Even some businesses - depending on their size - don't bother with those.

Qualitative (anecdote experience) and quantitative (probabilities) metrics are neat stuff and there are distinct areas in IT where they matter (TCO calculations come to mind as an example). The risk is that other factors impede your ability to bring those metrics to the forefront, like money, scale, or business need. Another way to say it is sometimes we know there is a risk, but we accept it (sometimes willingly sometimes not). So I'm not saying the odds are irrelevant, I just acknowledge that the odds don't override meeting the need.

I don't know why you locked the thread.


To quote Barney Stinson, "Legen... Wait for it!"

I took a break from it for the last week because I was tired of the pedantic nitpicking about the performance of mirrors.


dary!

Next time please bring discussions of why something was or wasn't locked to PM. If you wish to reply to the portions of this post that were about why this thread was locked, do so in PM. That goes for everyone. As all this does is derail a thread that already derailed once before.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3598
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 7:10 pm

I don't get this. We're still discussing 10^15 over 10^16 instead of best practices?
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
morphine
Grand Admiral Gerbil
Silver subscriber
 
 
Posts: 10091
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 10:08 pm

Scrotos wrote:Ok, I have no idea who uses Areca RAID controllers in business. Maybe for custom build stuff along with the LSI MegaRAID controllers? Most of the big vendors I've dealt with, Dell and HP, have their own solutions, the PERC and the SmartArray lines. That being said, here's what I have found. I will post the results/raw numbers/graphs when I get a chance to send the spreadsheet I've been working on to my home address so I can upload it to my geoblocked-from-work web space so y'all can download it.


IIRC, PERC are rebadged controllers from an OEM. Dunno about the SmartArray stuff. Most of it seems to work it's way back to LSI in the end. I don't know who uses Areca cards in business either honestly. It has been a good solid card though and I would certainly recommend it to anyone looking at building a serious setup for home or small business. I'll have to pull down ioMeter and get some data. As a single data point though, an 8 disk RAID6 setup will sustain 250MB/s write and about 2GB/s read. Mind you that is running iozone with direct I/O and array caches fully enabled. The controller has a 256MB cache.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1737
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 22, 2013 10:32 pm

morphine wrote:I don't get this. We're still discussing 10^15 over 10^16 instead of best practices?


Well, if I may say, both numbers are valid estimates for different types of drives. It doesn't look like Ryu was cherry picking anything on purpose, but if anything, bringing attention to the fact that 10^15 drives also exist is equally important to the discussion. It's another data point for those still interested in it. For those not interested they can ignore it, or not :-? ..

SecretSquirrel wrote:IIRC, PERC are rebadged controllers from an OEM. Dunno about the SmartArray stuff. Most of it seems to work it's way back to LSI in the end.


Quite right! LSI pretty much owns the market; Intel, HP, Dell, IBM etc. all seem to tap them for controllers. Here is a list I often refer to when I'm curious what I'm getting in a server: http://forums.servethehome.com/raid-con ... odels.html

The nice thing is that almost all of these cards can be cross flashed back to vanilla LSI controllers with LSI firmware. Not something I would do for a customer's server but finding a rebranded 9260 8i with a BBU for $200 on ebay because no one knows it's true value after a little elbow grease is pretty darn nice!
Tachyonic Karma: Future decisions traveling backwards in time to smite you now.
Convert
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3134
Joined: Fri Nov 14, 2003 6:47 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Sat Mar 23, 2013 10:03 am

Benchmark result post updated with a link to the raw data.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Sat Mar 23, 2013 10:26 am

Convert wrote:Quite right! LSI pretty much owns the market; Intel, HP, Dell, IBM etc. all seem to tap them for controllers. Here is a list I often refer to when I'm curious what I'm getting in a server: http://forums.servethehome.com/raid-con ... odels.html

Their ubiquity means they're generally well-supported on Linux too. The SAS controller on the server we put together at work a while back had decent support out-of-box since it was based on a fairly common LSI chipset.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38099
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Sun Mar 24, 2013 11:03 pm

Convert wrote:Well, if I may say, both numbers are valid estimates for different types of drives. It doesn't look like Ryu was cherry picking anything on purpose, but if anything, bringing attention to the fact that 10^15 drives also exist is equally important to the discussion. It's another data point for those still interested in it. For those not interested they can ignore it, or not :-? ..


I agree the lower numbers are relevant. Generally people don't buy enterprise drives for home use and these same people probably don't use SAS drives spinning at 15K either. Likewise they probably don't use enterprise grade controllers. However threads are littered with people using relatively large consumer drives in RAID configurations with their "reliable" on-board RAID controllers. While the Internet (or browser... :o ) can tell you anything you want to know it doesn't take the place of people sharing their real life examples and outcomes in practice.

Having said that I think the discussion could be viewed as extremely helpful for those users who may be thinking about utilizing a particular RAID configuration. In addition to what RAID configurations are widely used and why, the thread gives them some things to think about when it comes time to purchase a new hard drive or even a HBA/RAID card.
Core i7 920 @stock - 6GB OCZ Mem - Adaptec 5805 - 2 x Intel X25-M in RAID1 - 5 x Western Digital RE4 WD1003FBYX 1TB in RAID 6 - Nvidia GTX 460
kc77
Gerbil Team Leader
 
Posts: 242
Joined: Sat Jul 02, 2005 2:25 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Mon Mar 25, 2013 4:41 pm

I'm looking to get a storage solution with 12+ bays that I can grow over time. Obviously, if the array will run into problems rebuilding, I'd spec for more bays with smaller drives instead of fewer bays with larger drives. That's the primary reason I started investigating the whole RAID 5/6 dead due to large drive claims.

I'm hoping it's a red herring, honestly, but I don't have enough reliable data points to draw any conclusions. I guess I could scour the storage review forums, but I know that there are several TR members who work with this kinda stuff so figured it'd be good to get their input.

My other alternative of course is to do 1+0 and not worry about the entire array being tapped for rebuilding. Better performance, too. Downside of course is that I throw away half my capacity for this solution. With enough spindles I can probably saturate a gigabit connection and I'm not performance-sensitive on this data, anyway, so I don't mind taking a performance hit going to RAID 6. And again, long rebuild times that degrade array performance aren't a huge concern, either.

So I thank y'all for your input! Especially SecretSquirrel, that's similar to the setup I was thinking of deploying myself.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Mon Mar 25, 2013 8:45 pm

Scrotos wrote:I'm looking to get a storage solution with 12+ bays that I can grow over time. Obviously, if the array will run into problems rebuilding, I'd spec for more bays with smaller drives instead of fewer bays with larger drives. That's the primary reason I started investigating the whole RAID 5/6 dead due to large drive claims.


My one comment here would be not to "grow" into it. Even with a good controller, expanding an existing RAID group is somewhat iffy. It is by far more dangerous for RAID5/RAID6 than a rebuild as it can potentially move every block on every disk to rebalance the load across all disks -- it depends on how the array calculates and stores the parity information. It is extremely painful and slow. My recommendation would be to fill all 12 slots and let your budget set the size of the disk. Either that or go RAID 10 and grow in four disk increments with each RAID group being independent and a seperate volume/network share. If you are really concerned about having some data on RAID6, you could also set up a 4 drisk RAID 10 group and an eight disk RAID6 group. Put your really important data on the RAID10 group and bulk data on the RAID6.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1737
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Tue Mar 26, 2013 6:41 am

BTW, mirror results are updated. The Dell PERC 5/i really sucked so I wondered if that was the controller or the drives so I tested it with the same 15K SAS drives from the HP SmartArray P400i. Yup. Totally the drives. Interesting to see just how massive a performance difference proper enterprise drives make. The PERC 5/i results were competitive with the P400i when using the same drives.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 29, 2013 8:35 pm

It occurred to me that I have some access to a real world results/anecdotes for a piece of customer SOHO equipment (instead of enterprise) doing RAID5. These results fit with what I figure a Gerbil here on TR might do. These are of course singular results with a relatively low drive count. As always statistics are a bitch and your mileage may vary.

4 Disk RAID 5 array:
Software Linux RAID 5

Drives: Non-enterprise
Model: WD1001FALS-00J7 (x4)
Capacity: 1,000,204 MB
URE: <1 in 10^15
MTBF: Unlisted
SMART Power On Hours: 40,004 (4.567 Years) and counting (8,760 hours in a year)
SMART Status
Dead drives within service time: 0
Array re-builds within service time: 0

The system in question is scheduled for replacement in the next few months.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3598
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 29, 2013 9:05 pm

It's a 3 TB array?

I should see if I can get some drive failure / rebuild numbers from work...
Z68XP-UD4 | 2700K @ 4.7 GHz | 16 GB | GTX 780 SLI | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 480 GB Extreme Pro
Waco
Gerbil Elite
 
Posts: 816
Joined: Tue Jan 20, 2009 4:14 pm

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Mar 29, 2013 9:41 pm

Waco wrote:It's a 3 TB array?


Aye, 3TB of usable space, 1TB of parity.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3598
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Sat Mar 30, 2013 12:43 am

I wouldn't expect that to fail in rebuild...like...ever. :lol:
Z68XP-UD4 | 2700K @ 4.7 GHz | 16 GB | GTX 780 SLI | PCP&C Silencer 950 | XSPC RX360 | Heatkiller R3 | D5 + RP-452X2 | HAF 932 | 480 GB Extreme Pro
Waco
Gerbil Elite
 
Posts: 816
Joined: Tue Jan 20, 2009 4:14 pm

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Apr 05, 2013 3:42 pm

SecretSquirrel wrote:
Scrotos wrote:I'm looking to get a storage solution with 12+ bays that I can grow over time. Obviously, if the array will run into problems rebuilding, I'd spec for more bays with smaller drives instead of fewer bays with larger drives. That's the primary reason I started investigating the whole RAID 5/6 dead due to large drive claims.


My one comment here would be not to "grow" into it. Even with a good controller, expanding an existing RAID group is somewhat iffy. It is by far more dangerous for RAID5/RAID6 than a rebuild as it can potentially move every block on every disk to rebalance the load across all disks -- it depends on how the array calculates and stores the parity information. It is extremely painful and slow. My recommendation would be to fill all 12 slots and let your budget set the size of the disk. Either that or go RAID 10 and grow in four disk increments with each RAID group being independent and a seperate volume/network share. If you are really concerned about having some data on RAID6, you could also set up a 4 drisk RAID 10 group and an eight disk RAID6 group. Put your really important data on the RAID10 group and bulk data on the RAID6.

--SS


What about expanding a RAID 10? It's supposed to be more "safe" because there are no parity calculations, ya?

Either way it seems like "don't worry about rebuilding 2 TB drives in a RAID 6" turned into "oh geez don't expand a RAID 6 with 2 TB drives, that's askin' fer trouble!" I assume it's more worser if you're at capacity when you grow rather than if you're at 300 GB and then add more drives.
Scrotos
Graphmaster Gerbil
 
Posts: 1043
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Fri Apr 05, 2013 7:19 pm

Scrotos wrote:
What about expanding a RAID 10? It's supposed to be more "safe" because there are no parity calculations, ya?

Either way it seems like "don't worry about rebuilding 2 TB drives in a RAID 6" turned into "oh geez don't expand a RAID 6 with 2 TB drives, that's askin' fer trouble!" I assume it's more worser if you're at capacity when you grow rather than if you're at 300 GB and then add more drives.


It's better to think about RAID 10 as adding redundancy to RAID 0. It doesn't check existing data against parity because none exists. It will only make sure that what exists on one side of the mirror is the same as the other. That's it. In some ways RAID 5 and RAID 6 address other issues not addressed by RAID 10. Generally RAID 10 and R5 or 6 are about addressing different workloads with different/additional concerns in mind.

In terms of expanding an existing array, I would agree it's usually a long process followed by a lot of worrying while the array is being rebuilt. However, there is a way around this...pooled storage. Remember how I said earlier it's best to break up your arrays after so many disks? Well if you add space say 5 or 6 disks at a time to a new array you can expand your pooled volume with little fuss. Nothing gets recalculated or moved on the previous array. The new array is merely bolted on. A resize of the logical volume to accommodate the new space will happen in seconds.
Core i7 920 @stock - 6GB OCZ Mem - Adaptec 5805 - 2 x Intel X25-M in RAID1 - 5 x Western Digital RE4 WD1003FBYX 1TB in RAID 6 - Nvidia GTX 460
kc77
Gerbil Team Leader
 
Posts: 242
Joined: Sat Jul 02, 2005 2:25 am

Re: Is RAID 5/6 dead due to large drive capacities?

Postposted on Sat Apr 06, 2013 10:05 am

Scrotos wrote:What about expanding a RAID 10? It's supposed to be more "safe" because there are no parity calculations, ya?

Either way it seems like "don't worry about rebuilding 2 TB drives in a RAID 6" turned into "oh geez don't expand a RAID 6 with 2 TB drives, that's askin' fer trouble!" I assume it's more worser if you're at capacity when you grow rather than if you're at 300 GB and then add more drives.


The problem with expanding any RAID array, beyond the array rebuild, is the filesystem that is already on it. When you expand the array, you have to expand the the partition containing the filesystem and the filesystem itself. Doing that with a consumer level RAID system scares me. I don't run a consumer level raid system and its a big enough pain that I found it easier to just build a second array when I need to expand and copy the data over.

I don't know my controller allows for expansion of a stripe or not. Ostensibly you could add another two drive mirror to an existing RAID10 and have it re-stripe everything, but that would require controller support.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1737
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Previous

Return to Storage

Who is online

Users browsing this forum: No registered users and 2 guests