Personal computing discussed

Moderators: renee, morphine, Steel

 
Duct Tape Dude
Gerbil Elite
Topic Author
Posts: 721
Joined: Thu May 02, 2013 12:37 pm

Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 5:00 pm

What are the performance/reliability differences between a consumer SSD that is manually overprovisioned to the same levels as an enterprise SSD?

For example, if I took a consumer 500GB or 480GB (512GB actual capacity) SSD and made a 400GB partition as the only partition, how would it perform/wear compared to an enterprise 400GB (I assume also 512GB actual) SSD?

I have seen a few benchmarks comparing enterprise SSDs to consumer SSDs (http://www.tweaktown.com/articles/5392/ ... ndex7.html)
But also some benchmarks at Anandtech showing noticeably improved performance on overprovisioned SSDs, for example: http://www.anandtech.com/show/6884/cruc ... gb-120gb/3

Given how much more expensive enterprise SSDs are and how much "newer" consumer SSDs are, if I had to deploy something for production use, would it be better to choose 2-3x as many consumer SSDs and RAID 1 or RAID 10 them together instead of getting fewer enterprise SSDs?
EDIT: Also, is it a good idea to RAID 1 the same SSDs? Isn't that kind of setting them both up to fail at the same time?

My workload consists of many small file accesses less than the sector size, with perhaps a 10:1 ratio of reads:writes.

Thanks!
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 6:38 pm

Production in what sense? What kind of availability do you need? What's the average GB writes/day?

Without those it's pretty hard to gauge whether you could get by with consumer products.
Victory requires no explanation. Defeat allows none.
 
Duct Tape Dude
Gerbil Elite
Topic Author
Posts: 721
Joined: Thu May 02, 2013 12:37 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 9:01 pm

Waco wrote:
Production in what sense? What kind of availability do you need? What's the average GB writes/day?

Without those it's pretty hard to gauge whether you could get by with consumer products.

I don't get how this matters... Let me rephrase my question: what difference would I see between an overprovisioned consumer SSD vs an enterprise SSD of the same technology (ie: two MLC drives)?

My specifications shouldn't interfere with the performance and reliability of these two scenarios. But if you're really dead set on numbers or something figure a few dozen GB a day with heavy write amplification because of the small write size. I need as many nines of availability as my budget can afford, and budget is finite. I am not asking for a hundred nines, just as many as is feasible for as long as possible given a finite sum of money.

Finally, is it better for me to get x enterprise SSDs or 3x consumer SSDs that I manually overprovision?
 
UberGerbil
Grand Admiral Gerbil
Posts: 10368
Joined: Thu Jun 19, 2003 3:11 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 10:16 pm

I can't say for sure without looking at the specs for whichever enterprise drive you're considering, but I wouldn't assume "factory" overprovisioning is the only difference. There could be a variety of other factors / features in any particular enterprise line (NAND binning, ECC, controller logic to further reduce write amplification, etc) some of which may also enhance endurance (and others -- like encryption or backup power -- which may not). Whether any of those matter to you vs the cost-effectiveness of consumer drives is a separate question.
 
Duct Tape Dude
Gerbil Elite
Topic Author
Posts: 721
Joined: Thu May 02, 2013 12:37 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 11:03 pm

UberGerbil wrote:
I can't say for sure without looking at the specs for whichever enterprise drive you're considering, but I wouldn't assume "factory" overprovisioning is the only difference.
...
NAND binning, ECC, controller logic to further reduce write amplification, etc) some of which may also enhance endurance (and others -- like encryption or backup power -- which may not)

Aha! This is more of what I was looking for. I have heard that most modern SSDs have enough caps in them to finish what they are doing in case of a power loss. I can understand the performance benefits of encryption acceleration on SSDs, but I don't need that my this case (and I think some consumer SSDs have this, right?).

If someone could explain ECC behavior on SSDs better, that'd be great (specifically the difference between consumer SSDs' ECC or lack thereof vs enterprise SSDs). I thought all SSDs had to have ECC of some sort.

As for better controller logic/NAND binning/other endurance-enhancing measures, would it enhance the lifespan of a drive by a significant factor? I ask because I can probably get a consumer SSD for 1/3 to 1/6 the cost of an enterprise SSD, or alternatively get a bigger SSD that I overprovision by a lot.

One more thing I've been wondering: are the enterprise SSD NAND chips and controllers "older" technology compared to the consumer ones? I would be wary of an old controller/NAND vs a new controller with better algorithms/performance, even if the old controller and NAND are battle-tested.
 
UberGerbil
Grand Admiral Gerbil
Posts: 10368
Joined: Thu Jun 19, 2003 3:11 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Thu Jan 30, 2014 11:27 pm

They all have ECC, but not all ECC is created equal -- if you have more bits to devote to it and/or more processing power in the controller, you can cope with more entropy. I'm on my way out the door, and don't have a lot details to pass along anyway, but StorageReview has been doing Enterprise SDD reviews for a while and may be able to find some of the details you're looking for either in their reviews of particular models or in their forums.
 
Ringofett
Gerbil
Posts: 10
Joined: Sat Sep 10, 2011 5:13 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 1:39 am

Regarding NAND binning and longevity, I'd point to TR's very own endurance review. I've no doubt commercial SSD binned NAND might last a little better, but holy cow, they've already pounded those with mind-boggling, warranty-voiding numbers of writes and only a handful of bad sectors, not a single bad die, and not a single drive failure. So, I don't know that it'd matter, unless you're at the LHC or someone similar producing petabytes and intentionally will abuse your drive for years.

As for other features, I'll cede to others, except to say as far as ECC goes, I love btrfs/zfs/ReFS as a second line of defense for critical files vs just built in ECC.
 
Captain Ned
Global Moderator
Posts: 28704
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 1:49 am

OP:

Are you thinking that if you leave 100GB outside of the partition table on a consumer drive the firmware will add that unpartitioned space to its pool of spare cells and thus "add reliability"? That'd take a custom firmware.
What we have today is way too much pluribus and not enough unum.
 
Duct Tape Dude
Gerbil Elite
Topic Author
Posts: 721
Joined: Thu May 02, 2013 12:37 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 9:45 am

UberGerbil wrote:
They all have ECC, but not all ECC is created equal -- if you have more bits to devote to it and/or more processing power in the controller, you can cope with more entropy. I'm on my way out the door, and don't have a lot details to pass along anyway, but StorageReview has been doing Enterprise SDD reviews for a while and may be able to find some of the details you're looking for either in their reviews of particular models or in their forums.

Ooh thanks, I will have a look!

Ringofett wrote:
Regarding NAND binning and longevity, I'd point to TR's very own endurance review. I've no doubt commercial SSD binned NAND might last a little better, but holy cow, they've already pounded those with mind-boggling, warranty-voiding numbers of writes and only a handful of bad sectors, not a single bad die, and not a single drive failure. So, I don't know that it'd matter, unless you're at the LHC or someone similar producing petabytes and intentionally will abuse your drive for years.

As for other features, I'll cede to others, except to say as far as ECC goes, I love btrfs/zfs/ReFS as a second line of defense for critical files vs just built in ECC.

TR's endurance review is what has me so excited about using consumer drives :)
It is incredible how long these things last even after being hammered. I like your file system suggestions. We do have control over that and we'll be using Ubuntu server probably. I've heard ZFS can be memory hungry though. Any opinions on the file systems you mentioned if I'm looking for a lightweight mix between reliability and speed (if speed takes a noticeable hit)?

Captain Ned wrote:
Are you thinking that if you leave 100GB outside of the partition table on a consumer drive the firmware will add that unpartitioned space to its pool of spare cells and thus "add reliability"? That'd take a custom firmware.

Yes, am I mistaken? How does Anandtech get better performance numbers on a smaller partition then, and how does Samsung overprovision their drives via their Magician software? AFAIK the Magician software just created an empty partition on my 840. But I could be wrong! Enlighten me!
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:00 am

@Cap'n -

I think you're mistaken. If you set aside a partition, and (here's the crucial part) never write to it, then the flash controller should treat the cells which are initially assigned to that partition as part of the free pool. They will get used during wear leveling, just like any other other cells that have never been written (or cells that have been TRIMmed). That's how SSD wear leveling is supposed to work. The bottom line with manual underover-provisioning is to never touch some percentage of the drive's capacity; the easiest way to ensure that is to create a partition you never use.

Blocks of flash cells don't map one-to-one with logical sectors. The drive firmware remaps and shuffles things around at will behind the scenes, and can even relocate already written sectors on the fly if they are stored in cells which appear to be getting marginal.

@DTD -

Enterprise storage is typically tested more rigorously and has a longer warranty, and has firmware that is tuned for use in RAID arrays instead of as a single drive. Enterprise hardware may also use components with a wider operating temperature range. So these are all factors that may need to be considered as well, depending on your use case.
Nostalgia isn't what it used to be.
 
Duct Tape Dude
Gerbil Elite
Topic Author
Posts: 721
Joined: Thu May 02, 2013 12:37 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:15 am

just brew it! wrote:
@Cap'n - I think you're mistaken. If you set aside a partition, and (here's the crucial part) never write to it, then the flash controller should treat the cells which are initially assigned to that partition as part of the free pool. They will get used during wear leveling, just like any other other cells that have never been written (or cells that have been TRIMmed). The bottom line with underprovisioning is to never touch some percentage of the drive's capacity; the easiest way to ensure that is to create a partition you never use.

@DTD - Enterprise storage is typically tested more rigorously and has a longer warranty, and has firmware that is tuned for use in RAID arrays instead of as a single drive. Enterprise hardware may also use components with a wider operating temperature range. So these are all factors that may need to be considered as well, depending on your use case.

Thanks, just brew it!. That's some very informative stuff. I thought the term was "overprovisioning" but if it's actually "underprovisioning," I hope you all know what I intended. I'll keep using "overprovisioning" for thread consistency though.

The RAID firmware is a good point, though I think for for my purposes I'd only be considering RAID 1--which I would imagine benefits less from this firmware than a RAID 0/5/etc. I could be wrong!

The operating temperature is a great point, I saw some press releases advertising better SSD endurance at 50C+. Do we know how hot the TR endurance benchmark is running? I can't find temps on the 300TB or 500TB updates. Judging by the setup here: http://techreport.com/review/24841/intr ... periment/4 I would think these are running pretty cool. Anyone care to guess? <30C? <40C?

Out of curiosity, when you say:
The bottom line with underprovisioning is to never touch some percentage of the drive's capacity
is there any way to reclaim this? A secure erase or similar?

Again, thank you all for your answers and discussion so far. I'm learning a ton!
 
BIF
Minister of Gerbil Affairs
Posts: 2458
Joined: Tue May 25, 2004 7:41 pm

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:23 am

To the OP:

If your read-to-write ratio is 10:1, I don't think you will ever wear out this drive. Barring a catastrophic electrical event outside your control, this drive will probably outlive you, your dog, your long-lived pet Macaw, and maybe half your children. I doubt you'll need to overprovision it, but sure, if it makes you feel better, why not? Like the others have suggested, set aside 5% or 10 % in a partition that you never touch.

I never buy an extended warranty on stuff. Overprovisioning is exactly like doing that, except you're not giving cash to anybody, until you fill up your newly-smaller drive and have to buy another one. :P
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:26 am

Duct Tape Dude wrote:
just brew it! wrote:
@Cap'n - I think you're mistaken. If you set aside a partition, and (here's the crucial part) never write to it, then the flash controller should treat the cells which are initially assigned to that partition as part of the free pool. They will get used during wear leveling, just like any other other cells that have never been written (or cells that have been TRIMmed). The bottom line with underprovisioning is to never touch some percentage of the drive's capacity; the easiest way to ensure that is to create a partition you never use.

@DTD - Enterprise storage is typically tested more rigorously and has a longer warranty, and has firmware that is tuned for use in RAID arrays instead of as a single drive. Enterprise hardware may also use components with a wider operating temperature range. So these are all factors that may need to be considered as well, depending on your use case.

Thanks, just brew it!. That's some very informative stuff. I thought the term was "overprovisioning" but if it's actually "underprovisioning," I hope you all know what I intended. I'll keep using "overprovisioning" for thread consistency though.

The RAID firmware is a good point, though I think for for my purposes I'd only be considering RAID 1--which I would imagine benefits less from this firmware than a RAID 0/5/etc. I could be wrong!

The operating temperature is a great point, I saw some press releases advertising better SSD endurance at 50C+. Do we know how hot the TR endurance benchmark is running? I can't find temps on the 300TB or 500TB updates. Judging by the setup here: http://techreport.com/review/24841/intr ... periment/4 I would think these are running pretty cool. Anyone care to guess? <30C? <40C?

Out of curiosity, when you say:
The bottom line with underprovisioning is to never touch some percentage of the drive's capacity
is there any way to reclaim this? A secure erase or similar?

Again, thank you all for your answers and discussion so far. I'm learning a ton!

D'oh, yes overprovisioning not underprovisioning. It's early, I haven't had all of my caffeine yet. :oops:

Yes, a secure erase should reset all of the cells to "free". A file system that supports on-the-fly TRIM will also do this for you, though some people argue that on-the-fly TRIM can negatively impact performance. (I've seen a number of Linux users recommend disabling automatic TRIM in the OS, and running a daily or weekly batch trim instead. Not sure if there's an equivalent to this in Windows.)
Nostalgia isn't what it used to be.
 
Waco
Maximum Gerbil
Posts: 4850
Joined: Tue Jan 20, 2009 4:14 pm
Location: Los Alamos, NM

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:29 am

Well if the only concern is performance of the array, assuming you're using a reasonably good RAID card, most of the small writes won't be an issue unless they're random. You could avoid that particular issue if you use a data-journaling file system.

Manually over-provisioning will provide nearly the same benefit as the drives that do the same in the enterprise space - the difference is that the firmware (and hardware) in enterprise drives goes through a LOT more validation before release for both reliability and long-term performance.

Assuming you do regular backups and have a pool of hot spares AND lots of mirroring you might be able to get away with it. Perhaps you could use ZFS to avoid most of the trouble since you're looking at a few GB a day + ~10x that in reads. You'll need a beefy server but it might be worth at least looking into.
Victory requires no explanation. Defeat allows none.
 
Captain Ned
Global Moderator
Posts: 28704
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:43 am

just brew it! wrote:
@Cap'n -

I think you're mistaken. If you set aside a partition, and (here's the crucial part) never write to it, then the flash controller should treat the cells which are initially assigned to that partition as part of the free pool. They will get used during wear leveling, just like any other other cells that have never been written (or cells that have been TRIMmed). That's how SSD wear leveling is supposed to work. The bottom line with manual underprovisioning is to never touch some percentage of the drive's capacity; the easiest way to ensure that is to create a partition you never use.

Consider me educated. For some reason I simply assumed that the firmware would only look to the spare cells built into the firmware at the factory.
What we have today is way too much pluribus and not enough unum.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 10:45 am

Captain Ned wrote:
Consider me educated. For some reason I simply assumed that the firmware would only look to the spare cells built into the firmware at the factory.

There certainly could be some cheap thumbdrives and memory cards that work this way, but the write endurance will suck. Not something you'd want to do with anything that claims to be a "SSD".

Edit: It's not just about replacing worn-out cells with spares. Wear leveling attempts to ensure that individual blocks don't get more than their share of writes, spreading out the wear across the entire device. In order to do this effectively, there needs to be a pool of blocks available that the controller can re-use (after them erasing, if necessary). Over-provisioning and TRIM both help increase the size of this pool. Some controllers (e.g. SandForce) get even more involved, doing things like on-the-fly lossless compression to minimize the number of physical flash blocks needed to store data (and thereby reducing write wear); so one block of filesystem data may not even occupy one block worth of storage in the actual flash chips if the data is compressible.
Nostalgia isn't what it used to be.
 
SuperSpy
Minister of Gerbil Affairs
Posts: 2403
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 11:56 am

Over provisioning isn't going to impact reliability much, it's mainly a performance tweak that allows the drive's wear-leveling a little extra time before it _must_ start reorganizing the block mapping (which will kill your performance). The drive is going to move blocks around based on their write count regardless of it being in the over provisioned area. The only thing that will change the lifetime of the flash is the amount of data you write to it every day, which won't change if you have more or less free space.
Desktop: i7-4790K @4.8 GHz | 32 GB | EVGA Gefore 1060 | Windows 10 x64
Laptop: MacBook Pro 2017 2.9GHz | 16 GB | Radeon Pro 560
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 3:03 pm

just brew it! wrote:
(I've seen a number of Linux users recommend disabling automatic TRIM in the OS, and running a daily or weekly batch trim instead. Not sure if there's an equivalent to this in Windows.)


Given the way Windows 8 and 8.1 carry out TRIM I'd say Windows is already doing exactly that. Defrag.exe controls TRIM now and is scheduled for weekly use.

Windows 7 understands/does TRIM, but can't be setup to manually TRIM AFAIK. Meanwhile Vista and older don't support TRIM at all.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 3:10 pm

Ryu Connor wrote:
just brew it! wrote:
(I've seen a number of Linux users recommend disabling automatic TRIM in the OS, and running a daily or weekly batch trim instead. Not sure if there's an equivalent to this in Windows.)

Given the way Windows 8 and 8.1 carry out TRIM I'd say Windows is already doing exactly that. Defrag.exe controls TRIM now and is scheduled for weekly use.

That makes a lot of sense, provided what it does is just issue TRIMs for unallocated space instead of actually trying to defragment.
Nostalgia isn't what it used to be.
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 3:49 pm

just brew it! wrote:
Ryu Connor wrote:
just brew it! wrote:
(I've seen a number of Linux users recommend disabling automatic TRIM in the OS, and running a daily or weekly batch trim instead. Not sure if there's an equivalent to this in Windows.)

Given the way Windows 8 and 8.1 carry out TRIM I'd say Windows is already doing exactly that. Defrag.exe controls TRIM now and is scheduled for weekly use.

That makes a lot of sense, provided what it does is just issue TRIMs for unallocated space instead of actually trying to defragment.


Yeah, identified SSDs have TRIM applied and platters are made contiguous.

How to Use "Optimize Drives" to Defrag HDD and TRIM SSD in Windows 8 and 8.1
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 5:20 pm

Interesting. When I asked these same answers in putting together a 8 samsung 840 pro 256 gb raid 10, the discussion more led to the raid controller does what it wants and your partition size might not make a difference. I think jbi, you said that? Would have to dig up the thread.

I don't know that anyone has details of exactly how unpartitioned space is used.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Manually Overprovisioned Consumer SSD vs Enterprise SSD

Fri Jan 31, 2014 5:27 pm

Might've been that the RAID controller was allocating all of the space to the RAID array up front? I vaguely recall that thread, but not the details.

Edit: I think the issue may have been RAID controllers' interaction (or lack thereof) with TRIM. This stuff gets even more complicated when you introduce another layer of abstraction between the filesystem and the storage device!
Nostalgia isn't what it used to be.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On