TR Forums

You have no choice anyway, at least if want one that's compatible with any devices.

Interface speed is NOT sustained drive write speed.

Captain Ned wrote:

Interface speed is NOT sustained drive write speed.

Bet you can find a bunch of mechanical HDDs advertising the exact same speed...

Speaking of which, I am surprised that SATA hasn't introduced a 12 Gb/sec variant yet. SAS drives (which use the same connector) can do 12 Gb/sec these days. I suppose it's become moot for consumer platforms since most are using M.2 drives for primary storage, and if there's a secondary device at all it is probably a mechanical HDD for bulk storage, where there isn't nearly as much benefit from a faster interface.

I just found the random numbers funny, that's all.

You still might want more than one or two quick drives, so M2 isn't a full replacement because of the floor space. I also wonder if in some cases there might be benefit to having hot devices away from the motherboard or PCIe cards. Anyway, 12Gbps maybe when/if dual actuator drives become more common?

meerkt wrote:

Anyway, 12Gbps maybe when/if dual actuator drives become more common?

My guess is that dual-actuator will remain a SAS-only enterprise feature. AFAIK the initial implementations treat dual-actuator drives as two separate LUNs (i.e. they effectively appear to be two drives, each with half the capacity). SATA standard does not have a good way of doing this without resorting to port multipliers, and historically those have had their own issues.

Sure, you could have drive firmware that effectively makes it behave like a self-contained RAID-0, but I'm not convinced there would be enough demand in the consumer space for such a beast to make it worthwhile for drive manufacturers to produce in volume, which in turn means they probably wouldn't be cost-effective. Performance-conscious consumers have pretty much all moved to SSDs, with mechanical HDDs relegated to bulk storage and backups (where the benefits of dual-actuator are less relevant).

A 20TB drive at 250-300MB/sec is going to be painful no matter what your uses are.

meerkt wrote:

A 20TB drive at 250-300MB/sec is going to be painful no matter what your uses are.

How many consumers would actually buy 20TB drives? I'd be willing to bet that nearly all high-capacity HDD sales are to enterprise users or for storing surveillance video. For the former, dual-actuator is a plus because it can roughly double the effective IOPS and bandwidth; for the latter you can double the number of simultaneous streams the drive can handle. So I suppose dual-actuator SATA might have a niche for surveillance, but that's probably about it.

What consumer use case is write speed limited?? Who cares if my Macrium image takes an hour longer to write because it's going to a spinner?

I wouldn't be comfortable knowing accessing all my data is 24+ hours away. Much more attractive if I can cut that in half.

meerkt wrote:

I wouldn't be comfortable knowing accessing all my data is 24+ hours away.

Well, if you want to copy 20 TB in seconds/minutes, you better go look for openings with Waco's employer.

Realistically, 4 hours would be cool, 8 okay, but I'll settle for 12-14 if we're talking 2020-2021.

Waco's employer is too far away and I hear the weather isn't that great.

Adjusting to the altitude is the hard part.

I gave up on RAID for this very reason. Built a 3TB Raid5 array years ago with Intel Matrix Raid. Took 22 hours(!) to initialize. Promptly broke it up and the disks are still lying unused. We (crazy hoarders) need a disruptive company in the HDD space desperately. One that delivers speed along with capacity with technology so revolutionary that Seagate and WD have no choice but to merge and play catch up for a few years just to survive. Please God, make Elon Musk dream of a world of limitless capacity. Make him see that people everywhere are running out of space and they don't want to store their precious family photos/videos in the cloud.

I initially started my Synology NAS with a pair of drives (capacity of one with one redundant drive). When a third drive came on sale, I installed it and allowed the Synology software to reconfigure the array on its own to a capacity of two with one redundant drive Sure, it took almost three days to accomplish this feat, but the NAS stayed available the whole time, so it wasn't a problem. On a later sale, I picked up a fourth drive and designated it as a hot spare, so the NAS will automatically add it to the array if one of the three on-line drives dies.

That's what bothers me. 3 days is too long. Ample time for another drive to fail and take the whole array with it. Not everyone is good at keeping current backups.

meerkt wrote:

I wouldn't be comfortable knowing accessing all my data is 24+ hours away. Much more attractive if I can cut that in half.

While it would certainly be cool (from a geek cred standpoint) to be able to move that much data that fast, I can't think of a consumer use case where I'd actually need to do that. Given that most other people almost certainly feel the same, what you're talking about here is a tiny niche market.

Igor_Kavinski wrote:

I gave up on RAID for this very reason. Built a 3TB Raid5 array years ago with Intel Matrix Raid. Took 22 hours(!) to initialize. Promptly broke it up and the disks are still lying unused.

I've never personally used Intel Matrix RAID... but with Linux MD RAID you can start using the array immediately while the array finishes initializing in the background. If Intel does not allow this, then that's a problem.

Igor_Kavinski wrote:

We (crazy hoarders) need a disruptive company in the HDD space desperately. One that delivers speed along with capacity with technology so revolutionary that Seagate and WD have no choice but to merge and play catch up for a few years just to survive.

See my previous post. Very few consumers need/want that kind of speed or capacity. All of the smaller, potentially disruptive HDD companies have been assimilated into WD, Seagate, and Toshiba at this point, and nobody else is going to want to sink that sort of R&D money into what is effectively a commodity industry. Future advances are going to continue to be evolutionary/incremental, and the drives incorporating any new tech will be costly enough that they will be marketed to enterprise and Cloud users.

Igor_Kavinski wrote:

Please God, make Elon Musk dream of a world of limitless capacity. Make him see that people everywhere are running out of space and they don't want to store their precious family photos/videos in the cloud.

Current drives are already more than large enough to store family photos/videos.

Igor_Kavinski wrote:

That's what bothers me. 3 days is too long. Ample time for another drive to fail and take the whole array with it.

That's why there's RAID-6 and ZFS. More than 1 level of redundancy makes the chance of data loss due to multiple failures much less likely, since it would take 3 failures within the rebuild interval to take out the entire array.

Many large-scale enterprise/cloud storage systems use arbitrary "k of n" erasure coding, where data is written in n stripes but any k stripes are sufficient to recover the original data. So up to n - k failures can be tolerated before the array loses data. The values of n and k are chosen when the array is set up, based on capacity, cost, performance, and reliability constraints.

This is also why all RAID (and RAID-like) solutions should be configured for automatic scrubbing - the system periodically scans the array, looking for latent bad sectors which were silently mis-written, and media defects which have developed since the last scrub. This greatly reduces the odds of getting a nasty surprise down the road. (My home-brewed NAS scrubs its arrays monthly.)

Igor_Kavinski wrote:

Not everyone is good at keeping current backups.

And that's a separate issue; RAID is not (and never was) intended to be a complete replacement for backups. RAID only protects you from individual drive failures. It does NOT protect you from user error, malicious tampering, catastrophic PSU failures that fry drives, fire, flood, theft, nearby lightning strikes, earthquake, tornado, etc.

We wouldn't really need drive level RAID if someone would only implement platter/head level RAID within the drive itself. Yes, it would be expensive but it would still sell well because some people would just prefer to avoid dealing with data loss more than others.

Igor_Kavinski wrote:

We wouldn't really need drive level RAID if someone would only implement platter/head level RAID within the drive itself. Yes, it would be expensive but it would still sell well because some people would just prefer to avoid dealing with data loss more than others.

That doesn't help with a lot of common drive failure modes. Head crashes frequently snowball across platters because the debris from the initial crash flies around inside the drive and causes additional head crashes on other platters. Furthermore, many of the other components - some of the electronics, the spindle motor, head actuator - would still be shared across multiple platters. You'd be nearly doubling the cost (or halving capacity) for only a marginal reliability increase.

I'm sure that the engineers can think of something better if only the management shifts their focus from profit to reliability. Currently, they don't care that much about reliability coz they know people can buy more drives to safeguard their data or send back the failed drives to them for data recovery.

They already have come up with something better that meets the needs of most consumers - it's called an "SSD". Future high capacity HDDs will be marketed almost exclusively to enterprise customers, where cost per byte stored is king.

Well, I'm still waiting for an affordable 4TB SSD to put into my laptop...

Another way to look at it:

Like it or not, the Cloud revolution has already happened. Most people stream their media over the internet, and store their stuff in the Cloud. So the focus for high-capacity storage solutions is on Cloud providers. And Cloud providers have enough redundancy that they would rather deal with occasional drive failures than pay more for the raw storage (as long as failure rates remain "reasonable"). These are the customers who are driving the bleeding edge of the HDD market now.

Even if HDD reliability increased 10-fold, Cloud providers would continue to build a lot of redundancy into their systems to ensure availability even in the event that an entire server goes down. Those erasure coding methods I mentioned a few posts back? When dealing with systems "at scale" they are often applied across servers (not just across drives), since a storage pool contains multiple servers anyway. So you can lose several servers before system availability is impacted or data is lost.

It's really a whole different way of looking at storage reliability, compared to what a typical home user deals with.

just brew it! wrote:

I can't think of a consumer use case where I'd actually need to do that.

It's not just about speed but also data survivability in cases of imminent failure. Tape might be slow but is unlikely to vaporize all data in one fell swoop. RAID is not always an option. I hope future HDDs, or an entirely new medium, could achieve reasonable speed/capacity ratios, or improved reliability.

Igor_Kavinski wrote:

Please God, make Elon Musk dream of a world of limitless capacity.

That might yet happen, if there are uses on Mars.

Igor_Kavinski wrote:

I'm still waiting for an affordable 4TB SSD to put into my laptop...

Decent 2TBs can be had for <$200. Depending on your definition of "affordable" you might have your 4TB in 2-3 years.

meerkt wrote:

I wouldn't be comfortable knowing accessing all my data is 24+ hours away. Much more attractive if I can cut that in half.

Any particular datum is just 5ms away (time it takes for the average hard-disk arm to move to a particular track).

Its like saying "My Harry Potter Book is 4 hours away" because it takes 4-hours to read... even if its sitting next to you.

---------------

Realistically, there is no reason for me to go for 20TB drives. I'd personally have more use out of multiple-arrays of smaller drives. Not necessarily for the rebuild time (though smaller capacity helps), but mostly because redundancy is important. 4x 6TB drives in RAID5-ish for 18TB of usable data will be significantly more reliable than a singular 20TB hard drive (with singular point of failure).

IBM's quantum supercomputing result was computed on hard drives by the way: which suggests that hard drives remain useful even in the realm of compute. It needed 64PBs for a 53-qubit simulation (128PBs for 54-qubits, and 256PBs for 55-qubits). As a hobby programmer, I'm curious how I can unlock the compute-potential of a ton of slow-TBs.

meerkt wrote:

Decent 2TBs can be had for <$200. Depending on your definition of "affordable" you might have your 4TB in 2-3 years.

By then, I might need 8TB or more in my laptop. I like to have all my stuff in one place rather than plugging/unplugging external drives and copying needed data.

Igor_Kavinski wrote:

meerkt wrote:

Decent 2TBs can be had for <$200. Depending on your definition of "affordable" you might have your 4TB in 2-3 years.

By then, I might need 8TB or more in my laptop. I like to have all my stuff in one place rather than plugging/unplugging external drives and copying needed data.

Your use case is an outlier. You're just going to need to wait for SSD prices to come down.

meerkt wrote:

It's not just about speed but also data survivability in cases of imminent failure. Tape might be slow but is unlikely to vaporize all data in one fell swoop. RAID is not always an option. I hope future HDDs, or an entirely new medium, could achieve reasonable speed/capacity ratios, or improved reliability.

The problem is that there just isn't a market for a tech that sits between disk and SSDs at the moment (also, you should see how much fun it is to repair a tape that snaps while running at full speed).

Cloud vendors are pushing HDDs down as cheap as they're going to get. Individual drive repair times are going to go up commensurately with drive size (mitigated slightly by faster drives over time). If you want fast repairs, you can't do it with traditional methods. It's a little surprising that distributed rebuild tech hasn't caught on more with people, but I think that's mostly because almost nobody wants to run 10+ drive arrays at home. Spreading the rebuild across every head gets you back to protection much faster, if at a bit of a cost of space. The copyback for a drive replacement takes just as long as a standard rebuild, though. There's not much way around that outside of cross-server / cross-disk erasure systems, and there aren't any good ones for home use that aren't insanely cumbersome or badly implemented.

There's not much way around that outside of cross-server / cross-disk erasure systems, and there aren't any good ones for home use that aren't insanely cumbersome or badly implemented.

Not to mention that home users generally don't want to spend much more than a car payment on storage, as opposed to the full cost of the car...

K-L-Waster wrote:

Not to mention that home users generally don't want to spend much more than a car payment on storage, as opposed to the full cost of the car...

Agreed, and usually less than that.