Supid N00b question: HDD Transfer speeds

[ArniVidar]

Hey guys.

I was wondering if anyone here could explain to me one thing about HDD transfer speeds in general.

*SATA1 claims 1.5Gbps, SATA2 claims 3Gbps and SATA3 claims 6Gbps.
*Motherboard is specified SATA2
*HDD's are all specified SATA2
So far so good.

So here's the dumb question part:

What dictates the ACTUAL copy speeds on these things?
If my drives' theoretical maximum transfer speed is 375MBps, why do I get a sustained ~25MBps speed for hours on end when copying between two drives?

I'm not complaining about the speed (I'm in no hurry) but I've never understood why the actual speed is so far from the SATA2 specifications, and I wonder if anyone here can explain it in human?

[JustAnEngineer]

The mechanical parts of your hard-drives aren't as fast as the interfaces that connect them to the motherboard. You might get closer to the theoretical 300+ MB/s throughput for a tiny amount of data that is transferred to or from the cache on the drive, but the drive itself isn't that fast.

Having said that, a good 1+ TB 7200-rpm hard-drive (WD Caviar Black, Samsung F3, etc.) should be three to four times as fast as your 25 MB/s number. That doesn't stress SATA2, let alone SATA3.

[ArniVidar]

Hmm.. okay, so there's absolutely no benefit for me to upgrade to SATA3 then? The drives themselves will still be holding the speed down to far below SATA1 speeds anyway?

[just brew it!]

Hmm.. okay, so there's absolutely no benefit for me to upgrade to SATA3 then? The drives themselves will still be holding the speed down to far below SATA1 speeds anyway?

As with most things in life, it ain't that simple...

The actual copy speed is dictated by the slowest interface in the chain, *and* the sustained transfer speed of the drives in question, *and* the seek time of the drives (if you're copying lots of small files, or the hard drive is badly fragmented).

As JAE notes, 25 MB/sec is awfully slow for a SATA drive. Unless you're copying thousands of small files, or you've got a fragmentation issue (either of which will force the drive to seek a lot), something doesn't seem right here.

[DPete27]

I'd like to add that spindle speed and platter density (age) are additional factors that affect transfer speeds. Not to mention that read speeds are usually different from write speeds. (You'd be reading from one drive and writing to the other in a transfer) Here is a TR article on "modern" 7200 rpm 3.5" hdd's. Even with the aforementioned variables it seems odd that your transfers are 25MB/s unless you're copying to a USB flash drive? JBI's metion about size of the files being transferred is also relevant here, the smaller the files, the slower it goes.

[ArniVidar]

As JAE notes, 25 MB/sec is awfully slow for a SATA drive. Unless you're copying thousands of small files, or you've got a fragmentation issue (either of which will force the drive to seek a lot), something doesn't seem right here.

Well, the 25MBps that I have been getting for the past couple of hours were me copying 210GB of 700MB AVI's from an internal 1TB WD Green onto a USB connected (external housing) 320GB WD Blue that's not exactly new, so I guess 25MBps isn't TOO bad. I THOUGHT I was copying to an internal drive

While I was seeing a rather steady sustained 26-32MBps speed to the external USB drive, I was seeing bursts of 60+MBps from the internal Green into cache followed by a time of 0MBps. I guess I never saw the full potential of the Green drive in that copy operation.

Is fragmentation really an issue in today's world? I thought current operating systems were more proficient in keeping the fragmentation to a minimum?

My specs, by the way, are:
Windows 7 Enterprise, x64, running on a Gigabyte P43-ES3G mobo w/Intel Core2Duo E8400 processor @ 1.94GHz (windows is slowing it down on it's own) & 2x2GB OCZ Reaper chips @ 1066MHz.
System disk is a 120GB Mushkin Enhanced Chronos SATA3 SSD, and then there are nine 320GB->1TB HDD's connected both internally and externally.
Nothing fancy, for sure, but the machine it self shouldn't be MUCH of a bottleneck while copying files, should it?

Right now I've started a 52.000 file, 208GB MP3 move from a USB external onto the internal Green. There's absolutely no sustained transfer going on in the intenal (just bursts of 30-60MBps) but there is a somewhat steady 12-28MBps transfer going from the USB. I see what you guys mean by the size and number of the files making a difference.

[DPete27]

Yup, just as I suspected, its the USB 2.0 connection thats limiting your transfers to 25MB/s, get a USB3.0 enclosure (and a USB 3.0 add-on card for your mobo) and you'll see that number triple

[TwistedKestrel]

Also if you plan on getting an SSD anytime soon, drives from the two most recent generations actually are capable of saturating SATA 2. They're the reason why we even HAVE SATA 3 right now. If that's not in the cards though, don't worry about it!

[DPete27]

Also if you plan on getting an SSD anytime soon, drives from the two most recent generations actually are capable of saturating SATA 2. They're the reason why we even HAVE SATA 3 right now. If that's not in the cards though, don't worry about it!

System disk is a 120GB Mushkin Enhanced Chronos SATA3 SSD

the OP already has an SSD

Also, the 1.93Ghz on your E8400 is idle throttling. If you watch it when you're opening a program or something it should shoot up to 3Ghz. Overclock that thing!! I have a Gigabyte EP43-UD3L and an E8400 running at 3.6Ghz no problems, just change the FSB to 400mhz in the BIOS (make sure to make the proper adjustments so your ram frequency stays the same)

[ArniVidar]

Yup, just as I suspected, its the USB 2.0 connection thats limiting your transfers to 25MB/s, get a USB3.0 enclosure (and a USB 3.0 add-on card for your mobo) and you'll see that number triple

Very cool, thank you. USB3 controllers are fairly cheap these days, so I might do that. But then again, the externals are basically drives that I don't access often, so speed decrease on that end is not a major issue.

All this speculation has gotten me very interested in doing some tests between the internal drives and see how those behave.

[ArniVidar]

Overclock that thing!! I have a Gigabyte EP43-UD3L and an E8400 running at 3.6Ghz no problems, just change the FSB to 400mhz in the BIOS (make sure to make the proper adjustments so your ram frequency stays the same)

Hahahaha... this machine has everything required for overclocking, like huge fans and heat sinks, but It's just a file server and Facebook computer that shares a studio with my bed, so I'd rather have some peace and quiet in here

[moresmarterthanspock]

My advice, influenced by OCD and ADHD, is to live life in the fast lane. Get a PCI-E raid controller card that will do all of the processing independent of the CPU, and add 4 drives in a RAID 5 setup. If only 1 drive fails, data recover is usually good. I won't go into the technical aspects of it, but basically, 4 hard drives working in unison to make data transfer rates absolutely insane compared to 1 mechanical hard drive. But that is in my mindset. Srtiped raid. Fast. Higher possibility of data loss because there are more hard drives increasing the chance that one will go bad. But forget data loss. Speed? I'm all for it, at any cost.

[ArniVidar]

Speed? I'm all for it, at any cost.

LOL. Something tells me you don't sleep next to your computer running then, or have a very high tolerance for night noise

[Ryu Connor]

I'd also like to note that eSATA is pretty amazing. Generally better throughput than USB 3.0 and Windows treats the disk as if it were an internal drive instead of an external (few niche advantages to that).

You might compare the price difference between eSATA enclosures and a controller versus USB 3.0 enclosure and controllers and decide.

[Captain Ned]

I'd also like to note that eSATA is pretty amazing. Generally better throughput than USB 3.0 and Windows treats the disk as if it were an internal drive instead of an external (few niche advantages to that).

Agreed. All of my external backups run on eSATA and it just hauls.

[just brew it!]

Is fragmentation really an issue in today's world? I thought current operating systems were more proficient in keeping the fragmentation to a minimum?

It's *less* of an issue, but not completely gone. Copying lots of small files (which you apparently *weren't* doing in this case) is more of a factor.

As already noted, 25 MB/sec is about right for an external USB 2 drive.

[ArniVidar]

I'd also like to note that eSATA is pretty amazing. Generally better throughput than USB 3.0 and Windows treats the disk as if it were an internal drive instead of an external

My motherboard has a couple of eSATA ports, and I believe at least one of my housings has an eSATA port as well. I'll have to consider that, thank you. I've never seen eSATA cables though. I hope they aren't $5M per foot like HDMI cables

It's *less* of an issue, but not completely gone.

Oh.. I haven't defragged a drive since I stopped using XP. Perhaps I should run a program through my drives Do you know if O&O still the king of the game, or will the internal Windows defragger actually do it's job properly now?

[just brew it!]

I'd also like to note that eSATA is pretty amazing. Generally better throughput than USB 3.0 and Windows treats the disk as if it were an internal drive instead of an external

My motherboard has a couple of eSATA ports, and I believe at least one of my housings has an eSATA port as well. I'll have to consider that, thank you. I've never seen eSATA cables though. I hope they aren't $5M per foot like HDMI cables

If the housing has an eSATA port, maybe it came with a cable and you just chucked it in a drawer? I *know* I've got a few... just need to find them!

Failing that, they're not too bad at Newegg, and are dirt cheap at Monoprice.

It's *less* of an issue, but not completely gone.

Oh.. I haven't defragged a drive since I stopped using XP. Perhaps I should run a program through my drives Do you know if O&O still the king of the game, or will the internal Windows defragger actually do it's job properly now?

Unless you routinely create and delete lots of small files, or install/uninstall lots of applications, it isn't a big deal. And IIRC Win7 might run a defrag automatically once a week unless explicitly disabled...?

[Captain Ned]

I hope they aren't $5M per foot like HDMI cables

http://www.monoprice.com/products/searc ... word=esata

Not even close. BTW, you can get fully standards-compliant HDMI cables there for prices that just kill a Best Buy floor rep's commissions.

[Ryu Connor]

Unless you routinely create and delete lots of small files, or install/uninstall lots of applications, it isn't a big deal. And IIRC Win7 might run a defrag automatically once a week unless explicitly disabled...?

Correct. Vista and 7 have the Defrag as part of the task scheduler. If you miss the scheduled time it will begin defragging any time the PC is idle (and stop any time you start using the PC).

[Buub]

Unless you routinely create and delete lots of small files, or install/uninstall lots of applications, it isn't a big deal. And IIRC Win7 might run a defrag automatically once a week unless explicitly disabled...?

Correct. Vista and 7 have the Defrag as part of the task scheduler. If you miss the scheduled time it will begin defragging any time the PC is idle (and stop any time you start using the PC).

Well I'm not all that confident in Windows built-in defragger. In the past, it was based on Diskeeper, which IMHO is the worst defragger on the market.

Never used O&O, but I'm sure it's decent for a free defragger.

King of the Hill for commercial defraggers is Raxco PerfectDisk. You can schedule it to just do defragging when you're not using the machine, and you can also tell it not to defrag unless you're fragmented over a certain percent, which keeps it from defragging needlessly (even though it doesn't have the infinitely recursive defrag problem that Diskeeper seemed to suffer from for years).

[JustAnEngineer]

I'll agree that 25 MB/s is typical for USB 2.0. I have a USB3 external dock that solved that problem.

[Captain Ned]

King of the Hill for commercial defraggers is Raxco PerfectDisk.

Been using it for years. Still not sure if it isn't a slavish devotion to the concept of defrag brought on by the original Norton Speed Disk I used back when computer rooms had raised floors.

[Ryu Connor]

Well I'm not all that confident in Windows built-in defragger. In the past, it was based on Diskeeper, which IMHO is the worst defragger on the market.

There's no appreciable gain to be had by paying money for a defragger. I doubt there's a gain to be had by getting a different defragger for free.

The current Defrag tool was built by the filesystem team for Vista and 7. It most likely does things you wouldn't like, like ignores fragments larger than 64MB. Vista had an option to make it defrag those too, but 7 has removed the pointless option.

[just brew it!]

I generally don't bother with defraggers on Windows unless someone complains that their XP system is running really slow, in which case I'll do a defrag immediately following the obligatory malware cleaning.

On the Linux side, I'm not aware of any mature defrag tools for ext3/ext4 filesystems, and the lack thereof does not seem to be a major issue (at least, not with any of the usage patterns I've encountered).

Aside from the bad old days of MS-DOS and FAT-based filesystems, I've generally been of the opinion that for most use cases frequent defragmentation is both unnecessary, and potentially harmful. By definition, defragmentation moves files around on your disk. Latent hardware failures (RAM, motherboard, or disk drive) can potentially corrupt files as they're being moved, and you wouldn't even know. A file you haven't touched for months or years could silently get corrupted in the middle of the night by your automatically scheduled defrag...

I also cringe when I hear people say things like "My system is acting really flaky lately, I'm going to defrag the hard drive to see if that helps." Or flash the BIOS. These are exactly the *wrong* things to try when a system starts acting up unexpectedly; there's a good chance you've got a hardware issue of some sort, and this increases the odds that a defrag could trash your filesystem (or that flashing the BIOS could brick your motherboard).

IMO defrag should be done sparingly -- monthly at most, unless you've got an atypical use case like (say) testing software installers for large applications, where you're creating and deleting hundreds or thousands of files on a daily basis. And it's probably not a bad idea to run a defrag after installing OS service packs. But that's about it.

On the eSATA issue, I'd not been using it much because the last time I tried messing around with it in Linux (2-3 years ago) the eSATA hot plug functionality was a little squirrely to get working right. It appears that eSATA hot plug functionality is quite good in recent kernels; I guess I need to find some eSATA brackets for the systems that don't have one on the motherboard, so I can switch my drive dock and external drives over (most of the external enclosures I own actually have USB and eSATA ports on them, but I've been using them in USB 2.0 mode).

[DPete27]

I hope they aren't $5M per foot like HDMI cables

Not even close. BTW, you can get fully standards-compliant HDMI cables for prices that just kill a Best Buy floor rep's commissions.

Good God yes, I buy HDMI cables for $7 or less. It's digital, which means no amounts of gold plating or voodoo magic is going to improve the image/audio quality. Best Buy is exploiting unknowing consumers by playing off the long time "laws" of analog signaling. $50 for an HDMI cable.... preposterous

[just brew it!]

I hope they aren't $5M per foot like HDMI cables

Not even close. BTW, you can get fully standards-compliant HDMI cables for prices that just kill a Best Buy floor rep's commissions.

Good God yes, I buy HDMI cables for $7 or less. It's digital, which means no amounts of gold plating or voodoo magic is going to improve the image/audio quality. Best Buy is exploiting unknowing consumers by playing off the long time "laws" of analog signaling. $50 for an HDMI cable.... preposterous

And that $5 cable even has the gold-plated contacts... or you can get this one for only $3.50 at Monoprice, with gold plated contacts and ferrite cores to reduce radiated EMI!

I remain totally amazed at how cheap Monoprice's cables are; and I've never had quality issues with any of their cables either. (Their electronics is another story... but they're still my "go to" vendor for Ethernet, USB, video, etc. cables.)

[ArniVidar]

LOL! We pay $80 for a mediocre 3ft HDMI cable here in Iceland.

[just brew it!]

LOL! We pay $80 for a mediocre 3ft HDMI cable here in Iceland.

Maybe you could get several people together and place a bulk order at Monoprice for a bunch of cables to amortize the shipping cost. I believe they ship internationally, but I have no idea how much shipping and customs duties to Iceland would add.

They also have incredibly low prices on Ethernet cables. You can get high quality 10 foot Cat6 Ethernet cables from them for about $2!

[thegleek]

What this thread needs is some pictures to go along with all those words!