Small File vs Big File Transfer Rates

All things storage here: hard drives, DVD RW drives, little wicker baskets.

Moderators: morphine, Steel

Small File vs Big File Transfer Rates

Postposted on Sun Jul 08, 2012 5:00 pm

Generally speaking is it true that smaller files transfer much slower than big files?
Assuming 2 similar HDDs, both about half full, both recently defragged, using the same type of interface - SATA3, SATA2...

I know it depends a lot on the size of the files but is there any general rule of thumb?

I ask because I see 100-150MB/s moving large files - .avi files, .wav, etc. - and as low as 10-20MB/s for small, 100K files (according to Windows transfer popup).

I'm sure the info I'm looking for is in the benchmarks, found all over the place but I just don't have the patience right now to sort through them...if you'll kindly indulge a lazy sort on a Sunday afternoon I'd really appreciate it. Thanks!
canoli
Gerbil XP
Silver subscriber
 
 
Posts: 332
Joined: Fri Jul 18, 2008 9:55 pm

Re: Small File vs Big File Transfer Rates

Postposted on Sun Jul 08, 2012 5:41 pm

Yes, mechanical drives are bad at random read/writes, but are good at sequential read/writes. Copying lots of smaller files is essentially random read/writes and will be slower because of the extra seek time required.
kumori
Gerbil Team Leader
Silver subscriber
 
 
Posts: 282
Joined: Sun Dec 18, 2011 12:11 am

Re: Small File vs Big File Transfer Rates

Postposted on Sun Jul 08, 2012 5:45 pm

I can't give a rule of thumb for the speed you should expect becuse it depends on too many different factors. In general though, you should expect a transfer of a buch of small files across file-system boundaries to take longer than moving a single big file of the same size. The reasons relate to the overhead in handling the directory entry and metadata setup for a bunch of small files instead of just one big file. In a mechanical HD, the small files can also be spread over the disk, even if the disk itself is not very fragmented.
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-770; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
chuckula
Gerbil Elite
Gold subscriber
 
 
Posts: 563
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Small File vs Big File Transfer Rates

Postposted on Sun Jul 08, 2012 5:56 pm

Yes, generally small files transfer at a much lower rate than large ones. This is because copying a file doesn't involve just copying the file data, it also causes the file's meta-data (directory entry, file allocation table, etc.) to get created/updated. This means that the hard drive is typically doing multiple seeks for each file that is copied; this is what kills the throughput on small files. The difference is significantly less on an SSD, since seeks are effectively instantaneous; you've still got the additional overhead of updating the meta-data (which will reduce throughput somewhat), but at least the meta-data update doesn't cause the drive to thrash.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Small File vs Big File Transfer Rates

Postposted on Mon Jul 09, 2012 5:05 am

Ah-ha...thank you all - so it's the metadata...every file has it, naturally the more files the more metadata that has to be written...makes perfect sense.

Thank you again. I was a little concerned seeing 10MB/s transfer speeds on a brand new drive. But because they're such small files - <100K - and I was copying 100s of them - I understand now why it took much longer than copying 1 large file.
canoli
Gerbil XP
Silver subscriber
 
 
Posts: 332
Joined: Fri Jul 18, 2008 9:55 pm

Re: Small File vs Big File Transfer Rates

Postposted on Mon Jul 09, 2012 7:51 am

Also keep in mind that on modern file systems adding a new file to a directory incurs a pretty big hit (at least relative to the effort needed to actually copy a small file) as the directories index has to be updated.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1583
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: Small File vs Big File Transfer Rates

Postposted on Mon Jul 09, 2012 9:08 am

SuperSpy wrote:Also keep in mind that on modern file systems adding a new file to a directory incurs a pretty big hit (at least relative to the effort needed to actually copy a small file) as the directories index has to be updated.

Yup. On the other hand, the indexing also makes meta-data lookups more efficient, so if you have directories with lots of files in them it is a net win (e.g. when copying a file, the system can figure out very quickly whether it already exists, to decide whether it needs to remove the old copy first). Modern file systems also do more caching, which helps. On the other OTHER hand, modern file systems are also journaled, which improves reliability but introduces additional complications when meta-data gets updated.

People tend to take file systems for granted, but it's a pretty involved topic, and an area where research is still ongoing... :wink:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Small File vs Big File Transfer Rates

Postposted on Mon Jul 09, 2012 9:15 am

just brew it! wrote:People tend to take file systems for granted, but it's a pretty involved topic, and an area where research is still ongoing... :wink:

And we've always been at war with WinFS.
It is one of the blessings of old friends that you can afford to be stupid with them. Ralph Waldo Emerson.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20276
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: Small File vs Big File Transfer Rates

Postposted on Tue Jul 10, 2012 3:12 pm

My general rule of thumb is use robocopy to copy large amounts of files or large files in Windows. The speed of copying files with the Windows GUI is inversely proportional to the number of files.
Flatland_Spider
Gerbil Elite
 
Posts: 852
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539


Return to Storage

Who is online

Users browsing this forum: No registered users and 2 guests