TR Forums

Generally speaking is it true that smaller files transfer much slower than big files?
Assuming 2 similar HDDs, both about half full, both recently defragged, using the same type of interface - SATA3, SATA2...

I know it depends a lot on the size of the files but is there any general rule of thumb?

I ask because I see 100-150MB/s moving large files - .avi files, .wav, etc. - and as low as 10-20MB/s for small, 100K files (according to Windows transfer popup).

I'm sure the info I'm looking for is in the benchmarks, found all over the place but I just don't have the patience right now to sort through them...if you'll kindly indulge a lazy sort on a Sunday afternoon I'd really appreciate it. Thanks!

Yes, mechanical drives are bad at random read/writes, but are good at sequential read/writes. Copying lots of smaller files is essentially random read/writes and will be slower because of the extra seek time required.

I can't give a rule of thumb for the speed you should expect becuse it depends on too many different factors. In general though, you should expect a transfer of a buch of small files across file-system boundaries to take longer than moving a single big file of the same size. The reasons relate to the overhead in handling the directory entry and metadata setup for a bunch of small files instead of just one big file. In a mechanical HD, the small files can also be spread over the disk, even if the disk itself is not very fragmented.

Yes, generally small files transfer at a much lower rate than large ones. This is because copying a file doesn't involve just copying the file data, it also causes the file's meta-data (directory entry, file allocation table, etc.) to get created/updated. This means that the hard drive is typically doing multiple seeks for each file that is copied; this is what kills the throughput on small files. The difference is significantly less on an SSD, since seeks are effectively instantaneous; you've still got the additional overhead of updating the meta-data (which will reduce throughput somewhat), but at least the meta-data update doesn't cause the drive to thrash.

Ah-ha...thank you all - so it's the metadata...every file has it, naturally the more files the more metadata that has to be written...makes perfect sense.

Thank you again. I was a little concerned seeing 10MB/s transfer speeds on a brand new drive. But because they're such small files - <100K - and I was copying 100s of them - I understand now why it took much longer than copying 1 large file.

Also keep in mind that on modern file systems adding a new file to a directory incurs a pretty big hit (at least relative to the effort needed to actually copy a small file) as the directories index has to be updated.

SuperSpy wrote:

Also keep in mind that on modern file systems adding a new file to a directory incurs a pretty big hit (at least relative to the effort needed to actually copy a small file) as the directories index has to be updated.

Yup. On the other hand, the indexing also makes meta-data lookups more efficient, so if you have directories with lots of files in them it is a net win (e.g. when copying a file, the system can figure out very quickly whether it already exists, to decide whether it needs to remove the old copy first). Modern file systems also do more caching, which helps. On the other OTHER hand, modern file systems are also journaled, which improves reliability but introduces additional complications when meta-data gets updated.

People tend to take file systems for granted, but it's a pretty involved topic, and an area where research is still ongoing...

just brew it! wrote:

People tend to take file systems for granted, but it's a pretty involved topic, and an area where research is still ongoing...

And we've always been at war with WinFS.

My general rule of thumb is use robocopy to copy large amounts of files or large files in Windows. The speed of copying files with the Windows GUI is inversely proportional to the number of files.