How I saved over 40GB and probably sped up disk access

— 11:00 AM on September 26, 2014

I was going to make this an Etc. post, but apparently that's not good SEO. Such is life Google.

Anyhow, after a long string of reviews without much time to catch my breath, I've been forced to pay attention to some neglected issues this week, including the fact that my 2TB RAID array is filling up. Doesn't help that I've also been downloading ~100GB worth of backups pulled from our old web servers.

My storage strategy in recent years has essentially been one of intentional neglect. Rather than trying to clean up my disks and make sure I don't have any extra gigabytes dedicated to unnecessary files, I've been counting on the fact that hard drive capacities tend to rise at a steady rate. If I upgrade the two drives in my main RAID 1 array every few years, I figure I shouldn't have to waste time managing the contents of that array in any great detail.

That approach has worked for the most part, but it's not ideal. Although hard drive capacities have continued to rise, it seems to me that the price per gigabyte hasn't dropped as much lately as in the past. (Maybe I'm just getting cheaper.)

There's another problem. Here's a look, for example, at my main data folder for the reviews I've done each year dating back to TR's beginnings.

I've stripped out uncompressed videos and camera RAW image files from the past few years, or the picture would be even more dramatic. Even so, my point remains. The amount of data I generate, both in this context and with family photos and video and such, is growing pretty rapidly.

And right now, I'd really rather not fork over the cash for a larger RAID array.

That leads me to a simple, old-school trick for saving storage space that I just deployed on my massive "archive" folder of review data, drivers, downloads, disk images, and more. The NTFS file system built into Windows has an optional compression routine. You can turn it on with the click of a checkbox.

I did so my with massive "archive" folder yesterday. This folder is housed on my RAID 1 of WD Green 2TB hard drives, and it took about 24 hours for my system to run through and compress the entire contents.

Although this folder contains a mix of compressible files (like spreadsheets) and incompressible ones (like JPEGs and AVIs), NTFS compression turned out to be a nice win overall. At the start, the directory of 558GB of data took up a full 558GB of disk space.  Afterwards:

I saved about 42GB simply by checking the box for NTFS compression.

NTFS compression isn't likely to slow down my access to these files, either. Way back when CPUs were tremendously slower and had fewer cores than today's processors, Microsoft's advice about NTFS performance focused on the balance between disk read/write speeds and CPU compression times. Given a fast enough CPU, it's possible that storing or retrieving compressed data could be faster if the spinning-platter disk had less total data to handle. I'll admit I haven't tested this theory, but nowadays, that tradeoff ought to be any easy win. Disks haven't gotten faster at nearly the rate CPUs have, and in my case, I have six cores with 12 threads mostly sitting idle the majority of the time.

That said, you wouldn't want to use NTFS compression on system files, like the Windows directory, or on entirely incompressible files, like a repository of MP3s or JPEGs. You won't save space by doing so, and you'll likely lose performance, since compression routines don't cope well with pre-compressed data.

Anyhow, that's how I saved over 40GB and probably sped up my disk access in the process. If space is tight and you're not ready to upgrade your storage capacity, NTFS compression is an easy, free, and painless way to free up some room.

Tip: You can use the A/Z keys to walk threads.
View options

This discussion is now closed.