TR Forums

Nearly a a decade and a half ago, I ripped my entire music collection (as it existed at that point) to WAV. The WAV files were encoded to MP3, and the resulting MP3s were stored on my home server for local playback, and for loading onto portable devices (initially optical media based, and eventually flash) for playback on the go. I've continued to add to it since then as I've acquired content.

The original WAVs were archived off to DVD-Rs (disk space was still kind of expensive back then), and never touched again... until last week.

Over the years I've fiddled with encoder settings, transcoded older files from MP3 to OGG instead of re-encoding directly from the source files, switched to FLAC for the raw rips, etc... the provenance of the encodes in much of the current OGG library is questionable. Did they come by way of MP3, and if so what where the encoder settings? Heck if I know. Replaygain was also applied haphazardly (and with the "apply directly to the audio samples" option instead of just setting the meta-data), before I really knew what I was doing.

The time has come to clean this mess up to get the best fidelity possible within the constraints of my hardware. I'm re-loading all of the WAV files from my original CD and vinyl rips, compressing to FLAC, keeping the lossless FLAC rips accessible on the network (instead of on a drawer full of DVD-Rs), and re-encoding all of the OGG files I use for mobile listening directly from the lossless sources.

I've just completed Phase I (reload of all of the raw CD rips from DVD-R archives). I'm about to start Phase II (reload the raw vinyl rips from DVD-R archives). Then I get to track down all the other raw rips that are backed up to external HDDs, or still resident on my server (I stopped archiving them to DVD-Rs some time back, when HDDs got cheap enough that burning to DVD-R was no longer worth the effort). Then comes The Great Re-Encoding... I've got two AMD FX systems (an FX-8320 and an FX-8350) at my disposal; I wonder how many days (and kilowatt-hours of power) it will take to turn all of the WAVs into FLACs, and all of the FLACs into OGGs?

This has also served as an interesting anecdotal data point for longevity of burned optical media. I just finished re-loading 168 DVD-Rs worth of WAV files containing the CD rips (over a half TB total) from discs which were burned anywhere from 5 to 13 years ago. Every single one of them read without incident. (Now I move on to the vinyl rips...) Disclosure: All discs were verified after burning, and if a batch of media had a tendency to exhibit repeated verify errors, I tossed it.

It is also amusing how ~700GB of data seems almost trivially small these days, when it was more like "OMG that's YOOOGE!" back in 2003.

I am very impressed you had the patience to burn so many DVDs, even if it was basically the only option at the time.

700 GB of data is still not an inconsiderable volume of data, though I imagine around here the users and readers are accustomed to lots of hard drive space.

What program are you using for your conversions? I typically use JRiver Media Center as it is multithreaded and supports a wide and customizable variety of formats, and allows you the option of choosing how many files to convert at once.

Acidicheartburn wrote:

I am very impressed you had the patience to burn so many DVDs, even if it was basically the only option at the time.

It was still small compared to the patience required to do the initial rips! (And that was my rationale for doing the DVD burns it in the first place -- "I never want to do THAT again!")

And the vinyl rips were an order of magnitude more painful than the CD rips. At least there weren't as many of those!

Acidicheartburn wrote:

700 GB of data is still not an inconsiderable volume of data, though I imagine around here the users and readers are accustomed to lots of hard drive space.

Yes, it's a "lot" of data, but a 1TB HDD is... what?... like $50 these days?

Acidicheartburn wrote:

What program are you using for your conversions? I typically use JRiver Media Center as it is multithreaded and supports a wide and customizable variety of formats, and allows you the option of choosing how many files to convert at once.

Ad-hoc Linux CLI scripts.

The scripts mostly exist already; I just need to create a higher-level script to scan all of the folders, clean up the file names (get rid of characters that are not valid for FAT32/exFAT file systems), and feed everything to pre-existing encoding scripts I've been using for years. The plan is to just divvy it all up, feed roughly 1/16 of the files to each core across the pair of 8-core systems, and let it run until it is done. I'm sure the temperature in my home office will go up a few degrees while it churns...

I'm myself surprised that you still have the patience to do the The Great Re-encoding. That is still a lot of work.

Funny, I'm currently waiting for 76 files to re-encode to 44khz 24 bit FLAC.  It's taking ages right now because I seem to be limited by my external HDD...

Considering you have the WAV files still and can put them all online (not that 700 GB is small), why are you actually re-encoding them? Why not use the raw WAV files going forward? Sure, FLAC saves some disk space but it isn't that much of a savings compared to WAV.

the wrote:

Considering you have the WAV files still and can put them all online (not that 700 GB is small), why are you actually re-encoding them?  Why not use the raw WAV files going forward?

 A 2.0 TB drive is just $50 these days.
http://pcpartpicker.com/products/intern ... 00,8000000

Since he's interested in encoding to FAT32-compatible file names, I assume that he has a portable music player, car audio system or phone where he'll be using some of these.

I am oddly enough ripping and re-encoding about 500 Blu-rays and DVDs at the moment putting them on the NAS for a HTPC at my folks place. Done 200 so far.

the wrote:

Sure, FLAC saves some disk space but it isn't that much of a savings compared to WAV.

FLACs are something like half the size of a WAV, or were the last time I did something like this.

bthylafh wrote:

the wrote:

Sure, FLAC saves some disk space but it isn't that much of a savings compared to WAV.

FLACs are something like half the size of a WAV, or were the last time I did something like this.

Yep, saves lots of valuable space.  Seeing as how I have 2.2 TB of music on a 3 TB that's almost entirely FLAC, there's no way I have enough HDD space for that in .wav.  I've seen conversion to FLAC reduce the file size to anywhere from 2/3 to 1/3 of the original size.  It varies, but at worst you can expect about 66% compression ratio.

yogibbear wrote:

I am oddly enough ripping and re-encoding about 500 Blu-rays and DVDs at the moment putting them on the NAS for a HTPC at my folks place. Done 200 so far.

I'm in the same situation. I didn't rip *any* of my movie collection at a decent quality back in the late 2000s because I didn't have the HDD space to support a high enough bitrate. Quantity over quality...ugh.

So now I have the fun task of ripping them one-by-one and encoding them nicely. Thankfully I'm limited 100% by the external USB 3.0 Blu-ray drive, but dammit, it's annoying to redo something.

My music collection? Thankfully I have the prescience to rip them all at 192 kbps MP3. It's good enough for my concert-damaged ears.

Waco wrote:

yogibbear wrote:

I am oddly enough ripping and re-encoding about 500 Blu-rays and DVDs at the moment putting them on the NAS for a HTPC at my folks place. Done 200 so far.

but dammit, it's annoying to redo something.

Quoted for truth.

Is there a reason why no one encodes their music to AAC (Advance Audio Coding) format?

Refresher for anyone who's forgotten what AAC means.

Advanced Audio Coding is an audio coding standard for lossy digital audio compression. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.

biffzinker wrote:

Is there a reason why no one encodes their music to AAC (Advance Audio Coding) format?

Refresher for anyone who's forgotten what AAC means.

Advanced Audio Coding is an audio coding standard for lossy digital audio compression. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.

.mp3 is ubiquitous and supported by pretty much everything.  I imagine .mp3 is good enough for people who don't care about lossless audio, and those who DO care about lossless aren't going to use AAC.  I have nothing against the format, however, and if JRiver Media Center could convert audio files to AAC I would definitely consider switching what's on my phone to that format.  I'd prefer to use FLAC on my phone but even with an add-in SD card I simply don't have the storage for the music I want on my phone in a lossless format.

The encoding part of it will go waaaaaay faster than reading all of those DVDs, which are absurdly slow by today's standards.  A lot of people have internet connections that are faster than you can read data off of DVDs.

I have FLAC copies of my music and also encoded the whole collection to mp3s for phone/portable player.  I don't know exactly how big your music collection is but my 732 hours of music (about 8000 tracks) only took about 4 hours to encode to MP3 with my 2500K, and encoding from WAV to FLAC is actually slightly faster than to mp3 in my experience.  Encoding to OGG is a bit slower, but still probably only 50% more time than FLAC.

These days it's really the video encoding that takes forever, especially if you want to do x265.  You basically will want a supercomputer for that.  

Acidicheartburn wrote:

Funny, I'm currently waiting for 76 files to re-encode to 44khz 24 bit FLAC.  It's taking ages right now because I seem to be limited by my external HDD...

How many simultaneous streams are you encoding? It is entirely possible that you're thrashing the drive with seeks reading the original files.

the wrote:

Considering you have the WAV files still and can put them all online (not that 700 GB is small), why are you actually re-encoding them? Why not use the raw WAV files going forward? Sure, FLAC saves some disk space but it isn't that much of a savings compared to WAV.

Well, once I re-encode them I would also like to back them up, either to an external hard drive or Blu-Ray discs. FLAC is lossless (so no fidelity degradation), and supports replaygain tags (which is an advantage over WAV).

For portable use, I can either fit a substantial chunk of the collection onto the player (Sansa Clip+ with Rockbox and a 128GB SD card) as FLAC lossless, or cram the entire collection onto the player with OGG (the Clip+ is the primary reason for having the OGG encodings).

biffzinker wrote:

Is there a reason why no one encodes their music to AAC (Advance Audio Coding) format?

Refresher for anyone who's forgotten what AAC means.

Advanced Audio Coding is an audio coding standard for lossy digital audio compression. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.

Yes, I'm familiar with AAC. OGG accomplishes essentially the same thing (better quality than MP3 at a given bitrate). All else being more or less equal, I'll tend to favor an open standard over a proprietary, patent-encumbered one; hence my choice of OGG for portable use. It is also implemented more efficiently than AAC on the Clip+, which helps battery life. (For the past few years I've used OGG for playback on my PC as well, but once this project is done I plan to shift to FLAC for non-mobile use.)

travbrad wrote:

The encoding part of it will go waaaaaay faster than reading all of those DVDs, which are absurdly slow by today's standards. A lot of people have internet connections that are faster than you can read data off of DVDs.

I have FLAC copies of my music and also encoded the whole collection to mp3s for phone/portable player. I don't know exactly how big your music collection is but my 732 hours of music (about 8000 tracks) only took about 4 hours to encode to MP3 with my 2500K, and encoding from WAV to FLAC is actually slightly faster than to mp3 in my experience. Encoding to OGG is a bit slower, but still probably only 50% more time than FLAC.

These days it's really the video encoding that takes forever, especially if you want to do x265. You basically will want a supercomputer for that.

The collection currently contains approximately 21,000 tracks. Some of those are duplicates (e.g. stuff I originally ripped from vinyl then subsequently re-purchased on CD, or bought from Amazon and downloaded the free MP3 "Autorip" version while waiting for the CD to arrive). Sorting out the duplicates is another aspect of this project. In most cases it should be pretty clear which copy should be kept; but in some cases, Amazon includes extra bonus tracks in the MP3 Autorip that aren't on the physical media so I can't just have a blanket rule of discarding the MP3 downloads when I have both versions of an album. There are also cases where there are multiple copies of the lossless CD or vinyl rips on the backups, because I subsequently discovered a glitch in the rip and went back and re-did it.

Reading in all of the DVDs is nearly done. I'm using two DVD drives in parallel to speed things up, and a custom script that automatically recognizes when one of my WAV backup discs is inserted, kicks off the copy, and ejects the disc when done. So all I need to do is reach over and insert another disc every time I hear a drive open. (Started out using 3 drives, but discovered that one of them was reading very slowly compared to the other 2, and also running very hot for some reason, so I stopped using that one.)

The time for a replaygain scan needs to be added to each encode as well... this goes faster than the actual encode, but does add a non-trivial amount of time.

Edit: The perfect record of error-free DVD-R reads has just been disrupted on one of the last DVDs of vinyl rips. Disc wouldn't read in one of the DVD drives, but read fine in the other one. Go figure.

@ JBI

I was running 4 files at once.  Not sure if that's overload or not for an external HDD.  I was getting well below the USB2 speeds I was connected with, though.

If you're reading multiple files in parallel the drive can't stream because it needs to keep seeking back and forth between the files. Random access workloads like that can easily reduce HDD performance by an order of magnitude or more.

Acidicheartburn wrote:

Funny, I'm currently waiting for 76 files to re-encode to 44khz 24 bit FLAC. 

Just an FYI: unless you are going to apply further digital processing to your recording, using 24bit files for simple playback purposes only achieves one thing: wasting storage space.
@JBI: impressive. 0.7TB ... that's a LOT of music.... even in 2016. Makes my 120GB seem meager by comparison. 
Didn't know the clip+ could handle 128GB cards. Always thought 32 was the limit. Good to know for the future.

Ifalna wrote:

@JBI: impressive. 0.7TB ... that's a LOT of music.... even in 2016. Makes my 120GB seem meager by comparison.

I've got 331 GB in ripped CDs plus 1.95 TB of concert ROIOs shared through DC++. Some of the people in my group have over 30 TB of shared concerts.

30TB? You mean video footage of concerts, right?
Difficult to imagine 30TB of audio data.

Every known audio recording (and there are many multiples for some shows) just for Pink Floyd totals somewhere around 3 TB. Genesis isn't far behind. If you're a completist, amassing 30 TB wouldn't be difficult (at least in finding the data). I also assume that these collections live on some server managed by a university post-doc and uses their fat pipe to amass & share such quantities of data.

Ifalna wrote:

@JBI: impressive. 0.7TB ... that's a LOT of music.... even in 2016. Makes my 120GB seem meager by comparison. 

That doesn't include the vinyl rips, or the stuff that never got backed up to DVD either. I'll have a final total in a day or two after I find everything and gather it together.

Ifalna wrote:

Didn't know the clip+ could handle 128GB cards. Always thought 32 was the limit. Good to know for the future.

Stock firmware has some internal limitations that effectively cap things at 32GB; you need Rockbox firmware to go larger. You also need to re-format the card to FAT32 on a Linux system; larger cards come formatted exFAT (which Rockbox can't handle), and Windows won't let you create FAT32 file systems on large media. It amuses me that I need to use Linux to create a Microsoft file system...

Heheh, I know that feeling. How often does some setup "derp" and I end up with some weird folder I can't delete. Booting into parted magic always solves these kind of problems for me.
Ofc my Clip+ is rockbox'd, stock Firmware was horrible.
@Captain Ned: Okay if you include multiples, I can see how you'll end up at these numbers.

just brew it! wrote:

Windows won't let you create FAT32 file systems on large media. It amuses me that I need to use Linux to create a Microsoft file system...

I've used BOOTICE in the past to create FAT32 file systems larger than Windows allows.
https://translate.google.com/translate? ... t=&act=url
(I'm using an older version of BOOTICE)

The chunk size for a 128 64 GB FAT32 filesystem ends up being 64 KB so if you wrote a 1 KB file your wasting 63 KB of space if I remember correctly.

Edit: Correction, 128 GB FAT32 filesystem nets you a 128 KB chunk size

(FAT32 formatted 32 GB filesystem as an example)

One thing I would note is that the Sansa Clip+ has been discontinued and the new model they came out to replace it doesn't doesn't have a Rockbox build and probably never will because it has so little memory and a different processor so it is essentially worthless for a large music collection.  You can still find some leftover stock of the Clip+ (for 3x the original price) but even those will soon be gone.

The battery in mine degraded to the point where I only had about 2-3 hours of battery life, and when I went to a look for a new one I realized you can basically get an android phone (with microSD) for the same price or cheaper now which supports just about any music format.  I preferred the physical buttons of the Sansa Clip+ for a portable player, but you can find some cables with buttons that will work (depending on the phone)

 I have about 100G on disc. I just put what I wanted to listen to next, no plan really, and it's a small part of my collection. Anyhoo what I do is so simple:

Stuff CD in player/burner.
Create dir with name of album and go in it.
cdparanoia -B

 To play:
Go in dir
xmms .

  I am terminally lazy ... oh no.

biffzinker wrote:

The chunk size for a 128 64 GB FAT32 filesystem ends up being 64 KB so if you wrote a 1 KB file your wasting 63 KB of space if I remember correctly.

Edit: Correction, 128 GB FAT32 filesystem nets you a 128 KB chunk size

Fortunately music files are large enough on average that the stupid chunk size does not end up wasting too much space on a percentage basis.

travbrad wrote:

One thing I would note is that the Sansa Clip+ has been discontinued and the new model they came out to replace it doesn't doesn't have a Rockbox build and probably never will because it has so little memory and a different processor so it is essentially worthless for a large music collection. You can still find some leftover stock of the Clip+ (for 3x the original price) but even those will soon be gone.

I picked up a spare right around the time they discontinued it.

travbrad wrote:

The battery in mine degraded to the point where I only had about 2-3 hours of battery life, and when I went to a look for a new one I realized you can basically get an android phone (with microSD) for the same price or cheaper now which supports just about any music format. I preferred the physical buttons of the Sansa Clip+ for a portable player, but you can find some cables with buttons that will work (depending on the phone)

My older one gets used in the car, or for listening at my desk at work, where I can keep it plugged in. I use the newer one for mobile listening. Eventually I will need to switch to using my phone, or carry an external battery pack for the Clip+

I've taken apart and repaired 1st gen Clip players (broken battery wires) a couple of times, but never a Clip+. I wonder if there's a source for the internal battery pack. The one in the original Clip was an oddball thing, little foil pouch with wires sticking out of it. I imagine the one in the Clip+ is similar.

just brew it! wrote:

The one in the original Clip was an oddball thing, little foil pouch with wires sticking out of it. I imagine the one in the Clip+ is similar.

So Lithium Polymer?

biffzinker wrote:

just brew it! wrote:

The one in the original Clip was an oddball thing, little foil pouch with wires sticking out of it. I imagine the one in the Clip+ is similar.

So Lithium Polymer?

I imagine so.