TR Forums

One of my friends has indicated that some of his older (pressed) audio CDs have started to show signs of deterioration (increased incidence of skips & errors), in spite of the fact that they are scratch-free and have been stored in a reasonable environment. I have a fairly sizeable CD collection -- 1000+ CDs, some of which date back to the 80s -- so this concerns me.

I've decided to rip all of my CDs to WAV format, for archival purposes. In the process of doing so, I will also re-encode my MP3 collection, since the quality of MP3 encoders has improved over the past few years. (No, this is not for file sharing... personal use only.) This also ties in with my desire to set up a home multimedia server (probably wireless?) at some point, but that's a topic for another thread...

The plan is to keep the MP3s accessible (so I can easily listen to them whenever I want), and archive the WAVs (since they are lossless).

A few questions and observations --

- Anyone have any thoughts on what the best archival medium for the raw WAV files would be? I'm on the fence right now between getting a DVD burner, or buying enough OEM hard drives to hold 'em all. The hard drives wouldn't have to all be online simultaneously... just fill one up, and swap in the next one. In fact, storing them off-site (e.g. at a friend's house) would probably be prudent. But is the archival life of a hard drive more or less than DVD-R discs?

- On the ripping side, it seems that Exact Audio Copy is pretty hard to beat. Paired with a drive that can report C2 error pointers (e.g. the TDK CDRW4800B I currently have in my main rig), it gives near-flawless rips, even from CDs that are in questionable condition.

- I've noticed what appears to be a quirk in EAC... seems that you can actually improve the effective ripping speed by lowering the configured ripping speed (under EAC -> Drive Options -> Offset/Speed). With the default config, it seems to start out really fast on the first track, but eventually drops down to around 4x. OTOH if I explicitly limit it to 16x, it seems to maintain a higher extraction rate overall. My guess is that it gets errors when trying to extract at the maximum speed, and drops down to 4x. Maybe by configuring EAC for a lower speed, the errors are prevented, and the "downshift" to 4x is avoided.

- For the MP3 encoding, I'm using the latest LAME, with the "--preset standard" option. The "--preset extreme" option is supposed to yield better quality, but I'm hard pressed to hear a difference... and the resulting MP3 files are significantly larger. Anyone have any opinions to offer on the various LAME presets and options?

- Yes, I know OGG format gives smaller files than MP3 for a given sound quality... but I have a portable that can't play OGG format, so OGG is not an option for me at this time.

- I'll probably be starting some additional threads on home multimedia servers / DVRs / HTPCs soon, since that seems to be where this is heading. But where to put the threads? I think we could use a separate forum for this sort of stuff... maybe a Multimedia forum? Hey Damage... you listening? Whatcha think?

The only problem I see with storing music on DVDs is that they can get what's called "laser rot." I'm not sure if this has been fixed or if it's even something to worry about now. I think a large 250 GB hard drive would be a better option unless you need something smaller.

I'm pretty impressed with the Blu-Ray DVD Disc technology that allows up to 27 GB on a single-sided disc. That may be worth waiting for to archive a lot of WAV files.

Use good media, Verbatim is the best <imo> for archiving. You won't have a problem with cd-r's going to **** on you if you use a quality disc / burner to begin with.

NeXus 6 wrote:

The only problem I see with storing music on DVDs is that they can get what's called "laser rot." I'm not sure if this has been fixed or if it's even something to worry about now. I think a large 250 GB hard drive would be a better option unless you need something smaller.

Well.. for storing the raw WAVs, a 250GB hard drive won't be anywhere near enough... I'd need several. But if you look at the $/GB cost of DVD-R (using quality name-brand media), it isn't that much cheaper (within a factor of 2) than OEM hard drives. So it all comes down to which would be more reliable over the long term.

I'm pretty impressed with the Blu-Ray DVD Disc technology that allows up to 27 GB on a single-sided disc. That may be worth waiting for to archive a lot of WAV files.

If "laser rot" is real, what leads you to believe that the Blu-Ray discs would be less succeptible?

I guess my current thinking on this is that DVDs would probably be more stable from an environmental/mechanical stnadpoint (i.e. don't have to worry as much about storage conditions)... but given reasonably good storage conditions, hard drives might be better?

(It may seem to some of you like I'm over-analyzing this problem... but look at it this way: I've probably got more $ invested in my CD collection than in my computers. And some of the CDs are out-of-print, and can't be replaced easily if something happens to them...)

b3n113 wrote:

Use good media, Verbatim is the best <imo> for archiving. You won't have a problem with cd-r's going to sh*t on you if you use a quality disc / burner to begin with.

Yes, I like Verbatim "blue" CD-Rs too. But I'm looking for something a little higher density than CD-R -- you have to understand that I've got a pretty large CD collection.

I suppose if the CD-Rs are all stored on spindles, they won't take up too much space... but consider for a moment that a single 250GB hard drive equals roughly 8 spindles (50 CDs each) of CD-Rs...

Also... Verbatim seems to have moved away from their "blue dye" CD-Rs lately. Are they still better in terms of stability, or are they now having their media contract manufactured by the same places that make all the other generic media?

just brew it! wrote:

If "laser rot" is real, what leads you to believe that the Blu-Ray discs would be less succeptible?

I guess my current thinking on this is that DVDs would probably be more stable from an environmental/mechanical stnadpoint (i.e. don't have to worry as much about storage conditions)... but given reasonably good storage conditions, hard drives might be better?

Well, it's something to think about if you plan to place that many WAV files on DVDs. Plus, factor in the time to burn that many files compared to just copying the files to several hard drives. For the truly paranoid out there, I suppose you could do both...

in the mean time, would it be ok to just make backup copies of the CD's that you most value/ scratchy, etc...

as for storing the wav's, i'll assume that each cd is about 500mb, so with over a 1000 cd's, you're talking roughly .5-1 terabyte. i think it would make more sense to get a few 250mb HD's. my only concern is if for some reason 1 hard drive goes bad, there goes half your collection of music.

Well, you could buy 5 x 250gb hard disks, in a RAID5 array (gives 1TB storage, with 250gb lost to parity, plus OS FS overheads). Should be ample to store 1000 CDs @ 650mb/CD, with a little extra. If you mean 1000 CD tiles, e.g. some titles on 2+ disks, add another hard disk.

This would give best protection for on-line access, although I dont know if there are IDE RAID-5 controllers... hot swap would be a good idea too, with a 6th on-line hot spare.

But maybe that's OTT! Using cheaper IDE drives you would have to power down and swap out drives as per your original idea.

The cost of a DVD writer is a lot lesser, and might be an idea to consider as well as hard disk storage, you are going to spend a year archiving to .WAV and then .MP3, so if you/a friend had a second PC, you could fill one hard disk, then stick drive in a second PC and start creating MP3's, and then copying .WAV to DVD.

Perhaps consider a DLT drive (DLT, SDLT, AIT etal) - starting at 40gb (uncompressed) from £800UKP to 1.6TB (£5200UKP) you could archive off to tape.

I guess it comes down to value you place on collection, plus funds available. Either way, its going to be a long haul archiving 1000+ disks to hard disk (at least 6months, woking on 4 disks per day, extracting, testing, checking quality, converting to MP3, checking quality), and TIME is going to be the biggest investment.

DVD rot will not be a problem for any created disks for a couple of years. What will be more of an issue is CRC errors. Even using quality disks, with quality DVD-RW, I get disks that depite passing vaerification show CRC errors a couple of days later, or cannot be read on another DVD-ROM...

Good luck!
Alan

Also... Verbatim seems to have moved away from their "blue dye" CD-Rs lately. Are they still better in terms of stability, or are they now having their media contract manufactured by the same places that make all the other generic media?

Yes and no, most of their spindles is made by another manufacturer, cant remember the name, but there are utilities for checking that. Just input a disc in the burner and run the util.
But they still sells their Azo ones in jewel cases and some spindles. If it says AZO, then its real verbatim, if it doesnt, then its the other oem brand.

You could run a lossless compression scheme on the wavs. That will get you down to half at least, which is nice if you are thinking of burning it. Monkey Audio Ape maybe.

Try bzip2 with those WAVs. I've heard it works great for compressing WAVs, and of course it's lossless .

bzip2 is more or less the best compression technology around, but unfortunately is also VERY processor-intensive. If he archived the files with bzip2, he wouldn't probably find anything cheap/powerful enough (I mean, to the likes of a wireless/set-top box) to decompress them on the fly.

The best choice for lossless compression would be the FLAC (Free Lossless Audio Coder), which can be found at http://flac.sourceforge.net

What is "best"? FLAC is open source, which has its own merits. However, I find that Monkey's Audio has superior comprssion ratios.

Let's see what I've go here... I just ripped a co-worker's reggae CD to my HD the other day.

CDImage.wav 758 MB (794,870,204 bytes)

CDImage.ape 483 MB (506,700,252 bytes)

CDImage.flac 501 MB (526,204,601 bytes)

Downloaded latest frontends from monkeysaudio.com and flac.sourceforge.net and used default settings.

I'll try max compressions in a bit and see what happens.

CDImage.ape 476 MB (499,789,376 bytes)

CDImage.flac 498 MB (523,152,526 bytes)

Flac was set to compression rate "8" via the front-end, and Monkey's Audio was set to "extra high".

[edit]

The .ape comes in at just under 500 MB (1MB = 1million bytes, as drive space is measured)

A quick calculation indicates that you should shoot for about a half-trillion bytes of storage. Mind you, this is assuming that all albumns rip to ~800 MB as this rather sizable album did. Very few of your buddy's albums will rip to this size. Most of mine come in at ~450.

My advice? Stay on the lookout for harddrive deals and build yourself a RAID5 array. Black Friday isn't too far away. Rebate deals of $70 or better on 160 MB drives are pretty likely. Four of those on a Highpoint RocketRAID404 (flashed with the lates BIOS for RAID5 support.) will give you space, reliability and scalability.

I don't think I want to do a RAID-5 for this, at least not for archiving the WAV files. Having the entire WAV archive online at all times is not a requirement, and it actually decreases the reliability versus an off-line solution -- one bad power surge, and it's all over!

Just tried bzip2 on a WAV file... compression ratio sucks (only around 5%). Bottom line: Most general-purpose lossless compressors are designed to compress "typical" computer data, like documents and executables. Digital audio does not have the same sort of patterns, and therefore does not compress very well with these algorithms.

Will definitely check out FLAC and Monkey's Audio sometime.

alewisA wrote:

I guess it comes down to value you place on collection, plus funds available. Either way, its going to be a long haul archiving 1000+ disks to hard disk (at least 6months, woking on 4 disks per day, extracting, testing, checking quality, converting to MP3, checking quality), and TIME is going to be the biggest investment.

I've generally been very happy with the extraction quality of EAC. I don't think I've ever had a bad rip that EAC didn't automatically flag. So I'm sort of inclined to trust it, and skip the testing/checking step; that would save a lot of time. It would also save me from having to listen to all of my wife's CDs (some of which just aren't my cup of tea); I figure if I'm gonna do this, might as well do her CDs as well so that they are all backed up.

Has anyone ever had EAC give a CD a clean bill of health, and susbequently discover errors in the extracted WAVs when listening to them?

just brew it! wrote:

I have a fairly sizeable CD collection -- 1000+ CDs, some of which date back to the 80s -- so this concerns me.

being a full time DJ i would need to hire a staff of 10 just to burn and
encode the 10,000-20,000 cd's i have lying around all over my house.

sigh. i only wish i could archive them all. i think this would be an impossible
task. 1,000 cd's i could see... anyone have any suggestion for me?

just brew it! wrote:

One of my friends has indicated that some of his older (pressed) audio CDs have started to show signs of deterioration (increased incidence of skips & errors), in spite of the fact that they are scratch-free and have been stored in a reasonable environment. I have a fairly sizeable CD collection -- 1000+ CDs, some of which date back to the 80s -- so this concerns me.

[SNIP]

- Anyone have any thoughts on what the best archival medium for the raw WAV files would be? ...

I have no comment re: DVD-R vs HDs but some CDR manufacturers make an 'archival' CDR specifically designed for long term storage (ie. up to 100 years if properly stored). These arent your bulk spindle type discs and will cost you quite a bit more (>$1) but that's what they're for.

Incidentally, there was a story on slashdot a couple days back regarding the storage life of CDs (or lack thereof). You may want to browse some of the comments for some interesting info.

http://slashdot.org/article.pl?sid=03/0 ... 37&tid=198

Good luck!

thegleek wrote:

just brew it! wrote:

I have a fairly sizeable CD collection -- 1000+ CDs, some of which date back to the 80s -- so this concerns me.

being a full time DJ i would need to hire a staff of 10 just to burn and
encode the 10,000-20,000 cd's i have lying around all over my house.

sigh. i only wish i could archive them all. i think this would be an impossible
task. 1,000 cd's i could see... anyone have any suggestion for me?

Since both gleek and brew have large CD collections (I have maybe 100 at most) and there are many ways to automate and verify the archival/conversion strategy. A little googling will turn up some nice ideas. I seem to recall one where :

ie. make a script that will
- rip the CD to wav twice (into two separate directories)
- encode wavs from each directory into MP3s
- compare checksums (ie md5sum) of resultant MP3
- if checksums are the same, then delete the raw wavs and wait for the next CD.

You get total walkaway capability and only have to visit your PC to pop in the next CD. For super redundancy, rip/encode the same CD on two or more different PCs then compare MP3 cheksums (again, can be totally automated). I'm assuming if there are pops/skips/anomalies then they will either 1) randomly occur, 2) or not be identical (either one will wreck the checksum).

If you find a track that consistenly fails the final checksum, it may be a scratch or something preventing a perfect, consistent rip and that track will have to verified seperately (ie. listening)

I've seen both thegleek and justbrewit on the Linux forum so both of you certainly have the tools to set something up quite easily

meanfriend wrote:

Since both gleek and brew have large CD collections (I have maybe 100 at most) and there are many ways to automate and verify the archival/conversion strategy. A little googling will turn up some nice ideas. I seem to recall one where :

ie. make a script that will
- rip the CD to wav twice (into two separate directories)
- encode wavs from each directory into MP3s
- compare checksums (ie md5sum) of resultant MP3
- if checksums are the same, then delete the raw wavs and wait for the next CD.

You get total walkaway capability and only have to visit your PC to pop in the next CD. For super redundancy, rip/encode the same CD on two or more different PCs then compare MP3 cheksums (again, can be totally automated). I'm assuming if there are pops/skips/anomalies then they will either 1) randomly occur, 2) or not be identical (either one will wreck the checksum).

If you find a track that consistenly fails the final checksum, it may be a scratch or something preventing a perfect, consistent rip and that track will have to verified seperately (ie. listening)

I've seen both thegleek and justbrewit on the Linux forum so both of you certainly have the tools to set something up quite easily

lol thanx. i'm well aware of automating the process.. prob is the storage
space it would take to store all those cds... even in mp3 form. it would
go well beyond the terabyte level... sigh

i suggest a SATA card and a bunch of ~120-250 gig hard drives. they dont have to be fast or anything.

up to a terabyte is there anything that is less $/GB? CDs dont count since they store such a small amount

thegleek wrote:

prob is the storage space it would take to store all those cds... even in mp3 form. it would go well beyond the terabyte level... sigh

at 256kbps MP3 a CD is ~100 megs. thats ~10 CDs per gig. thats ~1000 CDs for 100gigs, or ~10,000 CDs for a terabyte.

sativa wrote:

thegleek wrote:

prob is the storage space it would take to store all those cds... even in mp3 form. it would go well beyond the terabyte level... sigh

at 256kbps MP3 a CD is ~100 megs. thats ~10 CDs per gig. thats ~1000 CDs for 100gigs, or ~10,000 CDs for a terabyte.

ok now do the math for the time it would take to be involved in such
a process like this.. even if u automate it, you'd still have to physically
change the cd's everytime its done.

You should check CD changer jukeboxes like those: http://www.crutchfield.com/S-jlZwxmE0Tc ... sp?g=54300

200 CDs for $180 and 400 CDs for $700.

Those have digital output. I wonder how easy would be to let them play CDs and have them recorded into WAVs... 200CDs is an average HD nowadays (~120GB if the average CD contains 600MB which is quite pesimistic).

I wonder if that "compulink" means you can play at speed higher that 1X... because having them played in real time would take ages, and that sucks even if you don't need to monitor the operation. OTOH Just brew it has spare computers to let them do that, but they should be folding instead

sativa wrote:

thegleek wrote:

prob is the storage space it would take to store all those cds... even in mp3 form. it would go well beyond the terabyte level... sigh

at 256kbps MP3 a CD is ~100 megs. thats ~10 CDs per gig. thats ~1000 CDs for 100gigs, or ~10,000 CDs for a terabyte.

For archival purposes, you don't want a lossy compression method like MP3 anyway. My plan is to archive the WAVs offline, and (eventually) have the MP3s available on some sort of home entertainment server that can be accessed from anywhere in the house.

muyuubyou wrote:

You should check CD changer jukeboxes like those: http://www.crutchfield.com/S-jlZwxmE0Tc ... sp?g=54300

200 CDs for $180 and 400 CDs for $700.

Those have digital output. I wonder how easy would be to let them play CDs and have them recorded into WAVs... 200CDs is an average HD nowadays (~120GB if the average CD contains 600MB which is quite pesimistic).

I wonder if that "compulink" means you can play at speed higher that 1X... because having them played in real time would take ages, and that sucks even if you don't need to monitor the operation.

I doubt they support > 1x playing. In any case, using one of these would probably preclude the automatic fetching of track info over the Internet... I really don't want to have to enter all of the track names by hand!

OTOH Just brew it has spare computers to let them do that, but they should be folding instead

Heh... yeah, I've already started ripping/encoding on a small scale, using my dually MPX box. I run the LAME encoder at reduced priority to prevent it from affecting whatever else I'm working on -- below normal, but above the priority of the folding client. It has been impacting my folding throughput a bit...

Update --

For archiving the WAVs, I'm leaning towards an external FireWire -> ATA adapter, like the one shown <a href="http://www.wiebetech.com/products.html"><b>here</b></a> (scroll about 3/4 of the way down to the FireWire DriveDock). That way I can use commodity PATA hard drives, but don't need to mess with opening the computer up and mounting/unmounting hard drives when they get full.

So far, I've ripped about 150 of my CDs (using EAC). I'm very pleased with the quality; I've listened to a number of the WAVs and have yet to find any sonic flaws. I've also started encoding the WAVs to MP3 (using LAME with the "--preset standard" option) for the planned home media server. The resulting MP3s seem to be essentially indistinguishable from the original WAVs.

My Folding@home throughput for Team TR will probably be dropping off for a while, as my 3 fastest CPUs are now being kept busy much of the time running LAME!

And, if I may go off on a tangent here for a moment...

<rant>

This is a perfect example of a legitimate use for CD ripping and MP3 encoding. But if the RIAA and MPAA get their way, all content will eventually be protected by some form of DRM, making it difficult and/or illegal (under the DMCA) to do this kind of thing. Preventing a legitimate purchaser from making a backup copy and/or converting content to another format for personal use is simply wrong!

While I don't agree with the RIAA's tactics (I still think they should be required to get a subpoena to force ISPs to divulge information about their subscribers), I do think that going after individual file traders is better than slapping crippling DRM on all content. In other words, go after the people who are breaking the law, not the technology. The problem is, I don't think the RIAA will stop there. Even if they manage to curtail the use of P2P networks for music swapping, it is quite likely that we will still have DRM (Palladium, anyone?) rammed down our throats.

Yes, the RIAA are a bunch of greedy scumbag weasels. But file traders are breaking the law, by giving away copyrighted content without permission of the copyright holder. Two wrongs don't make a right!

And ultimately, it is the legitimate users of the technologies that are caught in the middle who will be inconvenienced.

</rant>

just brew it! wrote:

And, if I may go off on a tangent here for a moment...

<rant>

This is a perfect example of a legitimate use for CD ripping and MP3 encoding. But if the RIAA and MPAA get their way, all content will eventually be protected by some form of DRM, making it difficult and/or illegal (under the DMCA) to do this kind of thing. Preventing a legitimate purchaser from making a backup copy and/or converting content to another format for personal use is simply wrong!

While I don't agree with the RIAA's tactics (I still think they should be required to get a subpoena to force ISPs to divulge information about their subscribers), I do think that going after individual file traders is better than slapping crippling DRM on all content. In other words, go after the people who are breaking the law, not the technology. The problem is, I don't think the RIAA will stop there. Even if they manage to curtail the use of P2P networks for music swapping, it is quite likely that we will still have DRM (Palladium, anyone?) rammed down our throats.

Yes, the RIAA are a bunch of greedy scumbag weasels. But file traders are breaking the law, by giving away copyrighted content without permission of the copyright holder. Two wrongs don't make a right!

And ultimately, it is the legitimate users of the technologies that are caught in the middle who will be inconvenienced.

</rant>

If only this kind of clarity and well-reasoned argument could be found in the RIAA thread in R&P