bitrot in Windows

Monopoly money comes in many flavors: 7, Vista, XP, 2K, ME, 98, etc.

Moderators: Flying Fox, Ryu Connor

bitrot in Windows

Postposted on Thu Jun 26, 2014 7:40 am

http://arstechnica.com/information-tech ... s/#image-1

After reading this article, I was wondering if anyone has had to deal with this and could suggest any solutions to these bitrotten files within Windows. It seems to me that NTFS was not designed to tackle this and I am not sure what can be done to correct it on any data files within Windows.
Laptop: HP Pavilion 17-e016dx. AMD A8-5550M, 4GB RAM, 750GB HDD, AMD Radeon HD 8550G integrated video, 17.3" display, 1600*900 (HD+) resolution, SD card reader, Windows 8.1 (DL Classic Shell)
riviera74
Gerbil Elite
 
Posts: 860
Joined: Mon May 29, 2006 6:14 am
Location: FM, FL, USA

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:00 am

riviera74 wrote:http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/#image-1

After reading this article, I was wondering if anyone has had to deal with this and could suggest any solutions to these bitrotten files within Windows. It seems to me that NTFS was not designed to tackle this and I am not sure what can be done to correct it on any data files within Windows.



Unless you have a good back up of that data, once corrupt you are SOL with NTFS. NTFS is an old filesystem and has many faults (bitrott is an old problem with it) when it comes to data integrity and isn't much better than Apple's HFS+. Both are seemingly held together with bubblegum and duct tape. You could always setup a linux file server and use btrfs if you are that concerned over it.
Deanjo
Gerbil XP
 
Posts: 373
Joined: Tue Mar 03, 2009 11:31 am

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:11 am

I have a fairly large collection of music that's been around since the late 90s, and some of the older files have been migrated between probably a dozen machines. I definitely have a non-trivial amount of files now that have signs of bit rot. Ever heard an old mp3 with a very tiny blip of static? Guess what, that's 1 frame broken screwing up a few milliseconds worth of audio data.

It amazes me that it's taken this long for file systems with full error correction to come about. I was hopeful Apple would move things along with their ZFS port, but with that abandoned, it looks now like Linux is the only consumer-ish place we can get these features, and even they are beta-level still.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1550
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:17 am

It is not just NTFS. Any "4th gen file system" as defined by the author, ext4, UFS2 were also mentioned.

The new ReFS is supposed to contain some of these next gen features to combat the issue. But it is at best a first "public beta" ever since it was introduced in Server 2012. It will take them a few iterations to sort out the kinks and to match the maturity of NTFS with respect to "last gen" features (there are some features of NTFS absent in the new thing, as they rushed things out).
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24293
Joined: Mon May 24, 2004 2:19 am

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:37 am

riviera74 wrote:It seems to me that NTFS was not designed to tackle this and I am not sure what can be done to correct it on any data files within Windows.

On traditional filesystems there isn't a way to correct it after it happens, other than restoring from a backup (assuming the backup isn't corrupted as well).

I'm considering using ZFS or btrfs for my upcoming file server upgrade.

I also believe that some of the errors that people blame on bitrot (or OS bugs) are actually being caused by flipped bits in RAM. Any time you copy a file from one place to another or run a HDD defrag, the data passes through RAM and is susceptible to corruption by RAM errors. Today's systems with 64-bit OSes and 16GB+ of RAM make this even worse, since large amounts of data can pile up in write buffers waiting to be flushed to disk; this increases the window where a flipped bit in RAM will corrupt data on disk. Sure, my insistence on ECC RAM for nearly all of my systems may be somewhat OCD; but the price differential versus non-ECC isn't that bad if you aren't using registered DIMMs.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:43 am

ReFS seems stable, it just lacks a lot of features that NTFS has. If you use Storage Spaces, you get bitrot protection for your data. Here's a decent rundown:
http://blogs.technet.com/b/askpfeplat/a ... se-it.aspx
Duct Tape Dude
Gerbil First Class
Gold subscriber
 
 
Posts: 116
Joined: Thu May 02, 2013 12:37 pm

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 9:45 am

Bitrot is mostly an issue with datacenter servers where you see frequent read/write cycles. For the vast majority of the people out there, memory issues are a larger problem with data integrity than file system issues.
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC-12800, EVGA 660Ti, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
Krogoth
Maximum Gerbil
Silver subscriber
 
 
Posts: 4395
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 10:20 am

Krogoth wrote:Bitrot is mostly an issue with datacenter servers where you see frequent read/write cycles. For the vast majority of the people out there, memory issues are a larger problem with data integrity than file system issues.

I would say for the above-average consumer, storage over a long period of time will give you the same probability as datacentre's frequent read/write cycles.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24293
Joined: Mon May 24, 2004 2:19 am

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 10:40 am

Make a parchive of valuable files if you are concerned. It will detect and fix these bit flip errors if they ever occur.
http://multipar.eu/

But yes RAM is a problem. With huge RAM capacities RAM errors are likely and ECC should probably be standard. Hard disks use ECC btw.
swaaye
Gerbil First Class
 
Posts: 174
Joined: Mon Apr 21, 2003 4:45 pm

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 12:18 pm

Yeah with as cheap as RAM became in the last few years it's sad ECC hasn't caught on more. I think it's partly Intel's fault for using it to segregate the enterprise and consumer markets.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1550
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 3:26 pm

SuperSpy wrote:Yeah with as cheap as RAM became in the last few years it's sad ECC hasn't caught on more. I think it's partly Intel's fault for using it to segregate the enterprise and consumer markets.

Yup. For non-registered DIMMs, the delta in raw materials cost is 12.5%, give or take (9 DRAM chips per bank instead of 8 ). There's really no excuse for not including ECC support on all but the lowest of budget platforms. This is one of the reasons I've stuck with AMD longer than most other PC enthusiasts.

Edit: And it is also the reason I'm not a big fan of using AMD's APUs on the desktop -- APU product line has no ECC support. For ECC you need to use AM2/AM3/AM3+.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Thu Jun 26, 2014 4:06 pm

At home I have a NA$4Free system that doubles as a VM host courtesy of VirtualBox + phpVirtualBox. By keeping the virtual disks on a ZFS volume, they're protect via the superior host file system.

Actually I've found ECC to be cheaper than regular RAM in some cases. A 16 GB stick of registered 1.35v DDR3 1600 Mhz memory was ~$125 where as the enthusiast 16 GB kits of two or four DIMMs were $140 to $160. Sometimes the market does weird things is you look around.

Intel needs to be called out on keeping this feature in the professional/server realm. I'd prefer this to be an option a consumer can make on their own without paying a price premium beyond the DIMMs.
Dual Opteron 6376, 128 GB DDR3, Asus KGPE-D16, Radeon 6970
Mac Pro Dual Xeon E5645, 48 GB DDR3, GTX 770
Core i7 3930K@4.2 Ghz, 32 GB DDR3, GA-X79-UP5-Wifi
Core i7 2600K@4.4 Ghz, 16 GB DDR3, Radeon 6870, GA-X68XP-UD4
the
Gerbil
Gold subscriber
 
 
Posts: 57
Joined: Tue Jun 29, 2010 2:26 am

Re: bitrot in Windows

Postposted on Sun Jun 29, 2014 11:59 am

While bitrot is a real thing, as HDDs' sectors are ECC protected, it is (fortunately) a rare event. So it is difficult that the problem Ars described was due to real bitrot.

The ECC protection should detect and recover most errors. Basically, three scenarios are possible:
- detected single bit error: it should be detected _and_ corrected. This evenience is logged by the HDD firmware but will not cause any harm, as the OS will receive correct informations;
- detected multi bit error: this will result in a "bad sector" error and the HDD firmware will refuse to pass the information to the OS;
- undetected multi bit error: this is the most dangerous condition. The HDD firmware will pass BAD data to the OS.

A "real" bitrot happens when a correctly-written information is altered due to external condition and/or due to the intrisic limits of magnetic storage. This kind of bitrot will generally alter a single bit, so that the ECC code will recover it harmlessy. A periodic check of HDD SMART data will highlight some ECC read errors, so the HDD can be swapped out before a real problem happens. Even multi-bit rot have good probability to be spotted by the ECC check, so the OS will receive a "bad sector" error and can re-read the same data from the RAID array.

The real problem is when a multi-bit rot is _not_ detected and so the OS can receive bad data. However, in normal condition (and in monitored systems) this should be a really rare event.

Another potential source of problem is described by the "bit error rate", which is the probability that, due to the intrisics limit of the magnetic process used to write on the media, the HDD will write bad data on the first attempt _and_ that the ECC code will detect but not correct the error. The bit error rate (BER) of common, consumer disk is one error each 10^14 bit written, or one unrecoverable bit error each 12.5 TiB of data (LINK: http://www.wdc.com/wdproducts/library/S ... 771438.pdf), while server-grade disk are 10x-100x times better.The probability that this error is not only unrecoverable, but also undetected is quite small. Any redundant-based RAID array level (eg: 1,10) should have no problem to recover from these kind of errors, while parity-based level (eg: 4,5,6) are in big troubles.

Anyway, a MUCH bigger problem is the probability that the to-be-written data are corrupted inside the RAM, and this likely is the problem observed by Ars. The point is that a data corruption that happens in RAM is totally undetectable by the HDD units: after all, the data arrived to it _already changed_ and the HDD's calculated ECC code will be consistent with the inconsistent (!) data written. ECC-protected RAM is therefore absolutely critical in storage units.

Don't let me wrong: end-to-end data checksumming is a very valuable feature, and high-level SAN units implement it. At the same time, ECC-protected servers should only very rarely suffer from bit rot and undetectable bit error (but avoid parity-based RAID, please). Let now see the same thing on the consumer side: as consumer PCs are without ECC ram and they have a single HDD, you can think that they will greatly benefit from the additional error recovery granted by data checksumming. In some manner, this is true: after all, Ars's article just proved that. However, a ZFS or BTRFS scrub on a non-ECC equipped system with even a single problematic memory location will lead to catastrophic data corruption: http://forums.freenas.org/index.php?thr ... zfs.15449/

To recap:
- Ars problem was likely caused by a flipped RAM bit (flipped _after_ the data checksum process, or during transfer to the SATA port);
- in a classic filesystem, this will cause a file corruption (as happened), while on a data checksumming filesystem the error will detected and (hopefully) corrected;
- using ECC RAM the problem is basically solved;
- in the event of a bad stick of non-ECC RAM, a scrub operation (on a scrub-capable filesystem, as BTRFS and ZFS) will catastrophically corrupt your data.

Now, a small performance-related detour. A COW-based filesystem will show BIG slowdown on rewrite-intensive workload when used on classic HDD. Database and virtual machine will peform much worse, due to the ever increasing file framments. I wrote something on the subject here (http://www.ilsistemista.net/index.php/l ... ml?start=5) and here (http://www.ilsistemista.net/index.php/l ... -look.html). I don't know about ZFS, but even with COW disabled BTRFS was noticeably slower the EXT4 or XFS.

So, what filesystem you should use for your storage?
- if you need a "cold storage" solution and the "experimental" label don't scare you, BTRFS + ECC memory is the way to go;
- if online data deduplication is a requirements, ZFS + 8/16/32GB ECC RAM is the best setup;
- if you want to run rewrite intensive application at full speed (eg: databases, virtual machines) use EXT4 or XFS (even better, if your application support it, directly use a LVM volume);
- if you plan to use non-ECC memory, stay with EXT4 or XFS.

Regards.
www.ilsistemista.net - test & bench :)
shodanshok
Gerbil
 
Posts: 25
Joined: Thu May 31, 2012 3:39 am

Re: bitrot in Windows

Postposted on Sun Jun 29, 2014 1:07 pm

Just a minor nit regarding the thread title... this really isn't a Windows-specific issue. Any OS which is using one of the commonly deployed current-gen (or older) file systems is vulnerable. Windows, Linux, OS X... all are at risk.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 2:56 am

just brew it! wrote:Just a minor nit regarding the thread title... this really isn't a Windows-specific issue. Any OS which is using one of the commonly deployed current-gen (or older) file systems is vulnerable. Windows, Linux, OS X... all are at risk.


Yeah, but that said Linux has multiple 5th gen file systems users can use. OS X at least has HFS+ now. Windows users are stuck with NTFS.

Does anyone think Windows 9 will incorporate ReFS? Which is basically asking does anyone think the problems with the initial release of ReFS will be overcome in time for it to make it into Windows 9?
Kougar
Gerbil Team Leader
 
Posts: 246
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 3:44 am

I haven't really had any problems with ReFS. It worked great while I was using it on my non OS drives.

I think they'll enable it for general use, but I doubt they'll allow it on the OS drive.
Last edited by Savyg on Tue Jul 01, 2014 3:45 am, edited 1 time in total.
Meow.
Savyg
Gerbil Elite
Silver subscriber
 
 
Posts: 613
Joined: Thu Aug 26, 2004 6:18 am
Location: Between desert and tundra

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 3:45 am

Kougar wrote:OS X at least has HFS+ now. Windows users are stuck with NTFS.

What does HFS+ have that prevents bitrot? It's a very old filesystem that's been hacked up along the way to try to keep current.
accord1999
Gerbil
 
Posts: 58
Joined: Thu Jul 22, 2004 6:25 pm

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 5:53 am

Kougar wrote:
just brew it! wrote:Just a minor nit regarding the thread title... this really isn't a Windows-specific issue. Any OS which is using one of the commonly deployed current-gen (or older) file systems is vulnerable. Windows, Linux, OS X... all are at risk.

Yeah, but that said Linux has multiple 5th gen file systems users can use.

No major distros use btrfs by default (though SuSE 13.2, scheduled to be released in the fall, supposedly will). Licensing prevents ZFS from being bundled with the Linux kernel; unless the license changes it'll probably never be widely deployed.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 6:56 am

accord1999 wrote:What does HFS+ have that prevents bitrot?


Nothing.

And I'm not sure what Kougar means by OS X has HFS+ now, as it always did. They've added a few features, but not for many releases now and they've never added checksums.

As JBI says, this is a widespread issue. Yes, Linux at least has checksumming FSes available, but as he says, they're not widely deployed by default. Most users are still using ext4.
Glorious
Darth Gerbil
Gold subscriber
 
 
Posts: 7837
Joined: Tue Aug 27, 2002 6:35 pm

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 8:41 am

just brew it! wrote:No major distros use btrfs by default (though SuSE 13.2, scheduled to be released in the fall, supposedly will). Licensing prevents ZFS from being bundled with the Linux kernel; unless the license changes it'll probably never be widely deployed.


Oh wow, I was under the impression btrfs was still beta-level software and was no where near production quality.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1550
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 9:10 am

SuperSpy wrote:
just brew it! wrote:No major distros use btrfs by default (though SuSE 13.2, scheduled to be released in the fall, supposedly will). Licensing prevents ZFS from being bundled with the Linux kernel; unless the license changes it'll probably never be widely deployed.

Oh wow, I was under the impression btrfs was still beta-level software and was no where near production quality.

Seems to depend on who you ask. I get the impression that the core file system implementation is stable, but some planned features are still missing. I think where users are more likely to encounter issues is in the supporting tools; e.g., I would be a little leery of relying on gparted to manipulate btrfs partitions until btrfs has seen more widespread use.

Side note: Interestingly, one of the companies really pushing btrfs forward is Facebook. They have hired the principal author of the current btrfs code base.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 9:33 am

just brew it! wrote:Side note: Interestingly, one of the companies really pushing btrfs forward is Facebook. They have hired the principal author of the current btrfs code base.

This will not end well.
It is one of the blessings of old friends that you can afford to be stupid with them. Ralph Waldo Emerson.
Captain Ned
Global Moderator
Gold subscriber
 
 
Posts: 20117
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: bitrot in Windows

Postposted on Tue Jul 01, 2014 9:44 am

Captain Ned wrote:
just brew it! wrote:Side note: Interestingly, one of the companies really pushing btrfs forward is Facebook. They have hired the principal author of the current btrfs code base.

This will not end well.

I am cautiously optimistic. At least they've got deep pockets, and a vested interest in seeing a production quality next-gen file system deployed on Linux.

If Facebook doesn't work out he'll probably end up working for Red Hat or IBM instead.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Thu Jul 03, 2014 3:45 am

I'd read something about OS X adopting a file system that had the capability so I wrongly guessed it was HFS+. It appears I had just gotten confused with previous efforts to get ZFS onto OS X, sorry for the mix up!

Anyone optimistic at all that ReFS will make it into Win 9? The more I read about these advanced file systems the more it's starting to seem like it won't.
Kougar
Gerbil Team Leader
 
Posts: 246
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: bitrot in Windows

Postposted on Thu Jul 03, 2014 5:59 am

Kougar wrote:I'd read something about OS X adopting a file system that had the capability so I wrongly guessed it was HFS+. It appears I had just gotten confused with previous efforts to get ZFS onto OS X, sorry for the mix up!

Anyone optimistic at all that ReFS will make it into Win 9? The more I read about these advanced file systems the more it's starting to seem like it won't.


Yeah, that's exactly the thing! They (MS and Apple) keep saying that their next release will have a next generation filesystem, but then it never actually does. Heh.

I mean, like you noted with OS X, they keep talking about it so it's easy to get all mixed up.

And, with MS, Longhorn (the project that was variously mutated, killed, and resurrected into Windows Vista and then Windows 7), there originally was going to be a new "WinFS" as major component. That obviously never happened either.

So, exactly like you said, the more you read about it, the more it seems like it won't happen. :wink:
Glorious
Darth Gerbil
Gold subscriber
 
 
Posts: 7837
Joined: Tue Aug 27, 2002 6:35 pm

Re: bitrot in Windows

Postposted on Thu Jul 03, 2014 10:28 am

Glorious wrote:And, with MS, Longhorn (the project that was variously mutated, killed, and resurrected into Windows Vista and then Windows 7), there originally was going to be a new "WinFS" as major component. That obviously never happened either.

IIRC WinFS was supposed to be based on MS SQL Server. That just seems like massive overkill to me (and a bit of "square peg in round hole" syndrome as well).
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37520
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: bitrot in Windows

Postposted on Thu Jul 03, 2014 10:53 am

just brew it! wrote:
Glorious wrote:And, with MS, Longhorn (the project that was variously mutated, killed, and resurrected into Windows Vista and then Windows 7), there originally was going to be a new "WinFS" as major component. That obviously never happened either.

IIRC WinFS was supposed to be based on MS SQL Server. That just seems like massive overkill to me (and a bit of "square peg in round hole" syndrome as well).

They were thinking about using a mssql backend because they are trying to overlay an object-relational model on top so they can do tagging/searching/metadata-ing. Don't think that ever flew and the file system guys ended up trying to reinvent the wheel. But I don't remember reading that WinFS will solve the file integrity issue. WinFS afaik functions at a higher level, with lower level support if necessary. The basic lower block (things like sectors, file/directory tables, etc.) was supposed to be some form of NTFS still. Of course we know how that turns out.

They do have a new file system out, ReFS. But it is v1 tech at best, realistically close to a public beta with all the missing feature and slowness when additional integrity checking is enabled.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24293
Joined: Mon May 24, 2004 2:19 am


Return to Windows

Who is online

Users browsing this forum: No registered users and 2 guests