Personal computing discussed

Moderators: renee, SecretSquirrel, notfred

 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Don't try this at home. Or at work.

Wed Mar 12, 2014 2:59 pm

Never, EVER run the following command as root:
find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b

The intent is to recursively produce a sorted listing of MD5 hashes for all files at or below the current directory.

What's wrong with this, you ask?

Well, if a subdirectory somewhere in the tree happens to contain an old copy of the /dev directory, this fails in rather spectacular fashion. Although we've taken care to exclude device nodes from the list of files to be processed (via the -type options), symbolic links to devices are still processed. And /dev/core is a symbolic link to /proc/kcore. And /proc/kcore is a pseudo-file that maps to the entire virtual address space of the CPU -- all 128TB of it (if you are on a 64-bit x86 processor)!

The initial outward symptom is that the script appears to hang, as md5sum attempts to digest the 128TB /proc/kcore.

If you let it continue running, several hours in really bad sh*t starts to happen. The evidence was in the system logs; messages like:
Mar 12 03:02:12 systemname kernel: [145860.816052] BUG: soft lockup - CPU#6 stuck for 22s! [md5sum:26719]

and a few hours after that:
Mar 12 08:31:51 systemname rtkit-daemon[5207]: The canary thread is apparently starving. Taking action.
Mar 12 08:32:07 systemname rtkit-daemon[5207]: Demoting known real-time threads.

Eventually the system just wedges and you need to push the reset button.

Fun times.
Nostalgia isn't what it used to be.
 
excession
Graphmaster Gerbil
Posts: 1262
Joined: Fri Dec 31, 2004 3:19 pm
Location: Nottingham, UK

Re: Don't try this at home. Or at work.

Wed Mar 12, 2014 3:09 pm

And this is why we all love Unix - because when you break your system you know EXACTLY what a muppet you were! :D
i5-4670 | Asus H87M-E | MSI GTX 960 | 8GB DDR3 @800 | WD Green 2TB | 850 EVO 250GB | CM Masterkeys Pro L White | MX518 (original!) | Hyper 212 Evo | 6TB Ubuntu/ZFS NAS
I was going to tell a Chemistry joke, but all the good ones argon.
 
JosiahBradley
Gerbil
Posts: 12
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:30 pm

This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:46 pm

JosiahBradley wrote:
This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.

I believe tar specifically refuses to recursively include an archive into itself (it skips the archive file), if it detects you're attempting to do that. You might be able to fool it with hard links or something though.
Nostalgia isn't what it used to be.
 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:46 pm

 
Scrotos
Graphmaster Gerbil
Posts: 1109
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:47 pm

jbi, does this still work?

#!/bin/sh
$0 &
exec $0
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:49 pm

Hrmmm.... seems to me that JBI just needs MOAR HARDWARE if generating an md5sum on a piddly little 128TB memory mapped file is causing issues! :P
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 6:51 pm

Scrotos wrote:
jbi, does this still work?

#!/bin/sh
$0 &
exec $0



Ah the classic forkbomb. If your distro is setup properly with per-user process limits, then hopefully the system contains the number of forked processes to a number that prevents it from screeching to a halt. Although your personal user account might be difficult to deal with since you won't be allowed to run any more programs once the bomb reaches your limit.
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 7:09 pm

chuckula wrote:
Scrotos wrote:
jbi, does this still work?

#!/bin/sh
$0 &
exec $0

Ah the classic forkbomb. If your distro is setup properly with per-user process limits, then hopefully the system contains the number of forked processes to a number that prevents it from screeching to a halt. Although your personal user account might be difficult to deal with since you won't be allowed to run any more programs once the bomb reaches your limit.

Just tried it. Yes, it still "works"... at least on Ubuntu 12.04 LTS.

It took a while to bring things down, but eventually the desktop became unresponsive and I could not Ctrl-Alt-Fx to alternate console sessions either. I had a "top" running in a terminal window, and the last update before things went south showed that the number of processes and the load average were both somewhere north of 15,000, and the system had started to hit the swap file pretty hard.

Edit: The unclean shutdown also caused the software RAID-1 array to drop out of sync. An automatic background re-sync kicked in as soon as the system was rebooted. And no, I did not try this on my main workstation! :lol:
Nostalgia isn't what it used to be.
 
JosiahBradley
Gerbil
Posts: 12
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 9:16 pm

Scrotos wrote:
web.mit.edu/~simsong/www/ugh.pdf


Is this a sick joke? Not sure if satire or actual idiots.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 9:31 pm

JosiahBradley wrote:
Scrotos wrote:
web.mit.edu/~simsong/www/ugh.pdf

Is this a sick joke? Not sure if satire or actual idiots.

It is rather dated (originally written in the 1990s). The dedication makes it pretty clear that it is (at least partially) tongue-in-cheek:
To Ken and Dennis, without whom this book would not have been possible.

(That's referring to Ken Thompson and Dennis Ritchie, co-inventors of UNIX.)

The "Anti-Foreword" by Dennis Ritchie is also a big clue.
Nostalgia isn't what it used to be.
 
ClickClick5
Gerbil Elite
Posts: 581
Joined: Fri Aug 15, 2008 8:28 pm
Location: Alaska
Contact:

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 9:49 pm

Never run "sudo rm -r /*" as a show that Linux will self delete itself…then actually enter your password and actually press enter. TO THE TAPE CLOSET I GO!
i7-5930k (4.1GHz), 32GB DDR4-2400, 2080Ti, GA-X99-UD3, Samsung 860 Pro 256GB (os/programs), Toshiba 5TB (games), WD VelociRaptor 500GB (games/scratch disk), Win 10 Pro x64
Check out my site for trance sets and other goodies! clickclick5.com!
 
keltor
Gerbil First Class
Posts: 180
Joined: Thu May 10, 2012 4:29 pm

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 10:12 pm

JosiahBradley wrote:
This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.


Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)
 
keltor
Gerbil First Class
Posts: 180
Joined: Thu May 10, 2012 4:29 pm

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 10:15 pm

find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b


This didn't take all THAT long on RHEL 6.4, 5.5, Ubuntu Server 12.04 or my Mac. I ran on all 4 as root. All of our machines except my Mac are pretty close to default except they are pretty locked down on the Firewall side.
 
JosiahBradley
Gerbil
Posts: 12
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 10:53 pm

keltor wrote:
JosiahBradley wrote:
This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.


Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)


It wasn't /dev causing the problem, but the resulting file definitely had 4G of data under /proc.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Thu Mar 13, 2014 11:53 pm

keltor wrote:
JosiahBradley wrote:
This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.

Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)

http://xkcd.com/1168/

keltor wrote:
find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b

This didn't take all THAT long on RHEL 6.4, 5.5, Ubuntu Server 12.04 or my Mac. I ran on all 4 as root. All of our machines except my Mac are pretty close to default except they are pretty locked down on the Firewall side.

I assume you were starting from a directory that didn't contain any nested copies of an old /dev directory though. In my case, that was the gasoline that provided the fuel for the match!

I dare you to create a symlink to /proc/kcore somewhere in that directory structure, then try running that command line again. :wink:
Nostalgia isn't what it used to be.
 
notfred
Maximum Gerbil
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Don't try this at home. Or at work.

Fri Mar 14, 2014 5:43 pm

For those that succeeded versus those that failed on the big tar, it's going to be down to permissions. As JBI mentioned, /dev/core is a symbolic link to /proc/kcore. But the permissions on /proc/kcore are 0400 and owned by root, so only if you tar as root do you get the hang as it digests the entire virtual address space.

Tar itself is pretty easy though. Almost daily I use
tar cf - dir1 --exclude-vcs | (cd dir2; tar xvf -)

It's a good recursive copy.
 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2726
Joined: Tue Jan 01, 2002 7:00 pm
Location: North DFW suburb...
Contact:

Re: Don't try this at home. Or at work.

Sat Mar 15, 2014 8:08 am

My worst screw up was at a clients site. I was my younger, but still, it was a basic mistake that I should have caught. I was working with them to implement a source code analysis tool that took a while to run. It was setup with a dedicated account which owned all the data it produced. That account set a number of options in its shell environment, one of which was $WORKDIR. To ensure that a run was clean and not polluted from prior artifacts, the analysis script had the following line in it.

/bin/rm -rf ${WORKDIR}/*


Since it needed to run weekly, it was put in the owning accounts crontab. A few days later, I got a very panicked call from my boss saying that the customer was seeing files all across their network disappear. See, while I had made one mistake, the customer had made another. The majority of their NFS, including NFS mounted home directories, had 775 permissions and all users were in the same group. I was on an airplane the next day. That was most unpleasant. That incident ingrained in me the need for defensive coding.

--SS
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Sat Mar 15, 2014 9:05 am

Any sort of recursive file processing gets very dangerous when network mounts are involved. It is generally prudent to include command line switches to prevent traversal of mount points (--one-file-system in the case of rm -r).

I've also had annoying things happen when importing file systems into a VirtualBox VM using VirtualBox's "shared folder" feature. One day at work I noticed that my VM was chewing up an awful lot of CPU time, and my workstation was hammering the network hard. A bit of investigation revealed that the indexing service (known as updatedb on Linux systems) was attempting to index the VirtualBox shared folder. Which in turn contained mount points to multiple Linux and Windows servers on the network. But because this was in a directory which was being imported via VirtualBox, the fact that some of the subdirectories were network shares (which updatedb would've normally skipped by default) was obscured. The VM was attempting to index the entire damn network!
Nostalgia isn't what it used to be.
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Don't try this at home. Or at work.

Sat Mar 15, 2014 5:50 pm

just brew it! wrote:
It took a while to bring things down, but eventually the desktop became unresponsive and I could not Ctrl-Alt-Fx to alternate console sessions either.

Edit: The unclean shutdown also caused the software RAID-1 array to drop out of sync. An automatic background re-sync kicked in as soon as the system was rebooted. And no, I did not try this on my main workstation! :lol:


That reminds me. I found this the other day.

Perform a gentle Linux restart: Ctrl + Alt + PrtSc (SysRq) + reisub (Linux gets frozen, what do you do?: http://www.jovicailic.org/2013/05/linux ... do-you-do/)

It requires three hands, and I'm not sure it would have helped. It's good to know though.
 
just brew it!
Administrator
Topic Author
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Sun Mar 16, 2014 8:29 am

Flatland_Spider wrote:
That reminds me. I found this the other day.

Perform a gentle Linux restart: Ctrl + Alt + PrtSc (SysRq) + reisub (Linux gets frozen, what do you do?: http://www.jovicailic.org/2013/05/linux ... do-you-do/)

It requires three hands, and I'm not sure it would have helped. It's good to know though.

I did not know about that. Turns out that Wikipedia has a complete list of supported "Magic SysRq" commands: http://en.wikipedia.org/wiki/Magic_SysRq_key

Not all of them appear to work on my Ubuntu 12.04 system, but most of them do.

As an aside, this is definitely something that needs to be taken into consideration if anyone is using Linux in a kiosk-style application with a keyboard attached. At least it is easy to disable with a system configuration option. But to disable it you need to be aware that it exists in the first place!

I also mentioned it to my Linux guru/evangelist friend, and (as expected) he already knew about it. Says he's used it quite a bit when he's done driver development (he's the author of some of the drivers used by MythTV).
Nostalgia isn't what it used to be.

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On