Don't try this at home. Or at work.

Where Penguins and Daemons chill together in the warmth of the Sun.

Moderators: SecretSquirrel, notfred

Don't try this at home. Or at work.

Postposted on Wed Mar 12, 2014 2:59 pm

Never, EVER run the following command as root:
Code: Select all
find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b

The intent is to recursively produce a sorted listing of MD5 hashes for all files at or below the current directory.

What's wrong with this, you ask?

Well, if a subdirectory somewhere in the tree happens to contain an old copy of the /dev directory, this fails in rather spectacular fashion. Although we've taken care to exclude device nodes from the list of files to be processed (via the -type options), symbolic links to devices are still processed. And /dev/core is a symbolic link to /proc/kcore. And /proc/kcore is a pseudo-file that maps to the entire virtual address space of the CPU -- all 128TB of it (if you are on a 64-bit x86 processor)!

The initial outward symptom is that the script appears to hang, as md5sum attempts to digest the 128TB /proc/kcore.

If you let it continue running, several hours in really bad sh*t starts to happen. The evidence was in the system logs; messages like:
Code: Select all
Mar 12 03:02:12 systemname kernel: [145860.816052] BUG: soft lockup - CPU#6 stuck for 22s! [md5sum:26719]

and a few hours after that:
Code: Select all
Mar 12 08:31:51 systemname rtkit-daemon[5207]: The canary thread is apparently starving. Taking action.
Mar 12 08:32:07 systemname rtkit-daemon[5207]: Demoting known real-time threads.

Eventually the system just wedges and you need to push the reset button.

Fun times.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Wed Mar 12, 2014 3:09 pm

And this is why we all love Unix - because when you break your system you know EXACTLY what a muppet you were! :D
i5-4670 @stock | Asus H87M-E| Asus HD6850 840/1100 | 8GB DDR3 @800 | Caviar Black 640GB | Agility 3 120GB | Unicomp Customiser 105 | Logitech MX518 | CM Hyper 212 Evo
I was going to tell a Chemistry joke, but all the good ones argon.
excession
Graphmaster Gerbil
Gold subscriber
 
 
Posts: 1171
Joined: Fri Dec 31, 2004 3:19 pm
Location: Nottingham, UK

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:30 pm

This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.
JosiahBradley
Gerbil In Training
Silver subscriber
 
 
Posts: 6
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:46 pm

JosiahBradley wrote:This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.

I believe tar specifically refuses to recursively include an archive into itself (it skips the archive file), if it detects you're attempting to do that. You might be able to fool it with hard links or something though.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:46 pm

Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:47 pm

jbi, does this still work?

#!/bin/sh
$0 &
exec $0
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:49 pm

Hrmmm.... seems to me that JBI just needs MOAR HARDWARE if generating an md5sum on a piddly little 128TB memory mapped file is causing issues! :P
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-770; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
chuckula
Gerbil Elite
Gold subscriber
 
 
Posts: 562
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 6:51 pm

Scrotos wrote:jbi, does this still work?

#!/bin/sh
$0 &
exec $0



Ah the classic forkbomb. If your distro is setup properly with per-user process limits, then hopefully the system contains the number of forked processes to a number that prevents it from screeching to a halt. Although your personal user account might be difficult to deal with since you won't be allowed to run any more programs once the bomb reaches your limit.
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-770; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
chuckula
Gerbil Elite
Gold subscriber
 
 
Posts: 562
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 7:09 pm

chuckula wrote:
Scrotos wrote:jbi, does this still work?

#!/bin/sh
$0 &
exec $0

Ah the classic forkbomb. If your distro is setup properly with per-user process limits, then hopefully the system contains the number of forked processes to a number that prevents it from screeching to a halt. Although your personal user account might be difficult to deal with since you won't be allowed to run any more programs once the bomb reaches your limit.

Just tried it. Yes, it still "works"... at least on Ubuntu 12.04 LTS.

It took a while to bring things down, but eventually the desktop became unresponsive and I could not Ctrl-Alt-Fx to alternate console sessions either. I had a "top" running in a terminal window, and the last update before things went south showed that the number of processes and the load average were both somewhere north of 15,000, and the system had started to hit the swap file pretty hard.

Edit: The unclean shutdown also caused the software RAID-1 array to drop out of sync. An automatic background re-sync kicked in as soon as the system was rebooted. And no, I did not try this on my main workstation! :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 9:16 pm

Scrotos wrote:web.mit.edu/~simsong/www/ugh.pdf


Is this a sick joke? Not sure if satire or actual idiots.
JosiahBradley
Gerbil In Training
Silver subscriber
 
 
Posts: 6
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 9:31 pm

JosiahBradley wrote:
Scrotos wrote:web.mit.edu/~simsong/www/ugh.pdf

Is this a sick joke? Not sure if satire or actual idiots.

It is rather dated (originally written in the 1990s). The dedication makes it pretty clear that it is (at least partially) tongue-in-cheek:
To Ken and Dennis, without whom this book would not have been possible.

(That's referring to Ken Thompson and Dennis Ritchie, co-inventors of UNIX.)

The "Anti-Foreword" by Dennis Ritchie is also a big clue.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 9:49 pm

Never run "sudo rm -r /*" as a show that Linux will self delete itself…then actually enter your password and actually press enter. TO THE TAPE CLOSET I GO!
i7-2600k (4.1Ghz), 16GB DDR3-1600, 6970 2GB, GA-Z68X-UD3H-B3, Recon3D Audio (PCI-E), WD 1TB Black (os/programs), WD 2TB Green (games), WD 3TB Green (movies/music), Win 8.1 Pro x64
Check out my site for trance mixes and other goodies! linserv.biz.tm
ClickClick5
Gerbil XP
Gold subscriber
 
 
Posts: 455
Joined: Fri Aug 15, 2008 8:28 pm
Location: Somewhere in a land the doctor says does not exist...

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 10:12 pm

JosiahBradley wrote:This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.


Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)
keltor
Gerbil First Class
 
Posts: 177
Joined: Thu May 10, 2012 4:29 pm

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 10:15 pm

Code: Select all
find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b


This didn't take all THAT long on RHEL 6.4, 5.5, Ubuntu Server 12.04 or my Mac. I ran on all 4 as root. All of our machines except my Mac are pretty close to default except they are pretty locked down on the Firewall side.
keltor
Gerbil First Class
 
Posts: 177
Joined: Thu May 10, 2012 4:29 pm

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 10:53 pm

keltor wrote:
JosiahBradley wrote:This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.


Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)


It wasn't /dev causing the problem, but the resulting file definitely had 4G of data under /proc.
JosiahBradley
Gerbil In Training
Silver subscriber
 
 
Posts: 6
Joined: Tue Sep 18, 2012 11:18 pm

Re: Don't try this at home. Or at work.

Postposted on Thu Mar 13, 2014 11:53 pm

keltor wrote:
JosiahBradley wrote:This reminds me when I naively attempted a system backup with " tar -cjf backup.tar.bz2 / ". It managed to tar the entire system including the entire contents of RAM at the time and itself. Surprisingly it worked.

Every version of tar I'm aware of backs up devnode files as devnodes and wouldn't make a copy of RAM. Tar is a tape control/backup program. (There is a command that DOES backup device file contents, but I don't really remember it anymore.)

http://xkcd.com/1168/

keltor wrote:
Code: Select all
find . -type f -or -type l | sort | xargs --delimiter=\\n md5sum -b

This didn't take all THAT long on RHEL 6.4, 5.5, Ubuntu Server 12.04 or my Mac. I ran on all 4 as root. All of our machines except my Mac are pretty close to default except they are pretty locked down on the Firewall side.

I assume you were starting from a directory that didn't contain any nested copies of an old /dev directory though. In my case, that was the gasoline that provided the fuel for the match!

I dare you to create a symlink to /proc/kcore somewhere in that directory structure, then try running that command line again. :wink:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Fri Mar 14, 2014 5:43 pm

For those that succeeded versus those that failed on the big tar, it's going to be down to permissions. As JBI mentioned, /dev/core is a symbolic link to /proc/kcore. But the permissions on /proc/kcore are 0400 and owned by root, so only if you tar as root do you get the hang as it digests the entire virtual address space.

Tar itself is pretty easy though. Almost daily I use
Code: Select all
tar cf - dir1 --exclude-vcs | (cd dir2; tar xvf -)

It's a good recursive copy.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Don't try this at home. Or at work.

Postposted on Sat Mar 15, 2014 8:08 am

My worst screw up was at a clients site. I was my younger, but still, it was a basic mistake that I should have caught. I was working with them to implement a source code analysis tool that took a while to run. It was setup with a dedicated account which owned all the data it produced. That account set a number of options in its shell environment, one of which was $WORKDIR. To ensure that a run was clean and not polluted from prior artifacts, the analysis script had the following line in it.

Code: Select all
/bin/rm -rf ${WORKDIR}/*


Since it needed to run weekly, it was put in the owning accounts crontab. A few days later, I got a very panicked call from my boss saying that the customer was seeing files all across their network disappear. See, while I had made one mistake, the customer had made another. The majority of their NFS, including NFS mounted home directories, had 775 permissions and all users were in the same group. I was on an airplane the next day. That was most unpleasant. That incident ingrained in me the need for defensive coding.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1715
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Re: Don't try this at home. Or at work.

Postposted on Sat Mar 15, 2014 9:05 am

Any sort of recursive file processing gets very dangerous when network mounts are involved. It is generally prudent to include command line switches to prevent traversal of mount points (--one-file-system in the case of rm -r).

I've also had annoying things happen when importing file systems into a VirtualBox VM using VirtualBox's "shared folder" feature. One day at work I noticed that my VM was chewing up an awful lot of CPU time, and my workstation was hammering the network hard. A bit of investigation revealed that the indexing service (known as updatedb on Linux systems) was attempting to index the VirtualBox shared folder. Which in turn contained mount points to multiple Linux and Windows servers on the network. But because this was in a directory which was being imported via VirtualBox, the fact that some of the subdirectories were network shares (which updatedb would've normally skipped by default) was obscured. The VM was attempting to index the entire damn network!
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Don't try this at home. Or at work.

Postposted on Sat Mar 15, 2014 5:50 pm

just brew it! wrote:It took a while to bring things down, but eventually the desktop became unresponsive and I could not Ctrl-Alt-Fx to alternate console sessions either.

Edit: The unclean shutdown also caused the software RAID-1 array to drop out of sync. An automatic background re-sync kicked in as soon as the system was rebooted. And no, I did not try this on my main workstation! :lol:


That reminds me. I found this the other day.

Perform a gentle Linux restart: Ctrl + Alt + PrtSc (SysRq) + reisub (Linux gets frozen, what do you do?: http://www.jovicailic.org/2013/05/linux ... do-you-do/)

It requires three hands, and I'm not sure it would have helped. It's good to know though.
Flatland_Spider
Gerbil Elite
 
Posts: 851
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Don't try this at home. Or at work.

Postposted on Sun Mar 16, 2014 8:29 am

Flatland_Spider wrote:That reminds me. I found this the other day.

Perform a gentle Linux restart: Ctrl + Alt + PrtSc (SysRq) + reisub (Linux gets frozen, what do you do?: http://www.jovicailic.org/2013/05/linux ... do-you-do/)

It requires three hands, and I'm not sure it would have helped. It's good to know though.

I did not know about that. Turns out that Wikipedia has a complete list of supported "Magic SysRq" commands: http://en.wikipedia.org/wiki/Magic_SysRq_key

Not all of them appear to work on my Ubuntu 12.04 system, but most of them do.

As an aside, this is definitely something that needs to be taken into consideration if anyone is using Linux in a kiosk-style application with a keyboard attached. At least it is easy to disable with a system configuration option. But to disable it you need to be aware that it exists in the first place!

I also mentioned it to my Linux guru/evangelist friend, and (as expected) he already knew about it. Says he's used it quite a bit when he's done driver development (he's the author of some of the drivers used by MythTV).
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37705
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer


Return to Linux, Unix, and Assorted Madness

Who is online

Users browsing this forum: No registered users and 2 guests