lower apache priority/system stability

Where Penguins and Daemons chill together in the warmth of the Sun.

Moderators: SecretSquirrel, notfred

lower apache priority/system stability

Postposted on Wed Oct 24, 2007 2:14 pm

So I was trying to re-setup my photo gallery on my server, and one of the steps was to re-import the 10's of 1000's of pictures that I have. That just runs by a script called by the gallery program. However, when it is running that, my server is then otherwise rendered non-operational. I started the procedure about 12 hours ago, and I still can't access anything on the server.

I am wondering if there is a way to lower apache2's priority such that ssh can still get (so I can stop said script if things break). Are there any "best practice" web server techniques that I should know about with reference to this? My server is an Athlon xp 2400 with 512mb RAM so it's not like it's completely outdated. It's run everything I've thrown at it without a problem, actually.

So yeah, any ideas/thoughts/comments?
Last edited by Usacomp2k3 on Mon Nov 05, 2007 12:20 am, edited 1 time in total.
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Re: lower apache priority

Postposted on Wed Oct 24, 2007 2:58 pm

Usacomp2k3 wrote:So yeah, any ideas/thoughts/comments?

As a baseline, have you tried running nice on the script process (or related processes -- whatever apache vessel is running the script if it's not forked cgi)? A priority of 19 would have it get out of the way of everything else in terms of CPU scheduling. Any I/O initiated or work done in another process (like your database) could still cause sluggishness, but making the original script nice 19 may allow it to get out of the way and throttle it enough to at least be tolerable.
bitvector
Grand Gerbil Poohbah
 
Posts: 3234
Joined: Wed Jun 22, 2005 4:39 pm
Location: Mountain View, CA

Re: lower apache priority

Postposted on Wed Oct 24, 2007 9:45 pm

Well it turns out that my server actually froze and that's why it wasn't responding. I'm going to try importing in much smaller chunks this time. 8)

bitvector wrote:
Usacomp2k3 wrote:So yeah, any ideas/thoughts/comments?

As a baseline, have you tried running nice on the script process (or related processes -- whatever apache vessel is running the script if it's not forked cgi)? A priority of 19 would have it get out of the way of everything else in terms of CPU scheduling. Any I/O initiated or work done in another process (like your database) could still cause sluggishness, but making the original script nice 19 may allow it to get out of the way and throttle it enough to at least be tolerable.

Would I have to do that manually? Also, that would just take affect on that spawned process, right? Not a permanent change.
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Wed Oct 24, 2007 9:46 pm

Sounds like maybe the script is ill-behaved when you throw a lot of files at it. Just because Linux is generally better-behaved than Windows doesn't mean you can't hose it pretty bad by doing stuff like chewing up all the virtual memory.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37710
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Wed Oct 24, 2007 9:59 pm

just brew it! wrote:Sounds like maybe the script is ill-behaved when you throw a lot of files at it. Just because Linux is generally better-behaved than Windows doesn't mean you can't hose it pretty bad by doing stuff like chewing up all the virtual memory.

Well it does process each picture when it adds it, so that's the cpu intensive part. I have it set right now to use netpbm as the toolkit for image manipulation, but gd and imagemagick are the other options.

PS, there are 12,171 pictures, here's a 'top' output
Image
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Wed Oct 24, 2007 10:07 pm

That doesn't look all that bad; load average is still less than 2.0, CPU isn't maxed out, and memory looks OK as well. The system really shouldn't be choking.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37710
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Wed Oct 24, 2007 10:11 pm

just brew it! wrote:That doesn't look all that bad; load average is still less than 2.0, CPU isn't maxed out, and memory looks OK as well. The system really shouldn't be choking.

That was a completely different calculation (I was deleting the album that it had tried to create), so not really representative of the original situation.

I never did figure out how to read the percentage things. Where did you get that 2 from?
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Wed Oct 24, 2007 10:19 pm

I was referring to the load average of 1.51. Basically it means that on average, you've only got 1.51 processes which are actively competing for CPU cycles. Which isn't bad.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37710
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: lower apache priority

Postposted on Wed Oct 24, 2007 11:35 pm

Usacomp2k3 wrote:Would I have to do that manually? Also, that would just take affect on that spawned process, right? Not a permanent change.

Right. It would only change the priority of the currently running process. If it spawns a bunch of processes that run for a while and quit, you can set the priority of the parent. New processes spawned from the parent will be inherit the parent's setting. It's still temporary in the sense that the next time that process starts up fresh it will be at the default priority. BTW, just to clarify since I said to "nice" the process... you use the "renice" command to change the priority of already running processes (there's two tools related to nice values, "nice" and "renice" -- nice spawns a program with a modified priority and renice changes the priority of a running program).

The fact that your system froze could point to any number of things (including hardware problems). Have you tried looking through syslog (and /var/log/messages on debian/ubuntu) to see if anything blew up?

BTW, here's a good article on load average calculation: http://www.teamquest.com/resources/gunt ... /index.htm

Short story is that the load average is a fancy time-dependent estimate of the number of running and runnable jobs (Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high). The three numbers you see are 1 minute, 5 minute and 15 minute averages.
bitvector
Grand Gerbil Poohbah
 
Posts: 3234
Joined: Wed Jun 22, 2005 4:39 pm
Location: Mountain View, CA

Re: lower apache priority

Postposted on Thu Oct 25, 2007 12:23 am

bitvector wrote:(Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high)

Ahh, I wasn't aware of that. I guess I can say I've learned my new thing for the day! :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37710
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: lower apache priority

Postposted on Thu Oct 25, 2007 8:52 am

just brew it! wrote:
bitvector wrote:(Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high)

Ahh, I wasn't aware of that. I guess I can say I've learned my new thing for the day! :lol:

That explains a lot. I used to have a debian machine with a pair of raptors in raid0 that would exhibit that exact issue under heavy disk load. I always thought it was something serious happening in the storage subsystem, as load averages would sometimes get as high as 12, but as I was running a vigorous benchmark against samba, I guess that should be expected.
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1583
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Re: lower apache priority

Postposted on Thu Oct 25, 2007 12:34 pm

bitvector wrote:The fact that your system froze could point to any number of things (including hardware problems). Have you tried looking through syslog (and /var/log/messages on debian/ubuntu) to see if anything blew up?

It froze again last night while trying to import a smaller set of pictures. Last time it had alot of stuff on the screen that didn't make any sense to me, but I'll take a picture to see if ya'll can help. I'll dig through some logs too.
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Thu Oct 25, 2007 7:45 pm

Yup, exact same thing on the screen today (in as best as I can tell):
Image
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Thu Oct 25, 2007 8:58 pm

Usacomp2k3 wrote:Yup, exact same thing on the screen today (in as best as I can tell):
Image


Hmm, kernel panic. And, it's crashing during vm handling, specifically during paging.

You have three possible reasons: kernel bug, bad memory, bad disk. I would opt for the middle one. It might be worth pulling down memtest86 and letting it run overnight.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1717
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Postposted on Thu Oct 25, 2007 10:28 pm

If I'm going to pull down my web server for testing, I guess I can do that. Just not right now. I'm gone for the weekend, so probably next week. So run memtest for some time, I can do that. The failures both happened during fairly heavy strain of the 250gb storage drive, as it was copying pictures from one directory to another and then resize each one twice. That was also the hard drive that was in the computer when the other one died.
If I were to do some sort of hard drive test, what would you recommend? I believe it's a WD drive, so I guess I could just use the manufacturers test.
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Fri Oct 26, 2007 7:31 am

Usacomp2k3 wrote:If I'm going to pull down my web server for testing, I guess I can do that. Just not right now. I'm gone for the weekend, so probably next week. So run memtest for some time, I can do that. The failures both happened during fairly heavy strain of the 250gb storage drive, as it was copying pictures from one directory to another and then resize each one twice. That was also the hard drive that was in the computer when the other one died.
If I were to do some sort of hard drive test, what would you recommend? I believe it's a WD drive, so I guess I could just use the manufacturers test.


I would probably go with the manufacturers tools. Unless the logs have other hints at drive problems (lots or errors in the logs), it is unlikely that it is the fault of the drive.

Best of luck tracking it down.

--SS
SecretSquirrel
Gerbil Jedi
Gold subscriber
 
 
Posts: 1717
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)

Postposted on Mon Nov 05, 2007 12:19 am

What I think I'm going to do is see if I can borrow a computer from work tomorrow (A core2duo with 2gb RAM). Then take the hard drives out of this one and put them into that one and do the imports while on the faster machine. In the mean time, I'm going to throw the ultimate Boot CD into the server and run a memtest and all that. I think I should also put a fan in there as all that is currently present are the cpu HSF and PSU. Putting one on the hard drives can't help but help out.
Usacomp2k3
Gerbil God
 
Posts: 21296
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL


Return to Linux, Unix, and Assorted Madness

Who is online

Users browsing this forum: No registered users and 2 guests