TR Forums

So I was trying to re-setup my photo gallery on my server, and one of the steps was to re-import the 10's of 1000's of pictures that I have. That just runs by a script called by the gallery program. However, when it is running that, my server is then otherwise rendered non-operational. I started the procedure about 12 hours ago, and I still can't access anything on the server.

I am wondering if there is a way to lower apache2's priority such that ssh can still get (so I can stop said script if things break). Are there any "best practice" web server techniques that I should know about with reference to this? My server is an Athlon xp 2400 with 512mb RAM so it's not like it's completely outdated. It's run everything I've thrown at it without a problem, actually.

So yeah, any ideas/thoughts/comments?

Usacomp2k3 wrote:

So yeah, any ideas/thoughts/comments?

As a baseline, have you tried running nice on the script process (or related processes -- whatever apache vessel is running the script if it's not forked cgi)? A priority of 19 would have it get out of the way of everything else in terms of CPU scheduling. Any I/O initiated or work done in another process (like your database) could still cause sluggishness, but making the original script nice 19 may allow it to get out of the way and throttle it enough to at least be tolerable.

Well it turns out that my server actually froze and that's why it wasn't responding. I'm going to try importing in much smaller chunks this time.

bitvector wrote:

Usacomp2k3 wrote:

So yeah, any ideas/thoughts/comments?

As a baseline, have you tried running nice on the script process (or related processes -- whatever apache vessel is running the script if it's not forked cgi)? A priority of 19 would have it get out of the way of everything else in terms of CPU scheduling. Any I/O initiated or work done in another process (like your database) could still cause sluggishness, but making the original script nice 19 may allow it to get out of the way and throttle it enough to at least be tolerable.

Would I have to do that manually? Also, that would just take affect on that spawned process, right? Not a permanent change.

Sounds like maybe the script is ill-behaved when you throw a lot of files at it. Just because Linux is generally better-behaved than Windows doesn't mean you can't hose it pretty bad by doing stuff like chewing up all the virtual memory.

just brew it! wrote:

Sounds like maybe the script is ill-behaved when you throw a lot of files at it. Just because Linux is generally better-behaved than Windows doesn't mean you can't hose it pretty bad by doing stuff like chewing up all the virtual memory.

Well it does process each picture when it adds it, so that's the cpu intensive part. I have it set right now to use netpbm as the toolkit for image manipulation, but gd and imagemagick are the other options.

PS, there are 12,171 pictures, here's a 'top' output

That doesn't look all that bad; load average is still less than 2.0, CPU isn't maxed out, and memory looks OK as well. The system really shouldn't be choking.

just brew it! wrote:

That doesn't look all that bad; load average is still less than 2.0, CPU isn't maxed out, and memory looks OK as well. The system really shouldn't be choking.

That was a completely different calculation (I was deleting the album that it had tried to create), so not really representative of the original situation.

I never did figure out how to read the percentage things. Where did you get that 2 from?

I was referring to the load average of 1.51. Basically it means that on average, you've only got 1.51 processes which are actively competing for CPU cycles. Which isn't bad.

Usacomp2k3 wrote:

Would I have to do that manually? Also, that would just take affect on that spawned process, right? Not a permanent change.

Right. It would only change the priority of the currently running process. If it spawns a bunch of processes that run for a while and quit, you can set the priority of the parent. New processes spawned from the parent will be inherit the parent's setting. It's still temporary in the sense that the next time that process starts up fresh it will be at the default priority. BTW, just to clarify since I said to "nice" the process... you use the "renice" command to change the priority of already running processes (there's two tools related to nice values, "nice" and "renice" -- nice spawns a program with a modified priority and renice changes the priority of a running program).

The fact that your system froze could point to any number of things (including hardware problems). Have you tried looking through syslog (and /var/log/messages on debian/ubuntu) to see if anything blew up?

BTW, here's a good article on load average calculation: http://www.teamquest.com/resources/gunt ... /index.htm

Short story is that the load average is a fancy time-dependent estimate of the number of running and runnable jobs (Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high). The three numbers you see are 1 minute, 5 minute and 15 minute averages.

bitvector wrote:

(Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high)

Ahh, I wasn't aware of that. I guess I can say I've learned my new thing for the day!

just brew it! wrote:

bitvector wrote:

(Linux also adds in processes blocked in uninterruptible sleep, so heavy I/O can make your load average go sky high)

Ahh, I wasn't aware of that. I guess I can say I've learned my new thing for the day!

That explains a lot. I used to have a debian machine with a pair of raptors in raid0 that would exhibit that exact issue under heavy disk load. I always thought it was something serious happening in the storage subsystem, as load averages would sometimes get as high as 12, but as I was running a vigorous benchmark against samba, I guess that should be expected.

bitvector wrote:

The fact that your system froze could point to any number of things (including hardware problems). Have you tried looking through syslog (and /var/log/messages on debian/ubuntu) to see if anything blew up?

It froze again last night while trying to import a smaller set of pictures. Last time it had alot of stuff on the screen that didn't make any sense to me, but I'll take a picture to see if ya'll can help. I'll dig through some logs too.

Yup, exact same thing on the screen today (in as best as I can tell):

Usacomp2k3 wrote:

Yup, exact same thing on the screen today (in as best as I can tell):

Hmm, kernel panic. And, it's crashing during vm handling, specifically during paging.

You have three possible reasons: kernel bug, bad memory, bad disk. I would opt for the middle one. It might be worth pulling down memtest86 and letting it run overnight.

--SS

If I'm going to pull down my web server for testing, I guess I can do that. Just not right now. I'm gone for the weekend, so probably next week. So run memtest for some time, I can do that. The failures both happened during fairly heavy strain of the 250gb storage drive, as it was copying pictures from one directory to another and then resize each one twice. That was also the hard drive that was in the computer when the other one died.
If I were to do some sort of hard drive test, what would you recommend? I believe it's a WD drive, so I guess I could just use the manufacturers test.

Usacomp2k3 wrote:

If I'm going to pull down my web server for testing, I guess I can do that. Just not right now. I'm gone for the weekend, so probably next week. So run memtest for some time, I can do that. The failures both happened during fairly heavy strain of the 250gb storage drive, as it was copying pictures from one directory to another and then resize each one twice. That was also the hard drive that was in the computer when the other one died.
If I were to do some sort of hard drive test, what would you recommend? I believe it's a WD drive, so I guess I could just use the manufacturers test.

I would probably go with the manufacturers tools. Unless the logs have other hints at drive problems (lots or errors in the logs), it is unlikely that it is the fault of the drive.

Best of luck tracking it down.

--SS

What I think I'm going to do is see if I can borrow a computer from work tomorrow (A core2duo with 2gb RAM). Then take the hard drives out of this one and put them into that one and do the imports while on the faster machine. In the mean time, I'm going to throw the ultimate Boot CD into the server and run a memtest and all that. I think I should also put a fan in there as all that is currently present are the cpu HSF and PSU. Putting one on the hard drives can't help but help out.