Stanford server issues

[farmpuma]

The SMP WU servers have been down since about noon EDT. The server status page has finally started updating again, so we can hope the SMP servers will be back up soon.

edit: It looks like the entire 171.64.65.xxx server group is down.

edit2: The server group came back up about 5:00 PM EDT. To Pegasus, yep, probably the same issue since .20, .103, and .106 serve GPU WUs.

edit3: Added "group" to edit#1 and edit#2. Added ".20" to edit#2.

[Pegasus]

For a while my ATI card wasn't able to turn in or get new WUs. Not sure if the GPU2 servers for ATI WUs were also down.

[Ragnar Dan]

Yeah they had a bunch down. I had trouble getting WU's up and down on my GPU for a while, but it finally downloaded a new one and one of the 2 needing uploaded made it. Still waiting since 13+ hours ago for a Linux SMP WU, though, on one box.

[Flying Fox]

Well, at least one of the servers was backed up for a while. My X2 box hard locked and required a reboot so I took the opportunity to change to notfred's VMware Folding appliance (can that even be called diskless anymore?). Fired it back up and it grabbed the backup files on my TFTP server. This afternoon it could not send the finished 2605 and could not get an assignment. Just checked it and everything seemed to be fine now. It even grabbed the A2 core and is now working on a 2662. We will see if the efficiency gains can be achieved on an "old" S939 X2.

I like notfred's new stuff. It picked up my old WU and worked through it and the new web frontend adds a lot of new features. This is probably the quickest way to get a VM-based SMP client up. I guess I can test that out a bit and make a recommendation. Thanks very much notfred!

[Ragnar Dan]

My Opteron finally grabbed a new WU about 28 minutes ago, anyway.

Ugh.

[pikaporeon]

my SMP's still not able to send, unfortunatly

[Ragnar Dan]

Did that ever get resolved, pikaporeon?

I've discovered a new problem Stanford has, in their folding software (fah6) under Linux, it appears, but I don't have time to recount the whole thing here. I'll post about it tomorrow.

[axeman]

Anyone else experiencing this ?

[farmpuma]

Yep, either they are fixing what went down Saturday or something really big hit the fan.

Unable to ping the 171.64.65.64 SMP server or fah-web.stanford.edu

edit: merged duplicate 4:20 EDT, 18 August 2008

[pikaporeon]

Did that ever get resolved, pikaporeon?

I've discovered a new problem Stanford has, in their folding software (fah6) under Linux, it appears, but I don't have time to recount the whole thing here. I'll post about it tomorrow.

My WU sent about an hour ago

[Ragnar Dan]

Stanford has had a massive power outage.

They also mention it on Dr. Pande's blog.

I haven't been able to connect to anything significant at Stanford for more than 90 minutes.

[Usacomp2k3]

They have more power outages than any academic setting that I've ever heard of. When I was at the dorms at UF, we never lost power, even during the bout of hurricanes a few years ago that knocked out all of the rest of the city of Gainesville and 50+% of the whole state.
Their facilities people should be ashamed of themselves.

[Ragnar Dan]

They have machines go down, and networking problems, but I don't recall any great number of power outages. This one, as mentioned on Pande's blog, is an external power outage:

Palo Alto is having a massive power outage which has brought down much of Stanford too. More info to come.

We will try to get this up asap once we get power back but it will be rough for a while.

BTW, the first link to foldingforum has a new post saying power is back up but they're waiting for the server rooms to cool off before starting all the machines.

[Usacomp2k3]

They have machines go down, and networking problems, but I don't recall any great number of power outages. This one, as mentioned on Pande's blog, is an external power outage:

Yeah, I guess you're right. It's mostly network problems, which is really sad too.

[fpsduck]

Groannn.
That's why I can't send my result or get new work :


[20:37:46] - Attempt #6 to get work failed, and no other work to do.
Waiting before retry.
[20:40:27] + Attempting to get work packet
[20:40:27] - Connecting to assignment server
[20:40:48] - Couldn't send HTTP request to server
[20:40:48] + Could not connect to Assignment Server
[20:41:10] - Couldn't send HTTP request to server
[20:41:10] + Could not connect to Assignment Server 2
[20:41:10] + Couldn't get work instructions.

[just brew it!]

Power is back on, but they are still waiting for the A/C to cool the server room down a bit before bringing the servers back up.

[Ragnar Dan]

I think most or all of the servers have been up for a while now, but I had to restart a couple of my clients to make sure they'd try without waiting umpteen hours before their next attempt to download a new WU. Now that they say they're starting to push the servers out to other campuses, they should probably allow the clients to try more quickly so as to minimize loss of folding time. At least once every 30 minutes should be allowed.

[SNM]

They have more power outages than any academic setting that I've ever heard of. When I was at the dorms at UF, we never lost power, even during the bout of hurricanes a few years ago that knocked out all of the rest of the city of Gainesville and 50+% of the whole state.
Their facilities people should be ashamed of themselves.

It's because they're in California. Power problems now are nothing like what they were ~2000, but the system still runs under heavy load a lot and a lot of the academic institutions are wired in so deep that the electric companies need to shut them down in order to work on power lines on a disturbingly common basis.