Stanford server issues

Come join the... uh... er... fold.

Moderators: just brew it!, farmpuma

Stanford server issues

Postposted on Sat Aug 16, 2008 3:40 pm

The SMP WU servers have been down since about noon EDT. The server status page has finally started updating again, so we can hope the SMP servers will be back up soon.

edit: It looks like the entire 171.64.65.xxx server group is down.

edit2: The server group came back up about 5:00 PM EDT. To Pegasus, yep, probably the same issue since .20, .103, and .106 serve GPU WUs.

edit3: Added "group" to edit#1 and edit#2. Added ".20" to edit#2.
Image Image
.* * M-51 * *. .The Whirlpool Galaxy.
farmpuma
Minister of Gerbil Affairs
Silver subscriber
 
 
Posts: 2311
Joined: Mon Mar 22, 2004 12:33 am
Location: Soybean field, IN, USA, Earth .. just a bit south of John .. err .... Fart Wayne, Indiana

Re: Stanford server issues

Postposted on Sat Aug 16, 2008 4:28 pm

For a while my ATI card wasn't able to turn in or get new WUs. Not sure if the GPU2 servers for ATI WUs were also down.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Stanford server issues

Postposted on Sat Aug 16, 2008 4:56 pm

Yeah they had a bunch down. I had trouble getting WU's up and down on my GPU for a while, but it finally downloaded a new one and one of the 2 needing uploaded made it. Still waiting since 13+ hours ago for a Linux SMP WU, though, on one box.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Stanford server issues

Postposted on Sat Aug 16, 2008 5:18 pm

Well, at least one of the servers was backed up for a while. My X2 box hard locked and required a reboot so I took the opportunity to change to notfred's VMware Folding appliance (can that even be called diskless anymore?). Fired it back up and it grabbed the backup files on my TFTP server. This afternoon it could not send the finished 2605 and could not get an assignment. Just checked it and everything seemed to be fine now. It even grabbed the A2 core and is now working on a 2662. We will see if the efficiency gains can be achieved on an "old" S939 X2.

I like notfred's new stuff. It picked up my old WU and worked through it and the new web frontend adds a lot of new features. This is probably the quickest way to get a VM-based SMP client up. I guess I can test that out a bit and make a recommendation. Thanks very much notfred! :D
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24240
Joined: Mon May 24, 2004 2:19 am

Re: Stanford server issues

Postposted on Sat Aug 16, 2008 5:41 pm

My Opteron finally grabbed a new WU about 28 minutes ago, anyway.

Ugh.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Stanford server issues

Postposted on Sun Aug 17, 2008 9:16 am

my SMP's still not able to send, unfortunatly
My Dreamcast PC thread
Main Rig: FX-6100 / Radeon 6950 / Velociraptor HD because old school rotations
pikaporeon
Graphmaster Gerbil
 
Posts: 1308
Joined: Mon Nov 19, 2007 4:42 pm
Location: Mississauga, ON, Canada.

Re: Stanford server issues

Postposted on Sun Aug 17, 2008 11:36 pm

Did that ever get resolved, pikaporeon?

I've discovered a new problem Stanford has, in their folding software (fah6) under Linux, it appears, but I don't have time to recount the whole thing here. I'll post about it tomorrow.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Assignment servers unreachable right now?

Postposted on Mon Aug 18, 2008 3:13 pm

Yep, either they are fixing what went down Saturday or something really big hit the fan.

Unable to ping the 171.64.65.64 SMP server or fah-web.stanford.edu

edit: merged duplicate 4:20 EDT, 18 August 2008
Image Image
.* * M-51 * *. .The Whirlpool Galaxy.
farmpuma
Minister of Gerbil Affairs
Silver subscriber
 
 
Posts: 2311
Joined: Mon Mar 22, 2004 12:33 am
Location: Soybean field, IN, USA, Earth .. just a bit south of John .. err .... Fart Wayne, Indiana

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:24 pm

Ragnar Dan wrote:Did that ever get resolved, pikaporeon?

I've discovered a new problem Stanford has, in their folding software (fah6) under Linux, it appears, but I don't have time to recount the whole thing here. I'll post about it tomorrow.

My WU sent about an hour ago
My Dreamcast PC thread
Main Rig: FX-6100 / Radeon 6950 / Velociraptor HD because old school rotations
pikaporeon
Graphmaster Gerbil
 
Posts: 1308
Joined: Mon Nov 19, 2007 4:42 pm
Location: Mississauga, ON, Canada.

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:28 pm

Stanford has had a massive power outage.

They also mention it on Dr. Pande's blog.

I haven't been able to connect to anything significant at Stanford for more than 90 minutes.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:31 pm

They have more power outages than any academic setting that I've ever heard of. When I was at the dorms at UF, we never lost power, even during the bout of hurricanes a few years ago that knocked out all of the rest of the city of Gainesville and 50+% of the whole state.
Their facilities people should be ashamed of themselves.
Usacomp2k3
Gerbil God
 
Posts: 21266
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:43 pm

They have machines go down, and networking problems, but I don't recall any great number of power outages. This one, as mentioned on Pande's blog, is an external power outage:
Palo Alto is having a massive power outage which has brought down much of Stanford too. More info to come.

We will try to get this up asap once we get power back but it will be rough for a while.

BTW, the first link to foldingforum has a new post saying power is back up but they're waiting for the server rooms to cool off before starting all the machines.
Last edited by Ragnar Dan on Mon Aug 18, 2008 3:46 pm, edited 1 time in total.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:45 pm

Ragnar Dan wrote:They have machines go down, and networking problems, but I don't recall any great number of power outages. This one, as mentioned on Pande's blog, is an external power outage:

Yeah, I guess you're right. It's mostly network problems, which is really sad too.
Usacomp2k3
Gerbil God
 
Posts: 21266
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 3:48 pm

Groannn.
That's why I can't send my result or get new work :


[20:37:46] - Attempt #6 to get work failed, and no other work to do.
Waiting before retry.
[20:40:27] + Attempting to get work packet
[20:40:27] - Connecting to assignment server
[20:40:48] - Couldn't send HTTP request to server
[20:40:48] + Could not connect to Assignment Server
[20:41:10] - Couldn't send HTTP request to server
[20:41:10] + Could not connect to Assignment Server 2
[20:41:10] + Couldn't get work instructions.
Life is short
fpsduck
Gerbil XP
 
Posts: 382
Joined: Fri Oct 05, 2007 4:13 am
Location: Unknown Planet called Earth

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 8:34 pm

Power is back on, but they are still waiting for the A/C to cool the server room down a bit before bringing the servers back up.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37400
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Stanford server issues

Postposted on Mon Aug 18, 2008 11:32 pm

I think most or all of the servers have been up for a while now, but I had to restart a couple of my clients to make sure they'd try without waiting umpteen hours before their next attempt to download a new WU. Now that they say they're starting to push the servers out to other campuses, they should probably allow the clients to try more quickly so as to minimize loss of folding time. At least once every 30 minutes should be allowed.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5353
Joined: Sun Jan 20, 2002 7:00 pm

Re: Stanford server issues

Postposted on Tue Aug 19, 2008 1:40 am

Usacomp2k3 wrote:They have more power outages than any academic setting that I've ever heard of. When I was at the dorms at UF, we never lost power, even during the bout of hurricanes a few years ago that knocked out all of the rest of the city of Gainesville and 50+% of the whole state.
Their facilities people should be ashamed of themselves.

It's because they're in California. Power problems now are nothing like what they were ~2000, but the system still runs under heavy load a lot and a lot of the academic institutions are wired in so deep that the electric companies need to shut them down in order to work on power lines on a disturbingly common basis.
Core i7 920, 3x2GB Corsair DDR3 1600, 80GB X25-M, 1TB WD Caviar Black, MSI X58 Pro-E, Radeon 4890, Cooler Master iGreen 600, Antec P183, opticals
SNM
Emperor Gerbilius I
 
Posts: 6206
Joined: Fri Dec 30, 2005 10:37 am


Return to TR Distributed Computing Effort

Who is online

Users browsing this forum: Exabot [Bot] and 2 guests