Quad CPU "crate farm" pics

Come join the... uh... er... fold.

Moderators: just brew it!, farmpuma

Quad CPU "crate farm" pics

Postposted on Fri Jul 09, 2004 12:58 am

Most of you folding fanatics already know about this, but I figured some of the new folders might want to see what a low-cost quad CPU "extreme folding" rig looks like. :D

System specs
CPUs: 1x Athlon XP 1700+ and 3x Athlon XP 2000+
Motherboards: 4 el-cheapo micro-ATX "integrated everything" mobos bought on clearance (1 FIC, 2 DFI, 1 MSI)
RAM: 3 sticks of PC-133 and 1 stick of PC-2100 from my junk closet
OS: LTSP 3.0
Case: 1 plastic storage crate from the local Ace Hardware, plus cable ties to mount the mobos :wink:

LTSP is designed to run completely diskless: It boots over the LAN, using DHCP and PXE. File storage and virtual memory swapping are also done over the LAN, to a single shared drive on a Linux server.

The server that hosts this mess is an old Athlon T-Bird 1.1GHz running Redhat Linux 8.0. And yes, the server folds too! :D

Without further ado, a couple of pics:

Image
Top view of the crate... the fans on the shelf above the crate keep the air circulating, to prevent the crate from overheating

Image
Close-up view... mounting the mobos facing each other saves a little space, making it possible to cram 4 mobos into the crate

Image
Screenshot taken from the Linux server, showing the crate in action

When I can find the time, I'll do a detailed article on how to build one of these beasts, including configuring LTSP.
Last edited by just brew it! on Fri Jul 09, 2004 1:06 am, edited 1 time in total.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Fri Jul 09, 2004 1:05 am

impressive.

i didnt even know such a thing was possible
You don't have to feel safe to feel unafraid.
danny e.
Maximum Gerbil
Gold subscriber
 
 
Posts: 4374
Joined: Thu Apr 25, 2002 3:09 pm
Location: Indonesia/Nebraska/Wisconsin

Postposted on Fri Jul 09, 2004 1:13 am

I was seriously considering housing my folding bots (franken and emkubed) at the office in a crate setup such as this. Thing is, we have this heavy duty plastic cart on wheels, and the idea was brought up. Why not make the crate mobile?

Of course, it wouldn't be mobile until we unplug from the wall and rely on a battery backup of some sort, but things are in the works, let me tell you.

I'm VERY interested in LTSP for this. Do share.
emkubed
Gerbil Elder
 
Posts: 5848
Joined: Mon Jun 02, 2003 9:28 am
Location: Limbo

Postposted on Fri Jul 09, 2004 1:41 am

What I'd really like to do is run through the entire setup again, using the latest version of LTSP (4.0) and a newer Linux distro, documenting everything as I go. The Redhat 8 + LTSP 3.0 combo still has some rough edges... like the NIS daemon not restarting automatically after a server reboot (which is fortunately a pertty rare occurence). I probably won't have time to do this until mid-August at the earliest.

If you want to give it a shot before then, download LTSP from ltsp.org; they've got some pretty good documentation on their site as well (though it is of course not specific to F@h). The hardest part was getting the NFS and NIS services both set up properly on the server; these are required in order for the LTSP nodes to share the server's disk.

There are two BIOS settings you need to change on the diskless nodes. First, you need to configure the BIOS such that the mobo will power up as soon as AC is applied to the PSU; this way you don't need to connect anything to the power switch header on the mobo -- the only physical connections to the mobo are the power supply and the network cable. The other change is to enable the onboard NIC's boot ROM, so that the board will boot over the LAN.

I test each board outside of the crate first, with a monitor and keyboard directly attached. Once it is booting LTSP and running the F@h client successfully, it gets mounted in the crate, and any further interaction with it is strictly over the network, using the rsh (remote shell) command.

Currently, I just power the nodes up, wait for them to boot, and start the folding client manually. I also monitor them manually, by occasionally checking their log files. Longer-term, I'd like to create some scripts that run on the server which will automatically fire up F@h when nodes are powered up, monitor the nodes & alert me if one of them goes down, etc...

Just out of curiosity emkubed, why do you want it to be mobile? So you can easily stash it somewhere out of the way? Throw in a wireless access point, and you could be untethered from the LAN as well! :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Fri Jul 09, 2004 2:54 pm

Some thoughts from my "crating" experience. I originally put mine together as a temporary location to house a couple of old MB's when I was still crunching UD. But they're still plugging away, now running F@H 24/7 for 2630.

I'm using an actual milk crate, not a storage crate like the kind that are sold at many office supply and discount stores. I found that the actual milk crate is exactly the right size to allow standard ATX MB's to slide into. Like JBI, I'm using cable ties to mount the MB's in the crate,

I'm running 3 MB's in the crate, since I was not diskless when I originally configured up, I have three older HD's mounted in my crate. There was room for two HD's on the back of the crate, but the third occupies space that would be needed for the fourth MB.

Right now, the crate has a P3 1GHz., a P3 933. The third board was originally running an XP1900, but is currently a XP-M 2400 OC'd to 2305.
I'm running the boards off two power suppllies, using a ATX Y splitter.

The picture is from back when it still had the Biostar & XP 1900 mounted over on the right hand side.
Image

And a close-up view at an angle-
http://home.comcast.net/~felispardalis/crate_b2.JPG

I've been toying with the idea of mounting the MB's on lengths of threaded rod, and mounting the PSU's off a piece of bar behind MB's.
Image
Foldin' foldin' foldin' keep those folders rollin'
Haggis!
Gerbil Team Leader
 
Posts: 258
Joined: Mon Apr 26, 2004 11:12 am
Location: Plymouth, MN

Postposted on Fri Jul 09, 2004 4:25 pm

Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.

What someone needs to design is a motherboard that simply contains a cpu slot, a memory slot and an ethernet port. That way you simply plug them in and they run using LTSP. Maybe some kind of general purpose diagnostics port would be useful for accessing the bios, but other than that I can't see what else would need to be on the motherboard.
To understand recursion, you must first understand recursion.
AMM
Minister of Gerbil Affairs
 
Posts: 2010
Joined: Tue Jun 10, 2003 9:12 am
Location: London

Postposted on Fri Jul 09, 2004 4:31 pm

AntiMagicMan wrote:Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.

What someone needs to design is a motherboard that simply contains a cpu slot, a memory slot and an ethernet port. That way you simply plug them in and they run using LTSP. Maybe some kind of general purpose diagnostics port would be useful for accessing the bios, but other than that I can't see what else would need to be on the motherboard.

"Integrated everything" micro-ATX boards are dirt cheap if you buy last year's model. I seriously doubt that someone would be able to build a product like you're describing, and sell it for less... especially given that it would be a low-volume "specialty" item.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Fri Jul 09, 2004 5:12 pm

Thanks JBI. Good stuff...

Newegg had some refurbished igp boards for under 30 awhile back but currently it's almost cheaper to go with a new ASrock board for $48. I just remembered that durons don't run on a 166fsb so why not use last years models. Since I out of habit exclusively buy Newegg, anyone know where you can pick up some cheapo integrated boards that are regularly stocked for say,..less than $35?

I've already got the ancient K6III to load LTSP onto... (though I gotz no linux skills, ...I will get through it over time :) )
RoninGyrbill
Gerbil Elite
 
Posts: 566
Joined: Tue Feb 03, 2004 10:12 pm
Location: Upstate

Postposted on Fri Jul 09, 2004 5:26 pm

just brew it! wrote:"Integrated everything" micro-ATX boards are dirt cheap if you buy last year's model. I seriously doubt that someone would be able to build a product like you're describing, and sell it for less... especially given that it would be a low-volume "specialty" item.


Yeah the fact that it would be produced in such low quantities was the main problem I could see. But from an efficiency point of view a whole computer just to fold does not make sense. I know a lot of people get a lot of satisfaction from folding, but I just can't stop thinking there must be a better way. Some kind of specialised hardware would be up for it.

Does anyone have any ideas about the algorithms used in folding and what particular bit of the cpu does folding use?
To understand recursion, you must first understand recursion.
AMM
Minister of Gerbil Affairs
 
Posts: 2010
Joined: Tue Jun 10, 2003 9:12 am
Location: London

Postposted on Fri Jul 09, 2004 6:17 pm

AntiMagicMan wrote:Does anyone have any ideas about the algorithms used in folding and what particular bit of the cpu does folding use?

Well, it really stresses the FPU. That's why newer CPUs with Extended 3DNow and SSE get such a huge performance boost on the Gromacs WUs, and why older CPUs without modern SIMD support (K6-x, P-II, and older) are so slow at it.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Sat Jul 10, 2004 8:05 am

That's amazing, really...I'm quite envious.

Do the fans on the shelf actually move any air that really helps cool the crate down?
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21403
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Sat Jul 10, 2004 10:48 am

AntiMagicMan wrote:Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.


I'd draw the distinction between unused and useless. Some of the ports/slots that aren't needed once you're up and running are still quite useful for the initial configuration and troubleshooting.

In my case, I didn't buy new equpiment to shove it in a crate. My old boards ended up in the crate as a way to get them back up and working on the cheap after they had been replaced in various upgrades.
Image
Foldin' foldin' foldin' keep those folders rollin'
Haggis!
Gerbil Team Leader
 
Posts: 258
Joined: Mon Apr 26, 2004 11:12 am
Location: Plymouth, MN

Postposted on Sat Jul 10, 2004 12:14 pm

derFunkenstein wrote:That's amazing, really...I'm quite envious.

Do the fans on the shelf actually move any air that really helps cool the crate down?

Well, the intermittent errors I was getting on one of the nodes stopped after I put the fans there. So based on circumstantial evidence, I'd say yes.

With the warmer weather these past few weeks, I've also been leaving the door of the closet cracked open a few inches, to keep things from getting too toasty in there.

(Haven't figured out how to read mobo sensors under LTSP, so I have no idea what the actual CPU temps are...)
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Sat Jul 10, 2004 1:18 pm

Hey JBI, what if you moved the crate up onto the wire shelving where you have the fans? Your airflow is currently occluded on three sides, but by getting some space under the crate you might be able to get enough natural airflow up and through to eliminate the need for the fans.
Image
Foldin' foldin' foldin' keep those folders rollin'
Haggis!
Gerbil Team Leader
 
Posts: 258
Joined: Mon Apr 26, 2004 11:12 am
Location: Plymouth, MN

Postposted on Sat Jul 10, 2004 2:12 pm

Haggis! wrote:Hey JBI, what if you moved the crate up onto the wire shelving where you have the fans? Your airflow is currently occluded on three sides, but by getting some space under the crate you might be able to get enough natural airflow up and through to eliminate the need for the fans.

Hmm... I'd probably need to disconnect all of the PSUs, and reroute the ATX power connectors up between the wires of the shelf. Getting the ATX power connectors on and off after the mobos are mounted in the crate is kind of a PITA... if my fingers were twice as long, that would help a lot! :lol:

On the plus side, I'd be able to move the PSUs underneath the shelf, which would tidy things up a bit. Sounds like a good idea.

Maybe as an interim solution, I could just elevate the crate a couple of inches by putting something under the corners. That would at least allow for some convection cooling through the bottom of the crate.

Edit: Cool, I got a mention on the front page. Damn... I'd better get started on that article! :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Sat Jul 10, 2004 7:49 pm

Could a system such as this be used to (very) quickly encode DVD->DivX? Are there distributed divx encoders?

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?
TheDVDMan
Graphmaster Gerbil
 
Posts: 1276
Joined: Fri Apr 30, 2004 2:34 pm
Location: Not TR anymore!

Postposted on Sat Jul 10, 2004 9:25 pm

TheDVDMan wrote:Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?


Yes. In fact my two SY7ISA+ boards are running off the same PSU. Depending on whether you have more time than money, or money than time, you can either build your own or order them prebuilt.

I'd be very, very careful about the PS you use. You're going to want to calculate what your load is going to be, and make sure you're not overtaxing it. I also wouldn't drive more than two MB's off a single PS.
Image
Foldin' foldin' foldin' keep those folders rollin'
Haggis!
Gerbil Team Leader
 
Posts: 258
Joined: Mon Apr 26, 2004 11:12 am
Location: Plymouth, MN

Postposted on Sat Jul 10, 2004 9:49 pm

Haggis! wrote:
TheDVDMan wrote:Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?


Yes. In fact my two SY7ISA+ boards are running off the same PSU. Depending on whether you have more time than money, or money than time, you can either build your own or order them prebuilt.

I'd be very, very careful about the PS you use. You're going to want to calculate what your load is going to be, and make sure you're not overtaxing it. I also wouldn't drive more than two MB's off a single PS.


wow...great links
Usacomp2k3
Gerbil God
 
Posts: 21313
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Sat Jul 10, 2004 10:41 pm

TheDVDMan wrote:Could a system such as this be used to (very) quickly encode DVD->DivX? Are there distributed divx encoders?

Should be doable. Assuming you've got more than one file you want to encode, you don't even need a distributed encoder... just do multiple files in parallel (encode one file on each node). Network bandwidth might be an issue though, since they're diskless (all of the file I/O is happening over the network). This will cease to be an issue in a year or two, when micro-ATX boards with integrated gigabit Ethernet become commonplace. :wink:

I actually thought about setting up the crate farm to do MP3 encoding, when I was re-ripping/encoding my entire CD collection a few months back. But my dually Athlon workstation does well enough for MP3 encoding, using 2 copies of LAME running in parallel. With both CPUs banging away, the encoding speed is within a factor of 2 of the CD rip speed, so I decided not to bother futzing around with the crate farm for this.

Another fun little project I'd like to do is a real-time fractal explorer program. Similar concept to <a href="http://www.softlab.ece.ntua.gr/miscellaneous/mandel/mandel.html">this</a>, but allowing you to zoom/fly through the fractal in real-time. This would be rather computationally intensive; the crate farm would be a near-ideal backend for something like this.

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?

Yeah, I was actually gonna do this. The plan was to cannibalize the ATX connectors off of a couple of dead PSUs, and splice them to good PSUs, more or less as shown on the page Haggis! linked. Then I was at the TigerDirect outlet store, and they had some generic ATX PSUs for cheap... so I took the path of least resistance instead. :wink:

I didn't realize that you could get pre-made ATX Y connectors. But at $21 a pop, you might as well just get another cheap PSU. :roll:

(Please note: Under normal circumastances, I do not advocate the use of cheap PSUs; you really do get what you pay for. But for a crate farm node -- which has no disk drives or video card -- even a cheap no-name unit should be able to handle the load just fine.)
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Sun Jul 11, 2004 2:28 am

I can't wait to read your article JBI. This looks like a good way to get a bunch of folding power on the cheap, and I have got the bug. It should be interesting, though I have no experience at all with linux.

Tell you what, you write that article, and I will build my own crate farm dedicated entirely to folding for team 2630.
CelJean
Gerbil First Class
 
Posts: 109
Joined: Mon Dec 22, 2003 4:23 pm
Location: Colorado

Postposted on Sun Jul 11, 2004 2:42 am

if you're making your own ATX splitter, I would suggest moving some +5 and grounds over from a molex to spread the load a bit.
SpotTheCat
Gerbilus Supremus
 
Posts: 12262
Joined: Wed Jan 29, 2003 12:47 am
Location: a regular hole

Postposted on Sun Jul 11, 2004 4:16 am

Tell you what, you write that article, and I will build my own crate farm dedicated entirely to folding for team 2630.


Hear that, JBI? Get cracking! :wink: :lol:
"God created man. Samuel Colt made them equal."

"Government does not tax to get the money it needs; government finds a need for the money it gets." - Ronald Reagan
eckslax
Gerbil Elder
 
Posts: 5332
Joined: Sat Oct 05, 2002 4:22 pm
Location: Vast Right Wing Conspiracy HQ

Postposted on Sun Jul 11, 2004 9:30 pm

Can't wait for the guide, joined the forums, loaded a linux box and now I am looking forward to figuring out the nuances of linux.

I have several spare systems that have parts missing sitting around that will network boot so this is a good use for them.
wolfsonbr
Gerbil In Training
 
Posts: 1
Joined: Sun Jul 11, 2004 4:08 pm

Postposted on Sun Jul 11, 2004 10:45 pm

wolfsonbr wrote:I have several spare systems that have parts missing sitting around that will network boot so this is a good use for them.

Absolutely!

Welcome to the TR forums!
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21403
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Sun Jul 11, 2004 10:55 pm

An excellent source for 486 - P3 700s is your local computer chop shop.

Just let them know that you're up for buying old working towers for say $15 a pop and I'm sure you can work something out. It saves them having to pay someone to trash them :P
So how fast is your car, anyway?

Oh, about 1200mhz.
Splinter
Grand Gerbil Poohbah
 
Posts: 3223
Joined: Fri Apr 25, 2003 1:28 pm
Location: Vancouver BC

Postposted on Mon Jul 12, 2004 3:47 am

Do you have to worry about grounding these things.. lol what kind of fire hazard do they pose not being secured in a metal case
Dirge
Gerbil Jedi
 
Posts: 1554
Joined: Thu Feb 19, 2004 3:08 am
Location: New Zealand

Postposted on Mon Jul 12, 2004 4:58 am

They are grounded by being connected to the power supply. And not being in a metal case is fine, because you don't want any electrical components to touch a metal case.

As for fire hazard... when was the last time your pc caught fire :p?
To understand recursion, you must first understand recursion.
AMM
Minister of Gerbil Affairs
 
Posts: 2010
Joined: Tue Jun 10, 2003 9:12 am
Location: London

Postposted on Mon Jul 12, 2004 9:28 am

Dirge wrote:Do you have to worry about grounding these things.. lol

The PSUs are properly grounded (via the 3-wire power cord), which in turn grounds the mobos (via the ATX power connector). Yeah, I'd be a little careful during dry weather (potential for static damage), but as long as you ground yourself by touching one of the PSUs before touching anything else it should not be a problem.

what kind of fire hazard do they pose not being secured in a metal case

Well, the material used to make printed circuit boards is fire retardant (i.e. will not burn once the source of ignition is removed). Furthermore, the mobos are suspended in the crate with cable ties; so in the unlikely event of a catastrophic meltdown (e.g. voltage regulators short out and self-destruct), the heat-generating components would not be in contact with anything flammable.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Postposted on Mon Jul 12, 2004 10:53 am

I just ordered a couple super cheap motherboards from compgeeks.com to go with a couple unused tbird processors I have laying around here at work. My plan is a tad different since I seem to be blessed(?) with numerous small HD's and other junk. I plan on setting up each board with good ol Win2K (we have corporate license here at work..we are a research facility after all :wink: ) and use a small hub plugged into just one port. I plan on using a small stackable shelf from a "Mart" since space is really not an issue for me. I will post some pics also in a week or so with this setup.

Gotta get those CPU's active!

Fold on....

By the way, if anyone is interested in about two large handfuls of old 4 and 8 MB SIMMS, they would be more than welcome to them - the pile is getting a bit comical...
pmeysemb
Gerbil First Class
 
Posts: 120
Joined: Tue Jan 21, 2003 3:16 pm
Location: Duluth, MN USA

Postposted on Mon Jul 12, 2004 11:22 am

pmeysemb wrote:I just ordered a couple super cheap motherboards from compgeeks.com to go with a couple unused tbird processors I have laying around here at work. My plan is a tad different since I seem to be blessed(?) with numerous small HD's and other junk.

Hehe... I have a few old HDs as well... going diskless was more of a space issue than anything.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37843
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Next

Return to TR Distributed Computing Effort

Who is online

Users browsing this forum: Bing [Bot] and 7 guests