Quad CPU "crate farm" pics

[just brew it!]

Most of you folding fanatics already know about this, but I figured some of the new folders might want to see what a low-cost quad CPU "extreme folding" rig looks like.

System specs
CPUs: 1x Athlon XP 1700+ and 3x Athlon XP 2000+
Motherboards: 4 el-cheapo micro-ATX "integrated everything" mobos bought on clearance (1 FIC, 2 DFI, 1 MSI)
RAM: 3 sticks of PC-133 and 1 stick of PC-2100 from my junk closet
OS: LTSP 3.0
Case: 1 plastic storage crate from the local Ace Hardware, plus cable ties to mount the mobos

LTSP is designed to run completely diskless: It boots over the LAN, using DHCP and PXE. File storage and virtual memory swapping are also done over the LAN, to a single shared drive on a Linux server.

The server that hosts this mess is an old Athlon T-Bird 1.1GHz running Redhat Linux 8.0. And yes, the server folds too!

Without further ado, a couple of pics:


Top view of the crate... the fans on the shelf above the crate keep the air circulating, to prevent the crate from overheating


Close-up view... mounting the mobos facing each other saves a little space, making it possible to cram 4 mobos into the crate


Screenshot taken from the Linux server, showing the crate in action

When I can find the time, I'll do a detailed article on how to build one of these beasts, including configuring LTSP.

[danny e.]

impressive.

i didnt even know such a thing was possible

[emkubed]

I was seriously considering housing my folding bots (franken and emkubed) at the office in a crate setup such as this. Thing is, we have this heavy duty plastic cart on wheels, and the idea was brought up. Why not make the crate mobile?

Of course, it wouldn't be mobile until we unplug from the wall and rely on a battery backup of some sort, but things are in the works, let me tell you.

I'm VERY interested in LTSP for this. Do share.

[just brew it!]

What I'd really like to do is run through the entire setup again, using the latest version of LTSP (4.0) and a newer Linux distro, documenting everything as I go. The Redhat 8 + LTSP 3.0 combo still has some rough edges... like the NIS daemon not restarting automatically after a server reboot (which is fortunately a pertty rare occurence). I probably won't have time to do this until mid-August at the earliest.

If you want to give it a shot before then, download LTSP from ltsp.org; they've got some pretty good documentation on their site as well (though it is of course not specific to F@h). The hardest part was getting the NFS and NIS services both set up properly on the server; these are required in order for the LTSP nodes to share the server's disk.

There are two BIOS settings you need to change on the diskless nodes. First, you need to configure the BIOS such that the mobo will power up as soon as AC is applied to the PSU; this way you don't need to connect anything to the power switch header on the mobo -- the only physical connections to the mobo are the power supply and the network cable. The other change is to enable the onboard NIC's boot ROM, so that the board will boot over the LAN.

I test each board outside of the crate first, with a monitor and keyboard directly attached. Once it is booting LTSP and running the F@h client successfully, it gets mounted in the crate, and any further interaction with it is strictly over the network, using the rsh (remote shell) command.

Currently, I just power the nodes up, wait for them to boot, and start the folding client manually. I also monitor them manually, by occasionally checking their log files. Longer-term, I'd like to create some scripts that run on the server which will automatically fire up F@h when nodes are powered up, monitor the nodes & alert me if one of them goes down, etc...

Just out of curiosity emkubed, why do you want it to be mobile? So you can easily stash it somewhere out of the way? Throw in a wireless access point, and you could be untethered from the LAN as well!

[Haggis!]

Some thoughts from my "crating" experience. I originally put mine together as a temporary location to house a couple of old MB's when I was still crunching UD. But they're still plugging away, now running F@H 24/7 for 2630.

I'm using an actual milk crate, not a storage crate like the kind that are sold at many office supply and discount stores. I found that the actual milk crate is exactly the right size to allow standard ATX MB's to slide into. Like JBI, I'm using cable ties to mount the MB's in the crate,

I'm running 3 MB's in the crate, since I was not diskless when I originally configured up, I have three older HD's mounted in my crate. There was room for two HD's on the back of the crate, but the third occupies space that would be needed for the fourth MB.

Right now, the crate has a P3 1GHz., a P3 933. The third board was originally running an XP1900, but is currently a XP-M 2400 OC'd to 2305.
I'm running the boards off two power suppllies, using a ATX Y splitter.

The picture is from back when it still had the Biostar & XP 1900 mounted over on the right hand side.


And a close-up view at an angle-
http://home.comcast.net/~felispardalis/crate_b2.JPG

I've been toying with the idea of mounting the MB's on lengths of threaded rod, and mounting the PSU's off a piece of bar behind MB's.

[AMM]

Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.

What someone needs to design is a motherboard that simply contains a cpu slot, a memory slot and an ethernet port. That way you simply plug them in and they run using LTSP. Maybe some kind of general purpose diagnostics port would be useful for accessing the bios, but other than that I can't see what else would need to be on the motherboard.

[just brew it!]

Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.

What someone needs to design is a motherboard that simply contains a cpu slot, a memory slot and an ethernet port. That way you simply plug them in and they run using LTSP. Maybe some kind of general purpose diagnostics port would be useful for accessing the bios, but other than that I can't see what else would need to be on the motherboard.

"Integrated everything" micro-ATX boards are dirt cheap if you buy last year's model. I seriously doubt that someone would be able to build a product like you're describing, and sell it for less... especially given that it would be a low-volume "specialty" item.

[RoninGyrbill]

Thanks JBI. Good stuff...

Newegg had some refurbished igp boards for under 30 awhile back but currently it's almost cheaper to go with a new ASrock board for $48. I just remembered that durons don't run on a 166fsb so why not use last years models. Since I out of habit exclusively buy Newegg, anyone know where you can pick up some cheapo integrated boards that are regularly stocked for say,..less than $35?

I've already got the ancient K6III to load LTSP onto... (though I gotz no linux skills, ...I will get through it over time )

[AMM]

"Integrated everything" micro-ATX boards are dirt cheap if you buy last year's model. I seriously doubt that someone would be able to build a product like you're describing, and sell it for less... especially given that it would be a low-volume "specialty" item.

Yeah the fact that it would be produced in such low quantities was the main problem I could see. But from an efficiency point of view a whole computer just to fold does not make sense. I know a lot of people get a lot of satisfaction from folding, but I just can't stop thinking there must be a better way. Some kind of specialised hardware would be up for it.

Does anyone have any ideas about the algorithms used in folding and what particular bit of the cpu does folding use?

[just brew it!]

Does anyone have any ideas about the algorithms used in folding and what particular bit of the cpu does folding use?

Well, it really stresses the FPU. That's why newer CPUs with Extended 3DNow and SSE get such a huge performance boost on the Gromacs WUs, and why older CPUs without modern SIMD support (K6-x, P-II, and older) are so slow at it.

[derFunkenstein]

That's amazing, really...I'm quite envious.

Do the fans on the shelf actually move any air that really helps cool the crate down?

[Haggis!]

Thinking for a minute about completely diskless folding farms, how many parts on a motherboard are completely useless. AGP slots, PCI slots, USB ports, PS2 ports, Video out ports etc.

I'd draw the distinction between unused and useless. Some of the ports/slots that aren't needed once you're up and running are still quite useful for the initial configuration and troubleshooting.

In my case, I didn't buy new equpiment to shove it in a crate. My old boards ended up in the crate as a way to get them back up and working on the cheap after they had been replaced in various upgrades.

[just brew it!]

That's amazing, really...I'm quite envious.

Do the fans on the shelf actually move any air that really helps cool the crate down?

Well, the intermittent errors I was getting on one of the nodes stopped after I put the fans there. So based on circumstantial evidence, I'd say yes.

With the warmer weather these past few weeks, I've also been leaving the door of the closet cracked open a few inches, to keep things from getting too toasty in there.

(Haven't figured out how to read mobo sensors under LTSP, so I have no idea what the actual CPU temps are...)

[Haggis!]

Hey JBI, what if you moved the crate up onto the wire shelving where you have the fans? Your airflow is currently occluded on three sides, but by getting some space under the crate you might be able to get enough natural airflow up and through to eliminate the need for the fans.

[just brew it!]

Hey JBI, what if you moved the crate up onto the wire shelving where you have the fans? Your airflow is currently occluded on three sides, but by getting some space under the crate you might be able to get enough natural airflow up and through to eliminate the need for the fans.

Hmm... I'd probably need to disconnect all of the PSUs, and reroute the ATX power connectors up between the wires of the shelf. Getting the ATX power connectors on and off after the mobos are mounted in the crate is kind of a PITA... if my fingers were twice as long, that would help a lot!

On the plus side, I'd be able to move the PSUs underneath the shelf, which would tidy things up a bit. Sounds like a good idea.

Maybe as an interim solution, I could just elevate the crate a couple of inches by putting something under the corners. That would at least allow for some convection cooling through the bottom of the crate.

Edit: Cool, I got a mention on the front page. Damn... I'd better get started on that article!

[TheDVDMan]

Could a system such as this be used to (very) quickly encode DVD->DivX? Are there distributed divx encoders?

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?

[Haggis!]

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?

Yes. In fact my two SY7ISA+ boards are running off the same PSU. Depending on whether you have more time than money, or money than time, you can either build your own or order them prebuilt.

I'd be very, very careful about the PS you use. You're going to want to calculate what your load is going to be, and make sure you're not overtaxing it. I also wouldn't drive more than two MB's off a single PS.

[Usacomp2k3]

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?

Yes. In fact my two SY7ISA+ boards are running off the same PSU. Depending on whether you have more time than money, or money than time, you can either build your own or order them prebuilt.

I'd be very, very careful about the PS you use. You're going to want to calculate what your load is going to be, and make sure you're not overtaxing it. I also wouldn't drive more than two MB's off a single PS.

wow...great links

[just brew it!]

Could a system such as this be used to (very) quickly encode DVD->DivX? Are there distributed divx encoders?

Should be doable. Assuming you've got more than one file you want to encode, you don't even need a distributed encoder... just do multiple files in parallel (encode one file on each node). Network bandwidth might be an issue though, since they're diskless (all of the file I/O is happening over the network). This will cease to be an issue in a year or two, when micro-ATX boards with integrated gigabit Ethernet become commonplace.

I actually thought about setting up the crate farm to do MP3 encoding, when I was re-ripping/encoding my entire CD collection a few months back. But my dually Athlon workstation does well enough for MP3 encoding, using 2 copies of LAME running in parallel. With both CPUs banging away, the encoding speed is within a factor of 2 of the CD rip speed, so I decided not to bother futzing around with the crate farm for this.

Another fun little project I'd like to do is a real-time fractal explorer program. Similar concept to <a href="http://www.softlab.ece.ntua.gr/miscellaneous/mandel/mandel.html">this</a>, but allowing you to zoom/fly through the fractal in real-time. This would be rather computationally intensive; the crate farm would be a near-ideal backend for something like this.

Would there be anyway to do this w/o a PSU for each mobo? Say, splitting the power connectors?

Yeah, I was actually gonna do this. The plan was to cannibalize the ATX connectors off of a couple of dead PSUs, and splice them to good PSUs, more or less as shown on the page Haggis! linked. Then I was at the TigerDirect outlet store, and they had some generic ATX PSUs for cheap... so I took the path of least resistance instead.

I didn't realize that you could get pre-made ATX Y connectors. But at $21 a pop, you might as well just get another cheap PSU.

(Please note: Under normal circumastances, I do not advocate the use of cheap PSUs; you really do get what you pay for. But for a crate farm node -- which has no disk drives or video card -- even a cheap no-name unit should be able to handle the load just fine.)

[CelJean]

I can't wait to read your article JBI. This looks like a good way to get a bunch of folding power on the cheap, and I have got the bug. It should be interesting, though I have no experience at all with linux.

Tell you what, you write that article, and I will build my own crate farm dedicated entirely to folding for team 2630.

[SpotTheCat]

if you're making your own ATX splitter, I would suggest moving some +5 and grounds over from a molex to spread the load a bit.

[eckslax]

Tell you what, you write that article, and I will build my own crate farm dedicated entirely to folding for team 2630.

Hear that, JBI? Get cracking!

[wolfsonbr]

Can't wait for the guide, joined the forums, loaded a linux box and now I am looking forward to figuring out the nuances of linux.

I have several spare systems that have parts missing sitting around that will network boot so this is a good use for them.

[derFunkenstein]

I have several spare systems that have parts missing sitting around that will network boot so this is a good use for them.

Absolutely!

Welcome to the TR forums!

[Splinter]

An excellent source for 486 - P3 700s is your local computer chop shop.

Just let them know that you're up for buying old working towers for say $15 a pop and I'm sure you can work something out. It saves them having to pay someone to trash them

[Dirge]

Do you have to worry about grounding these things.. lol what kind of fire hazard do they pose not being secured in a metal case

[AMM]

They are grounded by being connected to the power supply. And not being in a metal case is fine, because you don't want any electrical components to touch a metal case.

As for fire hazard... when was the last time your pc caught fire ?

[just brew it!]

Do you have to worry about grounding these things.. lol

The PSUs are properly grounded (via the 3-wire power cord), which in turn grounds the mobos (via the ATX power connector). Yeah, I'd be a little careful during dry weather (potential for static damage), but as long as you ground yourself by touching one of the PSUs before touching anything else it should not be a problem.

what kind of fire hazard do they pose not being secured in a metal case

Well, the material used to make printed circuit boards is fire retardant (i.e. will not burn once the source of ignition is removed). Furthermore, the mobos are suspended in the crate with cable ties; so in the unlikely event of a catastrophic meltdown (e.g. voltage regulators short out and self-destruct), the heat-generating components would not be in contact with anything flammable.

[pmeysemb]

I just ordered a couple super cheap motherboards from compgeeks.com to go with a couple unused tbird processors I have laying around here at work. My plan is a tad different since I seem to be blessed(?) with numerous small HD's and other junk. I plan on setting up each board with good ol Win2K (we have corporate license here at work..we are a research facility after all ) and use a small hub plugged into just one port. I plan on using a small stackable shelf from a "Mart" since space is really not an issue for me. I will post some pics also in a week or so with this setup.

Gotta get those CPU's active!

Fold on....

By the way, if anyone is interested in about two large handfuls of old 4 and 8 MB SIMMS, they would be more than welcome to them - the pile is getting a bit comical...

[just brew it!]

I just ordered a couple super cheap motherboards from compgeeks.com to go with a couple unused tbird processors I have laying around here at work. My plan is a tad different since I seem to be blessed(?) with numerous small HD's and other junk.

Hehe... I have a few old HDs as well... going diskless was more of a space issue than anything.