Folding Benchmark CD

Come join the... uh... er... fold.

Moderators: just brew it!, farmpuma

Folding Benchmark CD

Postposted on Sun Aug 13, 2006 9:03 pm

See the link in my sig, there's the new Folding Benchmark CD that I promised about 2 months ago - sorry it took so long.

Here's how it works:
The CD boots up and downloads both the folding client program plus the cores required all directly from Stanford (as required by their license) so the internet connection speed will not affect the result.
It then starts folding a WU per processor until it folds 1 complete step
It then takes the measurement for that WU, stops that one and starts the next.
If there are multiple processors it will leave the last WU running to provide load on the processors until all are complete and then shut them all down.
It benchmarks 4 WU types - Tinker, Amber, Gromacs and Gromacs3.3.

Results are available on the screen and on a web page, just point your browser at the machine being benchmarked. The URL is published on the screen as the machine starts up so you can tell what it is. Individual results are produced for each WU, average results are calculated across the 4 WU types and a total is calculated by multiplying the average by the number of processors.

Enjoy, but please don't meltdown my server :)

FYI this uses GPL code so if you want to mirror it you also must mirror the source file as well as the .iso. Other than that, you don't need the source file and it's so big that you really don't want to try downloading it from my poor server.

This uses a lot of new stuff that I am going to now work into the Folding CD generator / Diskless HOWTO / Folding USB stick and that should fix the problem with core 82 that has been reported recently.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Sun Aug 13, 2006 10:02 pm

:o You rock man. Did you happen to PM damage with this?
Usacomp2k3
Gerbil God
 
Posts: 21301
Joined: Thu Apr 01, 2004 4:53 pm
Location: Orlando, FL

Postposted on Sun Aug 13, 2006 10:02 pm

So the cores are downloaded, what about the WUs? If it is also from Stanford can we guarantee we can always get the same WU? Or it completely does not work that way that I am just blowing smoke here?
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24439
Joined: Mon May 24, 2004 2:19 am

Postposted on Mon Aug 14, 2006 8:17 am

WUs are on the CD so you will always fold the same WU and in fact it is the same steps on the same WU. Benchmarking completes before the WU are done so it never tries to send anything back to Stanford, even if it did they are old WUs that have already been folded and the results sent back so you wouldn't earn any points or mess up any of Stanford's results.

I've not PM'd Damage yet, hope some of the other gerbils here ensure it works before we take up any of his time.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Mon Aug 14, 2006 8:30 am

Good job. :)

Is this benchmark CD app portable to other servers without special stuff like mkisofs and such? May be we can host it off team2630.com and use it as our opening salvo of Folding Update : Reloaded too? :roll:
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24439
Joined: Mon May 24, 2004 2:19 am

Postposted on Mon Aug 14, 2006 10:08 am

Yes, the download is a straight link to a .iso image so can be put anywhere. It's only when you come to build it that you need all the funky stuff. Please note that you also need to provide the source package as well to comply with the terms of the GPL.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Wed Oct 11, 2006 11:09 pm

Holy poop. :o I didn't know this update was out there.

We're already thinking about how to benchmark folding on GPUs, but this is a huge step forward for CPUs. My schedule is packed right now, but I will have to give this a shot soon.

I take it this uses multiple cores if they're available? Up to how many?
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Wed Oct 11, 2006 11:35 pm

Ok, looks like detection doesn't work right on quad-core processors. notfred, think you can help with a fix? I'd like to use this as a benchmark in future CPU reviews.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Thu Oct 19, 2006 10:40 am

I definitely chose the wrong couple of weeks to go on vacation, sorry about that.

In terms of processors / cores, there are no limits although it needs to be recognised by the linux kernel. The kernel used is linux-2.6.17.1 I can try updating to a newer one and see if it is recognised correctly. Unfortunately I lack a quad-core to test on, could you ask your contacts if they know what linux version is required to support this processor?
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Mon Oct 23, 2006 4:44 pm

notfred wrote:I definitely chose the wrong couple of weeks to go on vacation, sorry about that.

In terms of processors / cores, there are no limits although it needs to be recognised by the linux kernel. The kernel used is linux-2.6.17.1 I can try updating to a newer one and see if it is recognised correctly. Unfortunately I lack a quad-core to test on, could you ask your contacts if they know what linux version is required to support this processor?


The Intel PR guys are trying to get the right answer from the guys there who might know. I assume the CPU detection mechanism you're using is tightly coupled with the kernel? Does it have a name? Also, they are asking what distro we're using. Not sure that's revelant, but it'd be nice to give them an answer.

Have you tried it with a newer kernel rev yet?

Thanks,
Scott
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Tue Oct 24, 2006 12:24 pm

There is no actual distro involved here, I suppose if you really want to give it a name it is the "notfred Folding Benchmark CD Distro", the important thing is it runs the vanilla kernel from kernel.org.

The way I detect the number of processors is just to grep through /proc/cpuinfo so this is pure Linux kernel stuff, explicitly:
Code: Select all
numprocs=`grep -c processor /proc/cpuinfo`


I'll have a try this evening at grabbing the latest kernel from kernel.org and see if I can put together a version with that for you to try.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Tue Oct 24, 2006 1:29 pm

Ok, that would help, because the Intel guys are pointing me to the Linux vendor at this point. :P
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Tue Oct 24, 2006 1:38 pm

Having a folding benchmark in CPU reviews would be awesome! Everytime a new CPU comes out I can't help but wonder how many PPD it would get me. A point per watt graph would be icing on the cake.
"I take sibling rivalry to the whole next level, if it doesn't require minor sugery or atleast a trip to the ER, you don't love her." - pete_roth
"Yeah, I see why you'd want a good gas whacker then." - VRock
dextrous
Gerbil Elite
 
Posts: 563
Joined: Mon Nov 22, 2004 1:49 pm
Location: Ooooooooooklahoma

Postposted on Tue Oct 24, 2006 7:59 pm

Damage - I've rebuilt with kernel 2.6.19-rc3 (dated 23rd October) and a test version is available at http://reilly.homeip.net/folding/test-bench.iso

If this still doesn't work, could you please point a browser at the benchmark client and then add "/cpuinfo" to the end, e.g. if it reports that it can be monitored at http://192.168.1.10 then point your browser to http://192.168.1.10/cpuinfo and it should download a file. If you can send that to me then I can take a look at why it isn't working properly. Hope there aren't any NDAs restricting this, if not PM me and we can try and work through some way of dealing with this.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Tue Oct 24, 2006 8:58 pm

Here's what it says:
---
Processor Detection
Processor 0 is an Intel(R) Core(TM)2 Quad CPU @ 2.66GHz
Found 1 processor(s)
---
Looks the same as last time to me.

The "cpuinfo" file downloads but is zero bytes when saved to disk. :(

I'm also getting an error while the benchmark runs:
---
Progress
Starting benchmark of Tinker WU
Hangcheck: hangcheck value past margin!
---

Dunno what that's about.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Tue Oct 24, 2006 9:20 pm

Is there a way to "hard-wire" a four-processor version? Perhaps a custom 4-way ISO could allow us to step around this problem for the time being?
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Tue Oct 24, 2006 9:59 pm

Not sure what the hangcheck timer message is, probably just a bug in the bleeding edge version of the kernel.

The cpuinfo file should be there, it works for me with firefox and that first bit of output that identifies the processor is from reading that file, were you using another browser?

That first bit of output is suggesting to me though that the cpuinfo is only showing one processor, so I suspect that the kernel may not properly schedule across the 4 cores if I hard code for 4 processors, I'm building one anyway - should have it ready first thing tomorrow.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Tue Oct 24, 2006 10:01 pm

notfred wrote:The cpuinfo file should be there, it works for me with firefox and that first bit of output that identifies the processor is from reading that file, were you using another browser?


Nope, was using Firefox. Also tried IE6, with the same result. Zero bytes.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Wed Oct 25, 2006 8:30 am

OK hardcoded to 4 processor version is at http://reilly.homeip.net/folding/4bench.iso

I've also had an idea of another way to detect how many processors, will have to try putting it together tonight.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Wed Oct 25, 2006 12:37 pm

Well, it's running, but as long as it's taking to finish a WU, I'm thinking it's only using one core. I'll let you know.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Wed Oct 25, 2006 12:59 pm

Yeah, it's definitely only using one core. Still running, first PPD number came in, for Tinker, and it's very low. Was worth a shot, I guess.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Wed Oct 25, 2006 9:29 pm

OK I think I have cracked it. It turns out that there are at least 2 different methods of an OS determining if the board is SMP:
1) Intel MultiProcessor Specification (both v1.1 and v1.4)
2) ACPI, specifically the MADT

I didn't have ACPI enabled in the linux kernel that all my folding stuff uses and so it was detecting SMP on my AMD X2 (on an Asus K8N-VM/CSM) through the Intel MultiProcessor Specification 1.4, but I bet that is not supported in the BIOS of the quad core board that you have. I've enabled ACPI and tested and it now detects SMP through ACPI on my rig.

I also found I had a bit of a problem with the debugs that I was giving you, so I have fixed those as well.

To make a long story short, new version is up on the benchmark webpage
http://reilly.homeip.net/folding/benchmark.html

If this doesn't work, do the web browser thing, but this time it will be http://192.168.1.10/cpuinfo.txt and also http://192.168.1.10/dmesg.txt (substitute the appropriate IP addresses of course).

Sorry for not finding this sooner.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Wed Oct 25, 2006 11:27 pm

Looks like it's working! w00t! Running on three boxes now...
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Wed Oct 25, 2006 11:33 pm

Ok, so here's the next hard question: Do you think this thing really reflects performance scaling on multiple cores accurately? It appears to be running two or four WUs at once, one per core, and then finishing the rest and coming out with a PPD number for each WU type. Is there more to it than that? How does it reward the fact that two or more cores are present?

EDIT: Ah, I see. You're running WUs on all cores simultaneously to make sure the cores are all loaded, and then you're multiplying the average PPD by the number of cores.

I guess that works, in a pinch. Explains this crazy high score I've got here. :)
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Thu Oct 26, 2006 12:59 am

For those of us still living in the Jurassic (dial-up) or using a USB wireless receiver can we still use the Folding Benchmark or do I need to be connected to the internet through a wired ethernet connection?

Cheers.

iamajai
iamajai
Gerbil Team Leader
 
Posts: 231
Joined: Thu Jul 07, 2005 2:02 pm
Location: Vancouver, BC

Postposted on Thu Oct 26, 2006 2:57 am

You could get a host computer to set up as a shared internet gateway on a switch..

Btw, damage... results on those quad cores? :P
Image
Nitrodist
Grand Gerbil Poohbah
 
Posts: 3280
Joined: Wed Jul 19, 2006 1:51 am
Location: Minnesota

Postposted on Thu Oct 26, 2006 8:37 am

Damage,
I'm open to other suggestions on how to calculate a benchmark figure, but I think this is quite realistic. After all we all fold on all the CPUs we have, don't we? :)

Maybe the only slightly non-realistic part is that it runs one instance of each WU type, where in actual fact you may get the same WU type across multiple processors (becasue Stanford tend to hand them out in batches) which could cause more contention. I guess this would matter the most in the case of HyperThreading where different WU types may exercise different parts of the same physical CPU (e.g. SSE2 vs x87).

In counter to that, the benchmark just awards standard points for the WUs, Stanford sometimes hands out point bonuses for things like large WU, e.g. there are 600 point Gromacs that are really a 300 point WU with a x2 bonus. So if you think you have a high score now, imagine double that if you got those 600 pointers!

I hope the 2 balance each other out, it really depends on the WU mix and bonuses that Stanford is handing out.

This is basically why I print the points per WU type - that lets you examine the individual core's strengths and weaknesses, plus the average at the end - a guide to what it should produce.

In the end, it's a benchmark and as long as it is consistent across systems that's really what counts. I do think it is missing a DGromacs though, that's on my TBD list (a long with a ton of other stuff....)
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Thu Oct 26, 2006 8:44 am

iamajai,
You need an ethernet network connection (ethernet card must be supported by Linux - most of them are) and something providing DHCP on that network segment.

If you have dialup probably the best thing is to do as Nitrodist suggested and use a host computer as a router - either through Windows Internet Connection Sharing or something like FreeSCO, CoyoteLinux, LRP etc (not sure which ones support modems).
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Postposted on Thu Oct 26, 2006 1:35 pm

notfred, I think your basic formula isn't terrible, but it does have a weakness in that it doesn't really address locality, memory access patterns, and resource sharing when running similar WU types across multiple cores.

I am pretty dubious, however, about doing the multiply by the number of front-ends given the presence of Hyper-Threading. I suppose an argument could be made for it.

My plan at present is to move forward with testing this version of the CD, and I'll simply explain how it works whenever I present any results. Folks can make up their own minds about the value of the numbers, but it's a darn sight better than nothing.
Scott Wasson - "Damage"
Editor - The Tech Report
Damage
TR Staff
Gold subscriber
 
 
Posts: 1690
Joined: Wed Dec 26, 2001 7:00 pm
Location: Lee's Summit, Missouri, USA

Postposted on Thu Oct 26, 2006 2:29 pm

I do leave the WU running to load the CPUs so that should keep the hyper-threading numbers down, but I agree on the weaknesses you highlight. Who knows, if you use it in an article someone may come up with a better way of doing things in a comment that I can then implement?

I suppose one test worth doing that I might try (if I ever get time) is to compare folding different WUs on each core with folding the exact same WU on each core. I need to spend some time first fixing up my other diskless folding things with what I have learnt here though.
notfred
Grand Gerbil Poohbah
 
Posts: 3736
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Next

Return to TR Distributed Computing Effort

Who is online

Users browsing this forum: No registered users and 5 guests