Looking for I/O bound compile test for benching

From Visual Basic to GNU C, this is the place to talk programming.

Moderators: SecretSquirrel, just brew it!

Looking for I/O bound compile test for benching

Postposted on Tue Jan 11, 2011 6:11 pm

Hey guys!

TR needs a new multi-threaded compile test for storage benchmarking, and it should be as I/O bound as possible, which leads me to thinking that it has to be something with lots of small files. Benchmark time shouldn't be too long if possible, with contemporary hardware (though a short run time is secondary). Any suggestions?
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
morphine
Grand Admiral Gerbil
Silver subscriber
 
 
Posts: 10065
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 1:43 pm

Since it's been a few days with no replies:
1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).
2) Do such compile tests actually exist? My understanding is that IO-bound compile tests are generally limited by the filesystem's ability to handle metadata, rather than the disk. So running on the Windows filesystem you'd probably be able to tell the difference between a modern SSD and a 5400-rpm laptop drive, but I don't think there'd be much difference in between. (Though, of course, if the point is to say "buy an SSD for compiling!" then that might be worth it.)
Core i7 920, 3x2GB Corsair DDR3 1600, 80GB X25-M, 1TB WD Caviar Black, MSI X58 Pro-E, Radeon 4890, Cooler Master iGreen 600, Antec P183, opticals
SNM
Emperor Gerbilius I
 
Posts: 6206
Joined: Fri Dec 30, 2005 10:37 am

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 1:57 pm

Why not simply use something (anything, really) that's compiled with the Boost uBlas templated matrix library with a make -j directive? The dependencies for a relatively simple (proprietary, alas) matrix math application has >400 header files. I'm sure that with a few more includes, we can get >1000 header files included.

www.boost.org

The problem for benchmarking will be having the disk cache emptied so that it's not "cheating." I generally compile with two different compilers for testing, and use gcc under windows with the cygwin, along with MSVC++ with the multiple compile flag being set for both.
duke_sandman
Gerbil
 
Posts: 21
Joined: Fri Sep 02, 2005 9:56 am
Location: Washington DC

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 2:52 pm

My problem with suggesting a project is that I am not entirely sure how much of the compile+link process is I/O and how much of it is the CPU. Both are equally important for compile+link. May be disabling optimizations and/or doing debug builds will reduce CPU involvement?
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24518
Joined: Mon May 24, 2004 2:19 am

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 5:14 pm

SNM wrote:1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait :) Good idea, nevertheless.

duke_sandman: that looks neat, though we'd prefer to use something a bit more prepackaged. Still, it's a pretty nice idea, keep'em coming.

FF: good idea on disabling optimizations and debug builds, that might help.

I did have one last-resort plan that might be worth exploring: I was thinking of using the "shotgun" approach and pick a reasonably-sized opensource app, then extracting to several directories (dir1, dir2, dir3, dir4), and firing up compiles all in parallel. How stupid is this idea?
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
morphine
Grand Admiral Gerbil
Silver subscriber
 
 
Posts: 10065
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 6:51 pm

morphine wrote:
SNM wrote:1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait :) Good idea, nevertheless.

Hmm. Test it before assuming that, I think it's <10 minutes on a big machine and is probably under 20 on a modern quad-core.
A quick Google shows it taking ~5 minutes a few years ago.
Core i7 920, 3x2GB Corsair DDR3 1600, 80GB X25-M, 1TB WD Caviar Black, MSI X58 Pro-E, Radeon 4890, Cooler Master iGreen 600, Antec P183, opticals
SNM
Emperor Gerbilius I
 
Posts: 6206
Joined: Fri Dec 30, 2005 10:37 am

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 7:14 pm

Boost has Visual Studio project files that you can use to get up and running quickly. So on Windows it should be relatively simple. Only thing I don't know is how CPU involved are the boost projects when they are compiled. Boost gets my vote as long as we can effectively exclude the CPU involvement.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24518
Joined: Mon May 24, 2004 2:19 am

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 9:30 pm

SNM wrote:
morphine wrote:
SNM wrote:1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait :) Good idea, nevertheless.

Hmm. Test it before assuming that, I think it's <10 minutes on a big machine and is probably under 20 on a modern quad-core.
A quick Google shows it taking ~5 minutes a few years ago.


It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

If you want to increase the amount disk I/O will impact the build time why not try building a gentoo kernel? The bzip of the kernel will have to be extracted and written to disk before the build starts and the ebuild will make sure everything happens the same way on each system. You could even make a small bootable usb key with the basic environment installed so you just boot up and emerge gentoo-kernel
ekul
Gerbil
 
Posts: 81
Joined: Thu Jan 17, 2008 1:25 pm

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 9:37 pm

From my playing with the Linux kernel compiles when I've built my Folding CD stuff, it's been more CPU bound than disk bound. Some of the big diskbound stuff I've come across in the past has been doing things like unpacking the kernel from it's tar file (only single threaded, but potentially could do multiple copies in parallel) and also the start of the glibc compile (although I think that might be single threaded).

It would be relatively simple to put together a Linux CD (kernel plus busybox plus a copy of the kernel tar.bz2 file) that boots, does a mkfs (ext4?) on a specified partition, looks at how many processors are in the system and then launches multiple untars in to multiple directories in parallel. That should get the I/O load up nicely. wrap it all up in a time command and for bonus points run iostat whilst it is running to ensure that the I/O system is getting a workout and it should be good to go and give the TR staff a boot and go method of benchmarking.
notfred
Grand Gerbil Poohbah
 
Posts: 3761
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 9:55 pm

ekul wrote:It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

Hmm... I don't build kernels much, but I don't think it took an hour the last time I built an Ubuntu kernel from scratch. (I may be mis-remembering, it was a few months ago.)

If there's concern about compilations being CPU bound, you could run multi-threaded builds on a 4- or 6-core system?
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37962
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 10:01 pm

just brew it! wrote:
ekul wrote:It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

Hmm... I don't build kernels much, but I don't think it took an hour the last time I built an Ubuntu kernel from scratch. (I may be mis-remembering, it was a few months ago.)


If you were building it from scratch I'd imagine you changed some config options; otherwise why bother? Building the config that ubuntu ships takes a substantial amount of time.
ekul
Gerbil
 
Posts: 81
Joined: Thu Jan 17, 2008 1:25 pm

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 10:18 pm

Given that there was a mention some time ago that OpenOffice had been suggested for this purpose and was under consideration, I presume that you've already ruled that out? Other than that, I don't really have any good Windows-based suggestions to make for something that might be more I/O-bound than CPU-bound.

In the non-Windows arena, I agree that the Linux kernel compile is ordinarily more CPU-bound than I/O-bound. I know that once upon a time X was reputed to take a huge amount of temp space to compile (ref.the knghtbrd quote, from the fortunes database, about a 4GB /tmp/ partition for exactly that purpose) - but I don't know whether or not it's still true in recent X versions. Still, if you're willing to go the *nix route in the first place, it might be worth looking into.

The trouble is that for the most part, including specifically for projects that most people are actually going to care about compiling (even for values of "most people" which ignore the people who will never compile anything at all), compilation is almost inherently bound more by the CPU than by anything else...
The Wanderer
Gerbil
 
Posts: 92
Joined: Tue Apr 22, 2003 12:20 pm
Location: Southern Maryland

Re: Looking for I/O bound compile test for benching

Postposted on Thu Jan 13, 2011 11:56 pm

Since there is a lot of CPU involved, is the compile test just to "throw another test" to the mix or there is a specific target in mind?
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24518
Joined: Mon May 24, 2004 2:19 am

Re: Looking for I/O bound compile test for benching

Postposted on Fri Jan 14, 2011 6:47 am

Flying Fox wrote:Since there is a lot of CPU involved, is the compile test just to "throw another test" to the mix or there is a specific target in mind?

Geoff's storage benchmarking, quite specifically, and multi-threaded at that.

The Linux boot cd + compile is a pretty good idea but I'd like to keep this in Windows if possible for simplicity. Right now I'm considering (among many), compiling OpenOffice or QT.

There are some very nice ideas in here, keep'em coming!
There is a fixed amount of intelligence on the planet, and the population keeps growing :(
morphine
Grand Admiral Gerbil
Silver subscriber
 
 
Posts: 10065
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)


Return to Developer's Den

Who is online

Users browsing this forum: No registered users and 2 guests