TR Forums

Hey guys!

TR needs a new multi-threaded compile test for storage benchmarking, and it should be as I/O bound as possible, which leads me to thinking that it has to be something with lots of small files. Benchmark time shouldn't be too long if possible, with contemporary hardware (though a short run time is secondary). Any suggestions?

Since it's been a few days with no replies:
1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).
2) Do such compile tests actually exist? My understanding is that IO-bound compile tests are generally limited by the filesystem's ability to handle metadata, rather than the disk. So running on the Windows filesystem you'd probably be able to tell the difference between a modern SSD and a 5400-rpm laptop drive, but I don't think there'd be much difference in between. (Though, of course, if the point is to say "buy an SSD for compiling!" then that might be worth it.)

Why not simply use something (anything, really) that's compiled with the Boost uBlas templated matrix library with a make -j directive? The dependencies for a relatively simple (proprietary, alas) matrix math application has >400 header files. I'm sure that with a few more includes, we can get >1000 header files included.

www.boost.org

The problem for benchmarking will be having the disk cache emptied so that it's not "cheating." I generally compile with two different compilers for testing, and use gcc under windows with the cygwin, along with MSVC++ with the multiple compile flag being set for both.

My problem with suggesting a project is that I am not entirely sure how much of the compile+link process is I/O and how much of it is the CPU. Both are equally important for compile+link. May be disabling optimizations and/or doing debug builds will reduce CPU involvement?

SNM wrote:

1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait Good idea, nevertheless.

duke_sandman: that looks neat, though we'd prefer to use something a bit more prepackaged. Still, it's a pretty nice idea, keep'em coming.

FF: good idea on disabling optimizations and debug builds, that might help.

I did have one last-resort plan that might be worth exploring: I was thinking of using the "shotgun" approach and pick a reasonably-sized opensource app, then extracting to several directories (dir1, dir2, dir3, dir4), and firing up compiles all in parallel. How stupid is this idea?

morphine wrote:

SNM wrote:

1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait Good idea, nevertheless.

Hmm. Test it before assuming that, I think it's <10 minutes on a big machine and is probably under 20 on a modern quad-core.
A quick Google shows it taking ~5 minutes a few years ago.

Boost has Visual Studio project files that you can use to get up and running quickly. So on Windows it should be relatively simple. Only thing I don't know is how CPU involved are the boost projects when they are compiled. Boost gets my vote as long as we can effectively exclude the CPU involvement.

SNM wrote:

morphine wrote:

SNM wrote:

1) Does the test need to run under Windows? If not, I suspect that the Linux kernel may come as close as you can get (though perhaps not, I'm no expert).

Yeah, though I suspect the Linux kernel is far too big to be practical as a benchmark. It's been years since I've compiled one but I remember it taking probably far, far longer than what Geoff is willing to wait Good idea, nevertheless.

Hmm. Test it before assuming that, I think it's <10 minutes on a big machine and is probably under 20 on a modern quad-core.
A quick Google shows it taking ~5 minutes a few years ago.

It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

If you want to increase the amount disk I/O will impact the build time why not try building a gentoo kernel? The bzip of the kernel will have to be extracted and written to disk before the build starts and the ebuild will make sure everything happens the same way on each system. You could even make a small bootable usb key with the basic environment installed so you just boot up and emerge gentoo-kernel

From my playing with the Linux kernel compiles when I've built my Folding CD stuff, it's been more CPU bound than disk bound. Some of the big diskbound stuff I've come across in the past has been doing things like unpacking the kernel from it's tar file (only single threaded, but potentially could do multiple copies in parallel) and also the start of the glibc compile (although I think that might be single threaded).

It would be relatively simple to put together a Linux CD (kernel plus busybox plus a copy of the kernel tar.bz2 file) that boots, does a mkfs (ext4?) on a specified partition, looks at how many processors are in the system and then launches multiple untars in to multiple directories in parallel. That should get the I/O load up nicely. wrap it all up in a time command and for bonus points run iostat whilst it is running to ensure that the I/O system is getting a workout and it should be good to go and give the TR staff a boot and go method of benchmarking.

ekul wrote:

It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

Hmm... I don't build kernels much, but I don't think it took an hour the last time I built an Ubuntu kernel from scratch. (I may be mis-remembering, it was a few months ago.)

If there's concern about compilations being CPU bound, you could run multi-threaded builds on a 4- or 6-core system?

just brew it! wrote:

ekul wrote:

It all depends on the config you build. A stripped down config for your particular machine with limited modules can take as little as a few minutes. A kernel fedora or ubuntu would ship can take over an hour or more to build even on fastest hardware.

Hmm... I don't build kernels much, but I don't think it took an hour the last time I built an Ubuntu kernel from scratch. (I may be mis-remembering, it was a few months ago.)

If you were building it from scratch I'd imagine you changed some config options; otherwise why bother? Building the config that ubuntu ships takes a substantial amount of time.

Given that there was a mention some time ago that OpenOffice had been suggested for this purpose and was under consideration, I presume that you've already ruled that out? Other than that, I don't really have any good Windows-based suggestions to make for something that might be more I/O-bound than CPU-bound.

In the non-Windows arena, I agree that the Linux kernel compile is ordinarily more CPU-bound than I/O-bound. I know that once upon a time X was reputed to take a huge amount of temp space to compile (ref.the knghtbrd quote, from the fortunes database, about a 4GB /tmp/ partition for exactly that purpose) - but I don't know whether or not it's still true in recent X versions. Still, if you're willing to go the *nix route in the first place, it might be worth looking into.

The trouble is that for the most part, including specifically for projects that most people are actually going to care about compiling (even for values of "most people" which ignore the people who will never compile anything at all), compilation is almost inherently bound more by the CPU than by anything else...

Since there is a lot of CPU involved, is the compile test just to "throw another test" to the mix or there is a specific target in mind?

Flying Fox wrote:

Since there is a lot of CPU involved, is the compile test just to "throw another test" to the mix or there is a specific target in mind?

Geoff's storage benchmarking, quite specifically, and multi-threaded at that.

The Linux boot cd + compile is a pretty good idea but I'd like to keep this in Windows if possible for simplicity. Right now I'm considering (among many), compiling OpenOffice or QT.

There are some very nice ideas in here, keep'em coming!