Dude, it's a double Etc. day! What the heck??
I've sent Cyril out on assignment, so news may be... nonexistent for the rest of the day. So here you have me.
I mentioned a WorldBench problem in my earlier Etc. post. This doesn't look like an issue that will be easily resolved, and it's one in a string of issues with big, scripted tests like that. I may end up dropping WorldBench or just reducing our use of it to a few of the most interesting tests. That might open up room for a few new tests in our CPU benchmark suite, which currently looks largely like what we used in our Lynnfield review, with the exceptions that I've switched to four newer games and updated some applications.
You guys often have suggestions for CPU tests we might try. Unfortunately, I often forget them between the time you make them and the time we go to re-work our test suite, which only happens a few times a year. Also, many of those suggestions present real problems for various reasons. I'm willing to do some of the work in sorting out a new test, but it has to be practical and workable. Here are a few criteria that help determine whether a test will work for us:
- The best tests are based on real-world applications and common usage models—things people do every day with their PCs. We're not looking for too many additional synthetic tests, and we don't want to serve as your personal lab service for your unique and perhaps really weird usage case. Sorry! (Although we might be persuaded to consider interesting apps for a workstation/server CPU test.)
- Any software licensing enforcement involved should be easy. We're going to be installing this program across four to six different test platforms at one time. Nazi DRM on a $500+ software suite is pretty much a non-starter. We even rule out some games on this basis. Free software is our favorite, but affordable software with reasonable licensing or free license keys for the press can work.
- We also need real-world workloads. Having access to a press/eval key of a mega-expensive application is no help if we don't have a data set for it that we can use. This is more of a problem in workstations and servers than on the desktop, though.
- Any tests should have a straightforward, repeatable means of timing operations or otherwise measuring performance. Without that, uh, benchmarking is difficult. We can key on output file creation/last modified timestamps in many cases, but not all. Some applications simply offer no means of measuring performance.
- Setting up and running the test should be clean, easy, and quick. We're talking about doing this test at least three times on perhaps 10-20 different CPUs. If it takes four hours each time, yeah, that adds up. We have a similar issue is the test setup is difficult and we're doing it across four to six different platforms.
- Time is of the essence. I'm asking now, and I'm gonna have to freeze up our CPU test suite shortly. Cruel, I know, but life is short and involves various deadlines and holidays and things.
That's about it, I think, but I reserve the right to add more conditions at will as your suggestions clarify our needs. Within that framework, I'm happy to take suggestions, although if they're just, "You should test Photoshop," well, thanks, Skippy, but there's more to it than that.