One of the projects I'm working on right now is testing a Core Duo-based ThinkPad laptop. As you may know, we've had mobile coverage around here, but mostly of components, not complete systems. I think it makes some sense to dip our toe into the water on laptop reviews, because we can't always test mobile processors and platforms outside of complete systems. Even when we can, say, throw a mobile processor on a desktop motherboard, we may not be testing something truly representative of a real mobile solution. My write-up of the Sharp M4000 WideNote was a start, but didn't incorporate any benchmarking.

I'm learning, however, that doing laptop testing well can get incredibly complicated in very short order, despite my plans to the contrary. I wanted to keep this laptop review fairly simple, so I decided to confine my testing to a few things: WorldBench for general performance, 3DMark03 to stress the crappy GMA950 graphics, Guild Wars for workable 3D gaming, and some battery life tests. That would give me a basic indication of performance, which I could compare to a similarly configured Pentium M laptop. I could then write about the system-specific quirks and design choices that make this laptop what it is. That sounded like a good plan at the outset, but life with mobile solutions is never that simple.

The trouble is this: mobile solutions offer multiple grades of performance, depending on how they are configured. The difference between no SpeedStep, basic SpeedStep, and really aggressive throttling can alter results in scripted tests like WorldBench significantly. These settings also change the battery life equation, as do many other things, including screen brightness, display and hard drive power-down timeouts, the use of wireless networking, and more. Trying to sort out all of the variables and conduct relevant tests isn't easy, especially since different laptops come with very different default settings for many of these things. The makers of MobileMark2005 suggest using a luminance meter in order to equalize various laptops' display brightness—not an item I have sitting on the shelf just yet.

Laptop makers complicate the setup process by installing "value-added" software on their systems, with tray icons sometimes stacked up a quarter of the way across the bottom of the screen. This ThinkPad's by no means an extreme case, but I spent some time agonizing over what to do: leave it all installed and test as is, with all of the slowdowns and unnecessary memory use intact? That would make some sense if I were testing the laptop as a stand-alone product, but this one came from Intel, and my goal was to use it as a sort of example of the Centrino Duo platform in action. Leaving all of the shovelware intact seemed a bit extreme. So did wiping the thing and starting over, particularly since Lenovo outfits it with all sorts of custom software and drivers for ThinkPad-specific features. In the end, I settled on a real-world approach that would mirror my own practice if I bought this system. I disabled a few apps that got in the way—Google Desktop, Picasa, and Diskeeper Lite's nagware tray icon—and left the rest of the ThinkPad software suite. I decided to turn off the Norton AntiVirus real-time protection feature during performance testing, as well. Testing again with NAV enabled might be an intriguing option for a dual-core system, but there's probably no perfect answer to the problem of pre-installed software.

At some point, you just have to choose what you're going to do and go with a hopefully representative set of tests and settings. The trouble, then, is the sheer number of hours involved in testing. A full WorldBench run on even a fast a laptop can take 12 to 18 hours (though the latter's only likely if you run into script hangups, which tend to happen on systems with software pre-installed by the manufacturer.) Testing at full speed and then again with SpeedStep/PowerNow enabled will take quite a while.

Then there's battery-life testing. The first step there is to run the battery out completely and recharge it again, conditioning it for later tests. This ThinkPad T60 comes with a double-sized battery hanging out of the back like a goiter, and running it entirely out of juice with MobileMark's conditioning test took about 10 hours. (That may sound like a long time, but MobileMark for some reason has long pauses built in where the display can go idle.) I expect it to take another 10 hours to fully recharge the thing. Then I can begin testing one of several workloads, including wireless browsing, DVD playback, and office productivity—all of which seem worthy of inclusion. Each test run and recharge cycle could take a full work day to complete. What happens if we go for three runs per workload? How about testing a couple of different power-saving profiles on each laptop? What about any manufacturer-provided power-saving options that differ from what Windows does? Perhaps my review will be out in time for Christmas!

These things would be manageable if I went into testing knowing exactly what I wanted to do, but I'm feeling my way through this first attempt, trying to decide what's worth doing, what's fair, and what isn't. As it stands, the experimentation is chewing up hours in great big chunks. Fortunately, I have other things to keep me busy, but my plans for producing a nice laptop write-up in short order are looking wobbly. I do think it's worth the time, though. Dynamic performance adjustments are standard-issue features in the mobile world, but they're making their way quickly into desktops and servers, as well. Getting a handle on these things will no doubt be important for all of our future testing. At least that's what I keep telling myself as I wait for this thing to recharge.

Tip: You can use the A/Z keys to walk threads.
View options

This discussion is now closed.