Vista testing begins... cautiously
At long last, we have the final version of Windows Vista in house and have begun doing some testing with it. I've been following Vista's development for quite a while, and we still have to work out some major issues with this operating system before we can trust it for everyday testing use. Among them:
- Figuring out how to conduct our hardware testing operations within the bounds of Vista's activation scheme. The issues involved here are sufficiently complex that I don't want to get into all of them right now, but it appears that running a lab setup like ourswith constantly changing hardware configscould be anywhere from inconvenient to nearly impossible given how Vista's activation and licensing schemes are structured and enforced. I've spent many hours sorting through the licensing issues and have yet to find a wholly acceptable solution. We're still exploring our options.
- Choosing which version of Vista to use. I'm not talking about Ultimate versus Home Premium, but 32-bit versus 64-bit (or more precisely x86 versus x64.) My inclination is to use the x64 version, but I'm not sure that's what the enthusiast community will embrace, given lingering questions about driver support for some devices. Another sticking point may be the purported requirement for signed drivers in Vista x64. 32-bit Vista will allow their installation, but the x64 version will not. Already, I've installed two sets of unsigned drivers on our very first Vista test system, just to get graphics and sound working properly. This problem might ease up as Vista becomes more widespread, but then again, we're often testing pre-release hardware with early drivers. I would prefer to test with a 64-bit OS, especially for CPUs, but that may prove rather difficult for us.
- Developing Vista-ready benchmarking procedures. Microsoft has been adding more and more self-tuning mechanisms to its operating systems, to good effect, and Vista is the apex of that effort. Vista features like SuperFetch try to make sure the right data is in memory when needed to ensure system responsiveness. The thing is, one has to account for the effects of such dynamic mechanisms when doing performance testing. When Windows XP was released, Microsoft published a useful paper on how to benchmark with that OS, and we used its recommendations in developing our own testing procedures. Despite the fact that Vista has been shipping to corporate customers for a while now and is about to ship to consumers, I don't believe Microsoft has published anything similar about benchmarking with Vista. The closest I've found is this blog on Windows performance, which promised information about "preparing a system for accurate, repeatable benchmarking" in "a couple of days"but hasn't been updated since December 14, 2006.
Given everything, our approach to Vista right now involves a lot of caution. This update to Windows long overdue, and I want to transition to it as soon as possible, but Microsoft has made some choices that complicate matters for us substantially.