Vista testing begins… cautiously

At long last, we have the final version of Windows Vista in house and have begun doing some testing with it. I’ve been following Vista’s development for quite a while, and we still have to work out some major issues with this operating system before we can trust it for everyday testing use. Among them:

  • Figuring out how to conduct our hardware testing operations within the bounds of Vista’s activation scheme. The issues involved here are sufficiently complex that I don’t want to get into all of them right now, but it appears that running a lab setup like ours—with constantly changing hardware configs—could be anywhere from inconvenient to nearly impossible given how Vista’s activation and licensing schemes are structured and enforced. I’ve spent many hours sorting through the licensing issues and have yet to find a wholly acceptable solution. We’re still exploring our options.
  • Choosing which version of Vista to use. I’m not talking about Ultimate versus Home Premium, but 32-bit versus 64-bit (or more precisely x86 versus x64.) My inclination is to use the x64 version, but I’m not sure that’s what the enthusiast community will embrace, given lingering questions about driver support for some devices. Another sticking point may be the purported requirement for signed drivers in Vista x64. 32-bit Vista will allow their installation, but the x64 version will not. Already, I’ve installed two sets of unsigned drivers on our very first Vista test system, just to get graphics and sound working properly. This problem might ease up as Vista becomes more widespread, but then again, we’re often testing pre-release hardware with early drivers. I would prefer to test with a 64-bit OS, especially for CPUs, but that may prove rather difficult for us.

  • Developing Vista-ready benchmarking procedures. Microsoft has been adding more and more self-tuning mechanisms to its operating systems, to good effect, and Vista is the apex of that effort. Vista features like SuperFetch try to make sure the right data is in memory when needed to ensure system responsiveness. The thing is, one has to account for the effects of such dynamic mechanisms when doing performance testing. When Windows XP was released, Microsoft published a useful paper on how to benchmark with that OS, and we used its recommendations in developing our own testing procedures. Despite the fact that Vista has been shipping to corporate customers for a while now and is about to ship to consumers, I don’t believe Microsoft has published anything similar about benchmarking with Vista. The closest I’ve found is this blog on Windows performance, which promised information about “preparing a system for accurate, repeatable benchmarking” in “a couple of days”—but hasn’t been updated since December 14, 2006.

Given everything, our approach to Vista right now involves a lot of caution. This update to Windows long overdue, and I want to transition to it as soon as possible, but Microsoft has made some choices that complicate matters for us substantially.

Comments closed
    • walkerbowe
    • 13 years ago

    Yeah you guys are right, i like how the new Vista looks, but i don’t really find that much new fuctions on it. That weren’t previously availiable for Windows XP, I still think it’s a nice change, even though staying with XP for awhile. Would be better at least till Vista matures more, on 3/4 of my hardware is supported, as of now.

    • Snake
    • 13 years ago

    “l[http://www.extremetech.com/article2/0,1697,2091592,00.asp< ]§ "r[http://www.wired.com/news/technology/0,72295-0.html< ]§ "r[http://news.com.com/Experts+Dont+buy+Vista+for+the+security/2100-1016_3-6154448.html< ]§ "r[http://www.pcworld.com/article/id,128669-page,1/article.html< ]§ "r[http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9009124< ]§ "r[http://www.cio.com/blog_view.html?CID=28427< ]§ "r[http://www.digg.com/tech_news/Wait_Don_t_buy_Windows_Vista_2<]§ "r[

    • provoko
    • 13 years ago

    Later this year, I’m sure x64 will be the way to go, driver and software support will be here. It was the same way in early Windows 95 days when it came to 16/32bits.

    • Krogoth
    • 13 years ago

    Here is my conclusion for Vista in its current shape.

    There are considerable improvements in threading and memory management.

    It is too bad it is offset by immature drivers and other annoying bugs. Aero isn’t as much as a overhead as I thought. It barely uses my video card and eats up like 30-50MBs of RAM. It is not that much larger then XP’s pretty GUI settings.

    • totoro
    • 13 years ago

    Oh look, the MS blog has been updated.
    They do note that “it’s been a long couple of days.” LOL

    • flip-mode
    • 13 years ago

    Haha. I think you should run a poll once a month on what percentage of the site’s readers are using Vista and when the poll approaches 50% then worry about making a Vista switch.

    Vista FTL!
    Crazy product activation FTL!
    DX10 FTFuture! a.k.a. when there’s actually *a* game LOL if not a few!
    Windows XP is dead, long live Windows XP

    Ah well, I know this is a tech site and has to be on the leading edge but I’m really gnashing my teeth at all the licensing B-fing-S that’s tied to Vista.

    The obvious solution: buy the few copies that you need to cover your conscience, throw them in a drawer, and get yourselves some cracked copies.

    • Nutmeg
    • 13 years ago

    It’s just total bullshit that there are limits to hardware configs/changes etc for this. You should be able to do what you want with it once you’ve bought it, put it on a new machine, then put it back on a different one etc. And having to reactivate it after buying a new card or whatever would really piss me off.

    • drsauced
    • 13 years ago

    I think you need to do an Open License with MS, which is what we do at work. We are an educational institution, but it is open for corporate customers, which I believe TR is. It seems to be cheaper than OEM versions, and doesn’t really care what it gets put onto. But like you, I can’t tell if ‘unlimited upgrade’ means upgrade to some other Vista or other hardware.

    Edit: there are provisions for doing images with Vista, say if you have 6 machines in test cycles, you can prepare 6 images on a network drive, and then use sysprep /generalize to wipe computer-specific information before an image installation:

    §[< http://technet2.microsoft.com/WindowsVista/en/library/791eb40e-6624-4217-b913-82a02e07465d1033.mspx?mfr=true<]§ I'm not particularly smart, but this could be a solution.

    • blastdoor
    • 13 years ago

    Sounds like a major PITA with no easy answers.

    Maybe a partial solution is to keep most of your benchmarks on XP, and do only reviews of DX10 video cards on Vista, and only when there are DX10 games worth testing.

    Sorry — not a very appealing suggestion, I know.

    • Proesterchen
    • 13 years ago

    As for the IA32 vs x86-64 issue, as much as I’d like to see number from an x86-64 setup, I will be running the IA32 version when I switch simply due to the current state of drivers, and believe that’s what most people will do for at least another 2 years, maybe longer.

    IA32 Vista numbers would be more interesting to me as they’d apply directly to my own setup. So if I had to chose one (testing & publishing both would be great, but also very time consuming), I’d therefore chose the IA32 version for the time being.

      • indeego
      • 13 years ago

      Most people will. 64-bit will be more interesting for the workstation set of benches thoughg{<.<}g

        • Proesterchen
        • 13 years ago

        _ Agreed. _

    • StashTheVampede
    • 13 years ago

    MSDN subscription? MSDN subscription would tie you to a volume license key, which you could use/activate as necessary.

      • Damage
      • 13 years ago

      MSDN subscriptions are a possibility. However, there’s still a lot of ambiguity about how the licenses are managed for MSDN and a few other drawbacks.

      One of those drawbacks is price: MSDN is licensed per person, and I believe it would be roughly $650-700 person per annum for the OS-only subscription. That adds up. More importantly, it may not buy us much.

      I’ll admit that I simply don’t know fully how licensing and activation will truly interact. The finer points of the wording of MS documents on this subject are key to whether MSDN would be a viable option for us. For instance, take this example:

      y[

    • codinghorror
    • 13 years ago

    Er.. you guys do know that the 30 day limit can be extended TWICE, perfectly legally, right? So the real limit is 90 days.

    * Press WinKey + R to open the Run window, or open Command Prompt.
    * Type the following command into the Open textbox, or from the Command Prompt: sysprep /generalize
    or
    slmgr.vbs –rearm
    or
    rundll32 slc.dll,SLReArmWindows
    * Click OK or press ENTER.
    * Reboot the computer, and Windows Vista will give users another 30 more days for users to activate Windows Vista.

    That’s right, this is built directly into Vista. I’m on about day 70 of my 90-day eval so far..

    • Bensam123
    • 13 years ago

    Buy a copy of Vista for each hardware configuration? 😛

    I’m not going to be upgrading to Vista for awhile just because of it being extremely overcumbersome to activate and re-activate each time I change a single piece of hardware. Not only that but there is limited support and I hear DRM is horrible on the new OS (something about not being able to have ISOs or MP3s in a certain edition).

    If I were you, I would just test everything in 64-bit XP as a sort of “changing to Vista” phase from now on. Once all the kinks get worked out then switch over to 64-bit Vista. I wouldn’t even bother with 32-bit Vista. We were supposed to be going 64-bit a long time ago, but here we are still discussing wether or not to do things in 32-bit or 64-bit.

    I have a Intel-640 and even that has 64-bit support.

    • indeego
    • 13 years ago

    Yeah…. Never activate it. For testing purposes shouldn’t that workg{

      • Damage
      • 13 years ago

      I’m amazed how many people think the 30-day limitation shouldn’t be an issue. It’s a huge inconvenience. Proper system setup takes time, and once it’s done, it’s not something I want to repeat unnecessarily. Especially for CPU reviews, we may carry results over for quite a while… well beyond 30 days. The whole idea that we have no other means to operate legally is… frustrating.

        • PetMiceRnice
        • 13 years ago

        Surely Microsoft should realize that a lot of websites and even magazines do hardware reviews on a regular basis. I think that as part of such a community, you should talk to Microsoft directly (in person if need be) and spell out all of your concerns. You never know what might happen.

          • Damage
          • 13 years ago

          We have talked with MS and will continue to do so. I’m hopeful but unsure whether they will find a way to enable us to operate without significant inconvenience.

            • My Johnson
            • 13 years ago

            If you can’t benchmark un-encumbered then how do the developers do it?

        • indeego
        • 13 years ago

        “Proper system setup takes time, and once it’s done, it’s not something I want to repeat unnecessarily.”

        Well how are you doing XP then? Imaging? sysprep?

        Script it with the drivers loaded during install. You have chipset and graphics, certainly those can be automatedg{<.<}g

          • Damage
          • 13 years ago

          Yes, imaging and sysprep, some, when it makes sense. More imaging of installs completed directly on the target systems. But the application setup, installation of patches, scripts, media files, etc. all take time. Ever installed six games and all of their associated patches, one after another? (GRAW’s 16-digit key is checked by a case-sensitive routine, FWIW.) Building an image, rolling it out, and then having to start over due to an arbitrary time limit is a huge PITA.

            • DrDillyBar
            • 13 years ago

            what about using ESATA and registry keys?

            • Damage
            • 13 years ago

            Are you familiar with the Vista activation scheme?

            • DrDillyBar
            • 13 years ago

            Well, I was refering to the games etc, not the OS itself. But I’ve played with Vista 3 times in total, and didn’t really enjoy it. 😉 So the finer details of your struggle are a little out of my experience ATM. But the fact that it uses the Hardware in the system is something I’m aware of.

            • Ryszard
            • 13 years ago

            That works (I use a separate partition for game installs that rarely gets blown away, rather than an external disk), but not for all games.

            With imaging and being careful about where your data goes, the 30-day (which seems like it can be 90-day) limit shouldn’t be a horrendous problem. And then there’s always MSDN as mentioned, if it can be procured.

        • davidedney123
        • 13 years ago

        Got wrong end of the stick – post deleted.

        Dave

        • davidedney123
        • 13 years ago

        Can you get an enterprise version and run your own licence server? Then as long as you don’t use more than the X number of vista licences you own simultaneously you should be able to reactivate at will, if I understand the arrangement for Vista Corporate properly.

    • Ryszard
    • 13 years ago

    As far as unsigned drivers on x64 goes, there’s a boot-time option to let that slide, so you can install what you wish in that respect. B3D will be running x64 wherever possible, if that’s of any comfort.

    I hear you on the rest of your concerns especially when it comes to the activation issue. I’m currently just planning on running inside of the 30-day grace for most testing systems, with a reinstall and setup when that runs out.

Pin It on Pinterest

Share This