Contemplating SpeedStep and Cool’n’Quiet in performance testing

Howdy all.  I’ve been hard at work in Damage Labs setting up new test systems with Windows 7.  This is a particularly agonizing chore for me, because I want to be sure to set up everything properly and perfectly the same (as much as possible) between the different systems.  Also, generally what happens is I go into this process looking to incorporate as many new benchmarks as possible, but then for various reasons (time constraints, poor application performance scaling, lack of counters for timing operations, software licensing/DRM restrictions), I end up using many of the same tests as in the past generation of results.  That kind of looks to be repeating itself in this new round of CPU tests, although I do expect to add a few new games, 7-Zip, and Windows Live Movie Maker, at least—along with new versions of a great many applications.

The question of the day, however, has to do with power management features.  Typically, we’ve left features like SpeedStep and Cool’n’Quiet disabled for our general performance tests, only enabling them when we do power efficiency testing.  We’ve disabled them for multiple reasons, mainly because they can affect performance results in some tests.  The picCOLOR application benchmark we’ve used for a while, for instance, does many short, quick operations spaced a little bit apart.  As a result, the CPUs don’t have time to ramp up their clock speeds for each operation, and the results come out lower than with power management disabled.

I also have a sense that, generally speaking, when cases like this occur, AMD processors are more likely to be negatively affected than Intel processors, in part because AMD chips tend to drop to lower clock speeds at idle and in part because of a history of real problems with AMD CPUs, CnQ, and performance.

The question is: Isn’t that fair game, though?  "Balanced" is the default power profile in Vista and Win7, and the vast majority of folks are going to want to have these power-saving features enabled on their systems in order to cut down on the noise, heat, and power consumption of their systems.  In a case like the picCOLOR benchmark, I think we have a simple solution: use a workload more like a real user would, with higher-resolution images and longer operations.  We can work with the software’s developer on that.  In other cases, well, most of our benchmarks already reflect real-world use pretty well and aren’t so affected by SpeedStep and Cool’n’Quiet.  If they are, the odds are pretty good that a real user might experience the same drop-off in performance, and even if it’s not perceptible, there’s no reason not to include it in our performance measurements.

I’m of two minds on this question.  The one sticking point that keeps me from making to switch and leaving power management features enabled is the possibility that our test results will be rendered unreliable in some cases, either due to big differences in outcome from one run to the next or to the occasional outright incompatibility between these features and a piece of software (which we’ve seen in some games in days past).  And I’m on a deadline here, so that prospect frightens me more than you might expect.  Still, it’s 2009, I’ve been using CnQ and SpeedStep on my own desktop and laptop systems for years, and I consider them integral features of a modern CPU.  Seems like it might be time to test ’em like we use ’em.

Hmm.  What do you all think?

Comments closed
    • indeego
    • 10 years ago

    It appears that the question addressed to your readers was answeredg{<.<}g

    • swaaye
    • 10 years ago

    Isn’t the only performance-related issue with these power saving techniques that of the delay in changing the clock rate from idle to load? That may screw up synthetic test results, but for real world apps it’s not an issue.

    • Sargent Duck
    • 10 years ago

    I’d be in favor of having power management features left ON. Back a few years ago I used to be all about overclocking and getting the best performance I can. Now, I’m perfectly content leaving my Core 2 duo 7300 (2.5Ghz) plugging away at stock.

    I leave my power management features on all the time, including when I’m gaming, and I suspect a lot of others do as well. Thus, I’d like to see the tests best mimic how most people have their desktops.

    And besides, remember the P4 3.8Ghz article you did? By leaving the power management feature on you found that the chip always ran around 3.2ghz or something only because it was too hot at 3.8! The nerve of Intel selling a 3.8Ghz chip when the fastest it would run was 3.6Ghz (or similar) due to heat. Had you disabled the power management I imagine you’d have skipped right by this important detail!

    • just brew it!
    • 10 years ago

    I think I’m generally in agreement with UberGerbil’s suggestion to disable power management for synthetic tests, and leave it enabled for everything else.

    One gray area is the F@h benchmarks. Does that qualify as a synthetic or real-world test? An additional wrinkle is that (on Linux at least) the default priority of the F@h client is low enough that it doesn’t kick CnQ out of power saving mode. (Not sure if this is the case on Windows as well, as I don’t run the F@h client on Windows any more…)

    • Ryhadar
    • 10 years ago

    Honestly if you were just to put the differences between CPU power saving features on vs off I’d be very happy. Of the few review sites that do this there certainly isn’t much of a difference between load, and with today’s CPUs it looks like there is only a slight difference between idle usage.

    From what little I’ve found usually idle varies only about 5-20W. In which case the extra $0.10-$0.40 I would be saving a month wouldn’t be worth the extra hassle I might have with applications with CnQ/SpeedStep enabled. I find the difference between CnQ/SpeedStep enabled/disabled is a far better metric because then it allows people to see for themselves if they see any performance detriment and whether or not it is worth the hassle.

    Though it would be nice if you made a note about some of the applications you test that do have an issue with CnQ/Speedstep.

    (sorry if that’s a little incoherent, I wrote this once only to have accidentally changed the page and lost what I originally wrote.)

    • Richie_G
    • 10 years ago

    Just for clarity here: for typical gaming speedstep or cool and quiet are not detrimental, or are they?

    At the moment I don’t seem to have any issues, but perhaps as games come out that strain my machine more I may need to look towards things like this to squeeze some extra performance.

    • blitzy
    • 10 years ago

    Keep it real, homie.

    • MadManOriginal
    • 10 years ago

    I think power saving options should be left on for as many tests as possible. Heck even if it does negatively affect a benchmark, too bad, that means it’s not implemented well and we ought to know that and have it reflected in benchmarks. Also by having power saving off for all benchmarks and on for power draw tests it makes the power draw tests less meaningful because they don’t jive with the performance tests.

    • Prototyped
    • 10 years ago

    A thought occurs to me. Since Turbo Boost on /[

    • Fighterpilot
    • 10 years ago

    Testing with everything on at least gives you a “worst case” scenario which is to be preferred over a “best case” result.
    That at least gives enthusiasts a little headroom to work with but accurately reflects what Mom and Dad are going to achieve.

    • ShadowTiger
    • 10 years ago

    As someone who disables all power saving features, uninstalls all unnecessary programs and disables unused system services, I would prefer benchmarks to show maximum performance. The “average user” does not run their hardware on a clean install of an operating system, they have tons of bulkware on their system that slows them down much more than their power saving mode. Thus calibrating benchmarks for “real” or “average” performance is just as arbitrary as synthetic benchmarks (but less reliable). Since most tests would be unaffected its not a big deal, but I would say that you should not assume people will have their power saving mode on for performance benchmarks.

    • human_error
    • 10 years ago

    I think you should leave CnQ and Speedstep on in tests where the cpu load varies quite a bit – they won’t kick in on cpu intensive tests which load them at 90-100% constantly so don’t enable them on those tests but do on games and other real-world scenarios which vary in their cpu usage.

    I would also leave the Intel cpu turbo mode on as well for all tests (i don’t think it needs speedstep does it?) as it will show if it makes much of a difference and only people pushing their overclocks to the limit will disable them on their cpus.

    I do like the idea of showing tests with these features on as they are real-world indicators of performance, however i would suggest logging (and maybe generating a chart showing) cpu speed changes to show not only the results of the testing but also how aggresive these features were – you could have a graph showing cpu speed over time, cpu usage (% of available) over time and total power usage (cumulative over the test) to show how much power the features actually save (and maybe heat as well, since that is the only reason i keep speedstep/cnq enabled on my machines).

    • KyleSTL
    • 10 years ago

    What about the inverse with Intel’s new automatic overclocking when certain numbers of cores are utilized? Wouldn’t this give Intel an automatic advantage for a give nominal clock rate?

    Edit: Sorry, I think I’ve just complicated the matter.

    • UberGerbil
    • 10 years ago

    I think I’d want to see the power-saving options /[

      • Damage
      • 10 years ago

      Not a bad idea, but I might extend that exception only to synthetic tests clearly affected by power management. For instance, looks like Stream is unaffected, but the cachemem-style latency walker we use is affected. I’ve already adjusted to disabling power management for that particular test.

        • flip-mode
        • 10 years ago

        Why not _[

          • Damage
          • 10 years ago

          Because then I can’t measure cache latencies accurately.

            • Lazier_Said
            • 10 years ago

            Cache latency tests aren’t real world usage so there’s no reason to constrain your testing to real world conditions.

            Application benchmarks at stock speeds should have power saving on in all cases.

            • nerdrage
            • 10 years ago

            q[

            • mattthemuppet
            • 10 years ago

            I have to say, I skip right over that page. Pretty graphs, but of no comparative interest to me at all. If you’re really worried about power management vs. no power management, do a comparison for your first review with the new equipment. Barring major architectural changes/ software updates I would imagine those figures would apply to every subsequent review /[

            • derFunkenstein
            • 10 years ago

            here I thought I was the only one.

    • flip-mode
    • 10 years ago

    Intel’s CPUs respond too quickly. They make short work of tasks, and speedstep is better than CnQ. But can it render kittens? What’s wrong with Intel?

    Seriously, I’d be in favor of at least showing the difference. I’d be in favor of you making one run through the benches with each system at bone stock settings – not touching the BIOS at all. That is perhaps as fair as it gets. Let CnQ and SpeedStep do their thing.

    • FubbHead
    • 10 years ago

    Leave it on for everything but the pure, raw-power performance tests, ie. leave on for games, benchmarks emulating workload, etc.

    • mdk77777
    • 10 years ago

    §[<http://www.lostcircuits.com/mambo//index.php?option=com_content&task=view&id=62&Itemid=42&limit=1&limitstart=16<]§ Excellent review of the AMD 965 BE using the cool and quite instead of leaving off. The results are rather dramatic and surprised the author.

      • mdk77777
      • 10 years ago

      Why not add a section in the power usage part of the review where you test the affect of power saving setting on a select number of benchmarks that you have already run with power savings disabled. That way your old reviews don’t become obsolete, but you test the new power settings to let people know the benefit or lack there of.

    • _Sigma
    • 10 years ago

    I assume two sets, with and without isn’t an option?

      • Damage
      • 10 years ago

      Anything that doubles your testing workload and doubles the number of results folks have to digest is…. horrible.

        • _Sigma
        • 10 years ago

        Very.

        But there could be a few select cases?

    • Dposcorp
    • 10 years ago

    Very good topic.
    As a former mechanic, I think in terms of testing sports cars.
    I know there are a few cars with gas saving techniques, including opening some valve / and throttle body jets at different loads, and some cars shutting fuel off to some of the cylinders.

    However, when testing a sports car, you go all out.

    When I wonder how a component will perform, I want to know the max performance possible, not performance that can vary with a board tweak, Bios Tweak , or specific OS. Tweaks should be disabled, especially when u consider we are gonna be in another Windows OS switching cycle, which may require the same hardware/software/game to be tested with more then one OS.

    Just my $.02

      • UberGerbil
      • 10 years ago

      But the “max performance possible” may /[

    • Usacomp2k3
    • 10 years ago

    I’d say use them for sure unless there are problems with reproducibility.

      • DrDillyBar
      • 10 years ago

      What he said.

Pin It on Pinterest

Share This