Whiterun
Let's start with something you'll find yourself doing quite a few times in Skyrim: walking up and down Whiterun, one of the large towns that serve as quest hubs and play a part in the main story arc. Our benchmark run involved walking up through the market area to Dragonsreach, the castle overlooking Whiterun, and then panning the camera to take in the scenery. We then walked back down through a different neighborhood to the main gate. We repeated this trajectory five times for each card, taking a measurement each time.

We started testing with the "high" detail preset, which seems to be what all the cards defaulted to. We'll do some further tinkering with other detail presets on the next page.

We've already established the benefits of looking at frame times rather than frame rates, both in our article, Inside the second: A new look at game benchmarking, and in our more recent Battlefield 3 performance comparison. So, we're going to start with frame times and then study average FPS numbers in context afterward.

If you're not familiar with our testing methodology, here's the Cliff's Notes version. Frame time data gives us a much better sense of overall smoothness and playability. In a perfect world, we'd want cards to spit out frames in about 16.7 milliseconds each (which would mean 60 frames rendered per second). Just as importantly, we'd want to ensure consistently low frame rendering times. Even momentary spikes in frame rendering times can translate into perceived choppiness—and overall FPS numbers can't capture that.

For example, imagine one hypothetical second of gameplay. Almost all frames in that second are rendered in 16.7 ms, but the game briefly hangs, taking a disproportionate 100 ms to produce one frame and then catching up by cranking out the next frame in 5 ms—not an uncommon scenario. You're going to feel the game skip, but the FPS counter will only report a dip from 60 to 56 FPS, which would suggest a negligible, imperceptible change. Looking inside the second helps us detect such skips, as well as other issues conventional frame rate data measured in FPS tend to obscure.

In the three graphs below, we've plotted individual frame rendering times for our cards across the duration of the Fraps run. We'll be comparing competing pairs of cards in each graph. Note that the faster cards produce more frames and thus slightly longer lines in the graphs.

Surprisingly, it doesn't look like there's much of a difference between, well, any of the cards we tested. The GTX 460 and 6850 seem to be about as fast as each other, for instance, and the GTX 560 Ti and 6950 don't appear to produce substantially lower frame times than their less powerful cousins. Across the board, we see frame times go up around two thirds of the way in. That's when we overlooked the town of Whiterun from the castle above.

We can use our frame time data to calculate the total number of frames that took longer than 40 ms to render across our five benchmark runs for each card. (40 ms per frame, in case you're wondering, would yield a sluggish 25 FPS average). Will this metric highlight greater differences between the products we're looking at?

Sort of. The GeForces have slightly fewer frame time spikes than the Radeons, but the spikes aren't numerous to begin with, and the differences between cards are relatively minimal.

As we've said, we want our graphics subsystem to deliver consistently low frame latencies. We can consider the big picture by looking at the 99th percentile frame times: the threshold below which 99% of frames are rendered. We think this may be our best overall performance metric. Simply put, the percentile calculation doesn't let unusually short frame times cancel out unusually high ones, unlike average FPS results can. At the same time, this calculation excludes the rarest and most extraordinary latency spikes—the 1%, if you will—so it better reflects overall playability.

This outcome confirms the surprising consistency we inferred from looking at the line graphs. On all of these cards, most frames take no longer than about 33-35 ms to render. (33-35 ms per frame works out to 29-30 frames per second, if the system maintains those frame times for a whole second.)

The 40-ms chart counts exceptional frame times, and the 99th percentile chart gives us typical maximum frame times. The chart below shows simple average frame rates per second, a metric that will be familiar to most folks.

Most frames should take no longer than 33-35 ms to render, but this latest chart tell us that, on average, you can expect frames to render in about 18-19 ms (corresponding to the 52-56 FPS frame rates above). That's quite smooth.

In fact, having played the game during testing, I'd go so far as to say even the 33-35 ms frame time peaks don't ruin fluidity too much, subjectively speaking. Yes, the game is smoother with sub-20-ms frame times, but Skyrim isn't one of those titles that requires instant twitch reactions and rapid, sweeping mouse movements. (During combat, you'll usually either slash at an enemy with a melee weapon or run backward while flinging fireballs at him.) Mouse movements seem to feel responsive even as frame times go up, too, which helps conserve a sense of fluidity even as slight choppiness pervades. We're going to test the "medium" graphical preset on the next page anyway, but keep these results in mind as we go forward.