New methods: some examples — continued
This real-world result nicely parallels our theoretical example from earlier, although the frame rate delay on the Radeon isn't nearly as long. Obviously, an FPS average won't catch the difference between the two cards here.
In fact, have a look at the two frame times following the 58 ms delay; they're very low. That's likely because the video card is using triple buffering, so the rendering of those two later frames wasn't blocked by the wait for the one before them. Crazily enough, if you consider just those three frames together, the average frame time is 23 ms. Yet that 58 ms frame happened, and it potentially interrupted the flow of the game.
Now, we don't want to overstate the importance of a single incident like that, but with all of these frame time data at our disposal, we can easily ask whether it's part of a larger pattern.
We're counting through all five of our 60-second Fraps sessions for each card here. As you may have inferred by reading the plots at the top of the page, the Radeons aren't plagued with a terrible problem, but they do run into a minor hiccup about once in each 60-second session—with the notable exception of the Radeon HD 6970. By contrast, the Nvidia GPUs deliver more consistent results. Not even the older GeForce GTX 260 produces a single hitch.
If you're looking to do some multiplayer gaming where reaction times are paramount, you may want to ensure that your frame times are consistently low. By cranking our threshold down to 20 ms (or the equivalent of 50 FPS), we can separate the silky smooth solutions from the pretenders.
Only two cards, the GeForce GTX 570 and Radeon HD 6970, produce nearly all of their frames in under 20 ms. If you're an aspiring pro gamer, you'll need to pick up a relatively fast video card—or just do what they all do anyway: crank the display quality down as much as possible to ensure solid performance.
Counting up frames over a 50-ms or whatever threshold is nice, but it doesn't really capture everything we'd like. We do want to know about those outliers, but what we really need to understand is how well a video card maintains that steady illusion of motion.
One way to address that question is to rip a page from the world of server benchmarking. In that world, we often measure performance for systems processing lots of transactions. Oftentimes the absolute transaction rate is less important than delivering consistently low transaction latencies. For instance, here is an example where the cheaper Xeons average more requests per second, but the pricey "big iron" Xeons maintain lower response times under load. We can quantify that reality by looking at the 99th percentile response time, which sounds like a lot of big words but is a fundamentally simple concept: for each system, 99% of all requests were processed within X milliseconds. The lower that number is, the quicker the system is overall. (Ruling out that last 1% allows us to filter out any weird outliers.)
Oddly enough, we want to ask the same basic question about gaming performance. We want our systems to ensure consistently low frame times, and doing so is arguably more important than achieving the highest FPS rate.
Happily, in this case, our 99th percentile frame time results pretty closely mirror the average FPS rates. The cards are ranked in the same order, with similar gaps between them. That fact tells us several things. Most notably, the cards with relatively high frames rates are also producing relatively low frame times with some consistency. The reverse is also true: the cards with lower FPS averages are delivering higher frame times. None of the cards are returning a bunch of strangely high or low frame times that would throw off the FPS average. As a result, we can say that these cards are doing a nice job of maintaining the illusion of motion at something close to their advertised FPS averages.
Also, I think this outcome somewhat validates our use of the 99th percentile frame times as a sort of complement to the usual FPS average. If all goes as it should, a video card delivering high frame rates ought to achieve predominantly low frame times, as well. Granted, this is a pretty geeky way to analyze the data, but you'll see why it matters shortly.