Slicing and dicing the numbers
Here's one reason why I wanted to press on with our look at frame time data. I believe the issue we're seeing here is largely independent of the micro-stuttering problem. If you look back at the individual frame time plots over the span of a whole test run, you can see the trend clearly. The multi-GPU solutions just tend to run into especially long frame latencies more often than the single-GPU offerings. There is some overhead associated with keeping two GPUs fed and synchronized in the same system, and it seems to lead to occasional trouble—in the form of frame times over our 50-ms threshold.
Having said that, if we lower the threshold to 20 ms, the multi-GPU solutions still look pretty good—especially the SLI pairs. That's true in spite of the jitter we know to be present. Micro-stuttering may cause other forms of visual pain, but the raw performance implications of it in terms of frame latency aren't always devastating, so long as the higher frame times in that oscillating pattern don't grow too large. Multi-GPU systems can still outperform a comparable single-GPU setup in an impactful way, in many cases.
Here's where I think my crazy metric imported from the world of server benchmarking begins to pay off. By asking how quickly the vast majority of frames were returned, we can sort the different GPU solutions in an order of merit that means something important, and we avoid the average-inflating effect of those low-latency frames in those multi-GPU jitter patterns.
Notice that with jitter present, the 99th percentile frame times now no longer neatly mirror the FPS averages. With our more latency-sensitive percentile metric, some of the rankings have changed. The Radeon HD 6970 CrossFireX config drops from second place in FPS to fourth place in 99th percentile latency, and the GeForce GTX 580 rises from ninth to sixth. Most dramatically, the Radeon HD 6870 CrossFireX team drops from a relatively strong FPS finish, well above the GeForce GTX 580, to last place, virtually tied with a single GeForce GTX 570. The 6870 CrossFireX team is paying the penalty for relatively high latency on every other frame caused by its fairly pronounced jitter.
I'm not sure our 99th-percentile rankings are the best way to quantify video card performance, but I am persuaded that, as an indicator of consistently low frame times, they are superior to traditional FPS averages. In other words, I think this metric is potentially more relevant to gamers, even if it is kind of eggheaded.
We should also pause to note that all of these solutions perform pretty well in this test. The lowest 99th-percentile frame times are just over 25 ms, which translates to 40 FPS at a steady rate. A video card slinging out frames every 25 ms is no bad thing.
To give you a sense of how this crazy percentile-based method filters for jitter, allow me pull out a breathtakingly indulgent bar chart with a range of different percentiles on it, much like we'd use for servers. I'm not proposing that we always produce these sorts of monsters in future GPU reviews, but it is somewhat educational.
You can see how some of the multi-GPU solutions look really good at the 50th percentile. They're churning out half of their frames at really low latencies. However, as we ramp up the percentage of frames in our tally, we capture more of those long-latency frames that constitute the other half of the oscillating pattern. The frame latencies for those multi-GPU solutions rise dramatically, even at the 66th percentile.
Heck, we could probably even create a "jitter index" by looking at the difference between the 50th and 99th percentile frame times.