Fraps vs. FCAT: Skyrim
We'll start with our Skyrim test because it's very repeatable and has proven to be almost impossible to complete without a few spikes in frame times. The numbers you'll see below come from both Fraps and FCAT captures from the exact same test run. I simply ran Fraps and the FCAT overlay together and recorded video on the capture system while benchmarking with Fraps. After the fact, I was able to watch for the start and end points of the Fraps test in the video and correlate them more or less exactly with the frames we analyzed to produce the FCAT results.
The outcome should give us a sense of what's happening at two points in the rendering process: when the game engine hands off a frame to DirectX (Fraps) and when the frame hits the display (FCAT).
The plots above show how closely correlated the Fraps and FCAT frame time distributions appear to be. Click through the buttons above to see the results for each config tested.
I expect some folks will be ready to give me the beating I so richly deserve for presenting the data in this way, by which I mean "in a really small image" and "without sufficient color contrast." I apologize. I was limited by both time and ability. And by Microsoft Excel, which should not escape blame. I fully endorse the use of "Ctrl + Mouse-wheel up" to zoom in on the frame time plots for better visibility.
What you should be seeing is that three of the four configs have very close correlations between the Fraps and FCAT numbers, and that plots for the FCAT results are much tighter, with less frame-to-frame variance than the Fraps numbers have. That suggests there's some natural variance in the dispatch of frames coming from the game engine (closer to where Fraps measures) that gets smoothed out by buffering later in the pipeline.
Now, that doesn't mean one set of results is "correct" and the other "incorrect." As far as we know, both are correct for what they measure, at different points in the pipeline. One thing we'll want to investigate further is those spots where the Fraps plot shows latency spikes that the FCAT plot does not. Keep that in mind for later.
On another front, FCAT looks to be giving us some important additional insight about the Radeon HD 7970 CrossFire setup: its Fraps results look like the other solutions' Fraps plots, but its FCAT output is much "fuzzier," with larger frame-to-frame swings. Curious, no? That's probably not a good outcome, but it does map well to our expectations that Fraps results may not capture the extent of the timing differences introduced by multi-GPU load-balancing.
Let's see how these data look in our latency-focused performance metrics.
Using a traditional FPS average, the SLI and CrossFire setups would appear to perform nearly twice as well as the single-GPU solutions. However, when we switch to the latency-oriented 99th percentile frame time, the Radeon HD 7970 CrossFire config proves not to be so hot. The 99th percentile frame time is just the threshold below which 99% of all frames were rendered; we can look at the fuller latency curve for a better sense of what went wrong.
The FCAT latency curve for the 7970 CrossFire config has that classic profile shown by multi-GPU micro-stuttering. About 50% of the frame times are inordinately low, and the other half are inordinately high. As we approach the 99th percentile on the FCAT latency curve, the 7970 CrossFire config's frame times climb to within a few milliseconds of the single 7970's. Uh oh.
Our measure of "badness" often acts as an anchor for us, preventing us from getting too bogged down in the weeds of other analysis. This metric adds up any time spent working on frames that take longer than a given threshold. Our primary threshold here, 50 milliseconds, equates to about 20 FPS. We figure any animation that dips below the 20 FPS mark is in danger of looking choppy. Also, 50 ms maps to three vertical refresh intervals on a 60 Hz display. The other two, 16.7 ms and 33.3 ms, map to 60 FPS with a single refresh interval and 30 FPS with two refresh intervals, respectively.
The truth is that all these solutions are incredibly quick in Skyrim, with only a few milliseconds spent above our main threshold by any of them. That jibes with our sense that all these cards ran this test pretty smoothly, with only an occasional hiccup in each case. You may also notice that the Fraps data tends to show more time spent beyond each threshold than FCAT does. That should be no surprise given the larger spikes visible in the Fraps plots. Let's see what we can make of that fact.