Single page Print

Zooming in on the problem

The three single-GPU solutions look fairly normal, but then they're the ones that don't look odd in the earlier plots.

A funny thing happens, though, when we get into the multi-GPU results.

The frames times oscillate between relatively long and short delays in an every-other-frame pattern. To one degree or another, all of the multi-GPU solutions follow this template. With apologies for the scrolling involved, let's illustrate more fully:

Wow. We've heard whispers from time to time about micro-stuttering problems with multi-GPU solutions, but here it is in captivity.

To be clear, what we're seeing here is quite likely an artifact of the way today's multi-GPU solutions work. Both AMD and Nvidia prefer a technique for divvying up the workload between two GPUs known as alternate frame rendering (AFR). As the name indicates, AFR involves assigning the first GPU in a team to render the even-numbered frames, while the second GPU handles the odd-numbered ones, so frames are produced by the two GPUs in an interleaved fashion. (AFR can also be employed with three or four GPUs, with one frame being assigned to each GPU in sequence.) SLI and CrossFire support other load-balancing methods, such as split-frame rendering, but those methods aren't favored, because they don't scale as well in terms of FPS averages.

Although the fundamentals are fairly straightforward, load-balancing between multiple GPUs isn't a simple job. Graphics workloads vary from frame to frame. Heck, graphics workloads vary down to the block and pixel levels; a frame is essentially a massively parallel problem in itself. The nature of that huge parallel workload can change substantially from one frame to the next, so keeping the timing synchronized between two GPUs doing AFR isn't easy.

Then, because only the primary video card is connected to the display, the frames rendered by the second GPU in an AFR team must be copied over to the primary GPU for output. Those frame data are typically transferred between the GPUs via the proprietary SLI and CrossFire interfaces built into high-end video cards. Dual-GPU cards like the Radeon HD 6990 and GeForce GTX 590 include an onboard version of that same interface. (Some low-end multi-GPU solutions do without a custom data channel and transfer frame data via PCIe.) Copying those frames from the secondary GPU to the primary takes time. In addition to completed frames, the contents of other buffers must oftentimes be transferred between GPUs, especially when advanced rendering techniques (like render-to-texture) are in use. Such information is usually shared over PCIe, as I understand it, which has its own transfer latencies.

The charts above demonstrate that the synchronization for some of the multi-GPU solutions is far from perfect. This revelation opens up a huge, messy set of issues that will take some time to cover in full. For now, though, we can start by identifying several basic problems raised by the jitter inherent in multi-GPU systems.

Obviously, one implication is that using an FPS average to summarize multi-GPU performance is perilous. In cases of extreme jitter, an FPS average may give too much credit to a solution that's suffering from relatively high latencies for every other frame. My sense is those longer frame times in the pattern would be the gating factor for the illusion of motion. That is, a solution producing frame times of roughly 20 ms and 50 ms in interleaved fashion would provide no better illusion of motion than a system with constant 50-ms frame times—at best.

The reality is probably somewhat worse than that, for reasons we've already discussed. The human visual system is very good at picking up visual irregularities, and too much jitter in the flow of frames could easily cause a form of visual interference that would be rather distracting or annoying. I can't say I've personally experienced this sensation in any acute way—and I do play games on our multi-GPU test rigs with some regularity—but I know some folks with multi-GPU systems have complained about micro-stuttering sullying their gaming sessions. (I am one of those folks who can see the rainbow effect in older DLP displays, so I know such things can be disruptive.)

That's just the tip of the iceberg, too. The complexity of this problem is even deeper than our frame time data alone can indicate. We'll address more of the issues shortly, but first, I still think a closer look at our frame time data can prove instructive.