Examining early DirectX 12 performance in Deus Ex: Mankind Divided

Deus Ex: Mankind Divided is one of the hottest games on the block right now. It also happens to be one of AMD’s Gaming Evolved titles, so it’s no surprise that the company is using it as a showcase for its graphics cards’ performance—especially now that a preview version of the DirectX 12 render path is available on Steam as a beta release branch. DX12 has offered performance boosts for AMD graphics cards in several titles now, so I was curious to see how the new renderer would affect Radeons’ performance in DXMD, as well.

After playing around with DXMD‘s DX12 preview last night, however, I wasn’t surprised to find some rough edges in this beta software. I noticed a lot of hitching and jerkiness in DX12 mode with GeForces and Radeons alike, so when AMD sent over some flattering average-FPS benchmark numbers as part of its press materials for the game, I knew it was time to pull out PresentMon and see just what the story behind that roughness was. You can see some of those numbers over at WCCFTech, for reference. Other sites, like ComputerBase, have also measured average-FPS results with Deus Ex‘s DX12 renderer.

Before we share some test numbers of our own, I want to make it clear that this is a quick-and-dirty look at one game’s performance, not a full review. While we tried to maintain the same rigor we do for our graphics card reviews, we didn’t test the full range of graphics cards or games we would have for a full review. Our goal here is just to get a sense of how this beta version of Deus Ex performs in a way that average FPS numbers just don’t offer. These results will most likely change once Deus Ex‘s DX12 renderer gets an official release on September 19.

Here are the specifications for our test system:

And here are the specifications and driver versions for each graphics card we used in our testing:

Here are the graphics settings we used for both our DirectX 11 and DirectX 12 tests:

Without any further ado, let’s see how this quartet of cards performs in Deus Ex: Mankind Divided.

DirectX 11 results

Under DX11, all of the cards we tested deliver solid-looking frame-time plots with only a few major spikes. Going by average FPS, the R9 Fury X and the GTX 1070 are closely matched in their weight class, while the Radeon RX 480 opens a bit of a lead on the GTX 1060 6GB. Our 99th-percentile frame-time graphs tell the most interesting story, however. The GTX 1070 and the Fury X need more than 16.7 ms to deliver 99% of their frames, suggesting more variance than average FPS alone can tell us. Meanwhile, the RX 480 and GTX 1060 both deliver 99% of their frames in about 30 ms.

Recall that if a graphics card delivers every frame in 16.7 ms, it’s maintained a perfect 60 FPS throughout our tests with no hitches. In a similar vein, a card would have maintained a perfect 30 FPS if it delivered every frame on a 33.3 ms interval. Any time spent beyond 16.7 or 33.3 ms is time that the average frame rate drops beneath 60 or 30 FPS. Let’s see just how much time each card spends working on tough frames now.

Our “time-spent-beyond-X” graphs are meant to show “badness,” or the amount of time in our one-minute test period where a card might have delivered less-than-fluid animation. The 50-ms threshold is the most notable one, since it corresponds to a 20-FPS average. We figure if you’re not rendering any faster than 20 FPS, even for a moment, then the user is likely to perceive a slowdown. 33 ms correlates to 30 FPS or a 30Hz refresh rate. Go beyond that with vsync on, and you’re into the bad voodoo of quantization slowdowns. And 16.7 ms correlates to 60 FPS, that golden mark that we’d like to achieve (or surpass) for each and every frame.

Between the RX 480 and the GTX 1060, the most interesting graphs may be those of the time spent beyond the 50-ms and 30-ms thresholds. The RX 480 spends slightly longer working on frames that take longer  than 50 ms and 33.3 ms to complete, and that time translates into slight-but-noticeable hitches during gameplay. The GTX 1060 spends barely any time past 33.3 ms working on frames, and it spends no time past 50 ms. That means it offers a perceptibly smoother gameplay experience, even if it’s not turning out as many frames as the RX 480 does. 

The GTX 1070 and the Fury X, on the other hand, are about as evenly matched as they come in this title. Neither card spends any time past 50 ms working on tough frames, and the GTX 1070 only has a vanishingly slight hiccup that takes it past 33.3 ms. Each card spends about three seconds of our one-minute test period working on frames that would drop frame rates below 60 FPS, too. Let’s see what switching over to Deus Ex‘s DX12 renderer can tell us about each card’s performance. 

DirectX 12 performance

So that’s a thing. Switching over to DXMD‘s DirectX 12 renderer doesn’t improve performance on any of our cards, and it actually makes life much worse for the Radeons. The R9 Fury X turns in an average FPS result that might make you think its performance is on par with the GTX 1070 once again, but don’t be fooled—that card’s 99th-percentile frame time number is no better than even the GTX 1060’s. Playing DXMD on the Fury X and RX 480 was a hitchy, stuttery experience, and our frame-time plots confirm that impression.

In the green corner, the GTX 1070 leads the 99th-percentile frame-time pack by a wide margin, and that translates into noticeably smoother gameplay than any other card here can provide while running under DirectX 12.

The Radeons’ lackluster 99th-percentile frame-time numbers in this test are corroborated by our “time-spent-beyond-X” graphs. In total, both of the red teams’ cards spend half a second working on frames that take more than 50 ms to render, and that means an unpleasantly rough gameplay experience while DirectX 12 is enabled. If I hadn’t been collecting data for this article, I would have immediately unticked the DirectX 12 checkbox in DXMD‘s options and gone back to the smooth sailing that the game’s DX11 mode offers on all of these cards.

Click over to the time-spent-beyond-33.3-ms graph, and we can see that both Radeons spend a full second working on frames that take longer than that to render. Again, that means we’re spending quite a bit of time below 30 FPS during our one-minute testing period, a noticeable blot on these cards’ smoothness. The Fury X’s monster shader array doesn’t seem to be much help here—it’s only a bit better off than the RX 480.

The GTX 1070 has a much better time of it in Deus Ex‘s DX12 mode. The card spends only a hair’s breadth of time past the 33.3-ms mark, and it spends 34% less time working on frames that would drop the frame rate below 60 FPS than the Fury X does. I had to double-check whether the GTX 1070 was actually performing this well compared to the Radeons, but the numbers don’t lie. It’s the performance champion in DXMD‘s DX12 mode with these settings, and not by a small margin.

Conclusions
I hate to toot TR’s horn here, but tests like these demonstrate why one simply can’t take average FPS numbers at face value when measuring graphics-card performance. We’ve been saying so for years. From our results and our subjective experience, it’s clear that the developers behind Deus Ex: Mankind Divided have a lot of optimizing to do for Radeons before the game’s DirectX 12 mode goes gold in a week and change. AMD’s driver team may also have a few long nights ahead, though in theory, DX12 puts much more responsibility on the shoulders of the developer.

It’s also clear that it’s too early to call a winner between the green and red teams for DirectX 12 performance in this beta build of Deus Ex, even if AMD seems to feel confident in doing so. The Radeon cards we tested perform poorly in our latency-sensitive frame-time metrics in DX12 mode, meaning that the Fury X’s hitchy gameplay stands in stark contrast to its respectable average-FPS result. Even if Nvidia isn’t shouting from the rooftops about Pascal’s performance in DXMD‘s DX12 mode right now, the green team has some kind of smoothness advantage despite the game’s beta tag. To be fair, we used different settings than AMD did while gathering its performance numbers, but we don’t feel like the choices we made would be much different than those the average enthusiast would have with this hardware.

The one bright spot for AMD among these early numbers is that the R9 Fury X and GeForce GTX 1070 are so closely matched in Mankind Divided once DX12 is out of the picture. The similarly-powerful GeForce GTX 980 Ti was the clear favorite in our latency-sensitive 99th-percentile FPS measure when we pitted it against the Fury X, but the Fury X has closed the gap in this one game, at least. It’s worth remembering that DXMD is a Gaming Evolved title, but even so, Fury X owners can enjoy a gaming experience that’s just as smooth as they would get with a Pascal-powered card, and that’s no small feat given AMD’s past history with DirectX 11 performance. We’ll just have to wait and see whether similar performance improvements are possible for Radeons running under Deus Ex‘s DX12 renderer once that software comes out of beta.