Re-examining the unusual frame time results in our Radeon RX 470 review

— 1:33 PM on August 12, 2016

It's never fun to admit a mistake, but we made a big one while writing our recent Radeon RX 470 review. That piece was our first time out on a new test rig that included an Intel Core i7-6700K CPU and an ASRock Z170 Extreme7+ motherboard. Once we got that system up and running, it delivered some weird-looking frame time numbers with some games. For example, the spikiness of the frame-time plot below didn't match any test data we had ever gathered before for Grand Theft Auto V, and we puzzled over those strange results for some time. We decided to go ahead and publish them anyway after doing some extended troubleshooting without seeing any improvement.

Compare that to a more typical GTA V result from our Radeon RX 480 review, as demonstrated by three GeForce cards running on an X99 testbed:

The spikiness caused by what turned out to be high deferred procedure call (or DPC) latency didn't seem to affect average framerates much (save one major exception), but it did worsen our 99th-percentile frame time numbers considerably. Given how much we use those 99th-percentile numbers in formulating other parts of our conclusions, especially value plots, the error introduced this way had considerable negative effects on the accuracy of several key parts of our review. The net effect of this error led us to wrongly conclude that the Radeon RX 480 8GB and the Radeon RX 470 were closely matched in performance, when in fact they're quite different.

Upon reflection, we should have stopped the RX 470 review at that point to try and figure out exactly what was going on, but the pressure of deadlines got the better of us. When these weird frame-time plots appeared once more in the preliminary testing work for our review of the Radeon RX 460, however, we had to acknowlege that something unusual was going on. Of course, that also meant that our published RX 470 review had problems that we had to deal with. We believe that in the interest of full transparency, it's important to explain exactly what happened, be clear about how we messed up, and resolve not to make the same mistakes in the future.

The issue
While I was testing the Radeon RX 460 and tearing my hair out over the wild frame time plots our cards were generating, TR code wizard Bruno "morphine" Ferreira brought the possibility of high DPC latency to my attention. DPC latency is usually of interest to musicians running digital audio workstations, where consistently minimal input lag is critical. Bruno pointed me to the LatencyMon app, which keeps an eye on DPC performance and warns of any issues, as a way of figuring out whether DPC latency was the root cause of this problem.

LatencyMon thinks something is up with my main desktop. I'll have to look into it soon.

I didn't capture any screenshots during my frenzied troubleshooting, but LatencyMon did show that our test rig wasn't servicing DPC requests promptly. Wireless networking drivers are generally considered the first place to look when troubleshooting DPC issues, and I use an Intel Wireless-N 6205 card in our test system. Oops. Even after disabling that wireless card, however, the issue persisted. After killing every potential bit of software that might have been causing the problem without getting any improvements, I took Bruno's suggestion of updating our motherboard's firmware. "The BIOS can't possibly be the cause!" I thought to myself smugly.

Pride goeth before a fall, of course, and the DPC latency issue vanished with the new ASRock firmware installed. The frame-time plots for GTA V began to resemble the flat, tight lines we've come to expect with modern hardware. I had to quash the urge to drive over the motherboard a few times and burn the remains before coming to grips with the fact that I would have to throw out large amounts of testing data.

So what happened? You see, ASRock sent us its Z170 Extreme7+ board during a brief period in which the company was promoting its ability to overclock locked Intel CPUs with a beta BIOS. I had hoped to explore that possibility with some ASRock motherboards and cheap CPUs, but Intel swiftly put the kibosh on the concept. We got busy with other work, and the beta firmware remained on the motherboard through our first attempts to test graphics cards with it. I don't know precisely what was wrong with this beta firmware that was causing it to wreak havoc on DPC latency, but updating the firmware did fix it, so Occam's razor suggests something weird was going on.

The fallout
Having solved the underlying problem, I now had to contend with the fact that I had published a very public and widely-read review that contained what seemed like reams of contaminated data. To see just how wrong I had been in my conclusions, I retested every title we had slated for our RX 470 and RX 460 reviews on our ASRock test rig, using the same settings we had initially chosen for our reviews.

As it turns out, high DPC latency doesn't affect every game equally, or at least not in a way that shows up in our frame-time numbers. While GTA V, Hitman, and Rise of the Tomb Raider all showed significant changes in average FPS and 99th-percentile frame times after a retest on the updated hardware,  Doom, Crysis 3, and The Witcher 3 did not. That second trio of games certainly felt more responsive to input after the critical firmware update, but the data they generated wasn't meaningfully different. We're talking fractions of milliseconds of difference in before-and-after testing, and those deltas are almost certainly imperceptible in real-life gaming. Given that behavior, we're confident that the numbers we generated for Doom, Crysis 3, and The Witcher 3 are representative of the performance of the Radeon RX 460, the Radeon RX 470, and the other cards we tested in those reviews. 

The resolution
Given the large differences in performance we saw with GTA V, Hitman, and RoTR, the only acceptable way to fix our mistake was to retest all of the cards in our Radeon RX 470 review from the ground up with those games. We've done that now, and as a bonus, we did that retesting with the same data-collection methods we just premiered in our RX 460 review.

As a result, we now have Doom Vulkan and OpenGL numbers for the RX 470 and RX 480, plus DirectX 12 numbers for Rise of the Tomb Raider and Hitman. We've also extensively re-written the conclusion of our Radeon RX 470 review to account for this new data, much in the same way that we crunched our results for the Radeon RX 460 and friends. We've accounted for the differences in our results there, so I'd encourage you to go read up on what changed.

If you read our original Radeon RX 470 review, we're deeply sorry to have misinformed you. We also extend our sincerest apologies to everybody at AMD and Nvidia for presenting incorrect information and misguided conclusions about their products. In the future, we'll strive to be both correct and swift with our reviews, but we also won't hesitate to delay a piece when clear warning flags are evident. We hope this clarification reinforces your trust in The Tech Report's reporting. If you have any questions or concerns, please leave a comment below or email me directly.

Tip: You can use the A/Z keys to walk threads.
View options