Reconsidering the overall index in our Radeon R9 Fury review

I've been pretty active over the weekend responding to questions in the comments section for our Radeon R9 Fury review.

As you may know, our value scatter plot puts the R9 Fury just behind the GeForce GTX 980 in our overall index of average FPS scores across our test suite. Some of you have expressed surprise at this outcome given the numbers you've seen in other reviews, and others have zeroed in on our inclusion of Project Cars as a potential problem, since that game runs noticeably better on GeForces than Radeons for whatever reason.

I've explained in the comments that we use a geometric mean to calculate our overall performance score rather than a simple average specifically so that outliers—that is, games that behave very differently from most others—won't have too big an impact. That said, the geomean doesn't always filter outlier results as effectively as one might wish. A really skewed single result can have a noticeable impact on the final average. For that reason, in the rush to prepare my Fury review, I briefly looked at the impact of excluding Project Cars as a component of the overall score. My recollection is that it didn't seem to matter much.

However, prompted by your questions, I went back to the numbers this morning and poked around some. Turns out the impact of that change may be worthy of note. With Cars out of the picture, the overall FPS average for the R9 Fury drops by 1.2 FPS and the score for the GeForce GTX 980 drops by 2.8 FPS. The net result shifts from a 0.6-FPS margin of victory for the GTX 980 to a win for the R9 Fury by a margin of 1.1 FPS.

Things are really close. This is why I said in my analysis: "That's essentially a tie, folks."

But I know some of you hang a lot of worth on the race to achieve the highest FPS averages. I also think the requests to exclude Project Cars results from the index are sensible given how different they are from everything else. So here is the original FPS value scatter plot:

And here's the revised FPS-per-dollar scatter plot without the Cars component.

Some folks will take solace in this symbolic victory for AMD in terms of overall FPS averages. Do note that the price-performance landscape isn't substantially altered by this shift on the Y axis, though.

We have long championed better metrics for measuring gaming smoothness, and our 99th-percentile FPS plot is also altered by the removal of Cars from the results. I think this result is a much more reliable indicator of delivered performance in games than an FPS average. Here's the original one:

And here it is without Project Cars:

The picture shifts again with Cars out of the mix—and in a favorable direction for the Radeons—yet the R9 Fury and Fury X still trail the less expensive GeForce GTX 980 in terms of general animation smoothness. I believe this result is much more notable to PC gamers who want to understand the real-world performance of these products. AMD still has work to do in order to ensure better experiences for Radeon buyers in everyday gaming.

Then there's the power consumption picture, which looks like so:

I didn't have time to include this plot in the review, although all of the data are there in other forms. I think it's a helpful reminder of another dynamic at play when you're choosing among these cards.

At the end of the day, I think the Cars-free value scatter plots are probably a more faithful reflection of the overall performance picture than our original ones, so I'm going to update the final page of our Fury review with the revised plots. I've looked over the text that will need to change given the shifts in the plot positions. The required edits amount to just a few words, since the revised scores don't change anything substantial in our assessments of these products.

Still, it's always our intention to provide a clear sense of the overall picture in our reviews. In this case, I'm happy to make a change in light of some reader concerns.

Tip: You can use the A/Z keys to walk threads.
View options

This discussion is now closed.