You may be interested to see the side-by-side high-speed videos we captured of our Skyrim test scenario, which illustrates the differences in smoothness between the 7950 and GTX 660 Ti.
Let’s pick up where we left off. Last week, we published a rematch of the classic battle between the Radeon HD 7950 and the GeForce GTX 660 Ti. We tested with a host of the latest games, including Borderlands 2, Hitman: Absolution, and Guild Wars 2.
Going into the contest, we thought the Radeon HD 7950 was the heavy favorite for a number of reasons. Among them is the fact that AMD has released new graphics drivers in recent weeks that promise a general performance increase. Also, the firm has worked closely with the developers of many of the holiday season’s top games, even bundling several of them in the box with the Radeon HD 7950.
To our surprise, though, the Radeon didn’t fare particularly well in our tests. Although it cranked out FPS averages that were competitive with the GeForce, the 7950 produced those frames at an uneven pace—our frame time plots for the Radeon were riddled with latency spikes. As a result, the Radeon’s scores were rather poor in our distinctive, latency-oriented performance metrics. Not only that, but our seat-of-the-pants impressions backed that up: play-testing on the GeForce felt noticeably smoother in many cases.
We noted that we’d taken various steps to ensure that the Radeon’s disappointing results weren’t caused by a hardware or software misconfiguration. Confident that our results were correct, we concluded that the most likely cause of the 7950’s poor showing was our transition to all-new games and testing scenarios. Our tentative conclusion: playing the latest games on the Radeon HD 7950 just isn’t as good an experience as playing on the GeForce GTX 660 Ti.
However, we did note the possibility that having upgraded our test rigs to Windows 8 might have played a role. I suppose we should have known that many of our readers would call for more testing in order to confirm our results. Quite a few of you asked us to run the same tests in Windows 7, to see if the Radeon’s performance problems are the product of the transition to a new operating system. Others asked us to re-run a familiar test scenario from one of our older articles to see whether the Radeon with the latest drivers would experience latency spikes in places where it previously hadn’t.
Those seemed like reasonable requests, and we were curious, too, about the cause of the 7950’s unexpected troubles. So we fired up our test rig, installed Windows 7 with the same drivers we’d used with Win8, and got to testing.
The results, naturally, are enlightening. Read on to see what we found.
Wait, er, latency what? What about FPS?
If you’re confounded by our talk of latency-focused performance metrics, you’re probably not alone. Gaming performance has been measured in frames per second since the dawn of time, or at least since the 1990s, when today’s PC enthusiast scene was first starting to form. However, FPS averages as they have been used tend to have some very big, very fatal flaws. We first explored this problem in the article Inside the second: A new look at game benchmarking. I recommend you read it if you want to understand the issues well.
For those too lazy to do the homework, though, let me extract a quick section from that article that explains the why FPS averages don’t tell the whole story of gaming performance.
Of course, there are always debates over benchmarking methods, and the usual average FPS score has come under fire repeatedly over the years for being too broad a measure. We’ve been persuaded by those arguments, so for quite a while now, we have provided average and low FPS rates from our benchmarking runs and, when possible, graphs of frame rates over time. We think that information gives folks a better sense of gaming performance than just an average FPS number.
Still, even that approach has some obvious weaknesses. We’ve noticed them at times when results from our FRAPS-based testing didn’t seem to square with our seat-of-the-pants experience. The fundamental problem is that, in terms of both computer time and human visual perception, one second is a very long time. Averaging results over a single second can obscure some big and important performance differences between systems.
To illustrate, let’s look at an example. It’s contrived, but it’s based on some real experiences we’ve had in game testing over the years. The charts below show the times required, in milliseconds, to produce a series of frames over a span of one second on two different video cards.
GPU 1 is obviously the faster solution in most respects. Generally, its frame times are in the teens, and that would usually add up to an average of about 60 FPS. GPU 2 is slower, with frame times consistently around 30 milliseconds.
However, GPU 1 has a problem running this game. Let’s say it’s a texture upload problem caused by poor memory management in the video drivers, although it could be just about anything, including a hardware issue. The result of the problem is that GPU 1 gets stuck when attempting to render one of the frames—really stuck, to the tune of a nearly half-second delay. If you were playing a game on this card and ran into this issue, it would be a huge show-stopper. If it happened often, the game would be essentially unplayable.
The end result is that GPU 2 does a much better job of providing a consistent illusion of motion during the period of time in question. Yet look at how these two cards fare when we report these results in FPS:
Whoops. In traditional FPS terms, the performance of these two solutions during our span of time is nearly identical. The numbers tell us there’s virtually no difference between them. Averaging our results over the span of a second has caused us to absorb and obscure a pretty major flaw in GPU 1’s performance.
The bottom line is that producing the highest FPS average doesn’t prove very much about the fluidity of in-game animation. In fact, even fancy-looking graphs that show second-by-second FPS averages over time don’t tell you very much, because tools like Fraps average the results over one-second intervals. Those plots simply make the same mistake over and over again in sequence.
Once we realized the nature of the problem, we decided to test performance more like game developers do: by focusing on the time in milliseconds required to produce each frame of the animation—on frame latencies, in other words. That way, if a momentary slowdown happens, we’ll know about it. This sort of analysis requires new tools and methods, which we’ve developed and refined over the past year. This article and other recent ones here at TR show those methods in action. We think they provide better insights into gameplay fluidity and real-time graphics performance than a traditional FPS-based approach.
The question of the hour is: Why does the Radeon HD 7950 struggle on this front in the current crop of games? Can switching back to Windows 7 alleviate the problem? Let’s have a look.
First up is my favorite game of the year so far, Borderlands 2. The shoot-n-loot formula of this FPS-RPG mash-up is ridiculously addictive, and the second installment in the series has some of the best writing and voice acting around.
Our game benchmarking methods are different from what you’ll find elsewhere in part because they’re based on chunks of gameplay, not just scripted sequences. Below is a look at our 90-second path through the “Opportunity” level in Borderlands 2.
As you’ll note, this session involves lots of fighting, so it’s not exactly repeatable from one test run to the next. However, we took the same path and fought the same basic contingent of foes each time through. The results were pretty consistent from one run to the next, and final numbers we’ve reported are the medians from five test runs.
We used the game’s highest image quality settings at the 27″ Korean monitor resolution of 2560×1440.
You can click the buttons above to switch between the results for the Radeon and the GeForce. Please note that we have only one set of results for the GeForce, and those are from Windows 8. The plots above show frame rendering times in milliseconds, so lower times are better.
As you can see, the plots for the 7950 in Win7 and Win8 look to be very similar, with little spikes to 40 ms or so throughout our test session. By contrast, the plot for the GTX 660 Ti is smoother; the frame latencies are generally lower, without so many high-latency frames interspersed throughout.
The GeForce is a little faster in the traditional FPS average, although the Radeons still reach over 60 FPS, which is usually considered sufficient.
The performance difference between the two cards becomes clearer in the latency-focused 99th percentile frame time. This number is simply the threshold below which 99% of all frames were rendered. The GTX 660 Ti produces all but the final 1% of frames in under 19.6 milliseconds—equivalent to about 50 FPS—while the Radeon’s threshold is over 30 milliseconds—or about 30 FPS.
The 99th percentile frame time is just one point on a curve, and here’s the curve. As you can see, the Radeon’s performance tracks with the GeForce’s until we reach the last 7% or so of the frames rendered. Those higher-latency frames are where the Radeon trips up, and from the plots, we know they’re distributed throughout the test session. Interestingly enough, the Radeon appears to perform marginally better in Windows 8, although the differences are pretty minor.
Our final latency-sensitive metric singles out those frames that take an especially long time to render. The goal of this measurement is to get a sense of “badness,” of the severity of any slowdowns encountered during the test session. What we do is add up any time spend rendering beyond a threshold of 50 milliseconds. (Frame times of 50 ms are equivalent to a frame rate of 20 FPS, which is awfully slow.) For instance, if a frame takes 70 milliseconds to render, it will contribute 20 milliseconds to our “badness” index. The higher this index goes, the more time we’ve spent waiting on especially high-latency frames, and the less fluid the game animation has been.
Again here, the 7950 falls behind the GeForce GTX 660 Ti, and going back to Windows 7 isn’t any help. That said, the Radeon doesn’t perform terribly; it only spends a small amount of time working on frames beyond our 50-ms threshold, so in-game animations should be relatively fluid. In fact, that’s what we experienced during our testing sessions. The game was playable enough on the Radeon, under both versions of Windows. The thing is, playing on the GeForce was simply a little bit smoother, both subjectively and by the numbers.
Why is that? For more insight into exactly what’s going on, we can zoom in on a small portion of our test run and look at the individual frame times. This is just a tiny bit of our overall data set, but it does help illustrate things. Click through the buttons to see each config’s results.
The 7950 suffers from these momentary hiccups. They’re not terribly dramatic, but still, the GeForce avoids them. Often, the long frame times are succeeded by several shorter frame times, as if the system is waiting on some long operation to complete the first frame and is ready with portions of the others once it finishes.
The presence of little hiccups of this size isn’t a big deal on a small scale, but it becomes more of an issue once we know that those longer frames comprise roughly 5-7% of the total frames produced, distributed throughout the test session. Given the size of the latency spikes here, they’re still not really a big problem, but they do bring the Radeon down—and make the GeForce look (and feel) like the superior solution.
Guild Wars 2
Once again, the move to Windows 7 doesn’t affect the Radeon HD 7950’s performance much at all. In fact, as with our Borderlands 2 test session, the 7950 looks to perform a little better in Win8, if anything.
Since the Radeon’s frame time plots show lots of latency spikes in this test scenario, here’s a small chunk of the test run, so we can see the problem up close.
Again, one or two bumps in the road like this wouldn’t be anything to worry about, but when those blips are everywhere, they begin to matter. Subjectively, the Radeon’s performance doesn’t feel horrible, but if you were play back to back versus the GeForce, you’d end up picking the GTX 660 Ti as the better option.
Our Sleeping Dogs test scenario consisted of me driving around the game’s amazingly detailed replica of Hong Kong at night, exhibiting my terrifying thumbstick driving skills.
This game’s graphics are intensive enough that we were easily able to stress these GPUs at 1080p.
Dropping back to Windows 7 doesn’t benefit the 7950 in this game, either. As in Win8, the 7950 posts a higher FPS average than the GeForce, but a small number of high-latency frames detracts from the 7950’s performance.
Assassin’s Creed III
Since the AC3 menu doesn’t lend itself to screenshots, I’ll just tell you that we tested at 1920×1080 with environment and shadow quality set to “very high” and texture and anti-aliasing quality set to “high.” I understand that the “high” AA setting uses FXAA HQ with no multisampling. This game also supports Nvidia’s TXAA, but Nvidia has gated off access to that mode from owners of Radeons and pre-Kepler GeForces, so we couldn’t use it for comparative testing.
Well, look at that. We finally have a difference of note between Win8 and Win7. For whatever reason, the 7950 struggles more in AC3 in Windows 8. Then again, the 7950 doesn’t perform especially well in this game in either version of Windows—not compared to the GTX 660 Ti.
In this game, Max Payne has sobered up and gotten a job with a shadowy government agency, yet somehow things still went totally sideways. He’s decided to stop talking about it so much, which is a relief.
I wasn’t sure how to test this game, since the object appears to be avoiding detection rather than fighting people. Do I test by standing around, observing guards’ patrolling patterns? Also, it seems that some areas of this game are much more performance-challenged than others, for reasons that aren’t entirely clear. Ultimately, I decided to test by taking a walk through Chinatown, which is teeming with people and seems to be reasonably intensive. I can’t say that good performance in this scenario would ensure solid performance in other areas of this game, though.
And we’ve finally found a good use for DX11 tessellation: bald guys’ noggins.
Although the Radeon’s FPS average is much higher than the GeForce’s, repeated latency spikes once again weigh it down, and rolling back to Windows 7 doesn’t offer any relief.
Both of these cards perform quite acceptably here, we should note. We’ve had to lower our “badness” threshold to 33.3 milliseconds since neither card surpasses the 50-millisecond mark for even a single frame. The thing is, the Radeon’s higher FPS average suggests it’s easily the faster solution for this workload, but some of its potential is wasted by the stubborn presence of higher-latency frames throughout the test run.
Medal of Honor: Warfighter
Warfighter uses the same Frostbite 2 engine as Battlefield 3, with advanced lighting and DX11 support. That’s fortunate, because I struggled to find any other redeeming quality in this stinker of a game. Even play-testing it is infuriating. I actually liked the last Medal of Honor game, but this abomination doesn’t belong in the same series. If you enjoy on-rails shooters where it constantly feels like you’re in a tunnel, bad guys pop up randomly, and your gun doesn’t work well, then this is the game for you.
The Elder Scrolls V: Skyrim
No, Skyrim isn’t the newest game at this point, but it’s still one of the better looking PC games and remains very popular. It’s also been a particular point of focus in driver optimizations, so we figured it would be a good fit to include here.
We did, however, decide to mix things up by moving to a new test area. Instead of running around in a town, we took to the open field, taking a walk across the countryside. This change of venue provides a more taxing workload than our older tests in Whiterun.
Note: do not aggro the giants.
This is one of the larger differences we’ve seen between Windows versions, and the 7950 definitely performs better in Win8. Notice, though, how much smoother the GTX 660 Ti’s frame time plot looks compared to either Radeon plot. Let’s zoom in for a closer look at a small slice of time.
For whatever reason, the 7950 doesn’t handle this new test scenario where we walk through the countryside very well.
Skyrim overtime: return to Whiterun
Some of you lamented the fact that our latest round of tests changed all of the games and test scenarios wholesale, so one couldn’t compare to familiar tests from past reviews to see whether the Radeon’s latency problems were introduced by recent driver updates or some other change. With that in mind, we’ve returned to our familiar test scenario where we make a loop around Whiterun. We’re using the same image quality settings that we did in our initial GTX 660 Ti review, although the OS and graphics driver revisions have changed.
Interesting. There isn’t much change from our older review. Let’s line up the numbers for a quick comparison:
|Zotac GTX 660 Ti AMP!||Zotac GTX 660 Ti AMP!||Radeon HD 7950 Boost||Sapphire
HD 7950 Vapor-X
|Sapphire HD 7950 Vapor-X|
12.11 beta 8
12.11 beta 8
|99th percentile frame time (ms)||17.8||16.5||17.7||18.3||18.0|
The Radeon HD 7950 appears to have regressed a bit with the move to newer drivers, with a slight drop in FPS averages and a corresponding increase in 99th percentile frame times. That’s true even though we’ve switched to a Sapphire 7950 card with a 25MHz higher Boost clock in our recent tests. Meanwhile, the Zotac GTX 660 Ti has improved somewhat with newer software and the move to Windows 8. The biggest change may be in its frame time plot, which looks tighter, with less variance than in our prior review.
However, the differences overall are very minor, and they appear to affect FPS averages and 99th percentile latency to similar degrees. As a bridge to the past, I think this outcome tells us nothing too major has changed, other than the way we’re testing Skyrim. Our basic hardware configs are working as expected, and the OS and driver changes haven’t introduced any new frame latency problems in this older test scenario.
The larger takeaway is that the results from both test scenarios are very likely valid. They’re just different. The Radeon HD 7950 handles the graphics workload in our Whiterun loop quite competitively, essentially matching the GeForce GTX 660 Ti, with nice, low frame latencies and relatively minor variance from frame to frame. However, the 7950 doesn’t process the more difficult workload in our cross-country test nearly as gracefully as its GeForce rival does.
Summary metrics and deep voodoo
Let’s see how the move to Windows 7 affects the 7950’s placement on our famous value scatter plots. As ever, we’ve converted our 99th percentile frame time results into FPS, so both of the plots below can be read the same, with the best values being closer to the top left corner of the plot area.
Predictably, there’s not much movement, because the 7950’s performance doesn’t change much from Win8 to Win7. Also, since Windows 7 doesn’t alleviate the latency spikes that plague the 7950 in many of these newer games, the Radeon continues to drop into a much less desirable position on the 99th-percentile plot than the GTX 660 Ti.
We should note that the overall performance number above is a geometric mean of the results from our seven game test sequences. (We’ve excluded Whiterun since we didn’t use it last time around.) We started using the geomean back in August in order to reduce the impact of outliers on our overall performance scores. In the past, games with vendor-friendly optimizations like HAWX 2 and DiRT Showdown tended to push the average a long way in one direction, so we excluded them from our calculations, prompting controversy. We were hopeful the switch to the geometric mean would curb outliers without manual intervention.
That said, some folks still objected to our inclusion of Assassin’s Creed III in our overall index, since it performs so much better on the GeForce than on the Radeon. I figured we might as well oblige them by taking a look at the overall scores with AC3 excluded, to see whether it moves the needle.
Well, things don’t change drastically, but excluding AC3 allows the 7950 to pull ahead of the GTX 660 Ti in our overall FPS average. In fact, that change produces virtually the same sort of outcome in the overall FPS numbers as we saw back in August, when the Radeon HD 7950 reference card with Boost edged out this same Zotac GTX 660 Ti AMP! card by a margin of several frames per second.
Trouble is, that doesn’t really matter. A moral victory in the borderline-meaningless FPS sweeps doesn’t overcome the fact that the Radeon HD 7950 has a persistent problem with high-latency frames across a range of test scenarios based on the latest games. The 99th-percentile frame times reflect that reality. Our latest round of tests shows that Windows 8 is not the problem. On the contrary, Windows 8 generally improves the latency picture somewhat.
When we first published our rematch between the 7950 and the GTX 660 Ti, we pinged AMD to ask if they could explain the Radeon’s struggles in recent games. AMD spokesman Antal Tungler told us that our article had “raised some alarms” internally at the company, and he said they hoped to have some answers for us “before the holiday.” He also noted that AMD is continually working to improve its drivers and that the 7950 does perform well in FPS-based benchmarks.
We’re hopeful that we may have a more detailed answer from AMD before too long, but in the interim, we have an even firmer grasp of the reality that caused us to recommend the GeForce GTX 660 Ti over the Radeon HD 7950 in our last article. Again, the outcome of our testing may be counter the expectations of many folks; they certainly weren’t what we expected when we set out to stage this rematch.
The tragedy here is one of wasted potential. The Radeon HD 7950 is, on paper, clearly a more powerful GPU than the GTX 660 Ti, with double the memory interface width and a theoretical edge in peak ROP rate and shader flops. AMD is giving you more hardware for your money when you buy a 7950. For whatever reason—and we suspect the main culprit is graphics driver software—the 7950 can’t convert that advantage into consistently smoother in-game animation. As one of my fellow TR editors pointed out to me the other day, this wouldn’t be the first time Radeon owners were let down by driver issues during the holiday rush. AMD was plagued by a painful series of driver issues last year, too.
To those who would claim that other “professional” review sites haven’t seen results like ours in their comparisons of the 7950 and GTX 660 Ti, I would simply respond: of course not. Virtually nobody else tests like we do. We’re working on persuading folks to focus on latency, to use a timer with finer granularity, but such changes are hard. They take time and effort. Heck, we may be nearly alone in using this approach for a long time yet.
The question you have to ask is what matters to you. Do you want the graphics card that scores best in the FPS beauty pageant, or do you want that one that gives you the smoothest gaming experience when you fire up a freshly downloaded game this Christmas? If you just want bragging rights, by all means, choose the Radeon HD 7950. If you’re looking for the friction-free fluidity that only comes from consistently quick frame delivery, though, our recommendation remains the GeForce GTX 660 Ti.