In a landscape littered with cookie-cutter war shooters and tightly scripted action-adventure games, titles like The Elder Scrolls V: Skyrim are a rarity. I suspect the very idea of letting players loose in a massive game world filled with hundreds of hours of content would give most game designers an aneurysm. Bethesda Softworks has been cranking out Elder Scrolls games in the same vein since 1994, though, and it’s gotten no shortage of accolades for them. Already, Skyrim has earned its place in Metacritic’s all-time top 10, with more favorable reviews than Portal 2 and the original Quake.
It’s not hard to see why. Skyrim‘s open world is a masterpiece of beautifully rendered forests, towns, moorlands, and snow-capped mountains. Yes, the huge open-world design does make for relatively formulaic quests, but the ability to explore freely and to develop your character however you please adds a dimension other titles simply lack. You can be anything from a thief to a mercenary to a master wizard, and the game lets you take sides in a huge civil war and turn the tide of battle. Oh, and did we mention you get to fight dragons?
To our knowledge, Bethesda hasn’t talked about giving the PC preferential treatment like EA DICE did during Battlefield 3‘s development. So, while Skyrim certainly looks beautiful, it may not quite harness all of the horsepower top-of-the-line GPUs have on offer. Nevertheless, we expect PC gamers will want a reasonably quick graphics card if they wish to bask in the game’s vistas and immerse themselves completely. They’d be doing themselves a disservice otherwise.
Since folks took a liking to our Battlefield 3 performance article earlier this month, we thought we’d put our assortment of mid-range graphics cards through the paces in Skyrim, as well. We’ve worked our benchmarking magic on the same six cards, with the GeForce GTX 460 and Radeon HD 6850 facing off below $150; the GeForce GTX 560 and Radeon HD 6870 duking it out at $180-190; and the GeForce GTX 560 Ti and Radeon HD 6950 1GB dueling just under $250.
Our questions are the same: which cards are needed to play the game at what graphical settings, and do either AMD or Nvidia GPUs have an advantage over the competition? Again, we weren’t content to jot down average frame rates. We’ve also used elaborate frame-time measurements to assess how smoothly the game runs on each card. Read on to see what we learned.
Our testing methods
Our testing setup, detailed below, should look familiar to folks who’ve read our Battlefield 3 article. It’s the same one; we’ve just updated drivers and loaded up Skyrim on it.
We’re still testing at a resolution of 1920×1080 throughout. Like BF3, Skyrim lacks a built-in, scripted benchmark, which forced us to test manually by playing certain portions of the game multiple times with each card. Covering multiple resolutions using that method would have been far too time-consuming. We chose 1080p because it seems to be considerably more popular: a quick look at Newegg’s listings shows considerably more 1080p monitors than panels with other resolutions.
As ever, we did our best to deliver clean benchmark numbers, with tests run five times per card. Our test system was configured as follows:
|Processor||Intel Core i5-750|
|North bridge||Intel P55 Express|
|Memory size||4GB (2 DIMMs)|
|Memory type||Kingston HyperX KHX2133C9AD3X2K2/4GX
DDR3 SDRAM at 1333MHz
|Memory timings||9-9-9-24 1T|
|Chipset drivers||INF update 220.127.116.115
Rapid Storage Technology 10.1.0.1008
|Audio||Integrated Via VT1828S
with 18.104.22.16800 drivers
|Graphics||XFX Radeon HD 6850 1GB (HD-685X-ZNFC)
with Catalyst 11.11 drivers
|Asus Radeon HD 6870 1GB 915MHz (EAH6870 DC/2DI2S/1GD5)
with Catalyst 11.11 drivers
|Gigabyte Radeon HD 6950 1GB 870MHz (GV-R695OC-1GD)
with Catalyst 11.11 drivers
|Zotac GeForce GTX 460 1GB (ZT-40402-10P)
with GeForce 285.79 beta drivers
|MSI GeForce GTX 560 1GB 870MHz (N560GTX Twin Frozr II/OC)
with GeForce 285.79 beta drivers
|Asus GeForce GTX 560 Ti 1GB 830MHz (ENGTX560 TI DCII/2DI/1GD5)
with GeForce 285.79 beta drivers
|Hard drive||Samsung SpinPoint F1 HD103UJ 1TB SATA
Western Digital Caviar Green 1TB
|Power supply||Corsair HX750W|
|OS||Windows 7 Ultimate x64 Edition
Service Pack 1
Thanks to Asus, Intel, Corsair, Kingston, Samsung, and Western Digital for helping to outfit our test rigs with some of the finest hardware available. XFX, Gigabyte, and MSI for supplying the graphics cards for testing, as well.
We conducted testing using the Catalyst 11.11 driver from AMD and the GeForce 285.79 beta driver from Nvidia. We left optional AMD optimizations for tessellation and texture filtering disabled. Otherwise, image quality settings for the graphics cards were left at the control panel defaults. Vertical refresh sync (vsync) was disabled for all tests.
We used the following test applications:
We used the Fraps utility to record frame rates while playing a 90-second sequences through each level we tested. Although capturing frame rates while playing isn’t precisely repeatable, we tried to make each run as similar as possible to all of the others. We tested each Fraps sequence at least five times per video card, in order to counteract any variability.
The tests and methods we employ are generally publicly available and reproducible. If you have questions about our methods, hit our forums to talk with us about them.
Introducing the graphics detail settings
We’ve included screenshots of Skyrim‘s video options screens on the next few pages, but let’s first take a little time to compare and contrast the different detail presets. What do they mean, and how do they impact image quality?
First, here’s an at-a-glance rundown of the three graphical presets we’ll be looking at today—”medium,” “high,” and “ultra high”—and the parameters they affect:
|Antialiasing||4X FXAA||8X MSAA||8X MSAA|
|Radial blur quality||Low||Medium||High|
|Water reflects||Land only||Land, objects,
trees, and sky
|Distant object detail||Medium||High||Ultra|
Just like with Battlefield 3, antialiasing is a standard feature, even at the “medium” setting. High texture detail is also on the menu for all three presets. Other settings degrade as you climb down the graphical preset ladder. In addition to what’s in the chart above, the presets impact the draw distance for things like characters, objects, lights, and vegetation—so fewer distant objects are displayed at the lower detail presets. The implications are obvious when we look at in-game screenshots:
You can click the thumbnails for full-sized versions of the shots in lossless PNG format. However, zooming in on certain parts of the scene gives us an even better sense of how the detail presets impact image quality and draw distance:
The water reflections don’t look too different even when we zoom in, but the changes in texture filtering quality, object detail, and object draw distance are immediately obvious.
Interestingly, because the “medium” preset uses FXAA antialiasing by default, it makes vegetation blend more smoothly into the scene than the “high” and “ultra” presets. Those higher-detail presets rely on multisampled antialiasing, which smooths over polygon edges more accurately than FXAA but doesn’t affect sprites and transparent textures. (We’ll be sticking to the presets for our tests, but users are free to mix and mix and match individual settings as they please in Skyrim‘s graphical options control panel.)
Another scene, this time in the town of Whiterun, gives us a better look at how the detail presets impact lighting and shadowing quality:
Here’s a part of the scene magnified 200%:
In this corner of the scene, some of the shadows are completely gone with the “medium” preset. You’ll have to switch to “ultra” to get all of the shadows, including those on that house at the right. Other points of note: the torch doesn’t render with the “medium” preset, but the flames inside do; foliage pixelation is again worse with the higher-detail presets; and distant textures look awfully blurry without anisotropic filtering in the “medium” mode.
From those two series of screenshots, I think we can take away that the “high” preset provides a good amount of visual fidelity, especially when you’re not magnifying part of the scene and looking for missing objects in the distance. Meanwhile, the “medium” preset really degrades visual quality, reducing detail and making textures look awfully blurry. Hopefully, our lower-end cards can handle Skyrim at the “high” preset.
Now that we have a firmer grasp on things, let’s get into our performance data.
Let’s start with something you’ll find yourself doing quite a few times in Skyrim: walking up and down Whiterun, one of the large towns that serve as quest hubs and play a part in the main story arc. Our benchmark run involved walking up through the market area to Dragonsreach, the castle overlooking Whiterun, and then panning the camera to take in the scenery. We then walked back down through a different neighborhood to the main gate. We repeated this trajectory five times for each card, taking a measurement each time.
We started testing with the “high” detail preset, which seems to be what all the cards defaulted to. We’ll do some further tinkering with other detail presets on the next page.
We’ve already established the benefits of looking at frame times rather than frame rates, both in our article, Inside the second: A new look at game benchmarking, and in our more recent Battlefield 3 performance comparison. So, we’re going to start with frame times and then study average FPS numbers in context afterward.
If you’re not familiar with our testing methodology, here’s the Cliff’s Notes version. Frame time data gives us a much better sense of overall smoothness and playability. In a perfect world, we’d want cards to spit out frames in about 16.7 milliseconds each (which would mean 60 frames rendered per second). Just as importantly, we’d want to ensure consistently low frame rendering times. Even momentary spikes in frame rendering times can translate into perceived choppiness—and overall FPS numbers can’t capture that.
For example, imagine one hypothetical second of gameplay. Almost all frames in that second are rendered in 16.7 ms, but the game briefly hangs, taking a disproportionate 100 ms to produce one frame and then catching up by cranking out the next frame in 5 ms—not an uncommon scenario. You’re going to feel the game skip, but the FPS counter will only report a dip from 60 to 56 FPS, which would suggest a negligible, imperceptible change. Looking inside the second helps us detect such skips, as well as other issues conventional frame rate data measured in FPS tend to obscure.
In the three graphs below, we’ve plotted individual frame rendering times for our cards across the duration of the Fraps run. We’ll be comparing competing pairs of cards in each graph. Note that the faster cards produce more frames and thus slightly longer lines in the graphs.
Surprisingly, it doesn’t look like there’s much of a difference between, well, any of the cards we tested. The GTX 460 and 6850 seem to be about as fast as each other, for instance, and the GTX 560 Ti and 6950 don’t appear to produce substantially lower frame times than their less powerful cousins. Across the board, we see frame times go up around two thirds of the way in. That’s when we overlooked the town of Whiterun from the castle above.
We can use our frame time data to calculate the total number of frames that took longer than 40 ms to render across our five benchmark runs for each card. (40 ms per frame, in case you’re wondering, would yield a sluggish 25 FPS average). Will this metric highlight greater differences between the products we’re looking at?
Sort of. The GeForces have slightly fewer frame time spikes than the Radeons, but the spikes aren’t numerous to begin with, and the differences between cards are relatively minimal.
As we’ve said, we want our graphics subsystem to deliver consistently low frame latencies. We can consider the big picture by looking at the 99th percentile frame times: the threshold below which 99% of frames are rendered. We think this may be our best overall performance metric. Simply put, the percentile calculation doesn’t let unusually short frame times cancel out unusually high ones, unlike average FPS results can. At the same time, this calculation excludes the rarest and most extraordinary latency spikes—the 1%, if you will—so it better reflects overall playability.
This outcome confirms the surprising consistency we inferred from looking at the line graphs. On all of these cards, most frames take no longer than about 33-35 ms to render. (33-35 ms per frame works out to 29-30 frames per second, if the system maintains those frame times for a whole second.)
The 40-ms chart counts exceptional frame times, and the 99th percentile chart gives us typical maximum frame times. The chart below shows simple average frame rates per second, a metric that will be familiar to most folks.
Most frames should take no longer than 33-35 ms to render, but this latest chart tell us that, on average, you can expect frames to render in about 18-19 ms (corresponding to the 52-56 FPS frame rates above). That’s quite smooth.
In fact, having played the game during testing, I’d go so far as to say even the 33-35 ms frame time peaks don’t ruin fluidity too much, subjectively speaking. Yes, the game is smoother with sub-20-ms frame times, but Skyrim isn’t one of those titles that requires instant twitch reactions and rapid, sweeping mouse movements. (During combat, you’ll usually either slash at an enemy with a melee weapon or run backward while flinging fireballs at him.) Mouse movements seem to feel responsive even as frame times go up, too, which helps conserve a sense of fluidity even as slight choppiness pervades. We’re going to test the “medium” graphical preset on the next page anyway, but keep these results in mind as we go forward.
Can we buff out those high frame times we saw earlier by turning down the detail? And is the “ultra high” detail preset a good match for higher-end cards? Let’s find out, starting with “medium” detail results.
The “medium” preset enables lower frame times as we overlook the town of Whiterun from the castle, but there’s still quite a jump from about 10 ms to nearly 30 ms when we start taking in the view. (That would correspond to a momentary frame-rate drop from 100 to about 33 FPS.) Considering what we saw in our image quality comparisons earlier, I’d say dropping down to the “medium” preset isn’t worth it.
What about “ultra” detail on the higher-end cards?
Unsurprisingly, our higher-end GPUs experience even greater increases in frame times at the highest detail preset. Not only that, but we’re looking at several hundred frames taking longer than 40 ms to render and at 99th-percentile frame times well over 40 ms for both cards. (Again, a 40-ms frame time corresponds to a 25 FPS frame rate.) The Whiterun walkabout is mostly playable at these settings, but it’s hard to get over the choppiness as we look around the castle.
We should note the 40-ms graph above potentially muddles things somewhat. Because both cards produce many frames that take longer than 40 ms to render, the card that produces the most total frames—in this case, the GeForce GTX 560 Ti—might be unfairly penalized. We can counteract this effect by raising the threshold and looking at the number of frames that take longer than 50 ms to render:
The Radeon is at a much greater disadvantage here, with 25 times as many frames over 50 ms as the GeForce. Again, though, both cards exhibit a lot of choppiness at these settings.
That’s enough exploration. How about some combat?
Skyrim will have you fending off wildebeests and clearing out dungeons of undead creatures on a regular basis. By far the most spectacular bits of combat in the game are fights with dragons, though. We just couldn’t pass up the opportunity, so we benchmarked our cards during a night-time dragon fight at Ancient’s Ascent, a location in the snow-capped mountains near the town of Helgen. Since the dragon’s attack behavior was hard to keep entirely consistent, we did seven runs per card instead of the usual five.
Again, we started testing at the “high” preset.
These line graphs tell us three things. First, there are a lot of rapid variations between long and short frame times in this test. These variations resemble multi-GPU-induced micro-stuttering we’ve measured in the past. Second, AMD cards seem to fare slightly better than their Nvidia siblings, at last when it comes to maintaining relatively consistent frame latencies. Third, faster cards clearly yield more consistently low frame times than slower ones; the uniformity we measured in our Whiterun test isn’t present here.
Dissecting the data further, we see the GeForce GTX 460 and 560 produce substantially greater numbers of exceptionally long frame times than the corresponding Radeons. The AMD cards stay ahead even when we remove outliers with our 99th-percentile calculation, though the spread between the 6850, GTX 460, 6870, and GTX 560 Ti doesn’t amount to much.
Looking at average frame rates gives no indication of the strange frame time inconsistencies we detected, and it actually puts one of the Nvidia cards, the GeForce GTX 560, ahead of its AMD rival. There’s clearly something strange at play here, and the FPS numbers don’t capture the problem.
Ancient’s Ascent—what’s with the jitter?
Before we look at results for other detail presets, let’s look into the cause of the strange frame-time jitter we measured.
Obviously, the jitter didn’t occur in Whiterun, so something about the dragon fight triggered it. The jitter was consistent throughout the test, so it wasn’t tied to the dragon’s attacks or the fireballs we shot at it. After a little brainstorming, we determined the jitter must have been caused by one of two factors—or a combination of them—in the Ancient’s Ascent test: the ongoing snow storm or the idle fire animation in our character’s hands, which indicates a fire spell is ready to cast.
We tried measuring frame times with our fire spell selected in Whiterun, but no jitter was apparent.
After that, we traveled to Winterhold, another location with an ongoing snow storm. First we measured frame times as we stared at the empty scene. Next, we unholstered our fire spell. Then, we pressed the R key to make the fire spell go away again:
Bingo. The snow storm doesn’t cause jitter on its own, and neither does the fire animation. Combine the two, though, and things go a little crazy. Here’s a close-up of what the jitter looks like:
As you can see, the jitter manifests itself as one long frame time, two medium frame times, and a short frame time, repeated in the same sequence ad nauseum. Subjectively, the variance wasn’t sufficient to disrupt the illusion of motion too much on the faster cards. That makes sense since, in our example above, the 6950’s frame times are consistently below 16 ms. However, we could definitely notice skips in what should have been continuous movement when sliding from side to side. The jitter was especially bad on the GeForce GTX 460, where frame times went from about 12 ms to 20 ms to 30 ms and back again repeatedly. (To put things in perspective, those frame times would correspond to frame rates of 83, 50, and 33 FPS.)
Ancient’s Ascent—further tinkering
Now that we’ve demystified that jitter issue, we can continue our performance testing, starting with a look at how the Ancient’s Ascent test plays out on the GeForce GTX 460 and Radeon HD 6850 with the “medium” graphical preset selected.
Turning down the detail level helps minimize the jitter, but the problem doesn’t go away entirely. The AMD and Nvidia cards are almost neck and neck, although the GeForce produces a slightly greater number of exceptionally high-latency frames.
What about the GTX 560 Ti and 6950 1GB in “ultra” mode?
Yes, the jitter is still there, but these cards produce surprisingly low frame times overall. They’re definitely playable at this setting—especially the Radeon, which has better average frame rates/times, a lower 99th-percentile threshold, and fewer high-latency frames. If it weren’t for the poor showing in Whiterun, we’d say 6950 and GTX 560 Ti owners should definitely be playing Skyrim at the “ultra high” setting.
We launched into this endeavor thinking Skryim wouldn’t put cards like the Radeon HD 6950 and GeForce GTX 560 Ti under too much strain. We weren’t quite right. Yes, for the most part, those cards have no trouble cranking out smooth, low frame times at 1080p with the “ultra high” graphical preset selected. However, particularly demanding scenes like the castle vista in Whiterun cause frame latencies to vault into noticeably choppy territory.
Some folks have dug up additional, undocumented eye-candy options in Skyrim‘s configuration files. We didn’t have time to play with those, but according to PC World’s coverage, they have a subtle but noticeable impact on the game’s visual fidelity. Bethesda may not have dazzled PC gamers with fancy DirectX 11 effects like EA DICE did with Battlefield 3, but Skyrim nevertheless has substantial amounts of visual ‘zazz—enough, perhaps, to warrant a graphics card upgrade for some players.
The good news is Skyrim‘s “high” detail preset is already plenty detailed, and it doesn’t stress cheaper cards like the GeForce GTX 460 and Radeon HD 6850 all that much. In other words, you’ll benefit from a fast GPU in this game, but a not-so-fast one will still deliver a great experience. As for which GPU vendor to choose, we saw consistently lower frame times during our dragon fight with the Radeons, but in cases where the strange snow-and-fire jitter issue didn’t come into play, the competitors were quite closely matched.
Before we go, we should mention that AMD released a beta Catalyst driver while we were putting the finishing touches on this article. The new driver promises performance improvements of 2-7% in Skryim, which could tip the odds further in AMD’s favor. In our experience, though, such quoted speed increases for driver updates apply only to best-case scenarios. We didn’t have a chance to re-test performance with the new drivers, but we did run a quick check to see whether the snowstorm-and-fire-spell jitter effect was still present, and it was.