Our first test case: Skyrim
We knew up front that finding solid answers to our questions about card-to-card variance might be difficult. For one thing, the R9 290X is particularly sensitive to ambient temperatures. A warmer environment can produce lower clock speeds, and cooler ambient temps can lead to higher clocks. Unfortunately, Damage Labs isn't set up for precise climate control. I'm lucky to have breathable oxygen most of the time. I did try to keep room temperatures from rising too high by cracking open a window when running a load test caused the room to heat up, but maintaining a perfectly steady environment just wasn't realistically possible.
Our solution for that problem was to take lots of samples, especially for our main test case, in Skyrim. (We chose this game because it's a reasonably taxing workload, which is also why we've used it for testing GPU power consumption in the past.) We monitored each card's vitals for 30 minutes while running the game with our character standing still in a particular spot. We then tested each card three times, to see how much clock speeds varied from run to run. That's 10.5 hours of testing at a minimum, just in Skyrim, to accommodate all of the configs we included. The actual testing time was much longer, since when we started, we were kind of clueless about the best way to proceed.
Our primary goal was to compare the performance of the retail and press sample 290X cards with the new Catalyst 13.11 beta 9v2 drivers, which attempt to equalize blower speeds. However, out of curiosity, we also decided to test our initial review unit and the HIS retail card with the older 13.11 beta 8 driver, which doesn't equalize fan speeds, to see how much of a difference that software change makes.
We've plotted a number of variables from our test sessions below. You can click the buttons beneath the plots to see the results from each card. The plots come from one of the three test runs, while the bar charts show the median results from three runs. Also, note that the unit of time on the X axis in the plots is seconds. Somehow, I failed to include the units when making the graphs. Too much sitting around in white fan noise has dulled my wits, apparently.
The card labeled "290X sample 1" is the review sample from AMD that we used in our R9 290X review. Sample 2 came to us a couple of weeks later, also from AMD, after we requested a second card for CrossFire testing. The HIS and Sapphire cards are the retail units. I've also included numbers from our GeForce GTX 780 Ti review sample for comparison.
You can see several things in the plots. Each 290X card starts out at a solid 1GHz. Then, its clock frequency drops and bounces around as the GPU reaches its temperature limits and PowerTune starts working to balance temperature, power draw, and operating frequency. The amount of time before the clock throttling begins varies, depending mostly on how warm the GPU was before we fired up Skyrim. To account for this variability, we clipped off the first five minutes (300 seconds) of the test period when calculating the average clock speed (or fan speed or FPS) for each run.
Click over to the HIS 290X's results, and the newer Catalyst driver that modified blower speeds has clearly paid off. The HIS card clearly runs slower with the older Catalyst 13.11b8 driver. By contrast, our initial review unit is largely unaffected by the change. You can also see that sample 1's clock speeds appear to be somewhat higher than the other cards'. If we plot the median clocks across three runs, here's how things line up:
That's a relatively large amount of variability across four copies of the same card, especially since we're running a popular game. Skyrim isn't a peak workload in terms of power consumption or thermals. You can see that the initial review unit is the fastest of the bunch, regardless of which driver we use, while the HIS card takes up the rear. The newer driver does help the HIS card make up some ground, but it still trails sample 1 by quite a bit.
The graph above is just a summary, though. This table will give you a sense of the clock speed variability from run to run, as the temperature in Damage Labs fluctuated.
|Clock speed (MHz)|
|Run 1||Run 2||Run 3||Median|
|Sapphire R9 290X||891||924||902||902|
|HIS R9 290X||882||893||889||889|
|HIS R9 290X - 13.11b8||845||832||835||835|
|290X sample 1||928||952||930||930|
|290X sample 1 - 13.11b8||947||958||957||957|
|290X sample 2||913||912||903||912|
|GTX 780 Ti||1005||1004||1005||1005|
Clock frequencies for the individual cards varied by as much as 22MHz during our Skyrim tests. Ideally, we'd impose stricter temperature controls and do even more testing. Heck, in an ideal scenario, we'd have a much larger selection of 290X cards to test, and I'd be conducting those tests from the temperature-controlled lower deck of my enormous luxury yacht anchored off the Yucatan coast, aided by a team of cheerleader interns. Sadly, that's not the case. Still, I think the trend for each card begins to become clear after several runs—and we have more data to review.
The new Catalyst drivers raise the HIS card's blower speed by over 300 RPM. Obviously, they're addressing a very real problem. Those drivers also raise the blower RPM slightly on sample 1, our first review unit. That means the 290X will be a little louder overall than our initial review indicated. (All of these tests were conducted in the 290X's default fan speed profile, not in "uber" mode.)
Notice the slight saw-tooth pattern on the plots for each of the 290X cards when used with the new drivers. The plots are much flatter with the 13.11 beta 8 drivers, which means the sound coming from the blower should be smoother and less variable, not quite as easy for the ears to notice. Presumably, when those little spikes happen with the new drivers, the GPU is reaching a thermal limit and needs additional help. The ensuing ramp up is abrupt, although the ramp down is more linear. Contrast that to the fan speed curve for the GTX 780 Ti, which is smoother than an Nvidia marketing pitch.
Also, in the "stuff you didn't expect" department, notice that the blower RPM for the GeForce GTX 780 Ti is higher than for any of the 290X cards, even though the 780 Ti is much quieter under load than the R9 290X. Nvidia's blower appears to have a slightly smaller diameter, but I'm impressed that it runs at substantially higher RPM and produces less noise.
Each plot starts with a flat line for the first couple of minutes, and then the FPS numbers start varying up and down at regular intervals. Why? Because after your charcacter stands still in Skyrim for a while, the game switches to an "attract mode" where the camera pans around him in a circle. The variance you see in the FPS plots is caused by that constantly changing perspective.
The clock speed variance we saw above translates into performance differences fairly predictably. The total gap in terms of FPS is fairly small, since we're testing at 4K resolutions where Skyrim doesn't run at really high frame rates. There is a 10% gap, though, between the HIS 290X and sample 1 with the 13.11 beta 8 drivers. With the newer drivers, where blower speeds are more even, the FPS gap narrows to about 5%. We might see larger differences at lower resolutions, where GPU speeds are likely to play a larger role than they do at 3840x2160. (At 4K, memory bandwidth has gotta be a big constraint.)
Thing is, small differences in performance can cost you a lot of money at the very top end of the graphics card market, with bragging rights on the line. The price difference between the Radeon R9 290 and 290X is $150, and the two were 3% apart in our overall performance assessment. That example's a bit extreme, but we saw a 12% difference between the GeForce GTX 780 and the GTX 780 Ti. The price difference between those cards is $200.
|Mozilla scraps Firefox for Windows 8's Modern UI interface||17|
|AMD could launch Hawaii-based FirePro card next week||6|
|Report: Steve Jobs wasn't keen on producing an Apple TV||6|
|Xbox One's Titanfall performance: not so great||74|
|TR subscriptions: our progress so far||72|
|Friday Night Shortbread||33|
|Friday night topic: Where is that plane?||137|
|WSJ: Microsoft, Google pressure Asus into shelving dual-OS tablet||35|