As you know if you have been following graphics hardware over the past six months or so, there is a rich thicket of issues surrounding benchmarking, driver optimizations, and next-gen graphics hardware. I will not attempt to bring you up to speed on all of them here. I believe Newell’s presentation summarized the relevant issues nicely and stated Valve’s position with little ambiguity, so I’ll present my pictures of those slides with some commentary as necessary.
The context of Newell’s remarks was ATI’s “Shader Days” event. Select members of the press spent the day here taking in presentations by employees of ATI, Microsoft, and SoftImage about the latest developments in DirectX 9-class shaders, with an emphasis on pixel shader hardware. The briefings were full of useful information about how innovations made possible by DirectX 9-class programmable graphics hardware promise to vastly improve visual fidelity in games and other interactive apps without compromising performance. The capstone of the day’s presentations was Newell’s talk, which was accompanied by a stunning demonstration of Half-Life 2’s source engine in action.
Newell began by establishing Half-Life 2’s credentials as a true Direct X 9 app.
Newell then talked about his consternation over techniques IHVs have used to skew benchmark results, a problem he considers serious, because it threatens the gaming experience for Valve’s customers.
As you can tell from looking at the list in the slide above, Newell was concerned particularly with some of the techniques NVIDIA has used in recent driver releases, although he didn’t exempt other graphics hardware makers from his complaints. He said they had seen cases where fog was completely removed from a level in one of Valve’s games, by the graphics driver software, in order to improve performance. I asked him to clarify which game, and he said it was Half-Life 2. Apparently, this activity has gone on while the game is still in development. He also mentioned that he’s seen drivers detect screen capture attempts and output higher quality data than what’s actually shown in-game.
Newell summed up his problem with bad benchmarks in a nice zinger line: “Our customers will be pissed.”
The results are striking, as you can see. I believe the results were recorded at 1024×768 in 32-bit color. This is with only standard trilinear filtering and no texture or edge antialiasing.
NVIDIA’s NV3x-derived chips are way off the pace set by the ATI DirectX 9-class cards. The low-end GeForce FX 5200 Ultra and mid-range GeForce FX 5600 Ultra are wholly unplayable. The high-end GeForce FX 5900 Ultra ekes out just over 30 fps, well behind ATI’s mid-range Radeon 9600 Pro (and yes, that score is from a 9600 Pro, not a stock 9600the slide was mis-labeled). The Radeon 9800 Pro is twice the speed of the GeForce FX 5900 Ultra.
Valve even ginned up a value-for-money slide to illustrate the problem with the current price/performance proposition for NVIDIA hardware.
However, NVIDIA has claimed the NV3x architecture would benefit greatly from properly optimized code, so Newell detailed Valve’s sojourn down that path. The company developed a special codepath for the NV3x chips, distinct from its general DirectX codepath, which included everything from partial-precision hints (telling the chip to use 16-bit floating-point precision rather than the default 32-bit in calculating pixel shader programs) to hand-optimized pixel shader code.
The “mixed mode” NV3x codepath yielded mixed results, with a fairly decent performance gain on the FX 5900 Ultra, but not near enough of a boost on the FX 5200 Ultra and FX 5600 Ultra.
Newell also expressed skepticism about the payoffs for NV3x-specific optimizations, noting that the optimization process was arduous, expensive, and less likely to produce performance gains as shader techniques advance. What more, he said, smaller developers are not likely to have the resources Valve was able to bring to bear on the problem.
He suggested one way of dealing with the issue would be to treat all NV3x hardware as DirectX 8-class hardware, which would cut down significantly on eye candy and new graphics features, but which could yield more acceptable performance on NV3x chips. Obviously, he noted, one could always cut down visual quality in order to achieve higher performance, but in the case of Half-Life 2, falling back to DX8 will require tangible sacrifices.
Oddly enough, even using the DX8 codepath, the previous-generation GeForce Ti 4600 outperformed the brand-new GeForce FX 5600 Ultra.
Finally, Valve’s top dog addressed potential accusations of bias head-on. The company recently signed a deal allowing ATI to distribute Half-Life 2 with its Radeon cards, but Newell made clear this was not a case of the tail wagging the dog.
To aid reviewers in evaluating graphics hardware, Valve will be releasing a new, Half-Life 2-based benchmark in the near future, with a full suite of tools and instrumentation to help reviewers (and end users) find performance bottlenecks. The test will output all kinds of useful data, including frame rate averages, deltas, and second-by-second frame rates. Reviewers will be able to record their own in-game demos for playback, and tools will include per-frame screen captures using either the graphics card or the DirectX reference rasterizer included in Microsoft’s DX9 SDK.
No doubt there will be repercussions from Valve’s unprecedented decision to address this topic in this way. We here at TR will be watching developments with interest, and we eagerly anticipate the opportunity give the new Half-Life 2 benchmark a spin. Valve’s efforts to build a useful, detailed set of benchmarking tools into its new game are exactly what we want and need from developers.
At the same time, while the NV3x cards’ weak performance numbers in Half-Life 2 with DirectX 9 are jarring, they don’t come as a complete surprise. We have seen NVIDIA go to great lengths to protect its NV3x hardware from proper scrutiny in prominent benchmark applications like 3DMark03 and Unreal Tournament 2003. Results from the few other DX9-class benchmarking tools available, like ShaderMark, have not bode well for NV3x. Valve’s frustrations seem to stem from the disconnect between its expectationsthat NV3x cards were capable DX9-class graphics hardwareand the apparent realitythe NV3x is a capable performer in DX8-class games, but it lacks the hardware resources (floating-point pixel shaders, register space) necessary to run DX9 games in real time.
We’ve had difficulty confirming this reality because of a lack of good tools to measure DX9 performance, and because NVIDIA has been very guarded with details of the NV3x chips’ internal configurations. It looks as though new games like Half-Life 2, and new benchmarking tools derived from them, are about to change all of that. We will have more to say about all of this soon, once we get our hands on the tests ourselves.