WELL, I CAN HARDLY BELIEVE we’re back on this subject again, but this saga just won’t die.
Let me try my best to bring you up to speed. First, there was a late, difficult birth for NVIDIA’s new line of GeForce FX chips, especially the stillborn GeForce FX 5800 Ultra. Then, NVIDIA voiced complaints and concerns about FutureMark’s new version of 3DMark, the DirectX 9-based 3DMark03. 3DMark’s publisher, FutureMark, fought back, and it was a hoota big, public brawl between hardware vendor and benchmark house.
Once GeForce FX review units finally trickled out to reviewers, folks started finding driver optimizations on the FX cards that reduced image fidelity to gain performance. NVIDIA’s spiffy new Detonator FX drivers seemed to fix that problem, and improved performance dramatically. But then ExtremeTech published an article exposing specific optimizations in the Detonator FX drivers intended to inflate 3DMark03 benchmark scores.
Soon, FutureMark released an audit report confirming the problems with the NVIDIA drivers and explicitly stating that NVIDIA was cheating. NVIDIA disagreed. Brouhaha ensued. ATI, for its part, was also caught optimizing its drivers. ATI owned up, pointed out that its optimizations didn’t change image output, and pledged to remove them from its next driver release anyhow.
FutureMark released a new build of 3DMark03 that disabled the disputed driver optimizations. (We benchmarked the build with optimizations versus the build without.) Dell then weighed in in support of 3DMark03. Presumably after some behind-the-scenes legal wrangling, FutureMark issued a joint statement with NVIDIA backing down from the “cheat” word.
And that’s where we are now. Phew. Thank goodness for hyperlinks. This story has been unfolding long enough to induce fatigue in certain quarters.
Even more background
If you’re old like me, you may remember the day when ATI was caught optimizing its Radeon 8500 drivers for Quake III Arena timedemos. The trick was simple: ATI’s drivers looked for an executable named “quake3.exe” and turned down texture quality when “quake3.exe” started. Kyle Bennett at the cold, HardOCP renamed “quake3.exe” to “quack3.exe” and ran some benchmarks. ATI was busted.
In a funny twist of fate, I got a tip earlier this week about NVIDIA’s Detonator FX drivers. The allegation: if you rename 3DMark03.exe to something else and run the benchmark with anisotropic filtering enabled in the drivers, test scores drop. In other words, NVIDIA appears to be using the same lame technique ATI did way back when: keying on the program’s filename in order to trigger benchmark “optimizations.” In this case, those optimizations appear to be a lower quality form of texture filtering than the anisotropic filtering method selected in the driver control panel. Many review sites like us benchmark cards with anisotropic filtering and edge antialiasing turned on, so these things do matter.
Keep in mind, now, that this goes beyond the eight specific Detonator FX “cheats” FutureMark identified in its audit report. This is in addition to all of those things, and this one works with the latest release version of 3DMark, build 330. It appears NVIDIA has taken yet another measure, which FutureMark didn’t catch in its audit, in order to boost scores in 3DMark03.
You probably won’t be surprised to learn that my tip came from ATI. The ATI folks say they were studying the Detonator FX driver’s new filtering routines, to see what they could learn, when they discovered this quirk. (Some have claimed NVIDIA was the original source for the Quake/Quack debacle, but I have no first-hand knowledge about that.) Regardless, we felt compelled to test the Detonator FX drivers to see if the allegations were true. So we gathered up a GeForce FX 5800 Ultra, a GeForce FX 5600, and a GeForce FX 5200 Ultra for a little benchmarking session. We brought along our typing fingers, so we could rename “3DMark03.exe” to “3DMurk03.exe” and see what happened. Our results follow.
Our testing methods
The benchmark results we’ll present come from testing conducted by our own Geoff Gasior. I also conducted extensive testing myself on a GeForce FX 5800 Ultra as we formulated the best means of testing for these driver optimizations. The results of Geoff’s tests served to confirm my own results. Since Geoff was able to test on a wider range of cards, we’ve elected to publish his results.
As ever, we did our best to deliver clean benchmark numbers. Tests were run three times, and the results were averaged.
Our test system was configured like so:
|Processor||Athlon XP ‘Thoroughbred’ 2600+ 2.083GHz|
|Front-side bus||333MHz (166MHz DDR)|
|North bridge||nForce2 SPP|
|South bridge||nForce2 MCP|
|Chipset drivers||NVIDIA 2.03|
|Memory size||512MB (2 DIMMs)|
|Memory type||Corsair XMS3200 PC2700 DDR SDRAM (333MHz)|
|Graphics card||GeForce FX 5200 Ultra 128MB
GeForce FX 5600 256MB
GeForce FX 5800 Ultra 128MB
|Graphics driver||Detonator 44.03|
|Storage||Maxtor DiamondMax Plus D740X 7200RPM ATA/100 hard drive|
|OS||Microsoft Windows XP Professional|
|OS updates||Service Pack 1, DirectX 9.0a|
The “Quality” setting was used with the Detonator FX 44.03 drivers.
The test system’s Windows desktop was set at 1024×768 in 32-bit color at an 85Hz screen refresh rate. Vertical refresh sync (vsync) was disabled for all tests.
We used the following versions of our test applications:
- FutureMark 3DMark03 Build 330
All the tests and methods we employed are publicly available and reproducible. If you have questions about our methods, hit our forums to talk with us about them.
3DMark03 overall score
We tested all of the new NV3x chips we could get our hands on, including the NV30, NV31, and NV34. Of course, we tested with 3DMark03 using two filenames: “3DMark03.exe” and “3DMurk03.exe”. In order to keep a control group of sorts, we tested both with and without anisotropic filtering enabled. As you will see, the “3DMark” and “3DMurk” scores are nearly identical when aniso is disabled. However, with 8X anisotropic filtering set in the driver…
…scores drop in “3DMurk.exe”. The GeForce FX 5800 Ultra’s scores suffer most dramatically, but all three of the GeForce FX chips are affected. In the following pages, you can see the scores on the four individual game tests that contribute the 3DMark03 overall score. After that, we’ll have a look at the differences in image output.
Game test 1
Game test 2
Game test 3
Game test 4
We’ve taken some screenshots using 3DMark03’s image quality tools in order see if there’s any output difference between “3DMark” and “3DMurk.” The differences are subtle, but unmistakable. Below are images taken from frame 480 of 3DMark03’s game test 3, Troll’s Lair, on a GeForce FX 5800 Ultra with both executable filenames.
Differences between the two are not immediately obvious here, even when you’re looking at the full-sized PNG images. 3DMark03 doesn’t present a lot of flat surfaces and bright lighting that might make detecting texture filtering quirks relatively easy. However, using some image editing tools, we can easily identify the differences.
The image below is the result of a mathematical “diff” operation between the two full-size screenshots linked above. We’ve enhanced the image using a histogram function in order to make the differences more visible. Here’s the result.
Texture filtering methods do indeed appear to have changed when we renamed the executable file.
To give you some context, the output from 3DMark03’s image quality tool is usually consistent from run to run. We captured a pair of screenshots from frame 480, both using “3DMurk03.exe”, and ran a diff between them. Not a single pixel was different. The differences between “3DMark” and “3DMurk” image output are very likely related to the performance differences we’ve observed.
Given the source for our tip about these benchmark-specific driver optimizations, we had some initial trepidation about bringing this issue out in the open. In light of that fact, it’s probably not appropriate for us to comment on the nature of these optimizations, whether they qualify as outright cheats, or any of that. We have already commented on many of the issues surrounding driver optimizations and popular benchmarks like 3DMark03, Quake III Arena, and the like. If you read all the linked items in the introduction to this article, you will have a good overview of the issues involved. We will leave it to the reader to draw his own conclusions.