Single page Print

Further NVIDIA optimizations for 3DMark03?


Quake/Quack, meet 3DMark/3DMurk
— 12:00 AM on June 6, 2003

WELL, I CAN HARDLY BELIEVE we're back on this subject again, but this saga just won't die.

Let me try my best to bring you up to speed. First, there was a late, difficult birth for NVIDIA's new line of GeForce FX chips, especially the stillborn GeForce FX 5800 Ultra. Then, NVIDIA voiced complaints and concerns about FutureMark's new version of 3DMark, the DirectX 9-based 3DMark03. 3DMark's publisher, FutureMark, fought back, and it was a hoot—a big, public brawl between hardware vendor and benchmark house.

Once GeForce FX review units finally trickled out to reviewers, folks started finding driver optimizations on the FX cards that reduced image fidelity to gain performance. NVIDIA's spiffy new Detonator FX drivers seemed to fix that problem, and improved performance dramatically. But then ExtremeTech published an article exposing specific optimizations in the Detonator FX drivers intended to inflate 3DMark03 benchmark scores.

Mmm. Fishy.

Soon, FutureMark released an audit report confirming the problems with the NVIDIA drivers and explicitly stating that NVIDIA was cheating. NVIDIA disagreed. Brouhaha ensued. ATI, for its part, was also caught optimizing its drivers. ATI owned up, pointed out that its optimizations didn't change image output, and pledged to remove them from its next driver release anyhow.

FutureMark released a new build of 3DMark03 that disabled the disputed driver optimizations. (We benchmarked the build with optimizations versus the build without.) Dell then weighed in in support of 3DMark03. Presumably after some behind-the-scenes legal wrangling, FutureMark issued a joint statement with NVIDIA backing down from the "cheat" word.

And that's where we are now. Phew. Thank goodness for hyperlinks. This story has been unfolding long enough to induce fatigue in certain quarters.

Even more background
If you're old like me, you may remember the day when ATI was caught optimizing its Radeon 8500 drivers for Quake III Arena timedemos. The trick was simple: ATI's drivers looked for an executable named "quake3.exe" and turned down texture quality when "quake3.exe" started. Kyle Bennett at the cold, HardOCP renamed "quake3.exe" to "quack3.exe" and ran some benchmarks. ATI was busted.

In a funny twist of fate, I got a tip earlier this week about NVIDIA's Detonator FX drivers. The allegation: if you rename 3DMark03.exe to something else and run the benchmark with anisotropic filtering enabled in the drivers, test scores drop. In other words, NVIDIA appears to be using the same lame technique ATI did way back when: keying on the program's filename in order to trigger benchmark "optimizations." In this case, those optimizations appear to be a lower quality form of texture filtering than the anisotropic filtering method selected in the driver control panel. Many review sites like us benchmark cards with anisotropic filtering and edge antialiasing turned on, so these things do matter.

Keep in mind, now, that this goes beyond the eight specific Detonator FX "cheats" FutureMark identified in its audit report. This is in addition to all of those things, and this one works with the latest release version of 3DMark, build 330. It appears NVIDIA has taken yet another measure, which FutureMark didn't catch in its audit, in order to boost scores in 3DMark03.

You probably won't be surprised to learn that my tip came from ATI. The ATI folks say they were studying the Detonator FX driver's new filtering routines, to see what they could learn, when they discovered this quirk. (Some have claimed NVIDIA was the original source for the Quake/Quack debacle, but I have no first-hand knowledge about that.) Regardless, we felt compelled to test the Detonator FX drivers to see if the allegations were true. So we gathered up a GeForce FX 5800 Ultra, a GeForce FX 5600, and a GeForce FX 5200 Ultra for a little benchmarking session. We brought along our typing fingers, so we could rename "3DMark03.exe" to "3DMurk03.exe" and see what happened. Our results follow.