Single page Print

Dissecting the 3DMark03 controversy

What good is a benchmark?

RECENTLY, FUTUREMARK RELEASED the latest version of its 3DMark benchmark, and the PC enthusiast world took notice. Previous versions of 3DMark had developed quite a following, in part thanks to publications like us using the tests in evaluating PC hardware. FutureMark claims over 1.5 million copies of the benchmark were downloaded within 72 hours of its release. Steve over at Fileshack told me 3DMark03's release caused the biggest spike in download traffic in Fileshack's history—even bigger than the last major Counter-Strike release.

Meanwhile, just as FutureMark was introducing its new benchmark, graphics heavyweight NVIDIA initiated a public relations campaign aimed at undermining 3DMark03's credibility as a benchmark and discouraging use of the test in the enthusiast press. NVIDIA's first move was to mail out a whitepaper outlining its criticisms of 3DMark03. NVIDIA asked members of the press not to redistribute this document, only to paraphrase or offer excerpts. The document registered some specific complaints about 3DMark03's methodology, but its primary thrust was an overall critique of FutureMark's approach to 3DMark03 and of synthetic benchmarks in general.

The impact of NVIDIA's PR push was immediate and impressive. A number of web sites published articles raising questions about 3DMark03—in some cases, unfortunately, repeating NVIDIA's claims about the test without attribution and without critical evaluation of those claims. Since that time, a number of players, including FutureMark themselves, have weighed in with responses to NVIDIA's criticisms.

During the past couple of weeks, I've talked with representatives of NVIDIA, FutureMark, and ATI about this controversy in an attempt to better understand the issues involved. Also, over the past few days, some intriguing new details about the architecture of NVIDIA's new GeForce FX chip have come to light, and those revelations may help explain why NVIDIA has objected so strenuously to 3DMark03's design. I'll try to cover what's happened and why it matters. Let's start with some background on FutureMark, NVIDIA, and the creation of 3DMark03.

FutureMark, NVIDIA, and the genesis of a conflict
FutureMark is a small company based in Finland whose business depends on two primary sources of income: sales of the "Pro" versions of its benchmarks to end users and sales of memberships in its beta programs to independent hardware vendors (IHVs) like AMD, Intel, ATI, and NVIDIA. The beta program has several membership tiers, with pricing tied to level of participation. Broad participation in the beta program has been key to FutureMark's success. The beta program member list on FutureMark's website reads like a Who's Who of PC performance hardware. Tier-one participants include ATI, AMD, Intel, and Microsoft. Other members include graphics players like Matrox, S3 Graphics, SiS, Imagination Tech, and Trident, plus PC OEMs like Dell and Gateway.

The months-long process of developing a new revision of 3DMark involves input and feedback from beta program partners about a series of design documents, alpha builds, and beta builds of the benchmark. As I understand it, NVIDIA had been a top-tier FutureMark beta program member during the development of 3DMark03 until the first of December, when NVIDIA's membership renewal came due. At that time, NVIDIA elected not to renew its membership. 3DMark03 was in the beta stage of development at this point, and was essentially feature-complete.

By all accounts, NVIDIA's decision not to renew its membership was triggered by its dissatisfaction with the 3DMark03 product and with FutureMark's responses to NVIDIA's input on 3DMark03's composition. Clearly the two parties had substantive disagreements over how 3DMark03 should be built. The questions now are, what were those disagreements, and who was right?

Was NVIDIA miffed because 3DMark03 wouldn't give its new GeForce FX chip a fair shake? Or because the test would disadvantage NVIDIA's current products in the GeForce4 line? Early benchmark results from 3DMark03 aren't as instructive one might expect. The HardOCP tested the GeForce FX versus the Radeon 9700 Pro, and results were mixed. In the first round of tests, the Radeon 9700 Pro won handily. A second set of tests with updated drivers from NVIDIA, however, showed the GeForce FX taking a narrow lead in the overall game score.

Our own testing with NVIDIA's current generation of 3D chips, the GeForce4 line, didn't look too good for NVIDIA:

But such things are to be expected when one's competitor is a technology generation ahead, especially in a benchmark that purports to be forward-looking. Besides, NVIDIA told me straight up its complaints aren't about 3DMark03's performance on its GF4 cards.