Evaluating the claims
When I began work on this article, I intended to offer my own attempt at an evaluation of the competing claims of FutureMark and NVIDIA. Now, Dave at Beyond3D has already offered an extensive evaluation with more detail than I was prepared to offer, so I will have to embarrass myself otherwise. I won't attempt to match all of his analysis, but I will try to offer my thoughts on the four basic tech issues I've identified among NVIDIA's complaints.

• Not enough multitexturing in game test 1 — Complaints about game test 1 are a large part of NVIDIA's case against 3DMark03. I'm compelled by NVIDIA's argument that FutureMark concentrated here on making a nice demo rather than a test representative of current games (and this first game test is indeed meant to represent current games). The percentage of pixels onscreen that are part of a skybox is probably a bit excessive. And anyone who's seen the 3DMark03 demo can see how FutureMark's developers could have taken a liking to the game 1 models and scene layout. This part of the 3DMark03 demo mode is really, really cool.

  However, FutureMark's gaffe doesn't seem too severe. Many current games are fill-rate bound, and many are bound by single-textured fill rate. Truly extensive multitexturing doesn't dominate the scene quite like one might expect. Take, for instance, the poor Matrox Parhelia with its quad texture units per pipe and massive theoretical texel fill rate; two to three of those units are doomed to sit idle. This is one reason why ATI's 8-pipe design for the R300 (Radeon 9700) chip makes so much sense.

  I'd prefer to have seen more complex geometry in this test to give it a little bit better balance. But to claim it's not representative of current games isn't entirely fair.

• The stencil shadow volumes implementation — We've already looked at FutureMark's response to NVIDIA's claim here, and we've seen benchmarks which prove the test isn't bound entirely by vertex shader performance. I'll leave the fight over the best methods of vertex skinning to graphics developers, but this one seems like a victory for FutureMark.

• Too much pixel shader 1.4 — This is a tough one, because it's an old fight (PS 1.1/1.3 vs. PS 1.4) between NVIDIA and ATI, yet it's a very current fight about the immediate future (the next 6-12 months or so) of 3D games.

  The primary reason pixel shader 1.4 has proven so useful to FutureMark is its ability to deliver per-pixel lighting effects in a single pass. As ATI pointed out to me in our conversation about 3DMark03, John Carmack's now-famous .plan file update on the subject describes PS 1.4's ability here:

  The fragment level processing is clearly way better on the 8500 than on the Nvidia products, including the latest GF4. You have six individual textures, but you can access the textures twice, giving up to eleven possible texture accesses in a single pass, and the dependent texture operation is much more sensible. This wound up being a perfect fit for Doom, because the standard path could be implemented with six unique textures, but required one texture (a normalization cube map) to be accessed twice. The vast majority of Doom light / surface interaction rendering will be a single pass on the 8500, in contrast to two or three passes, depending on the number of color components in a light, for GF3/GF4

  So PS 1.4 allows for single-pass rendering with per-pixel lighting, while pixel shader 1.1 and 1.3 require multiple passes to achieve the same effect. FutureMark apparently found the same thing in developing 3DMark03. I should note that no one has credibly claimed pixel shader 1.3 would reduce the number of rendering passes required versus PS 1.1 in 3DMark's game tests.

  However, it's quite possible the introduction of tools like high-level shading languages and all the advanced features of DirectX 9-class hardware could cause a rather sharp break between DX8-class games and really out-there DX9-only games with gobs of complex pixel shaders of the 2.0 variety. In this case, PS 1.4 would never see widespread use.

  I tend to think we will see an earnest transition period in which a mix of 1.1, 1.4, and 2.0 pixel shaders along the lines of those used in 3DMark03 will be common practice, with different rendering paths used for different types of hardware. FutureMark had to do some guessing here, and they haven't yet been proven wrong.

• Not enough DirectX 9 — The simple reality is, FutureMark could only go so far in making a "full DX9" test. DirectX 9 is a young API, and the tools are just now coming together. In light of this fact, 3DMark03's Mother Nature test seems like a decent first crack at a DX9 scene, and the procedural shaders in the PS 2.0 feature test are the kinds of complex shader programs one would hope to see. I asked FutureMark a couple of questions about the 2.0 shaders used in 3DMark03. The answers are worth repeating here.

  TR: How many instructions long are the pixel shader programs in the Mother Nature and PS 2.0 tests?

  FutureMark: We haven't published the actual shader code, but I can reveal that the ps2.0 test's procedural texture generation shaders are about as much as you can fit into a 2.0 pixel shader.

  TR: Did FutureMark use Microsoft's High Level Shading Language or a similar tool in developing any of 3DMark03's tests?

  FutureMark: All shaders are written in the assembly like shader language. This is because HLSL produces a pretty optimized shader code, but you can optimize even further manually.

  I have to admit, I'd rather see more and better shader programs written in HLSL, compiled at runtime, and running onscreen concurrently on the various cards. However, those are respectable answers for a first-generation DX9 benchmark. 3DMark03 isn't the end-all, be-all DX9 test, but it seems like a reasonable start. FutureMark's point about game test 4's workloads being designed for DX9 class hardware is persuasive, as well.