Post by Boooooommm:

This is simple: I don't trust 3DMark benchmark result because they don't ultimately show the performance on real games available in the market! They do not use engines that are, will or were used in games, so it really doesn't tell you exactly what your results will be in real games. I would consider it more a RAW POWER benchmark than anything else. As to nVidia and ATi cheating with their drivers the answer is simple: If you call optimizing cheating, then you folks must hate faster cards. It's a bit ironic how people think these current events can be called cheating. Anyway, there is no gain in optimizing drivers for a test that will not really provide a considerable view on how a card will perform in real game situations. In this case, both are cheating. SO nobody here is saved from the sin. Futuremark doesn't make a reliable benchmark (even though it is respectable today) because it doesn't make the cards run on Real Game situations. nVidia optimized for marketing reasons and ATi... well... ATi's life with drivers is optimizing. So all of them are completely unreliable if this is the case. Nobody, unfortunately is safe.

quote " Yawn. Impending death of 3dmark on horizon. Bring on the games."

Let me say this just once more for the Morons who don't seem to get it. IT IS NOT FUTUREMARKS FAULT Nvidia is CHEATING. There is nothing wrong with 3DMark03 as a Benchmark of DX9 Capabilities. I wouldn't be a bit surprised to learn that Nvidia cheated in the hopes of getting caught just to make Futuremark look bad and discredit their benchmark. Futurmark didn;t want to kiss Nvidia's ass in the first place and Nvidia dropped out of the Beta program. The rest is just sour grapes on Nvidia's part because they found out that Futuremark wouldn't cater to thier NV30 rendering path.

and as for using Games as benchmarks... As soon as you use a Game to test a Video Card it becomes a BENCHMARK itself. If it becomes popular enough, It becomes a target for cheating. Any GAME BENCHMARK can be cheated just as easily as 3DMark03 if the IHV wants to do it. "Use Games" is not an answer.

NVIDIA is the culprit here, NOT Futuremark. I wish some of you Lamers would get a clue... (or a Brain)

It is cheating the consumers. However, is it cheating that most software writers optimize their software to operate better with Pentium based processors? AMD may not be as fast, but if they have to emulate and still outperform the competition no one seems to cry foul. Likewise with the driver optimizations. Both ATI & Nvidia need to spend a little more time correcting their drivers to work with the various types of software programs and spend less time worrying about the benchmarks. The benchmarks will follow!

For anyone defending Nvidia in this cheating nonesense should be shot. Same for ATI if they did the same amount of b.s. now. What Nvidia has done is cheat the consumer. Plain deception. To cheat on a benchmark and then say that very benchmark you cheated in is not worth it's salt; then cheat again another 2 times, explains to me that Nvidia does not give a rats ass about consumers. For the lemmings that want to see all this as harmless optimizations from Nvidia, should take some drugs and call Dr.Spock in the morning.

this is funny...not defending either ati or nvidia...but, is ati in the beta program for future mark...nvidia i guess didnt want to pay fees to be part of the beta program...maybe thats why ati felt they need to say sorry (not coz they care and loved you loyal ati fans) and nvidia doesnt...they both just trying to maintain there image as the two top contenders in graphics...all they want is our hard earn cash...i think both companies make great video cards...benchmark to me is just to see if i my graphics card can handle the games being released today...damn i dont sit and play games and say damn im loosing x amount of fps...who cares whether they optimise...we're gonna optimise it ourselves anyways...there is no real life comparison when it comes to benchmarks just like cpus...your not really gonna see the numbers...of course there is a difference when your making comparisons with dinosaur components...for the record i think ati is a better video card but, like nvidia coz i dont have to optimise vid card myself...muahahahahah!!! with ati optimising i still have to optimise...this tells u i dont care!!!

ERIC'S OPINION
When this whole sorry spectacle first broke, there were those who claimed nVidia couldn't possibly be doing anything wrong. When Futuremark issued a formal statement calling it a cheat, there were those who said nVidia wasn't really cheating, just optimizing. Now nVidia is apparently doing the exact same things that ATI was castigated for a couple of years ago--fudging image quality during benchmarks to garner higher scores. Many of the voices who are defending nVidia today were the same voices who called for blood from ATI because of Quack vs. Quake. One wonders just how long a self delusion can continue before the truth runs head on into inescapable fact.

In the final equation, though, this changes very little. Given how wantonly nVidia has distorted the issue to serve its own interests, and how pitifully Futuremark has responded to a blatant lobotomy of its one and only product, can anyone give any credence whatsoever to any benchmark nVidia issues? Futuremark's reputation has been utterly destroyed here, and the very idea of synthetic benchmarks at all has been called--quite rightly in my opinion--into question.

That nVidia would stoop so low is disappointing but not surprising. That it is being allowed to get away with it by so many is.

#221,

I think you meant to say that "Confusing opinions with fact are what HardOCP is good at".

What did they use this as a rebuttal for ET's analysis of nVidia's 3dmark issues? That ET was sour because of not getting to review Doom 3. The "fact" was that they didn't get benchmark Doom 3. It is also fact that they were investigating this before the Doom 3 benchmarks occurred, and that the issues are independently verifiable and reproducible. Those facts were not recognized by the contested editorials at HardOCP. Picking and choosing facts to talk about does not make opinions fact, and proposing purely speculative opinion in response to facts is not unbiased. Nor is treating one IHV (ATI with Quack) differently and far more negatively than another (nVidia with 3dmurk), especially when the first case was prompted by the latter (nVidia). The difference with 3dmurk is that the prompting was not kept secret at all, and Tech-Report isn't treating the IHVs differently.

Their reviews are typically done by Brent or Sean, and those people tend to write non biased articles. The site is represented by Kyle, and I believe Steve, and it is their comments that are the focus of the disgust of many people. While the latter two are busy ignoring facts and stating opinions only consistent with one IHV's PR, the site is biased: Sean and Brent can be said to be unbiased so far only because their reviews don't follow the same practice of ignoring facts...the way the IHV looks should be determined by facts, and if that is done fairly and without it isn't bias.

Mister Wasson, you disappoint me.

run 3dmuck on two equally priced cards from nvidiati (not those 500$ cards. 200$ is more then enough),get the results
and then run whatever games you like and compare the results with the murk ones.
if the results are about the same then nvidia is the wrongdoer.
and if not ,then 3dmark is not not so good for benchmarking any more, isnt it?

once this graphzilla fight is over, world peace will be achieved and all problems will be over.

keep it in proportion people.

I did the same test as you did by renaming to 3DMurk03 with AF8X and I also ran it before and got exactly the same fps both times. I do have a FX 5800 Ultra using the 44.03's. I just ran the nature test and got 17.4fps while it was plain 3DMark03.exe and got 17.4fps after renaming it 3DMurk03.exe. I bet them test are fake and you just wanting hits on your site by publishing these bogus figures. I will conduct these tests again when I get the FX 5800 Ultra too. I be they will be the same before and after renaming 3DMark03.

"....why don't you read Carmack and Tim's opinions about the matter? ...."

Why? Who died and made them God?

This article sides with Kyle at HardOCP.
It says the blame lies with FutureMark's taking "Fees".
Taking 'fees' from the very companies (Nvidia for example) that FutureMark software is performing "impartial objective" tests on.

ipKonfig on purse power: the FutureMark controversy

So umm... what happens if you rename the Quake3 EXE file to 3Dmark03.exe?

In fact... what happens if you rename any 3D game/program to 3Dmark03.exe?

Try it... I wanna know if things look screwy.

What are 'optimizations'? Basically, the programmers find patterns in the execution of the code and add more intelligence to their own code to take advantage of it. In its essence, this is truly no different than compressing a file.

Benchmark optimizations (be they from actual games or from benchmark suites) are inherently risky, since the output (the set of frames) is identical from run to run.

This is equivalent to testing pkzip with the exact same file, over and over again. If you wanted to, you could compress the entire file down to zero bits, since from its name can assume what output to generate.

Evidently, neither NVIDIA nor ATI would ever go so far. My point is that any optimizations should be looked at critically, and that even identical output doesn’t mean that gross cheating isn’t taking place.

Conversely, optimizations done to the quake3 engine (for example) that result in improvements across all / most benchmarks of games based on the quake3 engine could be taken to be ‘valid’ optimizations.

Furthermore, if my imagined quake3 engine optimization were to modify the output slightly, that would not seem to me to be that big an issue. (lossy compression of images is evidently something we appreciate in jpegs, for example)

However, in a ‘future benchmark’ like 3dmark03, I do not believe that aggressive optimizations should be used AT ALL, since you end up optimizing heavily for an estimate of what games will be.

After all, optimizing aggressively for one game (e.g. Dungeon Siege) does not mean that you have optimized for all other games as well (all shooter games, RPG games, racing games, etc.).

To those who discount ‘future benchmarking’ as a whole and care only about benchmarks from actual games:

Simply because the problem space is complex doesn’t mean that we should simply give up.

An apt analogy can be found in economics, which also has a set of measures that you could decide to discount entirely: ‘So what if the GDP grew by X%, and the unemployment rate grew by Y%? That doesn’t mean that I should sell my shares and buy government bonds’. Evidently, only a fool would whole-handedly knock all the information that economic measures give us, only because they are uncertain.

Similarly, although a benchmark like 3Dmark03 is not perfect, and could in fact be so skewed that it were made useless, it still is one of the only measures we have of the ability of these cards (which are being marketed as ‘DirectX 9’ cards) to run DX9 code.

Here is how it breaks down: You can disregard the estimate because you do not plan on playing any future titles on your card, or because you have tangible evidence that it is skewed.

You can’t reasonably discredit all ‘future benchmarking’ for being inherently weaker than the benchmarks of today’s games, since by doing so you evidently show that you do not understand the actual purpose of the ‘future benchmark’ itself.

P.S.: as I understand it, there were games being demoed at E3 recently that are using DX9 shaders and whatnot. If DX9 is going to be used in games within a year (or six months at least) then I would say that testing for it is relevant today.

You guys might want to check out Quake 3's image quality using the latest 44.03 dirvers from nvidia....

Might only be my imagination, but comparing 40.xx to 44.03, 44.03 seems to be 'optimizing' for Quake 3....

Plastic Surgeon, before you mouth off, read what IpKonfig article said. Here's a snippet. I think it points finger in right direction.........
+++++++++++++++++++++++++++++++++++++++++++++

Subject: PURSE POWER (FutureMark Controversy)
Date: 07 June 2003
Author: Brian Hall

Buttered Bread

FutureMark's bread, in the short and medium term, is buttered on the manufacturer side. The pre-release "Beta" membership fee can be perceived as a kind of blackmail by FutureMark. It certainly leaves wide open the suspicion that those who pay get special treatment and information, and those who don't will have to take their chances. (Theoretically, it could even go so far as to involve deliberate crippling of performance on competitors' products. Remember Dr. DOS?) So FutureMark gets Money and Power. On the other hand, manufacturers know that their fees make a substantial contribution to FutureMark's profit margin, and can press for special consideration. ......

I see two posibilities here.

1) nVidia feels that 3dMark's settings are incorrect, so they have thier driver detect 3dMark and force it down what they feel is the approprate rendering path. The higher scores actually represent what happens in games, which are more gracefully path-selected and/or are also forced down the correct path. In other words, nVidia is "cheating" in a way that actually benifits the public with faster games and more realistic 3dMark scores.

2) As above, except the forced path exists only to improve 3dMark scores, and has nothing to do with actual gameplay. Other specific paths may exist to improve other common benchmarks as well. In other words, nVidia is "cheating" purely to inflate benchmarks.

Given the extreme sublilty of the pic differences, I think it's possible that nVidia is only doing this to work around path-selection issues. This may have been the only way to do it, since they're not in the beta program and were until recently hostile with Futuremark.

It seems more likely to me that this is a strait up cheat. But I'm not going to become a fanATIc just yet.

Luckily for me, I'm not looking to upgrade video at all for a while!

-- LiquiTJ]

(If for some reason anyone wants to reply personally, send a shack message to Liquid.)

This article sides with Kyle at HardOCP.
It says the blame lies with FutureMark's taking "Fees".
Taking 'fees' from the very companies (Nvidia for example) that FutureMark software is performing "impartial objective" tests on.

ipKonfig on purse power: the FutureMark controversy

I start to think that there are a lot of fanATIcs, then I realize it is just the same folks copy and pasting their post on board after board after board. Quite humorous. I guess they are really are starved for attention and need folks to agree with them somewhere.

You guys need to wake up and smell the coffee as this situation is about more than your precious ATI or NVIDIA, it is an indictment of a tool that the entire industry has put value in and there seems to be none left in it, if there ever was.

"The way it's meant to be cheated...."
Should be Nvidia's new motto. They make me laugh. Waht a joke of a company now.

The only way Nvidia and to a lesser degree ATI, will regain the trust of the buying public is to publish the app specific "optimisations" for each driver update. A separate tab for these optimisations should be included on the control panel, with an option to switch each on or off.

For the record I have a 9700 pro and a 4600.

ATI cheated, too (But not as mutch as nvidia did [about 2% performane gain])
...
Ati-Statement:

"The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST."

Read all @ http://www.theregister.co.uk/content/3/30880.html

And it already begins (from HardForums):

Kyle, are you going to start checking other games/benchmarks to see if they're affected?

Response:
At this point I think it is bascially couter productive to spend a bunch of time on going back over bencharks that are years old, but I do think it is very much well worth our time making sure the tools we currently use are solid.

Wait...who said anything about YEARS OLD. How about your most RECENT video card reviews?

and

(To Kyle) In all honesty though, how hard would it be to rename a few common benchmark game executables and compare the timedemo results?

Response:
Then by all means, get to it and let us know what you find. I have other things to do that concern where we are going instead of where we have been

So basically, instead of getting his hands dirty and POSSIBLY show that nVidia may have cheated on other game tests, SOME of which he contends are VALID, he shrugs it off and basically says they will be benchmarking different in the future, and don't give a damn about the past.

You'd think that information regarding currently benchmarked games COULD effect how he'd want to benchmark in the future, but NOOO...not going to take any chances at hurting his relationship with NV.

At least he can keep taking pot-shots at FM...those REALLY boost his credibility.

*sigh*

ok, so the scores so far are:
Nvidia = zero credibility
ATI = zero credibility
3dfark = zero credibility
many other benchmarks = zero credibility
Blind masses = never underestimate their appeal for meaningless graphs

As usual, I'll buy based on how well my favourite games run and all-round compatibility. It's not very hard. Oh wait, Nvidia has the "the way it's meant to be played" EA program happening. Just another way to break the competition through "optimised" games. OK then, I'll just get a console. Gamecube, Xbox or PS2? Hell, let's get all 3 and forget about PC games altogether. Yeah.

#188 lol. Then ATI goes and does it again. Didn't you read the futuremark audit? They caught ATI as well, sure its not as much as nvidia, but its an 8% increase on one gaming part of 3dmark03.

They both do it, and both after doing it in the past. They wont ever stop. Get real

Well it seems that ATI is doing the same thing by another mean (depends on where you enable AF, ie driver or 3DMark) and can gain around 10%. Seems Ati forgot to talk to you about that ;)

Plus inquiétant, on voit que les performances sont loin d'êtres identiques chez ATI lorsque l'on force l'anisotropic via l'application ou via les drivers, ce dernier mode étant près de 10% plus performant.
http://www.hardware.fr/news/lire/06-06-2003/

183 quit complaining.
Nvidia is engaging in legitimate biz practise. It's making sure its cards do great in benches, beat the competition. That's biz, baby. Relax.

If Nvidia confessed and said they were sorry, they probably wouldn't be in this mess.
Gamers forgave ATI, becuase they said they were sorry, and wouldn't do it again. Nvidia, being all high and mighty, has decided that they do not need to say sorry.

That, and I don't trust the french at all.

Reviewers Condemn Video Tweaks

ATI tweaked its new video card software to make Quake run faster. The company was only trying to please gamers, but instead it upset just about everyone. By Andy Patrizio.

By Andy Patrizio

2:00 a.m. Nov. 28, 2001 PST

Video card maker ATI Technologies is trying to smooth some ruffled feathers after it was discovered the software for its newest video card manipulated the performance of a popular gaming benchmark.

The software drivers for ATI's Radeon 8500 card were tweaked to make Id Software's Quake 3 Arena game run faster. The driver turned down the quality of the textures in the game and increased the frame rate.

But the tweak didn't affect any other games, and ATI has been angrily accused of trying to fudge benchmarks. Quake 3 is the de facto standard benchmark used by hobbyist websites and computer magazines to test the performance of video cards and PC systems.

See also:
Discuss this story on plastic.com
Pokey Man Big in Japan
Free Card Game Is Such a Deal
Murder Game Caught on Video
Read more Technology news

Even John Carmack, the highly respected lead programmer of id Software, chimed in on the controversy. "Making any automatic optimization based on a benchmark name is wrong," he wrote in a note posted to the Web. "It subverts the purpose of benchmarking, which is to gauge how a similar class of applications will perform on a tested configuration, not just how the single application chosen as representative performs."

The driver tweaks were discovered by HardOCP, a hobbyist site. They were unearthed when testers renamed the Quake 3 executable and noticed that performance dropped. The testers realized the driver was looking for that particular file name.

ATI has since released a new driver that doesn't just crank up Quake 3 performance, but cranks up all games that use the Quake 3 engine, including the newly released Return to Castle Wolfenstein.

ATI claims the tweaks were intended to merely improve the performance of a very popular game, not to enhance the card's ranking in performance benchmarks.

"(The accusations have) caught everybody here totally off guard," said David Nalasco, technical marketing manager for ATI. "Our driver guys were totally in shock. We thought we were making the experience better for the user."

"When you're developing a driver, you want to look at the applications people are going to use with your product," he said. "In the case of the 8500, it's a gaming-oriented product. So you want to start with what you think is most important and work from there."

But the issue has left some people a little red-faced, and not just ATI. Maximum PC put the Radeon 8500 on the cover of its November issue, saying it "beats GeForce 3 and scores a perfect 10!" In its January 2002 issue, Maximum PC will re-examine the Radeon 8500 with the new drivers.

"Maximum PC doesn't approve of these tactics," said Jon Phillips, the magazine's editor in chief. "There should be an implicit trust between ISVs and hardware vendors and journalists that that kind of thing isn't changed without notification."

Phillips said ATI has suffered more in the eyes of his readers than his staff.

Anand Lal Shimpi, who runs the popular AnandTech website, also ran a glowing review before the driver issue was uncovered. Like Phillips, he said the company has egg on its face.

"It damaged them a lot in the eyes of their end users," said Shimpi. "These are their customers, and they are a company being branded as a cheater. As a reviewer I've seen everything. This is just another I-can't-believe-they-did-this."

"Die hard ATI fans were saying this is going too far," Shimpi added. "It's not murder, but it's still cheating. It's still a betrayal of trust I think, and it really comes down to, 'did they consciously make a decision to do this?'"

ATI said the driver tweaks were made simply to optimize the performance of a popular game, not monkey with the benchmarks.

Dean McCarron, co-president of market research company Mercury Research, said he has more driver tweaks than he can count. In the past, video card drivers were tweaked to perform well with Ziff-Davis' 3D Winbench, which was a popular graphics benchmarking tool a few years back. But most users don't care about driver tweaks, he said.

"Essentially, as long as the driver works, probably on the order of 90-plus percent, users won't care," he said. "You get their attention when it doesn't work."

Personally I don’t give a crap who is the best NVIDIA or ATI. I do not base my judgment based on the freaking LOGO. I buy what fits my needs. It is simple is that. I have had both ATI and NVIDIA products before. Loyalists are the reason why company's producing crap in the first place. Why you might ask? Well it is very simple. Because you are supporting them, by buying there crap. Maybe it is time to start basing your judgment on the product itself and not the LOGO. Doing so will defiantly putt pressure on the other team and will force them to do better.