The benchmarking game
I half-expected TR's Ryu Connor to write this article. At the end of a product review, have you ever asked yourself, "what do all these benchmarks mean?" Leave it to Real World Technologies' Dean Kent
, a true doyen of this industry, to throw down the gauntlet.
What I am referring to is benchmarks. It seems that far too many people have completely forgotten, or perhaps never understood, the purpose of benchmarks. How many times have we seen motherboards or processors reviewed where at least half of the entire article is benchmark numbers. These numbers are then compared by the authors, manufacturers and consumers to results from previous tests or even from other publications. Though many think this is valid and useful information, the problem here is that subtle differences between the tests, whether hardware or software, can have a measurable effect on the results. This makes attempts to compare individual components very difficult, if not impossible, and always questionable. The only truly valid way to compare components is to keep as many of the other system components the same as is possible, and ensure that BIOS settings and drivers are similarly configured.
Michel de Montaigne couldn't have said it any better. Whether you agree with him or not, this is a gem, folks. I think the VIA 686B South Bridge controversy and the whole MSI K7T266 Pro soap opera have really brought this to a head. What he writes is so true, though. Hardware websites are a dime a dozen, and we are now at the point where anyone with a computer can publish a review. One has to ask, "who watches the watchmen?" The entire benchmarking situation has become a bit like the alphabet soup of boxing title belts. There are so many different interests tugging at each other but in the end, what can we do about it? The sad truth is that reviewing methodologies are a work in progress and will likely remain that way for the foreseeable future. Read the rest of this three page polemic
to find his stirring conclusion.