A correction: We flubbed networking test results

We recently discovered that the networking test results in all reviews where I used the ntttcp benchmarking tool—including my motherboard, small form factor, and chipset reviews—are invalid.


Here’s what happened. When I first started using ntttcp, the client and server systems had comparable hardware, and the reported throughput and CPU utilization on each side was very similar. This led me to assume that ntttcp was reporting the same results on both ends, which is not the case. Ntttcp throughput is fairly consistent across the client and server machines, as one might expect, but CPU utilization is not. Therefore, my reported CPU utilization numbers are not correct.

The question of the networking throughput numbers is more complicated, because the data reported by the client and server machines is definitely related. Although it would have been preferable for me to have reported the throughput numbers from the client machines, I believe the throughput numbers gathered from the server are still a useful basis for comparison. At no point did I mix my data collection methods, reporting numbers from a client machine in one case and from the server in another, while testing for a single review or a group of related reviews in which I reused data.

This problem affects only reviews conducted by me. The other reviewers at TR were reporting ntttcp results correctly, and their results should still be considered valid.


Since my ntttcp CPU utilization results are not valid, I’ve updated the affected reviews with a link to this news post, and I’ve removed any ntttcp CPU utilization results.


Fortunately, ntttcp CPU utilization didn’t factor heavily into the overall conclusions for most reviews. In cases where networking performance was an issue, the focus was usually on throughput rather than CPU utilization. However, our recent Athlon 64 chipset comparison drew several conclusions based on ntttcp CPU utilization, so I’ve retested and updated the article contents and conclusions to reflect the corrected results. The new results haven’t changed our final recommendations. NVIDIA’s ActiveArmor networking does come out looking a little bit better, but it still doesn’t function quite like it should. NVIDIA says it has new drivers, still in the works, with a fix.


It kills me that I’ve been presenting invalid results, and I apologize for any confusion those results have caused. We try to get things right every time, and this is something I’d hoped would never happen. When it does happen, all we can do is be honest with you all about the problem and try to fix the affected articles the best we can. Although few of our testing applications present similar opportunities for mis-reporting results, we will try to be more diligent in the future in ensuring consistent and correct testing methods and results in all of our reviews.

Comments closed
    • My Johnson
    • 14 years ago

    Long Live Tom’s Hardware! Long Live the Doctor!

    • 5150
    • 14 years ago

    No a big thing, just further proof that TR is the best site around.

    • albundy
    • 14 years ago

    *[

    • Ricardo Dawkins
    • 14 years ago

    Good and great…but the site must move on…!

    • PerfectCr
    • 14 years ago

    BURN HIM AT THE STAKE!! 🙂

    J/K Nice catch and thanks for that. Good job! That’s integrity, you don’t see much of that anymore.

    • Usacomp2k3
    • 14 years ago

    Talk about TR fanboys. Except in this case, it is well deserved.

    Thanks guys.

    • Spotpuff
    • 14 years ago

    I WILL NEVER TRUST ANOTHER TECH REPORT REVIEW AGAIN!111!111

      • Krogoth
      • 14 years ago

      OMG t3h SARCASM!

    • BiffStroganoffsky
    • 14 years ago

    You’ve been bad. Have your gf put on her cat suit and give you twenty whacks!

    • BenWang
    • 14 years ago

    Another reason to read techreport! Unbiased honesty, unlike some other big sites out there!

    We’re all human and make mistakes, I hope the TR team can just look past this and move on to greater heights.

    Cheers

    • Xylker
    • 14 years ago

    Wow, I am sorry to read that, but impressed that you have published a front page correction. As users of the data we are impacted by the error, but not so much, in my opinion, that there is a serious question as to the usefulness of the whole review. It ethernet performance was the /[

    • zgirl
    • 14 years ago

    People make mistakes, it could have even been faulty testing software reporting numbers you thought were right.

    By letting us know and re-evaluating the reviews affected just re-affirms TR commitment to providing accurate and unbiased reviews.

    There is a reason I usually base my final decision off of what you guys report.

    • R2P2
    • 14 years ago

    Next week: Dissonance admits that his name is really spelt “Jeff”. 😉

      • jobodaho
      • 14 years ago

      Very nice.

      • eitje
      • 14 years ago

      excellent! ;D

    • Samlind
    • 14 years ago

    It just occured to me – has T– /[

    • JediNinjaWizards
    • 14 years ago

    Totally awesome…keep up the good work!

    • data8504
    • 14 years ago

    Si, si senors. Great job.

    • davidedney123
    • 14 years ago

    This is why I love TR – Confessing like a man to something that many review sites would consider acceptable anyway.

    We all drop a bollock from time to time, and it’s how we deal with it that makes the difference. Cracking job!

    Dave

    • ozymandias
    • 14 years ago

    Can you tell us which articles are changed (apart from the quoted ones in your post)? That would make it easier for us to find them (and perhaps reconsider certain hardware)

    • Samlind
    • 14 years ago

    I’d suggest a formal audit – that is nothing goes up until the review is approved by someone else who wasn’t unvolved with doing it and who asks enough questions to understand how the testing was done and critiques it. The auditor puts his reputation on the line too – and assumes responsibility for his audit and by association with the results of the test.

    It doesn’t have to be complicated – reading the test results and 15 minutes of conversation will do wonders. The auditor should come with a critical eye, asking questions intending to make the output as accurate as possible in a reasonable length of time. No result could be posted without audit.

    This can be a little humiliating to do, but I guarantee it will raise the level and accuracy of the results. After a while you get comfortable with it and begin to rely on it.

    Of course if you don’t do any of this, you are still the best review site on the web… so you must be doing something right.

      • FireGryphon
      • 14 years ago

      All of TR’s tests are ‘publically reproducible’, so in theory they can be audited by anyone who has the hardware.

      Going by your suggestion, who would be in charge of auditing: people from the community, other TR staff… who?

        • Samlind
        • 14 years ago

        y[

    • totoro
    • 14 years ago

    That’s awful, Geoff.
    I echo #13’s well put statement.
    TR rocks!

    • njpoz
    • 14 years ago

    While the error is unfortunate, I applaud you for not only discovering it, but openly admitting to the problem and reexamining all your data and conclusions based upon the error.

      • continuum
      • 14 years ago

      Hear hear.

      • Phattie
      • 14 years ago

      Errors, especially semi-trivial errors like this one, are unavoidable, if lamentable. I applaud your forthright confession of the error, and your rapid remediation of the issue in the most recent and applicable articles.

      Thank you for your hard work, and for your honesty.

    • ylyad
    • 14 years ago

    Integrity is the key: bravo for putting that on the homepage

    • Irascible
    • 14 years ago

    Sheesh Louise. Talk about a bunch of sycophants! TR commits an egregious and unforgivable blunder and all I read is: “TR rules because TR admitted they suck!”.

    s[

      • data8504
      • 14 years ago

      Forgive me – but sheesh louise is from Catdog?

    • just brew it!
    • 14 years ago

    Ouch.

    And I agree with the other posters — this is one of the reasons TR is my favorite tech web site: having the integrity to admit when you’ve screwed up. Everyone makes mistakes; but not everyone admits it.

    • hardwarenewbie
    • 14 years ago

    LOL, Nice marketing strategy on TR’s part.

    Anyways, keep up the good work. 🙂

    • indeego
    • 14 years ago

    How did you discover thisg{

      • FireGryphon
      • 14 years ago

      I’ve been thinking about this, too.

    • Tupuli
    • 14 years ago

    Karl Rove and I suggest you highlight Iraqi beheadings next time one of these mishaps occur. The Jackson trial, or the next missing-attractive-blonde-girl-of-the-week would also make good distractions.

    Anyway, kudos for displaying the correction on the front page.

    /Karl Rove is still a bastard though.

    • FireGryphon
    • 14 years ago

    This is the sort of down-to-earth honesty that keeps TR at the top of my list for hardware review sites. Few sites would have the balls to post this on the front page, but you fellas do, and that says a lot. Thanks for being real when so many other sites aren’t.

    Now that the error is corrected, it’s all cool.

    Keep up the good work guys! 🙂

    • The H
    • 14 years ago

    TR rules, period.

    The important thing is that you guys are honest and straightforward with us and that’s something I can’t say of a large number of ‘big’ websites these days.

    • TO11MTM
    • 14 years ago

    Kudos for admitting the issue and not sweeping it under the rug. There’s a number of reasons why this website is the only one I consistenlty go to for Hardware news, and this is one of em.

    • Palek
    • 14 years ago

    This honesty is one of the reasons why TR is my favourite hardware site.

    • eckslax
    • 14 years ago

    Thanks for being up-front about these things. Keep up the good work! 🙂

    • firerules16
    • 14 years ago

    This is why I like TR… they’re honest, and they are not afraid to admit when a mistake has been made. Although some may disagree, I for one applaud you and appreciate your honesty. Good job 😀

Pin It on Pinterest

Share This