A correction: We flubbed networking test results


— 12:00 AM on June 14, 2005

We recently discovered that the networking test results in all reviews where I used the ntttcp benchmarking tool—including my motherboard, small form factor, and chipset reviews—are invalid.

Here's what happened. When I first started using ntttcp, the client and server systems had comparable hardware, and the reported throughput and CPU utilization on each side was very similar. This led me to assume that ntttcp was reporting the same results on both ends, which is not the case. Ntttcp throughput is fairly consistent across the client and server machines, as one might expect, but CPU utilization is not. Therefore, my reported CPU utilization numbers are not correct.

The question of the networking throughput numbers is more complicated, because the data reported by the client and server machines is definitely related. Although it would have been preferable for me to have reported the throughput numbers from the client machines, I believe the throughput numbers gathered from the server are still a useful basis for comparison. At no point did I mix my data collection methods, reporting numbers from a client machine in one case and from the server in another, while testing for a single review or a group of related reviews in which I reused data.

This problem affects only reviews conducted by me. The other reviewers at TR were reporting ntttcp results correctly, and their results should still be considered valid.

Since my ntttcp CPU utilization results are not valid, I've updated the affected reviews with a link to this news post, and I've removed any ntttcp CPU utilization results.

Fortunately, ntttcp CPU utilization didn't factor heavily into the overall conclusions for most reviews. In cases where networking performance was an issue, the focus was usually on throughput rather than CPU utilization. However, our recent Athlon 64 chipset comparison drew several conclusions based on ntttcp CPU utilization, so I've retested and updated the article contents and conclusions to reflect the corrected results. The new results haven't changed our final recommendations. NVIDIA's ActiveArmor networking does come out looking a little bit better, but it still doesn't function quite like it should. NVIDIA says it has new drivers, still in the works, with a fix.

It kills me that I've been presenting invalid results, and I apologize for any confusion those results have caused. We try to get things right every time, and this is something I'd hoped would never happen. When it does happen, all we can do is be honest with you all about the problem and try to fix the affected articles the best we can. Although few of our testing applications present similar opportunities for mis-reporting results, we will try to be more diligent in the future in ensuring consistent and correct testing methods and results in all of our reviews.

Tip: You can use the A/Z keys to walk threads.
View options

This discussion is now closed.