We begin with a few synthetic memory performance tests because the memory subsystem's performance affects our understanding of what's going on in the rest of the benchmarks.
Sandra's memory performance tests are multithreaded and take full advantage of the dual-socket Opterons' NUMA memory subsystem. Yow.
The Xeon systems are at a distinct disadvantage due to their dual-channel DDR333 memory, but that's about as fast as it gets for Xeons currently. Some Xeon motherboards offer dual-channel DDR2 400, but that's not likely much faster than DDR memory at 333MHz. The Pentium XE 840 achieves much higher throughput thanks to its dual channels of low-latency DDR533 memory.
Linpack lets us get a quick visual on the size and speed of the L1 data and L2 caches on these processors, and there's nothing unexpected here.
The Opteron's copious memory bandwidth is tied to its low memory latencies, which come courtesy of its built-in memory controller. Because the memory controller runs at the speed of the CPU, the faster the processor, the lower the memory access latency. The multi-socket Opterons pay a small latency penalty here, but nothing major. The Xeons, on the other hand, have a rough time of it.