The non-profit Standard Performance Evaluation Corporation (SPEC) has released an update of its popular CPU benchmarking tool, fittingly dubbed SPEC CPU2017. The new version contains 43 tests arranged into four suites that test CPUs' integer and floating-point performance under a multitude of workloads. Optionally, CPU2017 can measure power consumption during testing and take advantage of the OpenMP multiprocessing API. Updates to the tool are particularly noteworthy because they don't happen often; the previous version was released all the way back in 2006.
Performance in SPEC's benchmarks are an industry-standard method of quantifying CPU performance across a wide range of tasks. The tool is based on code elements taken from real-world applications, rather than synthetic code or kernels. The benchmark is distributed as source code, which must be compiled for the host system by the tester. The test can run on ARM, Power, SPARC, and x86 architectures, and requires 16 GB of system memory and 250 GB of storage space. Supported plaftorms for CPU2017 are AIX, Linux, macOS, Solaris, and Windows.
The benchmark has distinct work latency and throughput tests. The SPECspeed 2017 Integer test and SPECspeed 2017 Floating Point suites are designed to run one copy at a time, while the SPECrate 2017 Integer and SPECrate 2017 Floating Point tests run multiple copies at once. The "speed" tests measure the amount of time needed to complete the tests within the suite and awards higher scores for shorter completion times. The rate tests measure throughput, or how fast a system can complete a given set of tasks.
The new tests eliminate the libquantum test component, which was believed to give Intel processors an advantage over the competition. The Intel C++ Compiler had "cracked" libquantum, which may have inflated overall benchmark scores by 5-10% on Intel CPUs.
The SPEC CPU2017 benchmark is available now on the group's website. The tool costs $1000 for new customers, $500 for an upgrade, and $250 for educational institutions.