SPEC updates its CPU benchmark for the first time in 11 years

The non-profit Standard Performance Evaluation Corporation (SPEC) has released an update of its popular CPU benchmarking tool, fittingly dubbed SPEC CPU2017. The new version contains 43 tests arranged into four suites that test CPUs' integer and floating-point performance under a multitude of workloads. Optionally, CPU2017 can measure power consumption during testing and take advantage of the OpenMP multiprocessing API. Updates to the tool are particularly noteworthy because they don't happen often; the previous version was released all the way back in 2006.

Performance in SPEC's benchmarks are an industry-standard method of quantifying CPU performance across a wide range of tasks. The tool is based on code elements taken from real-world applications, rather than synthetic code or kernels. The benchmark is distributed as source code, which must be compiled for the host system by the tester. The test can run on ARM, Power, SPARC, and x86 architectures, and requires 16 GB of system memory and 250 GB of storage space. Supported plaftorms for CPU2017 are AIX, Linux, macOS, Solaris, and Windows.

The benchmark has distinct work latency and throughput tests. The SPECspeed 2017 Integer test and SPECspeed 2017 Floating Point suites are designed to run one copy at a time, while the SPECrate 2017 Integer and SPECrate 2017 Floating Point tests run multiple copies at once. The "speed" tests measure the amount of time needed to complete the tests within the suite and awards higher scores for shorter completion times. The rate tests measure throughput, or how fast a system can complete a given set of tasks.

The new tests eliminate the libquantum test component, which was believed to give Intel processors an advantage over the competition. The Intel C++ Compiler had "cracked" libquantum, which may have inflated overall benchmark scores by 5-10% on Intel CPUs.

The SPEC CPU2017 benchmark is available now on the group's website. The tool costs $1000 for new customers, $500 for an upgrade, and $250 for educational institutions.

Comments closed
    • Klimax
    • 4 years ago

    Mandatory link to RWT discussion:
    [url<]http://www.realworldtech.com/forum/?threadid=169334&curpostid=169334[/url<]

    • DancinJack
    • 4 years ago

    I wonder if they included any AVX2/AVX512 here at all. I didn’t see anything in a cursory look of the website. I’m kind of doubting it.

      • willmore
      • 4 years ago

      That would be a task for the compiler. If it can make use of those instructions then it’s free to do so.

      If you are asking if the benchmarks come with assembly optimizations using those specific instruction sets, then I would highly doubt it. That would make the job of the compiler much more difficult.

      I’ve not yet looked at it, but it may rely in part on existing system libraries and those have been used as a vector for inserting heavily tuned code into benchmarks before.

      • chuckula
      • 4 years ago

      OpenMP directives + a good compiler can, at least some of the time, produce AVX2/AVX-512 code.

      OpemMP is also often used to provide directives that allow the compiler to multi-thread certain portions of code without having to actually manually code the multi-threading using pthreads or the like.

      Of course, all of this requires testing and YMMV based on the compiler & hardware.

      • John Henning
      • 4 years ago

      SPEC CPU2017 is distributed as a source code benchmark, using portable source code wherever practical. There are #ifdefs as needed, but no assembly code.

      That being said . . . as noted by another commenter, it is up to the compiler. Check out the posted results; for example this result
      [url<]https://www.spec.org/cpu2017/results/res2017q2/cpu2017-20161026-00028.html[/url<] mentions something called '-xCORE-AVX2' (and you can click on the flag for a brief description) -john henning (one of the developers for SPEC CPU2017)

    • chuckula
    • 4 years ago

    It’s gonna be EPYC!

      • maxxcool
      • 4 years ago

      Well *I* thought it was funny 😛

        • chuckula
        • 4 years ago

        EPYC is officially launching today. You’d think some people would appreciate a little hype.

    • Meadows
    • 4 years ago

    Does this mean we can expect TR to add it to the suite?

      • Sargent Duck
      • 4 years ago

      One would think so…

      • Beahmont
      • 4 years ago

      Sure. Just as soon as you give Jeff the $1000 to buy the thing.

        • chuckula
        • 4 years ago

        Nah, only $250 is required for TR Tech to get the academic license.

          • Beahmont
          • 4 years ago

          Honest Question: Are you sure that they will let their software be used by and for commercial entities? Because TR is, to the best of my knowledge, a commercial entity. I know quite a few software companies specifically forbid review sites from using educational licences.

            • chuckula
            • 4 years ago

            No, they’d make TR pay full price.

            But I was able to weasel my academic credentials into a few discounts here and there that were probably not 100% approved once upon a time. Ah.. the bad old days.

Pin It on Pinterest

Share This