The non-profit Standard Performance Evaluation Corporation (SPEC) has released an update of its popular CPU benchmarking tool, fittingly dubbed SPEC CPU2017. The new version contains 43 tests arranged into four suites that test CPUs' integer and floating-point performance under a multitude of workloads. Optionally, CPU2017 can measure power consumption during testing and take advantage of the OpenMP multiprocessing API. Updates to the tool are particularly noteworthy because they don't happen often; the previous version was released all the way back in 2006.
Performance in SPEC's benchmarks are an industry-standard method of quantifying CPU performance across a wide range of tasks. The tool is based on code elements taken from real-world applications, rather than synthetic code or kernels. The benchmark is distributed as source code, which must be compiled for the host system by the tester. The test can run on ARM, Power, SPARC, and x86 architectures, and requires 16 GB of system memory and 250 GB of storage space. Supported plaftorms for CPU2017 are AIX, Linux, macOS, Solaris, and Windows.
The benchmark has distinct work latency and throughput tests. The SPECspeed 2017 Integer test and SPECspeed 2017 Floating Point suites are designed to run one copy at a time, while the SPECrate 2017 Integer and SPECrate 2017 Floating Point tests run multiple copies at once. The "speed" tests measure the amount of time needed to complete the tests within the suite and awards higher scores for shorter completion times. The rate tests measure throughput, or how fast a system can complete a given set of tasks.
The new tests eliminate the libquantum test component, which was believed to give Intel processors an advantage over the competition. The Intel C++ Compiler had "cracked" libquantum, which may have inflated overall benchmark scores by 5-10% on Intel CPUs.
The SPEC CPU2017 benchmark is available now on the group's website. The tool costs $1000 for new customers, $500 for an upgrade, and $250 for educational institutions.
Mandatory link to RWT discussion:
[url<]http://www.realworldtech.com/forum/?threadid=169334&curpostid=169334[/url<]
I wonder if they included any AVX2/AVX512 here at all. I didn’t see anything in a cursory look of the website. I’m kind of doubting it.
That would be a task for the compiler. If it can make use of those instructions then it’s free to do so.
If you are asking if the benchmarks come with assembly optimizations using those specific instruction sets, then I would highly doubt it. That would make the job of the compiler much more difficult.
I’ve not yet looked at it, but it may rely in part on existing system libraries and those have been used as a vector for inserting heavily tuned code into benchmarks before.
OpenMP directives + a good compiler can, at least some of the time, produce AVX2/AVX-512 code.
OpemMP is also often used to provide directives that allow the compiler to multi-thread certain portions of code without having to actually manually code the multi-threading using pthreads or the like.
Of course, all of this requires testing and YMMV based on the compiler & hardware.
SPEC CPU2017 is distributed as a source code benchmark, using portable source code wherever practical. There are #ifdefs as needed, but no assembly code.
That being said . . . as noted by another commenter, it is up to the compiler. Check out the posted results; for example this result
[url<]https://www.spec.org/cpu2017/results/res2017q2/cpu2017-20161026-00028.html[/url<] mentions something called '-xCORE-AVX2' (and you can click on the flag for a brief description) -john henning (one of the developers for SPEC CPU2017)
It’s gonna be EPYC!
Well *I* thought it was funny 😛
EPYC is officially launching today. You’d think some people would appreciate a little hype.
Does this mean we can expect TR to add it to the suite?
One would think so…
Sure. Just as soon as you give Jeff the $1000 to buy the thing.
Nah, only $250 is required for TR Tech to get the academic license.
Honest Question: Are you sure that they will let their software be used by and for commercial entities? Because TR is, to the best of my knowledge, a commercial entity. I know quite a few software companies specifically forbid review sites from using educational licences.
No, they’d make TR pay full price.
But I was able to weasel my academic credentials into a few discounts here and there that were probably not 100% approved once upon a time. Ah.. the bad old days.