UL yanks some Huawei 3DMark scores after cheating revelations


On the heels of an in-depth analysis at Anandtech, UL Benchmarks (formerly Futuremark) has delisted Huawei's P20, P20 Pro, Nova 3, and Honor Play devices from its public 3DMark Sling Shot Extreme benchmark database. The delisting comes after Anandtech's discovery that Huawei's customized version of the Android operating system was cheating on common benchmarks by engaging a "Performance Mode" that artificially increased clock speeds and power consumption to deliver inflated scores.


Source: UL Benchmarks

UL independently confirmed that Huawei devices were engaging in this benchmark inflation by testing the P20 Pro, Nova 3, and Honor Play phones using its publicly-available 3DMark binary and an internal version that is not released to the public or OEMs. The firm discovered that the public binary of 3DMark turned in scores as much as 47% higher than those from its private application, despite there being no difference in code between the two. That difference in scoring confirms that the devices are detecting executable names to trigger "Performance Mode" rather than adapting to the test by way of dynamic voltage and frequency scaling or other heuristics that would presumably benefit all applications, not just benchmarks.

That detection and boosting behavior breaks the benchmarking rules UL sets for manufacturers. Among other conditions, the company expressly forbids this kind of app-name detection to deliver special priority to its benchmarks. Manufacturers can still implement this kind of special performance mode on their devices, but UL says those modes must be off by default and can only be enabled by device owners. Huawei has a path to redemption in the eyes of UL after this sanction, at least. According to a statement received by the firm, Huawei plans "to provide users with access to "Performance Mode" so they can use the maximum power of their device when they need to." 

This isn't the first time mobile device makers have come under fire for cheating on common benchmarks. The practice was condemned in a wide-ranging 2013 article by Anandtech that revealed Samsung, LG, Asus, and HTC were all putting their fingers on the scale in at least some benchmarks. We can only hope that mobile device companies still tempted to try gaming the system for temporary gain will be deterred by the eternal prospect of being named and shamed.

Tip: You can use the A/Z keys to walk threads.
View options