UL yanks some Huawei 3DMark scores after cheating revelations

On the heels of an in-depth analysis at Anandtech, UL Benchmarks (formerly Futuremark) has delisted Huawei's P20, P20 Pro, Nova 3, and Honor Play devices from its public 3DMark Sling Shot Extreme benchmark database. The delisting comes after Anandtech's discovery that Huawei's customized version of the Android operating system was cheating on common benchmarks by engaging a "Performance Mode" that artificially increased clock speeds and power consumption to deliver inflated scores.

Source: UL Benchmarks

UL independently confirmed that Huawei devices were engaging in this benchmark inflation by testing the P20 Pro, Nova 3, and Honor Play phones using its publicly-available 3DMark binary and an internal version that is not released to the public or OEMs. The firm discovered that the public binary of 3DMark turned in scores as much as 47% higher than those from its private application, despite there being no difference in code between the two. That difference in scoring confirms that the devices are detecting executable names to trigger "Performance Mode" rather than adapting to the test by way of dynamic voltage and frequency scaling or other heuristics that would presumably benefit all applications, not just benchmarks.

That detection and boosting behavior breaks the benchmarking rules UL sets for manufacturers. Among other conditions, the company expressly forbids this kind of app-name detection to deliver special priority to its benchmarks. Manufacturers can still implement this kind of special performance mode on their devices, but UL says those modes must be off by default and can only be enabled by device owners. Huawei has a path to redemption in the eyes of UL after this sanction, at least. According to a statement received by the firm, Huawei plans "to provide users with access to "Performance Mode" so they can use the maximum power of their device when they need to." 

This isn't the first time mobile device makers have come under fire for cheating on common benchmarks. The practice was condemned in a wide-ranging 2013 article by Anandtech that revealed Samsung, LG, Asus, and HTC were all putting their fingers on the scale in at least some benchmarks. We can only hope that mobile device companies still tempted to try gaming the system for temporary gain will be deterred by the eternal prospect of being named and shamed.

Comments closed
    • DoomGuy64
    • 1 year ago

    In other news, mad scientist Frankenstein resurrects dead duck to constantly quack 3 times.

    Nvidia honors this achievement by naming their next graphics architecture Frankenstein’s Monster. New architecture to start at $5000, but it achieves 10,000 bungholio marks. The chip’s new Frankenstein AI cores brings resurrecting dead ducks to the masses. Soon there will be a duck on every corner quacking 3 times. Nvidia fans never question whether or not this is sane, and massively pre-order the untested new card.

    Huawei cheating soon forgotten after dead duck pollution problem.

    • kitsura
    • 1 year ago

    Chinese companies cheating? Umpossible!

    • blastdoor
    • 1 year ago

    Independent tech journalism is important.

    • pirate_panda
    • 1 year ago

    In addition to the ATI Quake3 cheating others have mentioned, don’t forget about the saga of [url=https://techreport.com/review/5226/further-nvidia-optimizations-for-3dmark03/<]Nvidia's cheating on the 3dMark03 benchmark, aka 3dMurk.[/url<] The moral of the story is, if there's a benchmark that's easy to cheat, someone (or everyone) will do it.

    • ronch
    • 1 year ago

    Huawei. Oh well.

      • gerryg
      • 1 year ago

      Many alleged (actual?) shenanigans they are accused (discovered?) to be doing on behalf of their home country. Google news about them, you’ll find it.

        • ronch
        • 1 year ago

        Yep.

    • Captain Ned
    • 1 year ago

    Quack quack quack.

      • jihadjoe
      • 1 year ago

      I’m disappointed at myself for how long I took to get this reference.

    • ludi
    • 1 year ago

    A cheating scandal involving 3DMark?

    What YEAR is it?

      • Klimax
      • 1 year ago

      Everything old is new again…

      • BobbinThreadbare
      • 1 year ago

      history is a flat circle

        • gerryg
        • 1 year ago

        duh, all circles are flat. like the earth

    • Redocbew
    • 1 year ago

    Inconceivable!

      • BillyBuerger
      • 1 year ago

      I do not think that word means what you think it means.

        • Mr Bill
        • 1 year ago

        +3 [quote<][url=https://www.youtube.com/watch?v=OHVjs4aobqs<]I do not think that word means what you think it means.[/url<][/quote<]

        • LeoScott
        • 1 year ago

        It’s a movie reference.

          • RAGEPRO
          • 1 year ago

          And his post is the next line.

      • UberGerbil
      • 1 year ago

      I am [url=http://www.reactiongifs.com/r/mshckd.gif<]shocked![/url<]

    • chuckula
    • 1 year ago

    Aren’t you just so proud of how the kids have grown up!
    — Intel & Nvidia

      • tipoo
      • 1 year ago

      They even have differences in GPU render quality like ATI vs Nvidia of Olde

      [url<]https://www.anandtech.com/show/13285/huawei-gpu-turbo-investigation/5[/url<]

        • DavidC1
        • 1 year ago

        The review does say that’s not Huawei’s fault, but ARM’s Mali core.

        But yes, shady tactics. Most megacorporations are resorting to such things, so they are unique however.

          • tipoo
          • 1 year ago

          I know, all of it just seems like a retread of those early ‘Aughts desktop days though.

      • bhtooefr
      • 1 year ago

      If it quack3s like a duck…

      • jihadjoe
      • 1 year ago

      To be fair, everyone cheated in Quack3 lol

      [url<]http://www.extremetech.com/extreme/58762-tech-watch-ati-quake-3-driver-manipulation[/url<]

        • chuckula
        • 1 year ago

        Why yes: [url<]https://techreport.com/review/3089/how-ati-drivers-optimize-quake-iii[/url<] But if I put ATi in there I would have been downvoted mercilessly by all the people who pretend that only Intel & Nvidia are capable of doing anything wrong.

        • DoomGuy64
        • 1 year ago

        Ati cheats in one game, removes cheat in next driver, community never lets it down.
        “This issue will be corrected in the next RADEON-family driver release”
        Nvidia cheats in hundreds of games, 3dmark, and eventually turns to outright performance sabotage of the competition.
        [quote<]"What got AMD seriously aggravated was the fact that the first step of this code is done on all AMD hardware: “‘Amusingly’, it turns out that the first step is done for all hardware (even ours) whether AA is enabled or not! So it turns out that NVidia’s code for adding support for AA is running on our hardware all the time – even though we’re not being allowed to run the resolve code! So… They’ve not just tied a very ordinary implementation of AA to their h/w, but they’ve done it in a way which ends up slowing our hardware down (because we’re forced to write useless depth values to alpha most of the time…)!”[/quote<] [quote<]Yep, this flat, hard-edged slab is one of the most highly tessellated objects in the scene.[/quote<] [quote<]That's right. The tessellated water mesh remains in the scene, apparently ebbing and flowing beneath the land throughout, even though it's not visible. [/quote<] Community conveniently ignores this is going on. Blames AMD for poor performance in Nvidia sponsored titles. Only occasionally gets upset when Nvidia no longer competes with AMD and starts sabotaging it's own last gen hardware to sell new hardware. Like when the 960 beat the 780Ti in games like TW3. Yeah, Quack3 everyone. Like a 20 year old game hack on dx8 hardware, back when ATI was it's own company, couldn't do opengl, and was even patched out is somehow still relevant. I guess in some people's mind, that one time somehow equals out and justifies Nvidia's constant and still ongoing bad behavior.

Pin It on Pinterest

Share This