Google retires outdated Octane JavaScript benchmark

Benchmarks are a crucial part of what we do here at TR. As fun as it can be to compare performance figures for competing widgets, any given benchmark has little value if it doesn't correlate to performance in a real workload. Google's V8 JavaScript engine team has taken a long look at its Octane benchmark and concluded that the test is no longer indicative of real-world JavaScript performance. The team has therefore chosen to retire the benchmark and stop recommending its use as a JavaScript performance test.

The Octane benchmark was born in 2012 in the wake of what Octane's developers describe as the waning usefulness of the SunSpider JavaScript benchmark. Octane consists of 17 individual tests originally intended to measure performance in a variety of workloads, including a kernel simulation test and a self-compiling Microsoft Typescript test.

Ever since JavaScript performance started playing a part in the web, virtual machine developers optimized their software specifically for the tasks present in Octane. That optimization was partially a natural evolution of JavaScript as the basis for complex web applications, or for server-side implementations of the language like Node.js. However, the Octane devs believe that many of those optimizations led to higher benchmark scores at the expense of performance in more general workloads. In some cases, even bugs in the benchmark were targets for VM optimization. Google's blog post has the nitty-gritty details and is worth checking out.

The developers go on to note that the loss of relevance thanks to over-optimization is not unique to Octane and specifically name Kraken and JetStream as fellow optimization targets. However, the team has not given up on JavaScript VM benchmarking and says it will continue to develop new tests that are more indicative of real-world performance.

Comments closed
    • xDoritox
    • 3 years ago

    Been using Geekbench 4 and Octane 2 as my main CPU benchmarks for over a year now. Damn!
    I wonder if AMD had anything to do with this.

    • TheSeekingOne
    • 3 years ago

    Coincidently, Anandtech’s evaluation of RyZen’s IPC is entirely based on this benchmark, out of all the single-threaded tests that anandtech used in their latest CPU review. If you look at the other web tests, Ryzen CPUs exhibit roughly the same single threaded performance as the Kaby Lake ones.

    [url<]http://www.anandtech.com/show/11244/the-amd-ryzen-5-1600x-vs-core-i5-review-twelve-threads-vs-four/7[/url<]

    • chuckula
    • 3 years ago

    It had a good ride.
    But Octane was out of gas.

      • JosiahBradley
      • 3 years ago

      TechReport should just hire you to place the puns even deeper inside the articles. Chief Punster.

        • EzioAs
        • 3 years ago

        Don’t you mean ‘fire’? 🙂
        I thought chuckula was already a paid commenter.

          • jihadjoe
          • 3 years ago

          The awesome thing here is chuckula is obviously very smart and knowledgeable. But instead of using that knowledge to be productive, he uses it to troll the comments section and make bad puns! Bravo!

          It’s like those old 3Dfx commercials about using the Voodoo3 for compute.

            • ColeLT1
            • 3 years ago

            [url<]https://www.youtube.com/watch?v=ooLO2xeyJZA[/url<]

        • Neutronbeam
        • 3 years ago

        It’s a punning plan, but why punish the rest of us?

        • Anovoca
        • 3 years ago

        Why? That is what drfish is here for.

          • drfish
          • 2 years ago

          Aww…

        • Anovoca
        • 3 years ago

        Also that wasn’t a pun, just a play on words.

          • JosiahBradley
          • 3 years ago

          Not sure if trolling??? Pun: noun – play on words….

            • Anovoca
            • 3 years ago

            Pun is a phonetic play on words. The phrase “out of gas” is an idiom; and Chuck didn’t even use it as an idiom, he used the phrase literally, which made it clever and therefore humorous. But the fact that it was a humorous play on words does not automatically make it a pun.

Pin It on Pinterest

Share This