Google retires outdated Octane JavaScript benchmark

— 5:00 PM on April 13, 2017

Benchmarks are a crucial part of what we do here at TR. As fun as it can be to compare performance figures for competing widgets, any given benchmark has little value if it doesn't correlate to performance in a real workload. Google's V8 JavaScript engine team has taken a long look at its Octane benchmark and concluded that the test is no longer indicative of real-world JavaScript performance. The team has therefore chosen to retire the benchmark and stop recommending its use as a JavaScript performance test.

The Octane benchmark was born in 2012 in the wake of what Octane's developers describe as the waning usefulness of the SunSpider JavaScript benchmark. Octane consists of 17 individual tests originally intended to measure performance in a variety of workloads, including a kernel simulation test and a self-compiling Microsoft Typescript test.

Ever since JavaScript performance started playing a part in the web, virtual machine developers optimized their software specifically for the tasks present in Octane. That optimization was partially a natural evolution of JavaScript as the basis for complex web applications, or for server-side implementations of the language like Node.js. However, the Octane devs believe that many of those optimizations led to higher benchmark scores at the expense of performance in more general workloads. In some cases, even bugs in the benchmark were targets for VM optimization. Google's blog post has the nitty-gritty details and is worth checking out.

The developers go on to note that the loss of relevance thanks to over-optimization is not unique to Octane and specifically name Kraken and JetStream as fellow optimization targets. However, the team has not given up on JavaScript VM benchmarking and says it will continue to develop new tests that are more indicative of real-world performance.

Tip: You can use the A/Z keys to walk threads.
View options

This discussion is now closed.