Google Has Retired the Octane JavaScript Benchmark Suite

Google’s Octane JavaScript benchmark has been put out to pasture due to developers focusing on optimizing for higher benchmark scores despite detrimental effects to actual real-world performance. The Octane web browser benchmark is a popular tool introduced in 2012 that tests software capabilities via cut-down versions of semi-realistic workloads but the change in JavaScript coding styles as well as addition of new libraries and frameworks ever since has affected the benchmark considerably. Just like SunSpider before it, the Octane JavaScript benchmark is not safe from being ‘gamed’ by developers. This is a natural cycle in the benchmarking scene, and that is why Safari developers created JetStream to replace SunSpider. Now it has come to a point where even bugs in the benchmarking program within the Octane benchmarking suite is being taken advantaged of such as the Box2DWeb benchmark to get better results. Using `<` and `>=` operators gave a ~15% performance boost in performance on that specific benchmark for example and it had no effect on real-world scenarios.

Octane was developed with the goal of pushing the JavaScript ecosystem and to make large gains in computationally-expensive JavaScript, which Google believes has now plateaued and the time to retire is right.

“The next frontier, however, is improving the performance of real web pages, modern libraries, frameworks, ES2015+ language features, new patterns of state management, immutable object allocation, and module bundling. Since V8 runs in many environments, including server side in Node.js, we are also investing time in understanding real-world Node applications and measuring server-side JavaScript performance through workloads such as AcmeAir.” from a statement posted on the V8 team’s blog, developers of Octane.