IE9 alleged to be cheating in JavaScript benchmark

We’ve all heard of graphics card makers optimizing their drivers for various benchmarks—some of you might recall the Quack story as one of the earlier examples. I think this might be the first time I’ve heard about the same thing happening in the world of web browsers, though. Believe it or not, Digitizor says a Mozilla engineer has found evidence that Internet Explorer 9 is “cheating” in the popular SunSpider JavaScript benchmark.

Where’s the incriminating evidence? Well, Mozilla’s Rob Sayre reportedly discovered that Internet Explorer 9 somehow completed one of the SunSpider sub-tests about ten times quicker than the competition. He went ahead and modified said sub-test, first adding a “true” to the code, then adding a “return,” both useless snippets that should have no impact—and indeed, they reportedly don’t in Google Chrome and Opera. In IE9, though, the modified code is said to execute 20 times slower.

Digitizor provides two explanations other than Microsoft cheating. The IE team could also have “unintentionally over-optmized” IE9’s JavaScript engine for SunSpider, it says, or this could simply be a bug. The site notes the first possibility is unlikely and the second would raise “a serious question about the robustness of the engine,” however.

The fact that a story like this would even crop up—let alone that IE9 might actually be up to some funny business in SunSpider—goes to show just how heated the browser wars have become. Personally, I’m still scratching my head over the whole thing. Are scores in a synthetic JavaScript test really that important?

Comments closed
    • HiTML5
    • 9 years ago

    I agree with Tamale’s points, and will add here my own informal developer’s anecdote regarding HTML5 and the present IE9 Beta:

    I have written a Javascript-intensive HTML5 application and released it initially as a Chrome Web App on Google’s new Chrome Web Store.

    This thing grinds through highly-optimized Javascript to compute every Scrabble word that can possibly be played to a given Scrabble board from a given rack of letters. This is a search problem, and so the amount of computational effort devoted to the interface is inconsequential compared to the internal search to find all words that can be made by placing rack letters among the pre-existing board tiles.

    This app, once loaded, runs entirely on the client, and so there is very little that will affect its performance other than raw Javascript execution speed.

    One of my benchmarks is to place three specific words on the Scrabble board and then type a specific set of seven letters, including one blank (a wild card that can represent any letter in Scrabble), onto the rack. This generates over 10,000 words, which are stored internally and displayed only when the user requests them, one at a time. Here are rough elapsed wall times for various browsers, from fastest to slowest on a 64-bit Vista PC:

    Safari 5.0.3 (7533.19.4): 5 seconds
    Chrome Beta (9.0.597.19 beta): 6 seconds
    Opera 11: 17 seconds
    IE9 Beta 9.0.7930.16406: 26 seconds
    Firefox 3.6.13: 45 seconds

    To give IE9 credit for improvement, IE9 is now much faster than IE8, for this particular test. I can’t reliably quantify the
    amount of improvement, but I think it is at least 50 percent.

    I must also list an important caveat: my Javascript was produced by the Google Web Toolkit compiler, and I believe that GWT is still in the process of optimizing its output for IE9, so it may be that there will be additional improvements for IE9 as a result of that.

    As a developer, I could not be more pleased that IE9 is finally addressing its Javascript performance issues. I have heard that they are still working on this, and have high hopes for even better results before the first stable version is released.

    • Bensam123
    • 9 years ago

    Yes, it’s not like most of the delay is determined by your ISP or anything, rather it’s about the 10s of MS it takes to draw your page.

    • pogsnet
    • 9 years ago
      • wibeasley
      • 9 years ago

      That Dead Code explanation makes sense to me.

    • indeego
    • 9 years ago

    And here I do everything I can to block most scripting from running, periodg{<.<}g.. Can we get benches on the cessation of scripting while still loading a site, or at least just allow primary toplevel scripts for sites to run javascriptg{

    • potatochobit
    • 9 years ago

    What kind of dumb question is that?

    if nvidia reported 200 FPS in games over AMDs 30FPS which would you use? most people dont have a clue what a ‘synthetic’ benchmark is
    so, of course fake benchmark tests are serious crimes.
    we are talking about millions of dollars at stake here.

    • l33t-g4m3r
    • 9 years ago

    l[

      • HisDivineShadow
      • 9 years ago

      I think people feel they’re at a point where the differences are not noticeable because of the fact that so many browsers perform poorly and websites are being built for the “worst”-common denominator. Chrome may run circles around IE9, but when all websites are built around IE8, it doesn’t matter so much. You’ll get acceptable performance a lot of the time from that crappy browser, so the higher performing one isn’t strictly necessary.

      Personally, I prefer Chrome. Not strictly because of its speed, but more for its stability. That is, I got tired of watching my IE8 tabs crash repeatedly for no apparent reason (ie., a common bug for IE8 back when I was using it). While trying Chrome (which was sandboxed like IE8), I liked that they reclaimed more of the vertical resolution than competing browsers by putting the tabs in the title bar I didn’t use for anything else and that when I restored it to less than fullscreen, part of the title bar came back for window manipulation. Smart things like that won me over.

      It doesn’t hurt that it’s routinely the fastest browser at just about anything a browser needs to do.

        • MadManOriginal
        • 9 years ago

        Chrome’s speed is powered by the life-intruding force of datamining for targeted advertising :p

      • raddude9
      • 9 years ago

      Very well put.

      These browser benchmarks are not about making your ordinary web sites run faster, its more about allowing web developers create entirely new web applications that have not been possible before.

    • Pettytheft
    • 9 years ago

    Who cares. I don’t have a webpage that loads any noticeably slower in any browser. The differences in benchmarks are barely noticeable.

      • ybf
      • 9 years ago

      True. Mozilla has much bigger problems with the features they’re trying out in the new beta of Firefox. They can make javascript run in 0 time, but if I can’t find the button I want that’s always been there I’m just going to give up on FF and use another browser.

        • axeman
        • 9 years ago

        if you futz with it you can get it back to something sane. number one, putting tabs on the bottom. Chrome has the tabs where the title bar would be, which saves space. FF4 still has a title bar, with the tabs below that. FAIL.com. Google’s layout makes sense since it saves vertical space. Mozilla’s layout is trying to copy Google’s while missing the point.

          • HisDivineShadow
          • 9 years ago

          Completely agree. If you’re going to copy Chrome’s placement of the bars, then copy them all the way and gain some benefit from it. Don’t go halfway. Go all the way. If you want to keep from alienating everyone, then make an option to switch back and forth for those people with higher resolution displays.

          • jstern
          • 9 years ago

          Back when Apple completely re did their browser, I hated it because it was pretty much exactly like chrome, and if I wanted to use something that looks like chrome, I would use chrome. Basically they killed their old look that I kind of liked, and it was like one less browser to have. But then I found out that the reason why they look so similar is because they use the same engine, webkit, the same same as firefox now. So that probably has more to do why they look similar, than Mozilla simply trying to copy it. In fact, they look significantly different to me. So far I prefer Chrome on Windows XP, really like the way it looks on it, and I prefer FireFox 4 on Windows 7. Note that I use FireFox 4 with the tabs on bottom, it looks much better to me like that, and which it was like that by default.

          (I’m no browser export, but I think it’s webkit, and before firefox was using the gecko engine.)

            • SoulSlave
            • 9 years ago

            l[

    • TaBoVilla
    • 9 years ago

    I can’t wait for the time when these are completly useless. I feel like all they are saying is “I have more megaheartz than you!.. Im betta!” which eventually became sort of useless too.

    browsers should be STABLE, COMPATIBLE, and somewhat speedy. If the first 2 are achieved, who cares if they doit in 3.44, 3.41, 3.3997.

    I feel this latest batch is IE9’s attempt at deviating eyes from their horrible recent track record and shrinking market share. Talk about microsoft botching market dominance in stuff by completely falling behind and failing to innovate and invest in their products: windows mobile, IE, Hotmail MSN etc.

    • SomeOtherGeek
    • 9 years ago

    Right, there is no, zero, zip point in the browser speeds. It is all about functionally.

    • Chrispy_
    • 9 years ago

    All of this PR and marketing over ‘browser speed’ is really starting to annoy me. It’s as pointless as locking your door when there’s a nine-foot hole in the wall next to it.

    *[http://www.clickhereifyouaregullible.com<]§ takes about 4 seconds to load on your computer. Of those 4 seconds, 3.9 seconds are *[

      • Corrado
      • 9 years ago

      I care more about how fast a browser loads from double click than I do how long it takes to load a page. Theres more latency in the net connection and server processing than there is on my end from a javascript perspective.

      • sschaem
      • 9 years ago

      Funny. When I moved from Ie6 to Chrome I experience a DRAMATIC performance boost.
      Page refreshed faster, but also scrolled better. Everything was smoother.

      You might not see this on a 3.2ghz I7 desktop processor with tripple channel overclock DDR3 and a 1gig 512bit GPU… but on an Atom those 1/10th of a second become ethernal!

      Note: I also use chrome on my desktop. Not for the speed, but for its UI.
      Google just knows how to build software just for me 🙂

        • axeman
        • 9 years ago

        This. The OP just proved in his case the bottleneck is the webserver or bandwidth of his connection. On slower devices, like netbooks, phones, etc, with limited processor power there are huge differences in how rendering engine performs. On the desktop side, there is significant differences in javascript performances which may not matter MOST of the time. If browser performance continues to increase, though, we will start to see AJAX used more and more heavily.

      • highlandr
      • 9 years ago

      The problem becomes when you get to really rich content sites, like gmail, facebook, etc. At work, gmail takes 3-5 seconds to load, and our connection is a dedicated 20Mbit link, so it is NOT the ISP. Of course, my work machine is an old P4, but the potential for improvement is still there.

      I’m not saying I care about JS speeds between browsers, I’m just saying that it will become more important as web content gets “richer.”

        • Chrispy_
        • 9 years ago

        That just proves my point =)

        An old and “slow” browser like IE6 shouldn’t take more than half a second to render a page of Gmail, even on an Atom. Your browser is waiting on *[

          • KilgoreTrout
          • 9 years ago

          I agree about the rich content sites. It is not just about rendering and loading a page, it is also about executing animations and handling dynamic content during the visit.

          Even IE8 is considerably slower than its competitors when it comes do doing a simple slide animation of an image. Javascript is becoming more and more common for doing these things, since jQuery and similar code libraries made it very simple do do even advanced animations, previously only attainable in Flash (for most web designers, at least).

          With CSS3 and HTML5 in conjunction with mobile platforms ahead the code execution efficiency of browsers will become even more important, I think.

          You can always argue that sites shouldn’t need animations and eye candy, but that’s like saying video games don’t need eye candy either. It may be true, but people in general want it anyway.

      • sonneillon
      • 9 years ago

      For normal browsing of the net I agree with you but overall I disagree with you as not everyone uses a browser just for surfing the net.

      Im a windows+linux administrator for a server hosting company and the systems we use at work we access through a browser. When I use IE8 I notice the pages at work tend to load a bit slower and when I click a button it feels slow to respond but FF+Opera load a few seconds faster and feel snappier. Considering how many different pages I go through at work during any given shift those few seconds per page add up over a course of a day.

      Now my personal beef is memory usage. I prefer FF but after a single day of work FF is usually using around 1GB of RAM where as IE and Opera all use a lot less.

      • eitje
      • 9 years ago

      q[

        • codedivine
        • 9 years ago

        Both valid points.

      • HisDivineShadow
      • 9 years ago

      I think the issue here is not the relevance of the benchmark, but the fact that the IE team is potentially cheating at it to regain mindshare. If true, it would speak to desperation on Microsoft’s part, which would be rather sad.

        • codedivine
        • 9 years ago

        I agree thats the main point of the article. My comment was tangential and not directly a reply to the article. But hey, this is the internet so I get to write tirades on any topic 😛

          • HisDivineShadow
          • 9 years ago

          Haha, true. Hence my reply! 😛

    • bittermann
    • 9 years ago

    “Are scores in a synthetic JavaScript test really that important?”

    Apparently to MS they are…

      • designerfx
      • 9 years ago

      only when they’re favorable to ms. otherwise it’s irrelevant to them.

    • sweatshopking
    • 9 years ago

    This is moot. I’m not convinced that means anything at all. activating code, that other browsers ignore, and then forcing IE to run it, makes for a stupid test.

      • kamikaziechameleon
      • 9 years ago

      Its deffinately a shortcut to reading all the code for sure, but then again this was just one guy messing around ya know. He only points out his findings and their obvious implications, there isn’t so much a conclusive tone to his discovery.

      • poulpy
      • 9 years ago

      Meh? That’s not what TFA says, all browsers run the same codepath.

      Because IE9 ran one test 10 times faster than anyone else they tried to change the signature of the said test by adding two harmless variations.
      The first contained a /[

    • Sargent Duck
    • 9 years ago

    To answer your question, nope.

    Reminds me of the Futuremark cheating that both ATI and Nvidia did to get higher scores. At the end of the day no one cared.

Pin It on Pinterest

Share This