For decades, Consumer Reports (CR) has acted as a fairly respected consumer watchdog, testing the claims of countless products and offering up assessments to help the public choose in a world filled with dubious advertising. When the magazine said that Apple's new MacBook Pro was lacking in the battery life department, there was general surprise. After some investigation, though, it appears that the testing conditions were unusual. CR's methodology may have been flawed, though it exposed a Safari bug in the process.
In a statement to TechCrunch, Apple said that CR used a hidden setting in Safari that disabled caching completely. CR explains in an update posted yesterday that the test it ran on the new MacBook Pro is the same one it uses for every other laptop. The lab disables caching in the machine's primary browser and then has it downloading a set of sites over and over again. These conditions were intended to more closely simulate the user going to new webpages over and over, rather than reading the same 10 sites from the cache over and over again.
Apple contends, however, that this method doesn't reflect real-world usage. It also states that "the use of this setting also triggered an obscure and intermittent bug reloading icons which created the inconsistent results in their lab." Apple says it asked CR to run tests with that setting enabled and the Macbook Pro consistently delivers the expected battery life under those conditions. The company also notes that the Safari icon downloading bug has been fixed. CR is now fully re-testing the MacBook Pro and will publish its findings.
The idea of "real-world usage" is a hard one to nail down accurately. Users intermittently consume media, browse, open and close the laptop, and plug it in for charging. While CR didn't want to test re-downloading the same site over and over, many users' browsing habits often mirror that. While I visit plenty of sites throughout a day's work and in my leisure time, the majority of my browsing is done on a tight core of the same sites which, as Apple notes, would take advantage of a browser's caching ability.
As a result of CR and Apple's cooperation, though, readers have a better understanding of what caused the disparity between the official estimate and tested results. Now, only time will tell if Apple's fix to Safari will sort out the battery life issues that end-users are reportedly experiencing.