Consumer Reports approves MacBook Pros after retesting

Consumer Reports (CR) has generally been a fan of the build quality and performance of Apple's computers. However, the consumer watchdog recently cautioned its readers against purchasing Apple's newest MacBook Pro due to discrepancies between Cupertino's published battery life specifications and the figures CR derived in its lab.

Apple and CR were eventually able to pin down and correct a bug related specifically to CR's testing methods, and the nonprofit agreed to rerun its tests. The results of the second round of tests are right in line with Apple's claims. One of CR's test machines even turned a battery life of over 18 hours with the corrections. With the new results in hand, CR now recommends the latest MacBook Pros to consumers.

For the unfamiliar, the issue arose from the fact that Consumer Reports disables browser caching on all machines it tests for battery life. That scenario inadvertently triggered a bug in Safari, which Apple has now fixed. Apple still doesn't believe that CR's testing methods are representative of real-world use, but it seems the sparring partners can at least expect mutually agreeable results from the testing firm's methods for now.

Having had to issue a mea culpa regarding test results ourselves last year, we'd hope this experience leads CR to exercise more caution about presenting wildly variable test numbers as conclusive evidence. Though the initial controversy may have led to a large upswell of interest, it ultimately seems harmful to CR's reputation for rigorous and data-driven testing. Perhaps this fiasco will result in a more conservative approach for the consumer watchdog in the future.

Comments closed
    • WaltC
    • 3 years ago

    Consumer Reports is an editorial magazine. It is not and never was “scientific.” It is purely opinion. And as we see constantly–not upsetting the establishment’s status quo seems very important to CR’s bottom line…;) Isn’t it obvious?

    A: Apple we hate your junk–it stinks.

    Q: So, OK, suppose we triple our ad budget for CR this year?

    A: Wow, you know thanks for pointing out what we missed! (Slaps forehead!) Sorry, yeah, sure we can recommend your crap *now*…! Oh, yeah! (And we promise in a solemn pledge to you that this won’t happen *every year*! Oh, no. Every 2-3 years on various things should be enough to help keep us in high cotton here at CR.) What an Industry Leader Apple is!

    • VincentHanna
    • 3 years ago

    TR criticizes CR for lacking journalistic integrity while making statements that imply something that is patently untrue:
    [quote<]Apple and CR were eventually able to pin down and correct a bug related specifically to CR's testing methods[/quote<] No. Apple and CR were eventually able to pin down and correct a bug related specifically to Safari browser's handling of content when a cashed version of an object is not available. Not related in any way to CR's testing method, other than the fact that CR, and rightly so, disables the cashe for their web browser testing.

    • ronch
    • 3 years ago

    You’re testing it wrong.

    • green
    • 3 years ago

    i think part of the problem with this whole thing was apple’s statement about cr’s testing method. apple noted that cr’s test method was not reflective of real world usage. so the general public, who don’t really follow reviews / methodologies, would think cr was stacking the deck against apple. and so the backlash from apple lovers, which in turn raised the ire of apple haters.
    of course, cr was simply running the same test it had done for laptops it had tested in the past. while people and sites cycling the news may have mentioned that cr had [i<]always[/i<] disabled browser cache (where it's mention was generally a 1-liner that may have also noted that was how cr had always tested thing), that was certainly not the takeaway of what was written and so the overall impression was that cr's testing had always been flawed and they aren't a reputable source of recommendations. while apple didn't outright say people shouldn't trust cr as a source of information, apple's pr department did a pretty good job of getting everyone to say it for them in the end, the only winner in this case was apple realistically, if apple wanted cr to test the laptop reflective of [i<]real world[/i<] usage then cr would be testing chrome instead of safari. that and they'd stream hours of random video content from youtube (faster achieved by downloading the same video over and over. with [i<]cache disabled[/i<]). that and streaming content from netflix and spotify, while randomly posting crap on facebook, loading random links from facebook, and visiting random shopping websites [quote<]Apple’s updated software is available through Apple’s Beta Software Program now, and will be rolled out in a full Software Update to all users in several weeks. According to Apple, the new software fixes a bug in Safari that caused the poor battery-life results in Consumer Reports testing.[/quote<] i'm a little suprised cr didn't quality the laptop "will be" recommended in serveral weeks given the update cr has on their laptop's apparently aren't available to the general public just yet (i'm quite certain the majority of the general public aren't in the beta software program). and so what people crurrently may not be reflective of what consumers will experience (at least for serveral weeks) as a little tongue-in-cheek comment, was the update to make safari's disable cache option do nothing?

      • RdVi
      • 3 years ago

      I up-voted you because I agree with what you wrote, but the way you wrote it made me think of you as a Bizarro world version of SSK. The thought was unsettling.

      • VincentHanna
      • 3 years ago

      Also, NOT disabling browser cashe would be a completely dishonest test, because at that point they are pulling from ram or harddisk and not the remote server, which is the entire point.

    • adisor19
    • 3 years ago

    Good. I’m glad TR has the same conclusion as I had : That CR jumped the gun with those initial bad results.

    TR should have added as well that Apple’s response was also a bit lame by trying to point the blame on CR’s tests if you read their statements in between the lines.

    It’s good that the bug has been fixed and hopefully both entities have learned something from this experience.

    Adi

    • willyolio
    • 3 years ago

    On the other hand, you can look to the VW Dieselgate thing. If someone gets wildly different results because their testing methodology is slightly different than others, it might be because the manufacturer knows what standardized testing conditions are and optimizes/cheats for those particular scenarios.

    The one guy who’s doing different or more real-world-like testing gets a completely different result. Certainly doesn’t mean it should be discounted.

    • Metonymy
    • 3 years ago

    Jeff, you use the phrase “conclusive evidence” but I’m wondering about that: I don’t have the original CR article handy, but I read it, and I didn’t think at the time that CR was hiding anything, but was very forthright about what they were experiencing. They have a history of changing their recommendations about products when the product has been altered (both for better or worse) and I think most of their readers expect that.

    Had they misrepresented their results, they would clearly be in the wrong. But they didn’t do that and if I’m correct about them being upfront about the basis of their recommendations, I fail to see this in as harsh a light as do you. And weren’t they pretty quick to present their positive results after the Safari change?

    To call it a fiasco seems somewhat extreme.

      • Rectal Prolapse
      • 3 years ago

      Agreed. CR did nothing wrong here, were upfront about it, and followed up with a retest. If they didn’t publish anything there is no guarantee Apple would do anything about it either, and could even have dragged their feet and no one would be the wiser.

      It’s a win-win for everyone – the consumer, CR, and even Apple for fixing the bug to save face. Non-zero sum! 🙂

        • adisor19
        • 3 years ago

        Nope. It’s a lose-lose for everyone. CR lost some credibility and Apple lost some reputation for their product.

        Adi

      • derFunkenstein
      • 3 years ago

      [url=http://www.consumerreports.org/laptops/macbook-pros-fail-to-earn-consumer-reports-recommendation/<]Here's the original[/url<] and you're correct. Consumer Reports was WAY up-front about it.

      • Jeff Kampman
      • 3 years ago

      We’re not accusing CR of “hiding” anything.

      What I take issue with is that the numbers were wacky as hell, CR admitted they were wacky as hell, but instead of saying “we’re not really in a good position to issue judgment on the product based on these wacky-as-hell numbers,” CR went ahead and broke with their standard method (averaging the numbers) and just cherry-picked one from the basket of results that seemed right to them for whatever reason before throwing them back in the pool with numbers that were arrived at using another method (averaging). That’s really questionable, in my mind.

      If you admit that you have suspect data, that should be a stop sign, not a license to go ahead and do whatever based on some editorial fiat that flies in the face of your established testing methods just so you can publish an article.

        • blastdoor
        • 3 years ago

        Well, those are good points too…

        • MrDweezil
        • 3 years ago

        [quote<]What I take issue with is that the numbers were wacky as hell, CR admitted they were wacky as hell, but instead of saying "we're not really in a good position to issue judgment on the product based on these wacky-as-hell numbers,"...[/quote<] This is a ridiculous statement. If they got weird numbers and verified their testing methods hadn't changed, they have every right to report that they got weird numbers and aren't comfortable recommending a product that is turning in weird numbers. None of this is set in stone. Once the product was updated they re-tested and changed their recommendation to reflect the product's current state. Not only were they not wrong, it seems like they handled it in the best possible way.

          • Jeff Kampman
          • 3 years ago

          Obviously CR has every right to report its observations.

          What I specifically take issue with (as someone who has to synthesize benchmark results as a large part of his job) is the decision to, as CR put it, “[report] the lowest battery life results, and [use] those numbers in calculating our final scores. It’s the only time frame we can confidently advise a consumer to rely on if he or she is planning use the product without access to an electrical outlet.”

          At that point, you have one set of notebooks in your test results (those whose battery life figures were arrived at by averaging) and another (those that you felt should have arbitrary battery life figure X). The data just isn’t cross-comparable.

            • MrDweezil
            • 3 years ago

            Imagine you were a website that has always measured graphics cards in reviews using fps numbers, and then one day you’re doing a review and see a card’s fps measures fine, but it looks and feels off. You’re both curious and through so you do some research and deep dives and discover the wonderful world of frame times. You measure these on this new card and find they’re all over the place.

            It may be breaking with your historical test result data, but I don’t think it would be unethical to report on it and conclude that while the new card measures fine on average, you aren’t currently comfortable recommending it due to issues not captured by your traditional measurement.

            • Jeff Kampman
            • 3 years ago

            You’re outlining what CR should have done, not what it did.

            • MrDweezil
            • 3 years ago

            Its like, exactly what they did. They recognized that the usefulness of their normal metric hinged on certain things being true (relatively consistent test runs/frame times). When they encountered a product where those things were not true, they explained their findings and didn’t recommend the product despite that normal metric being in an acceptable range.

        • sreams
        • 3 years ago

        Yes, but the “wacky as hell” numbers were not a result of a problem with CR’s testing methodology. They were a result of Apple’s bug and are -entirely- Apple’s fault. We don’t need CR to put on Nerf gloves and baby Apple over their own problems.

          • JalaleenRumi
          • 3 years ago

          This exactly. I don’t know why people keep arguing about it and making it sound like it’s CR’s fault while Apple has accepted their mistake and fixed it.

          If nothing was wrong with the device and it was only CR’s fault and their testing methodology then why did Apple respond quickly and provided the fix?

          Adisor19 has been fanboing over Apple for a while now and trying to accuse CR of some ‘wacky’ business or something but to hear the same coming from Jeff? That is disappointing.

          The reason we come to this site is because we trust the writers here and we trust that when you test something, no matter who it belongs to, be it Apple or Intel or NVidia or AMD or whoever, we believe that you will provide us the results that you ‘got’ instead of what everyone else ‘expected’.

        • adisor19
        • 3 years ago

        It’s ok Jeff, I know you’re trying to be objective here but you gotta remember who your core audience is on this site 😉 So yeah, expect everyone to boo your conclusion lol

        Adi

    • Chrispy_
    • 3 years ago

    A bug in Safari is not CR’s fault at all. The credibility of the test is that they use the same settings for all laptops with no exceptions. If Apple’s product failed where [b<]EVERYTHING ELSE RUNNING THE SAME TEST[/b<] did not, that is Apple's fault and only Apple's fault. This is Apple's modus operandi; "The problem is not us, we are perfect. The problem is you, so shut up and keep giving us your money"

      • jihadjoe
      • 3 years ago

      You’re -x-ing it wrong!

    • derFunkenstein
    • 3 years ago

    I don’t necessarily agree with the conclusion here. The original CR article notes that they went to Apple before publishing and Apple had nothing to say because they didn’t “understand” (CR’s word) the problem. That’s pretty conservative, but at the same time, CR doesn’t answer to Apple. They answer to their readers.

    Unlike the TR RX 470 review that had a setup problem (beta UEFI for overclocking locked CPUs that never went final), CR actually found a bug in production software. Would Apple have fixed it if the results weren’t public? Who knows.

    I’m personally biased maybe, because I’m grappling with Apple over a PDFKit bug in [url=https://tidbits.com/article/16966<]both Sierra 10.12.2 and iOS 10.2[/url<], and they're being a pain in the ass.

      • Chrispy_
      • 3 years ago

      I hate dealing with Apple. They are always a pain in the ass, because they’re so full of themselves. Luckily I don’t have to do it too often as part of my day job.

      • Jeff Kampman
      • 3 years ago

      That’s not really what was said at all. The exact wording is “Apple declined to comment on our test results until they better understand the issue, but emailed this statement: ‘Any customer who has a question about their Mac or its operation should contact AppleCare.'” The implication of that statement is that an investigation was ongoing (and, in fact, one was).

      Editorial prudence in that case would be to withhold judgment until Apple had finished its investigation, not to jump through the hoops CR did to justify going ahead and publishing anyway. By jumping the gun, CR ultimately produced a fiasco that didn’t need to happen.

        • derFunkenstein
        • 3 years ago

        I dunno, man. I strongly feel stuff with open questions can and should be published with the open question right out there for everyone to see. As you plainly quoted, that open question was still out there. They plainly concluded with “Consumer Reports has shared diagnostic files pulled from all three computers with Apple in the hope that this will help the company diagnose and fix any problem. We will report back with any updates.” I personally think that CR is being more than courteous to Apple when they buy their hardware right off the shelf.

        Stuff like that gets reported all the time in all kinds of publications. The weird 4KB IOMeter graph for the [url=https://techreport.com/r.x/2016_10_17_960Pro2TB/sustained.png<]Samsung 960 Pro[/url<] springs to mind.

          • Jeff Kampman
          • 3 years ago

          The issue is not really the “open question,” the issue is the commingling of data arrived at by one method (averaging) for a large part of the testing lineup and another (just pick a number) for the Apple products when you’re synthesizing CR ratings or whatever. That’s the questionable part of this whole process.

            • derFunkenstein
            • 3 years ago

            I have two points:

            1.) You said they should have halted publishing based their testing results. The only party that benefits from that is Apple. I believe every manufacturer should have its feet held to the fire.

            2.) They did not cherry-pick a single result in [url=http://www.consumerreports.org/laptops/macbook-pros-fail-to-earn-consumer-reports-recommendation/<]the original article[/url<]: [quote<] For instance, in a series of three consecutive tests, the 13-inch model with the Touch Bar ran for 16 hours in the first trial, 12.75 hours in the second, and just 3.75 hours in the third. The 13-inch model without the Touch Bar worked for 19.5 hours in one trial but only 4.5 hours in the next. And the numbers for the 15-inch laptop ranged from 18.5 down to 8 hours. [/quote<] I don't see where they made up a singular number. I'm also not a CR subscriber so maybe it's behind a paywall.

            • Jeff Kampman
            • 3 years ago

            In CR’s words: “However, with the widely disparate figures we found in the MacBook Pro tests, an average wouldn’t reflect anything a consumer would be likely to experience in the real world. For that reason, we are reporting the lowest battery life results, and using those numbers in calculating our final scores. It’s the only time frame we can confidently advise a consumer to rely on if he or she is planning use the product without access to an electrical outlet.”

            They didn’t make anything up, but they did pick numbers and they did (apparently) commingle them with other numbers that were arrived at in a different way for their first set of ratings. That’s a bad practice.

            • derFunkenstein
            • 3 years ago

            I missed that somewhere, so that’s my bad. Apologies. You’re right. An average and a single result aren’t compatible with one another.

            I do think they did the right thing by publishing their findings, though. That was their experience, and based on their experience they couldn’t recommend the MacBook Pro. They published in the article pretty much every result and painted a clear picture that battery endurance wildly fluctuated, and included the caveat more than once that Apple was looking into it.

            • sweatshopking
            • 3 years ago

            I haven’t been coming to TR because I’ve been finding it basically a collection of PR pieces with little analysis, and hadn’t planned on logging in anymore, but thought I should make this comment, having read this, as I think it helps me understand why I feel the site has changed so drastically since scott left:

            CR was 100% right. Not publishing that data would have hurt them as an organization, and would have only benefitted apple. We’re not talking about an hour or two. we’re talking less than 25% of other tests. this isn’t a situation where averaging made sense, and had they not it would have hurt their organization. People would have been upset when CR suggests they can get 10 hours of battery life on their device and it’s pulling in less than 4. they’d also wonder why they’re only rated for 10 when they’re getting 16. An average would have been moronic and entirely irrelevant to anything any consumer would ever experience. Suggesting they should have used the same method of rating for a clearly broken and buggy device is ludicrous. Now that it’s fixed they’ve adjusted. Fine.

            • Srsly_Bro
            • 3 years ago

            SSK is on point.

            • adisor19
            • 3 years ago

            “People would have been upset when CR suggests they can get 10 hours of battery life on their device and it’s pulling in less than 4.”

            LOL No. Nobody in their right mind disables Safari caching unless he or she is a webdev or has a very specific need.

            “People” would have never noticed this.

            Adi

            • VincentHanna
            • 3 years ago

            And yet, the bug would have persisted.

            Unnoticed? Perhaps.
            Misattributed to other settings/issues? Maybe.

            But every time safari loaded a webpage fresh, the bug still would have been there.

            • adisor19
            • 3 years ago

            Sure, so let me ask you this : if a tree falls in a forest and there is no one there to hear it ? Does it matter ?

            Adi

            • JalaleenRumi
            • 3 years ago

            There are many things ‘People’ don’t use or ever notice but QA goes through everything and forces the developers to fix bugs that will NEVER be discovered by the ‘people’ but they never want to take chances. Even if a group of people go into that grey area and find those bugs, it will be all over the internet and ‘memes’ will haunt the company for years.

            My point is, just because average user will never experience it, does not mean it won’t be discovered and when it is found, people will put on the internet about how they found a funny bug and how CR’s credibility is in question as they didn’t report such a thing and that means they didn’t know what they were doing when they reported that everything was fine.

            It is not CR’s headache if something is not working. Its Apple’s fault or any other company. CR just tests them products and publish their analysis and accordingly they report to their customers whether they should or shouldn’t use the product. Its not like CR is going out of their way to target Apple or anything. But hey, CR is NOT like putting a gun to their heads and preventing them from buying Apple products. No. They just want to save their behind, so that when ‘people’ figure out that problem with Apple device, CR will at least say ‘See, I told you so.’

            Damn what am I doing. Its useless. Fanbois will never understand. Adi.

            • VincentHanna
            • 3 years ago

            SSK IS DEAD ON!

            • ludi
            • 3 years ago

            If the standard test suddenly goes from producing reliable and consistent results to producing a completely wacky spread, it’s customary to notify all parties and then give reasonable time to explore the problem and possible solutions before publishing final conclusions. How do you know the problem isn’t caused by a bug in YOUR methodology, as opposed to the other party?

Pin It on Pinterest

Share This