Consumer Reports stops recommending Surface mobile devices

Microsoft's Surface line of mobile devices has a lot of fans, including our own Bruno "morphine" Ferreira. Consumer Reports (CR) has traditionally been part of that fan club, giving various devices a place on its recommendations list. However, the magazine has now removed the "recommended" designation for all mobile Surface models after receiving tens of thousands of complaints from readers. The news is headlined by a two-year problem report rate of 25%, suggesting that a quarter of all Surface owners will experience problems with their devices by their second year of ownership.

Surface Laptops with 256 GB and 512 GB SSDs and Surface Books with 128 GB and 512 GB drives previously had a place on CR's recommended lists. Microsoft has only recently joined the ranks of computer manufacturers, and this is the first year that the magazine had enough data to "estimate predicted reliability for the company's laptops."

Microsoft disputes Consumer Reports' findings, stating its "real-world return and support rates for past models differ significantly from Consumer Reports’ breakage predictability." CR says that while hardware from the Redmond software giant has performed well in laboratory testings, the site's readers have had trouble with Microsoft's devices at higher rates than with those of other brands. The site's latest surveys tabulate results from almost 91,000 readers that own tablets and laptops.

Comments closed
    • tipoo
    • 2 years ago

    Except for the Book, the return rate of 3% for other products seems contrary to ConsumerReports percent

    [url<]https://www.theverge.com/2017/8/14/16142490/microsoft-surface-consumer-reports-memo-return-rates[/url<]

    • WaltC
    • 2 years ago

    I stopped recommending Consumer Reports decades ago…;) Sad, that some people still can’t discern the difference between opinion and “science.” CR is 100% opinion. That’s not a bad thing, of course; it’s just that opinions are like armpits–everyone has at least two…

      • cynan
      • 2 years ago

      Sure, like just about any other review. What would go a long way to making it “scientific” would be if CR, in sufficient detail for replication by a 3rd party, discloses their testing methods. As far as I know, no such details are made available.

      Because of the larger sample size, at first glance, it would seem that recommendations based on reader surveys (such as the decision to delist Surface products as “recommended) bare more rigor. But again, until more information is disclosed (in particular the number of readers who actually own surface products participating in the survey in this case), there are potential pitfalls to accepting these recommendations.

      TLDR: “Trust us, we know how to review/rate stuff” is just not very scientific.

    • chuckula
    • 2 years ago

    Hrrm… recommendation removed on the same day The Threaderipper launches?

    Not a surprise. Any system that lacks ThreadRipperiness is now obsolete and can’t be recommended.

    That goes double for your so-called “smart” phones.

    • barich
    • 2 years ago

    I haven’t seen significant hardware problems with Surface devices. However, I’ve seen plenty of driver and firmware related bugs that took far too long to be ironed out (especially considering Microsoft has the benefit of making both the OS and the hardware). And to the consumer, those are “problems.” Microsoft probably doesn’t count them as such.

    Frankly, though, I’d agree with the consumer definition. If you can’t undock your Surface Book without power cycling it, or your WiFi keeps disconnecting, would you consider that a reliable device even if an eventual firmware or driver update fixes it without a hardware repair?

    • swaaye
    • 2 years ago

    I’ve had two Surface Pro 4 units. Upgraded from an i5. But I’ve really liked both of them. The only problem I’ve run into is my i7 model has a backlight flicker if I run it at <10% output. That is irritating when I (rarely) run it at that level.

    But overall I’ve found these machines to be very solid. I’m particularly impressed the rubber/plastic keyboard dock doesn’t fall apart in some manner after a zillion opening/closing/de/re-attachments. And that magnetic charger jack seems very durable too.

      • Flying Fox
      • 2 years ago

      This does not seem to include the SP4, but the new Surface Pro 2017 and the Surface Books and Surface Laptops.

        • Drachasor
        • 2 years ago

        They say that there are no laptops or tablets from Microsoft that they recommend now. So that includes the Surface Pro 4.

    • deathBOB
    • 2 years ago

    This makes sense in light of MSs numerous firmware updates to fix issues like wifi or hibernation.

      • Voldenuit
      • 2 years ago

      Or detaching the Surface Book from its base.

    • deruberhanyok
    • 2 years ago

    I can’t say I’m too surprised. My own experience with Surface devices has been mixed. In isolation they were fine, but as soon as I tried using an external display I’d get any number of freezes, display corruption, etc, regardless of whether a dock was involved or directly connected via mDP. And then the seemingly random times when it stopped taking any input (keyboard, mouse or touch) and needed a hard restart.

    We had purchased one or two to test out for corporate use and it mirrored my own experience with them. After a week or two I remember saying “well, they’d be completely fine as long as our users don’t mind having to reboot the system twice a day because it froze in the middle of doing something, they’d be great!”

    Thankfully we didn’t buy them.

    • Andrew Lauritzen
    • 2 years ago

    Curiously, do CR try to account for self-selection bias on higher priced devices at all here?

    I certainly know people who have had Surface devices break but anecdotally it doesn’t seem like the landslide that these numbers would suggest. That said, I’m willing to trust the data if it is appropriately controlled.

      • ikjadoon
      • 2 years ago

      Anandtech came to the same conclusion after their Surface Book review. This is a real problem.

      >The Surface Book offers some of the nicest hardware in any detachable convertible. I think despite the price Microsoft is going to have customers for this. The question is will they get the bugs sorted out. The first firmware update fixed a lot, but there are still too many outstanding issues to recommend the Surface Book at this time.

      [url<]http://www.anandtech.com/show/9767/microsoft-surface-book-2015-review/9[/url<]

        • Andrew Lauritzen
        • 2 years ago

        Uhh… that’s not exactly related to long term reliability, or indeed to CR’s definition of reliability at all.

      • tipoo
      • 2 years ago

      What I’d like to see is if they only select laptops over 2000 dollars, and then see if 2 year survival rates fare any better against Apple.

      Dell and Lenovo and such make some high end laptops, but the vast majority sold are sub 500 dollar craptops (not trying to be classist – just that there are very literally almost no decent laptops under 500), that will likely make a difference in reliability scores.

      • willmore
      • 2 years ago

      They send their surveys out to a sample of their subscribers and ask them to fully answer the survery. So, that should remove a great deal of self selection bias in their data.

      I’ve gotten about a dozen survey requests from them and I’ve filled all of them out that applied to anything I owned wether I had an issue with it or not. The only ones that I did not complete were either for things I don’t own or when they had an issue with the freaking survery software where it kept asking the same set of questions over and over. I finally abandoned that one–only to get numerous followup emails asking me to complete it. So, their methodology is pretty good.

    • Morgartjr
    • 2 years ago

    We removed all of the surfaces at our company and stopped buying them from MS as well. We had huge failure rates with high temps, phantom touches, blue screen issues that reimaging couldn’t fix. After months of conference calls with ms, we cancelled our orders and sent them all back.

      • ikjadoon
      • 2 years ago

      The blue screens on /r/surface always cracked me up. How does a Microsoft OS running on a Microsoft laptop get BSODs while running Microsoft PowerPoint? It hit the top of the subreddit for two days.

      If the company says, “This isn’t a first-generation product”…but it is a first-gen product…don’t believe the company.

    • brucethemoose
    • 2 years ago

    I wonder how many of those complaints are general Windows problems (meaning they would happen on any PC, not just a Surface).

    • ddarko
    • 2 years ago

    This article leaves out the most interesting part of Microsoft’s statement that they gave Consumer Reports:

    “Microsoft’s real-world return and support rates for past models differ significantly from Consumer Reports’ breakage predictability,” Microsoft said in an emailed statement. “We don’t believe these findings accurately reflect Surface owners’ true experiences or [b<][i<]capture the performance and reliability improvements made with every Surface generation[/b<][/i<].” Read between the lines and Microsoft seems to be admitting their Surface devices had quality problems in the past but it's gotten better. That may well be true but Consumer Reports predicted reliability is based on extrapolating past data into the future. Their method is going to have lag - it's not going to immediately pick up improvements in quality. All the same, this is a sensible and reasonable method of statistical analysis.

      • blastdoor
      • 2 years ago

      All good points!

      I figure that the burden of proof is on Microsoft to show that they’ve really made the improvements they claim. Earning a good reputation is, and should be, hard. It is, and should be, even harder to recover from a bad reputation.

      Remember Microsoft — fool me once shame on you, fool me twice shame on me.

      • ikjadoon
      • 2 years ago

      Oh, they definitely did. Anandtech wouldn’t even recommend the Surface Book only because of the crazy high bugs. And how often does Anandtech actually bluntly say, “we don’t recommend you purchase this”??

      >The Surface Book offers some of the nicest hardware in any detachable convertible. I think despite the price Microsoft is going to have customers for this. The question is will they get the bugs sorted out. The first firmware update fixed a lot, but there are still too many outstanding issues to recommend the Surface Book at this time.

    • willmore
    • 2 years ago

    Yo, SSK, remember these?

      • Neutronbeam
      • 2 years ago

      I BET HE DOES!!!

        • willmore
        • 2 years ago

        Ha, he drove by but didn’t stop.

    • Flying Fox
    • 2 years ago

    My take on this:
    – How much of those 91000 use a Surface mobile device? Let’s say 1000, then the sample size may still be relatively small. I don’t see “tens of thousands” in the original article.
    – How much of the issues are software? If it is a problem with Windows, then other brands may stand to have similar issues. So it is not that they are worse than other Windows mobile devices. Windows Updates have a habit of breaking stuff, which is not an excuse but it adds noise to the build quality of the devices.

    One way this can be mitigated is to have longer warranty terms. So people who bought the extended warranty should be well covered, but not everyone buys that.

    • cynan
    • 2 years ago

    That’s sort of like basing hard drive reliability from Newegg reviews. Better than nothing, but obviously, something like the published Backblaze reliability statistics is much sounder data on which to base an informed opinion.

      • shank15217
      • 2 years ago

      91000 is a pretty large sample size, not sure what you are getting at. Also Microsoft it’s probably hiding the truth due to surface bring popular among IT deployments.

        • DPete27
        • 2 years ago

        I think he’s trying to point out that more people will complain about a “broken” product than will bother to take any interest in a [CR]survey of the same product that is working perfectly fine.

        [Add] Also: [quote<]The site's latest surveys tabulate results from almost 91,000 readers [b<]that own tablets and laptops.[/b<][/quote<] Notice that doesn't say "readers that own [b<]SURFACE[/b<] tablets and laptops."

          • cynan
          • 2 years ago

          Yes. In addition to your point about selection bias (ie, who bothers to complete the surveys), What is the denominator on which that 25% problem rate is based? I also don’t see “tens of thousands” in the original article.

          At the risk of being pedantic, here’s the crux of the problem:

          1) A sampling issue :

          [quote<]The site's latest surveys tabulate results from almost 91,000 readers that own tablets and laptops.[/quote<] How many of those 91,000 even owned Surface products? MS has sold millions of Surface tablets. Was it 5%? If so, that implies these findings are based on less than 5,000 Surface tablets. Are the vastly smaller number of CR readers who have bought Surface products representative of the average Surface customer and usage environment? With a larger sample, on average, this is more likely to be the case. Backblaze goes through a lot of HDDs but maybe not that many more relative to the total number of HDDs sold. However, their data was still likely based on way more units. 2) A "test" control issue: And given that these findings were based on a relatively small number of Surface products, were these, and competing mobile products purchased by CR readers, treated uniformly, used in similar environments, etc? It would be ridiculous to assume so, or at least to claim knowledge that that was the case. With the Backblaze data on HDDs, it is not much of a stretch to presume that each HDD, within and between brand/model of HDD, was handled the same as the next.

            • blastdoor
            • 2 years ago

            Those issues affect CR’s reviews of all competing products too, not just Surface. Yet many of the competing products fare far better than Surface.

            You would have to argue that Surface users are just bigger complainers than users of competing products, or that Surface users operate in very different environments than users of competing products. Those things are possible, but not likely.

            • cynan
            • 2 years ago

            Yes. It is reasonable to assume that such issues would impact all CR reviews [i<]edit[/i<] and competing products. The point was that these issues are mitigated at least to some degree with more systematic "testing", larger data samples (ie, Backblaze vs Newegg review data for HDD reliability - see my OP)

            • Andrew Lauritzen
            • 2 years ago

            It’s not clear that those issues affect all products *equally* though. It’s definitely a reasonable hypothesis that people will complain more about – for instance – their $3000 surface laptop breaking than their $300 walmart special, even if the real incidence rates are similar.

            Also as this very article demonstrates, it’s possible – even likely – that they simply don’t have enough data on any single other product to draw conclusions. Apparently the Surface products were in this list for a long time before they decided they had enough data to judge longer term reliability… if they just got to that with Surface, it’s very likely that very few other Windows laptops will *ever* get enough data, so it’s possible they are just riding under the radar perpetually.

            [Edit] It seems like they actually only collect reliability data based on *brand* with no consideration of SKU or even product type. I get why they do that for practical reasons but IMO that severely limits the utility of the data when applied to *specific* products.

            Obviously comparing things like a Surface and Macbook is likely to be more legitimate here as they are more similarly priced (and purchased) products, but I’m not sure these conclusions are really meaningful beyond that.

            • ludi
            • 2 years ago

            Right, because a broken $300 laptop can be replaced ten times for every $3000 laptop that breaks once. When you pay a dozen or more Benjamins for a laptop, there’s a certain expectation of quality.

      • w76
      • 2 years ago

      Knock online reviews if you want, but I’ve rarely gone wrong following this logic and almost always been burned when I dared to ignore it. [url<]https://xkcd.com/1098/[/url<] But hey, it's your own money/data/etc!

        • cynan
        • 2 years ago

        I’m only knocking the utility of online reviews compared to better, less bias prone sources of data (on reliability). As stated, they are “better than nothing” when these better data don’t exist.

        Perhaps the poor reception has to do with the nihilistic lack of utility of pointing out, by analogy, the potential caveats with CR survey data, given that this may very well be the best data available on Surface reliability available to consumers

        Also, this may be obvious, but ironically, the basis for the humor in the comic you linked is the generally poor validity of online reviews: It is because they are so prone to bias that you need to ignore all but the extreme “effect sizes” (eg, only choosing products with 4-star reviews, or higher given sufficient sample size) in order for them to be of any utility. This happens when making any inference from heavily biased data (you need a strong signal to separate it from a lot of noise)

      • Luminair
      • 2 years ago

      what you don’t realize is that CR has statisticians on staff, and your brain reading newegg reviews does not.

      also, I’d love for organizations to collect and release their laptop fleet failure rates the way backblaze does. but nobody does. so now what

      • ikjadoon
      • 2 years ago

      Nah. Anandtech came to the same conclusion two years ago. They’re buggy as hell:

      >The Surface Book offers some of the nicest hardware in any detachable convertible. I think despite the price Microsoft is going to have customers for this. The question is will they get the bugs sorted out. The first firmware update fixed a lot, but there are still too many outstanding issues to recommend the Surface Book at this time.

      [url<]http://www.anandtech.com/show/9767/microsoft-surface-book-2015-review/9[/url<]

    • ludi
    • 2 years ago

    Marketing: “Break it as thin as possible.”

    Engineering: “Sorry, what was that?”

    Marketing: “We said, ‘Make it as thin as possible’.”

    Engineering: “Yeah, we heard you right the first time. If we could just have an 0.2 inch for–”

    Marketing: “No.”

      • CuttinHobo
      • 2 years ago

      Engineering: “But we can double battery life by–”

      Marketing: “No. …Oh and while we’re on the subject, solder the battery.”

        • tay
        • 2 years ago

        The new surface laptop is a ball of glue. You can’t even open it without destroying the electronics

          • Axiomatic
          • 2 years ago

          Thats the #1 reason I didn’t buy one. Been a computer engineer for 30 years now. I don’t own anything that I can’t service myself. Yeah I know thats not the norm but the “ball of glue” is real on this product per the iFixit tear downs.

          • tipoo
          • 2 years ago

          It’s insane that you can’t so much as physically disconnect the battery connector, without ripping through the alcantara top layer. Think of getting some water on the keys – disconnecting the battery for a few days could be the difference between laptop life or death (or dead keys).

          Afaik, Microsoft hasn’t made it clear what they can do if someone needs that replaced after a repair, or how they handle their own repairs.

            • ludi
            • 2 years ago

            Very likely the repair kits, official or unofficial, will have to include a replacement top surface.

          • usernam3
          • 2 years ago

          Oh yeah. Data recovery is fun (luckily the SSD turned out to be removable M.2 unit). MS’ response to support request: “You should had backed up to OneDrive” (suckers). Seems like our business has also abandoned the Surface line (with few exceptions to influential hipster wanna-be’s that may be lacking on common sense but surely not style).

      • Drachasor
      • 2 years ago

      I got a Surface Pro 4 in December since at the time there wasn’t really a ton of other options where the stylus was good — not that I found anyhow. It was for taking notes in classes.

      I’ve found it to be a fragile machine, far more than any other laptop, phone, or tablet I’ve ever owned. They really did sacrifice a lot just to be thin and it doesn’t seem worth it to me. The customer service experience was terrible as well.

Pin It on Pinterest

Share This