Report: 3.5% of world’s March NAND supply lost to outage at Samsung fab

I just had a hard drive fail on me, and I'm not looking forward to buying more spinning rust for my desktop. Unfortunately for me, it doesn't look like flash memory prices will be falling any time soon. TechNews of Taiwan reports that Samsung's massive NAND flash memory plant in Pyeongtaek, South Korea, surprisingly suffered a roughly 30-minute power outage on March 9. Due to the outage, Samsung reportedly lost between 50,000 and 60,000 wafers, or around 11% of its flash production for March.

TechNews goes on to say that number amounts to around 3.5% of the world-wide flash memory production for this month. The site remarks that foreign investors don't expect the accident to have a major impact on Samsung's profitability or future operations, owing to the fact that Samsung apparently has "sufficient inventory" of flash memory. The idea that a flash factory suffered a 30-minute power outage and that the effect was the loss of its pending production is a little surprising. Nevertheless, we've only recently seen SSD prices drop and this event could change that.

Comments closed
    • albundy
    • 2 years ago

    because toshiba, micron, hynix, etc have absolutely never produced nand. give me a break.

    • ptsant
    • 2 years ago

    Yeah, I got the message: my next SSD will cost 3 times more per GB than the previous one.

    • tootercomputer
    • 2 years ago

    My first thought reading this was, how did a 30 minute outage cause such a loss. I know nothing, nothing about NAND production, so anybody know?

      • stdRaichu
      • 2 years ago

      I’m not an expert and I’m simplifying greatly, imagine [url=http://tvtropes.org/pmwiki/pmwiki.php/Main/EndangeredSouffle<]opening an oven door before the soufflé is ready[/url<]; the soufflé collapses and you have to start all over again with fresh ingredients. Similar sort of deal with silicon, 'cept with something like a 64 layer NAND wafer you've got to go through that oven at least 64 times [i<]just right[/i<] in order to get your soufflé, and if any any time you're a few degrees off the right temp, or you keep it in for a few minutes too long on any one of those oven runs, your soufflé is once again ruined, as are all the other soufflés waiting in other ovens and whatnot along the production line. There's a reason why modern soufflé factories take years to build and are hugely expensive. There's very little time on the production line where things won't be being processed to some degree, the process takes at least a couple of days for each soufflé depending on the complexity of the dish (so at any one time the production line will have several days worth of output on it) and a mistake at any one of a hundred different steps is purportedly enough to write off a whole batch of soufflés. Incidentally, does anyone know if fabs tend to grow their own silicon ingots or do they tend to get get made elsewhere? That's another process that takes a long time and is extremely time, speed-and-temperature critical. Those things are made as a single crystal - any variations in that production process and it won't grow into a single crystal and bang goes your chance at getting an ingot. <goes off to find a [i<]How it's Made[/i<] episode about ICs and/or soufflés>

        • YellaChicken
        • 2 years ago

        +1 for answering a question I was going to ask too, +1 for simplifying with soufflés and +1 for “purportedly”. Thank you sir.

        • anotherengineer
        • 2 years ago

        I think ingots are made at a separate facility, and it would probably get sliced there or sliced at the fab, not sure and don’t have proof, but doping silicon and making an ingot is a completed different process that would require a different plant. I suppose it could be on the same property though.

        However, 11%, so a 30 minute outage basically caused roughly a 3 day production outage, seems high to me, but I have no idea of that process.

          • Wirko
          • 2 years ago

          Surprisingly, intel buys wafers. At least it did several years ago.
          [url<]http://download.intel.com/newsroom/kits/chipmaking/pdfs/Sand-to-Silicon_22nm-Version.pdf[/url<] - page 4

          • davidbowser
          • 2 years ago

          Caveat – 20 years ago when I worked for Lucent.

          Silicon ingots were made near the raw material source (clean sand) so that there was less expense in shipping tons of sand. There was also a factor of fuel (natural gas) pipeline availability near the sand source. Cost balance that by location and you have yourself a silicon ingot factory.

        • tootercomputer
        • 2 years ago

        That makes sense. Great analogy.

      • Ifalna
      • 2 years ago

      The processes are temperature and time sensitive.

      Now imagine just one simple thing: Wafer is in the oven, heated to 800°C to allow particle migration to desired depths. Normally the temp would ramp up in a controlled manner, stay up for a defined period of time and then ramp down, also controlled and predictable.

      Enter a 30 minute break ANYWHERE in that cycle and the wafer is gone. Why? Because you no longer know how far the particles have migrated, so you no longer know how thick your doped layers are. Impossible to calculate with sufficient accuracy, impossible to measure.

      ESP bad if the wafer stays in too long. Overexposure to heat destroys carefully placed layers.

    • Mr Bill
    • 2 years ago

    Coincidence? [url=https://www.nytimes.com/2018/03/15/us/politics/russia-cyberattacks.html<]Cyberattacks Put Russian Fingers on the Switch at Power Plants, U.S. Says[/url<] More at NPR HereAndNow... [url=https://www.npr.org/2018/03/16/594371939/u-s-accuses-russia-of-cyberattacks-on-energy-infrastructure<]Report: Russian Hackers Had The Ability To Shut Down U.S. Power Plants[/url<]

    • Froz
    • 2 years ago

    Everyone is assuming that the only reason for the issues were the power outage and Samsung somehow doesn’t have any safety measures against that. Well, I’m pretty sure power outages happen there regularly. But just as with networking, fallback solutions sometimes do not work correctly and often you learn about it when the issue already happened. That’s what must have happened here.

    What is more interesting to me here is that one fabric provides 1/3 of the world’s flash memory production. I mean, that really feels incredible to me that something that can be found nowadays pretty much in every home has so concentrated production.

    • Duct Tape Dude
    • 2 years ago

    Fabs of course need a reliable supply of clean air and electricity, lest something like this happens. In college my microelectronics professor told me a story about a fab from years ago:

    After suffering a power outage in the area, one particular fab had had enough. They decided to install enough diesel generators to run the plant continuously at the first sign of a power outage. It was a large, expensive project as they took seemingly everything into account: failover systems, generators, fuel tanks and distribution–it was the only one of its kind in the region.

    Everyone else just had a few wimpy batteries to hold them over for a few minutes (clearly buying generators was a waste of money). How could such an expensive project be worthwhile?

    But sure enough, one day, the power went out, for real this time. Everyone was fine for the first few minutes, but after batteries ran out, this one fab stood alone with the lights on. Their diesel generators saved their entire production run.

    A week or two later, the same fab noticed a sharp drop in yields. They traced everything… was their equipment damaged in the power outage? Did something change in their manufacturing process? Then their maintenance division, tasked with scheduled air filter changes, noted the ones they were pulling were black with soot.

    During the outage, the diesel exhaust was directly upwind of some air intakes, and had spread throughout the entire facility’s filters. The entire fab had to be shut down for weeks while they changed ALL the air filters and purged their filtration system.

    I think I’m supposed to put a punch line here. Something something clean power. Maybe you can do one better.

      • chuckula
      • 2 years ago

      They just need solar power and to locate the fab in Philadelphia.

      Because It’s Always Sunny In Philadelphia (even at 3 AM).

      • stdRaichu
      • 2 years ago

      [quote<]I think I'm supposed to put a punch line here. Something something clean power. Maybe you can do one better.[/quote<] This was the fab that made Volkswagen ECUs, right?

      • cynan
      • 2 years ago

      Considering more than 60% of America’s power still comes from burning fossil fuels, I think this is a pretty good illustrative example of why electric cars may not be quite as clean as some may think. (not that most electricity production exhaust is likely to be anywhere near as dirty as diesel generator exhaust, but nonetheless).

        • pandemonium
        • 2 years ago

        That’s a non-starter.

        That’s like saying that crops grown without harmful pesticides are just as dangerous to consume as those grown with because they both get their energy from the sun.

        • Usacomp2k3
        • 2 years ago

        That’s why we need more nuclear plants. Far and away the best source of power from a TCO standpoint.

          • Mr Bill
          • 2 years ago

          A Geothermal field and plant might be just the ticket. Only, the groundwater would not be lovely and pure. A single well (Vulcan) at the CalEnergy Salton Sea facility generates 80MW of thermal output. Its pretty awesome to stand near that well. The ground shakes like a coal train is coming through.

        • cygnus1
        • 2 years ago

        A very large, growing portion of that is from natural gas. Natural gas power is less polluting per mile driven than the average gas engine.

        • wierdo
        • 2 years ago

        “an EV driving on electricity in the U.S. today is equivalent to a conventional gasoline car that gets 80 MPG, up from 73 MPG in our 2017 update.”

        [url<]https://blog.ucsusa.org/dave-reichmuth/new-data-show-electric-vehicles-continue-to-get-cleaner[/url<] Some places are even hitting 259mpg equivalent. That's the nice thing about centralized energy production, you only have to modernize at the source, so things are just getting better every year.

      • Klimax
      • 2 years ago

      Punch line: Always mind where exhausts pit.

      • Kougar
      • 2 years ago

      Yeah, they should have used fuel cells. Won’t compete with gas generators for cost, but they work best at scale with just water as the byproduct. Samsung could certainly have afforded a few hours of standby fuel cell power had it wanted.

      • specialworks
      • 2 years ago

      It certainly sounds like years ago before air-pollution in major cities became much better covered in the media and started to affect children of the political classes.

      what’s most telling with hindsight is how clean the air must have been normally around the plant for the filters to not be able to filter out some of the largest polluting particulates going!

      Sounds like a f**k up waiting to happen…

    • Concupiscence
    • 2 years ago

    Another day, another convenient power loss that keeps NAND prices from dipping.

      • DavidC1
      • 2 years ago

      They probably figured it wasn’t a big deal since prices will rise anyway, lol.

      When it rains, it pours. There’s enough evidence to believe things happen in a cyclical fashion. So much so that it almost looks like a conspiracy when its not.

      • gmskking
      • 2 years ago

      No problem, I just won’t buy.

      • DPete27
      • 2 years ago

      And I’m sure they’ll use this to justify a 10%-15% price increase for years to come.

    • chuckula
    • 2 years ago

    You think Samsung would have had a battery backup!

    But then you realize that they made their FAB so thin that the batteries couldn’t fit without catching fire.

      • EzioAs
      • 2 years ago

      The jokes write themselves.

      You’re just a medium, chuck! You don’t deserve these upvotes!

        • chuckula
        • 2 years ago

        You’re right. I’ll write Samsung a check for all my shilling profits today.

          • Wirko
          • 2 years ago

          Currently the shilling is used as a currency in four African countries of Kenya, Tanzania, Uganda and Somalia. [Wikipedia]

      • Chrispy_
      • 2 years ago

      To the frontpage with you!

    • Topinio
    • 2 years ago

    How?

    How is Samsung so vulnerable to power failures? Why can’t the production line have sufficient power protection to be able to park in-process wafers safely until power gets restored?

      • cynan
      • 2 years ago

      They sold all of their 18650s to Tesla?

      • gecko575
      • 2 years ago

      ^ This

      • DragonDaddyBear
      • 2 years ago

      From reading Jeff’s article about his trip to Global Foundries it appears that the power requirements to sustain operations are quite large.
      [quote<]Powering the EUV equipment on top of all of the other tools in the fab is another challenge in itself. Painter told us that GlobalFoundries gets its power from dual 115 kV lines from the New York grid, and it uses about 80 MW of power on average. Any interruption to that supply would prove crippling for the facility, so Fab 8 has 40 MW of short-term power spread across flywheels, uninterruptible power supplies, and diesel generators. Painter noted that the site had "just enough" electrical headroom to accommodate its EUV expansion.[/quote<] Although that is not Samsung, I figure it reasonable to assume that Global Foundries isn't alone in their need for a lot of electricity.

        • the
        • 2 years ago

        It sounds like Samsung just let everything die.

        While having power to continue operations is some thing, battery backing only critical stages/equipment could permit throwing away fewer wafers. By knowing where in the production line each were and keeping them in their known state until the line could resume would be more advantageous. A smaller number stuck mid-stage would need to be thrown out. Additional testing of these waters would also be prudent.

          • stdRaichu
          • 2 years ago

          80MW is a [i<]massive[/i<] amount of power, and there are loads of activities inside fabs that require the power to be kept up for the duration of a long process - for example, doping the wafers will typically take ~8hrs at a very precisely controlled temperature. Bizarrely enough I just found an odd s/h listing for someone [url=http://www.kitmondo.com/80mw-diesel-power-plant/ref456167<]selling an 80MW diesel plant[/url<]; Four 20MW generation plants Each comprising of 12 diesel units So... 48 diesel engines in total. That mentions the Perkins 4016TAG2A which can be seen [url=https://www.perkins.com/en_GB/products/new/perkins/electric-power-generation/diesel-generators/1000002717.html<]here[/url<] - a 61 litre V16 which weighs 8 tons and is over 4m long. And you need ~48 of those. Don't think I've even seen a data centre genny setup anywhere [i<]near[/i<] as big as that. Not trying to say that buying a setup like this might not be worth it in the case of a fab, but it's certainly no simple matter. More likely Samsung has a contract with the grid supplier where they're compensated in the event of power loss.

            • Waco
            • 2 years ago

            Woah – I think I know the source of that diesel setup. I’m surprised it’s on the market if it’s the one I think it is.

            • morphine
            • 2 years ago

            Oh, don’t tease us. Let’s hear it.

            • Waco
            • 2 years ago

            If the sale goes through I’ll follow up. 🙂

            • MOSFET
            • 2 years ago

            That unit is a long way from New Mexico 🙂

          • Waco
          • 2 years ago

          Actually, it sounds like you don’t understand the logistics for this kind of power redundancy or what happens in a fab (and how time-critical a huge number of processes are). It’s not trivial, and no amount of redundancy can cover long power outages upstream.

          My site has ~20 MW redundant power to the main building and even with all of the UPS systems (flywheels, batteries, etc) we can’t keep production going for longer than 10-15 seconds depending on the current load. The biggest systems (~12 MW peak) can’t even be fed by UPS in any reasonably affordable way even for short periods of time – so any blip on both feeds takes down the whole thing.

          It’s [i<]not[/i<] easy.

          • ludi
          • 2 years ago

          Uninterruptible power in this scenario requires about 30 seconds of battery or intertia storage while the gensets are spooled up and synchronized, including all of the maintenance that requires, in addition to an enormous generator hall and fuel storage and the maintenance [i<]that[/i<] requires. The capital equipment and maintenance costs for such a system might actually exceed your losses for one-off events that occur every few years (at most), and losses are potentially insurable or written off against taxes at much more favorable rates than capital equipment depreciation on a massive redundant power setup. If life and death are not hanging on the balance, letting the plant go dark and trashing the lost product may actually be the more economic option.

            • UberGerbil
            • 2 years ago

            Indeed, and that’s assuming everything goes perfectly. If there’s any hitch in the switchover, you may lose everything in production anyway, which means you spent all that money on redundancy and then got [i<]nothing[/i<] back for it. And it's not like you're going to be pulling the plug on an operational line regularly to test it, either. I can easily imagine them running the numbers and deciding it wasn't worth it. Especially when the occasional failure helps jack up prices for whatever they have in inventory. (It helps their competitors more, of course, which is why nobody goes out of their way to have these kinds of accidents, whatever the conspiracy theorists want to tell you).

            • JustAnEngineer
            • 2 years ago

            Most optional standby generators meet the same standard as emergency generators: be ready to accept load in less than 7 seconds.

            • Waco
            • 2 years ago

            Small ones, sure. The big multi-MW generators take 15-30 seconds to start, run up to speed, and sync up with the RUPS/batteries.

            • anotherengineer
            • 2 years ago

            Depends on the type of generator, some could take even longer. We can bring an 80MW unit online in under a minute, possibly in 10 seconds, depending on the mode it’s operating 🙂

            But like above, yes, I used to work at a smelter/metal refining plant, and it typically consumed 125-135MW, I think the smelter alone had six v-8 diesel backup generators, which basically powered emergency lighting and pumps, and that’s it, same with the other plants. Power came in on a 230kV line, basically right from the transfer station.

            To have a redundant 150MW backup generator wouldn’t have not been economical for the rare occasion a blackout happened.

            • JustAnEngineer
            • 2 years ago

            The site that I work at uses more than 250 MW. The critical loads connected to standby generators are less than 2% of the total.

            If you’re near one of the big 12 or 16-cylinder Caterpillar generator sets when it starts up, it will definitely get your attention. Hearing protection is recommended.

Pin It on Pinterest

Share This