Sony sandwiches DRAM into image sensors for better captures

Stacking layers of silicon is all the rage these days, and Sony is getting in on the party with a new image sensor for smartphones at the ISSCC conference. The company is using 3D fabrication to layer a 125MB DRAM cache in between a conventional backside-illuminated image sensor and the accompanying processing circuitry. That DRAM buffer solves some annoying problems with smartphone image capture and lets the sensor capture more frames per second of slow-motion video.

Those who have already used a device with an electronic shutter will be familiar with the "rolling shutter" effect, wherein the line-by-line readout of data from the sensor causes fast-moving objects to appear tilted or otherwise distorted. Sony says the DRAM buffer built into this sensor can read out the entire 19.3MP raw image in 1/120 of a second. That's four times faster than the company's own IMX318 sensor, and it does appear to have a major positive effect on action photos.

Putting a DRAM cache in close proximity to the sensor also lets the device capture slow-motion 1920×1080 video at up to 1000 frames per second. Most of today's phones can only capture 1920×1080 slow motion at around 120 FPS, so that improvement could offer a whole new window into fast action. The company demonstrated this new sensor's prowess with an impressive demo reel:

Sony didn't offer any information on commercial implementations of this technology, but given the popularity of its Exmor image sensors in high-end smartphones, we'd expect handsets with much-improved action and slow-mo capture to begin appearing on the market relatively soon.

Comments closed
    • psuedonymous
    • 3 years ago

    Embedded DRAM means per-pixel processing structure (to sample and store the pixel values in the adjacent DRAM cells). Per-pixel processing means you can do some super-cool stuff, like Contrast-sensing cameras (e.g. Inilabs’ DVS) and built-in image thresholding (cuts down on pre-processing if you’re using these as tracking cameras), and maybe even phase-based time-of-flight (if you can clock that hardware off an external source) e.g. Kinect 2.

    • GrimDanfango
    • 3 years ago

    It seems like Sony is so far ahead of the game with image sensors these days. It must almost be enough to convince a few professionals to loosen their death-grip on their beloved Canon/Nikon kit.

    Of course it would help immensely if Sony would put 1/100th the same R&D resources into developing their cameras’ user interfaces. I love my A6000 for the amazing results, utterly preposterous mirrorless-phase-detect autofocus times and 11fps burst fire (with autofocus between each shot)… but dear lord, I think they actually went backwards with the interface design.

      • Airmantharp
      • 3 years ago

      Sony would have to bother to put out a decent semi-pro mirrorless with usable ergonomics or upgrade their aging A-mount line for their DSLTs to get professional interest piqued- and then they’d have to follow through with a real support program.

      It’s not that they can’t do it, because by God that A99 II is a fine looking camera and that E-mount lineup is looking spectacular (if you don’t need anything particularly long), but God help them if they cross the streams and get one coherent system together!

    • TheEmrys
    • 3 years ago

    The ramifications of this technology is huge for the Sony camera business. When this tech is dropped into their fixed lens, mirrorless, or DSLT bodoes, it could be a real game changer in terms of gettimg a global shutter for video. Moreover, it might make buffer optimization for their large sensor (42mp in FF, 50-100mp in Medium Format) more efficient.

      • Airmantharp
      • 3 years ago

      This is what Canon is focused on as well; not so much for their still cameras, but to get true global shutters for their Cinema EOS line, which competes with Sony on up to RED and Arri.

      Of course, if Sony puts this in their smaller sensors, up to the 1″ mark, this technology will be in everything in the next few years regardless of what Canon does (since they buy sensors for their compacts from Sony, like everyone else).

      Hopefully we’ll get fully electronic shutters in the near future that both function globally and do not come with a DR/noise penalty that electronic first-curtain shutters come with today.

    • Hattig
    • 3 years ago

    The 960fps video capture is very nice, but I presume the reason it is limited in duration each time is because it fills up the embedded DRAM.

    The video encoding happens on the main SoC’s ISP (you’d need a ~1GB/s link to the image sensor though, thinking about 1080p120 capture, but 4-channel MIPI seems to support that), so you’d store about 20 1080p frames of data before filling that DRAM up. Maybe a bit longer more if you capture in 4:2:2 or similar (saves about 1/3rd bandwidth). That video indicates you can have around 30 consecutive frames of 960fps capture within a normal capture stream.

    Obviously this year’s 128MB DRAM cache on an image sensor is 2020’s 512MB DRAM cache.

      • the
      • 3 years ago

      But the 2020 cache will be for 4K resolution so buffer time won’t change at 512 MB capacity.

      • Airmantharp
      • 3 years ago

      And this all assumes you’re doing pixel binning- i.e., you’re just reading out a 1080p etc. ‘grid’ of pixels, and not downsampling the whole sensors output.

      And that reduces quality, *considerably*. They’ll need to back this up with upgraded processors to get the full potential out of tiny phone sensors.

        • chµck
        • 3 years ago

        That may not be as difficult as you might think.
        Take a look at the Nokia 808 or the Lumia 1020. Both had 41MP sensors that used the full sensor for 1080p video.
        The 808 is from 2012.

    • DPete27
    • 3 years ago

    What do you suppose the file size of a 1080p 1000fps video is?
    [add] That’s a rhetorical question. Obviously you can calculate it by extrapolating 1080p-60 video size per second. 5 seconds of 1080p-10000 is equivalent to 1:23 of 1080p-60.

    IMO, smartphone videos are already too big at 1080p-60. I usually drop my video settings to 720p @ 30fps.

      • Froz
      • 3 years ago

      A slow-motion video of 1080p recorded at 1000 fps will have exactly the same size as 1080p video recorded at 30 fps, for the same length of playback. The first video will just take a lot shorter time to record.

        • meerkt
        • 3 years ago

        I suppose it will be far less, because the difference between frames is smaller.
        Video codecs are generally based on storing the deltas: “following the motion” of the scene and extrapolating a frame from previous ones, then storing the difference from the actual frame, subject to the target quality settings.

        • David
        • 3 years ago

        That’s a silly measurement, though. You’re not going to record a 1000fps video for .03 seconds.

        Slow mo videos are huge. A camera like the Phantom will fill hundreds of gigs, at sub 1080 resolution, in seconds. A camera phone isn’t going to do that since it’s 1k fps and not 10 or 20k, and the codec won’t be as good, but it’s still going to be much larger than a standard 30fps video.

      • psuedonymous
      • 3 years ago

      With the tiny deltas between frames when recorded at high speed, a 1000FPS video will be a LOT more compressible than a 30FPS, or even 60FPS, video. You can greatly drop your I-frame rate. Motion vectors are the bomb, yo.

    • the
    • 3 years ago

    The concept of putting a DRAM buffer on an image sensor is something I had a long conversation about at RSNA last year. This was in the context of medical imaging using x-rays so things like 2 bit ECC on the DRAM were considered as well as a lead layer in the interposer. Things like this need to be considered since it is literally a target for high energy photons that can flip bits. The neat thing is that while the company had their own custom ISP block manufactured by TSMC on bulk silicon, the sensor itself was manufactured on exotic wafers (I want to say tellerium based) that they fabbed themselves. Really neat stuff as the bonding between two wafers of different types requires unique TSV techniques.

    Sony here is likely using a different process for the DRAM and sensor to do the stacking. It looks like Sony solved on this hard problem. Very impressive.

    • drfish
    • 3 years ago

    A phone packing that camera might finally get me to drop Window Mobile. o_0

      • chµck
      • 3 years ago

      I was already planning to dump Windows 10 Mobile when Nokia releases their new android phones after MWC2017.
      But now I need to know when we can expect these new sensors in devices.

        • Voldenuit
        • 3 years ago

        Aren’t the new ‘Nokia’ phones just Chinese Android phones with a nokia badge slapped on?

        I’m not expecting them to have the same… Finnish.

          • chµck
          • 3 years ago

          HMD, located in Finland, is run by one of the ex-heads of nokia product production and soon to be ex-microsoft head of mobile devices .
          They also have exclusive IP rights to nokia & certain microsoft IP
          I would say that it’s the combination of former nokia parts that have been laundered through microsoft and brought back together.
          More here:
          [url<]http://economictimes.indiatimes.com/opinion/interviews/nokia-handsets-to-ride-brand-recall-hmd-ceo-arto-nummela/articleshow/55808902.cms[/url<] [url<]http://www.nokia.com/en_int/news/releases/2016/05/18/nokia-signs-strategic-brand-and-intellectual-property-licensing-agreement-enabling-hmd-global-to-create-new-generation-of-nokia-branded-mobile-phones-and-tablets[/url<]

      • derFunkenstein
      • 3 years ago

      I bought a Lumia 928 dirt cheap a while back just to see what Windows Phone was like. For a phone that’s pushing four years old (at the time, 2 years old when someone sold it to me on Swappa), I’m really impressed with the camera. And that’s not even one of the super-fancy camera models.

    • Voldenuit
    • 3 years ago

    Saw ‘sony’ and immediately read that as ‘Sony sandwiches DRM into image sensors’.

      • Wirko
      • 3 years ago

      Yeah, must be a typo in the headline.

      • Chrispy_
      • 3 years ago

      Not just me then.

      • BurntMyBacon
      • 3 years ago

      I’m told by my peers that it says DRAM, but I can’t see it.

Pin It on Pinterest

Share This