Google RAISR upsamples thumbnails for massive bandwidth savings

When we're browsing the web with our phones, bandwidth is usually at a premium. Even carriers that promise unlimited bandwidth often pin an asterisk to the end of that statement that reminds us that they don't really mean that. Google's new image compression tool might just help save you some of those precious 4G bytes. I'd like to introduce you to Google's most recent application of its machine-learning technology RAISR, or Rapid and Accurate Image Super-Resolution.

RAISR upsampling takes a thumbnail of an image and uses machine-learning algorithms to try and figure out what detail went into the original, larger version. This in turn allows Google to serve upsampled images that use up to 75% less bandwidth of the original, while still retaining most the visual detail.

Right now, Google is only applying the technology to its Google+ service, and even then only to a subset of the images on the service. This will change as Google starts to roll the technology out to more of its services "in the coming weeks."

If you've been watching HBO's Silicon Valley, this looks a bit like life imitating art. There's no word if Google is going to keep this technology for itself or release it to the public like Brotli, but it'd be neat to see it used in other image-heavy services like Facebook and Amazon.

Comments closed
    • Major-Failure
    • 3 years ago

    Hm, a machine “intelligently” deciding how to save bandwidth, it’s not like this ever caused Billions of unusable documents. You know, except in this on small case of Millions Xerox machines producing seemingly correct scans with barely noticeable number swaps.

    [url<]http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning[/url<]

    • GTVic
    • 3 years ago

    So collectively all of our phones and web servers now consume an extra 1.21 gigawatts of power to do all of this processing … which drastically slows down the web browsing experience that this was intended to speed up?

    • Waco
    • 3 years ago

    Please, please, someone make a new plugin for DirectX and OpenGL to do this for textures!

      • chuckula
      • 3 years ago

      While not identical, this scheme seems to have some similarities to old-school mipmapping.

        • Waco
        • 3 years ago

        Kinda, but that’s for downsampling. I want to bump up texture detail beyond what games offer without 35 GB texture packs and wasted VRAM. πŸ˜›

    • chΒ΅ck
    • 3 years ago

    Just replace .gif with .webm and i’ll be happy until 2050.

    • brucethemoose
    • 3 years ago

    “try and figure out what detail went into the original, larger version”

    That’s the idea behind the old superres filter (the same one you see in MadVR):

    [url<]https://github.com/zachsaw/MPDN_Extensions/blob/master/Extensions/RenderScripts/Shiandow.SuperRes.cs[/url<] I spent a ton of time researching upscalers to clean up a certain much-loved TV series, and I can tell you these "neural network" image doublers are no joke. It's freaking black magic. NNEDI3 is an older one y'all have probably heard of, and that already looks way better than sinc-based scaling. More recently, Waifu2X (which is what I used) has emerged as a image doubler for animation, and it's nothing short of miraculous. Did Google publish the source code? I'd love to stuff this into a vapoursynth plugin and see how it works with live action video...

    • jts888
    • 3 years ago

    If RAISR can actually reliably get solid PSNRs, why hasn’t this already been put forward as a compressed image format standard, where the residual deltas can be selectively compressed and included?

    It seems like there would be a lot more to gain as an stored image format than as a mere un-downscaler filter on viewer apps.

    • juzz86
    • 3 years ago

    One quarter, America. Not one fourth. Haha.

      • Meadows
      • 3 years ago

      You can’t write “quarter” into “1/4” however, and it wouldn’t be Google if there wasn’t a maths expression in there somewhere.

        • juzz86
        • 3 years ago

        Oh I’m just being a grizzly old pedant Meadows, haha. It was the only thing that really jumped out at me about the whole story, to be honest πŸ™‚

    • meerkt
    • 3 years ago

    A better general purpose strategy would be to finally start adopting a better-than-JPEG format:
    BPG (based on H265).

    Specs and info: [url<]http://bellard.org/bpg/[/url<] Visual comparison: [url<]http://xooyoozoo.github.io/yolo-octo-bugfixes/[/url<] All previous purported better-than-JPEG formats (JPEG2K, HDP/JXR, WebP) were to my eyes worse than JPEG or the same, besides at abysmal quality settings. BPG is the first to look better.

      • morphine
      • 3 years ago

      <xkcd_standards.jpeg>

      • LostCat
      • 3 years ago

      Probably true, but getting devs to actually adopt it seems unlikely.

      • jts888
      • 3 years ago

      HEVC/h.265 is a litigation minefield, with separate groups trying to license IP rights:

      [url<]https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#Patent_License_Royalties[/url<] The forthcoming AOMedia Video 1 (AV1) format is allegedly competitive with HEVC and being backed by most of the heavy hitters of new media (Google, Microsoft, Mozilla, Intel, AMD, Nvidia, ARM, Netflix, Amazon, Cisco, ...) who want lower bandwidth and no royalties. An improved (and universally hardware accelerated) image format based on video I-frame encoding is more likely to spring from here IMO.

        • jensend
        • 3 years ago

        To add to your point:

        Xiph/Mozilla’s Daala codec, which is one of the sources for AV1, has had all kinds of new ideas. Some of those ideas ended up causing impractical complications when combined with motion compensation for video and thus won’t be used in AV1.

        But [url=https://jmvalin.ca/papers/icip2016.pdf<]for still images[/url<], Daala [url=https://jpeg.org/downloads/aic/wg1n73041_icip_2016_grand_challenge.pdf<]already does about as well as HEVC/h.265/BPG[/url<], considerably better than WebP, JPEG2K, etc, and could be considerably further improved without bumping into any patent minefields. Note also that while JPEG with normal encoding options is beaten by a lot of other formats, only Daala and BPG are noticeably better than a smart JPEG encoder at any but the lowest bitrates they tested. JPEG isn't going away any time soon.

        • meerkt
        • 3 years ago

        Isn’t everything? πŸ™‚ But everyone still use/used GIF (before it expired)/JPEG/MPEG4 ASP/H264…

        If Daala intra can work as well, that’s good too.

          • jts888
          • 3 years ago

          Being a non-niche/toy web format requires being supported by default in all the major browser vendors plus the default image viewers and file manager shells on Windows and OS X.

          The companies who make and distribute said products are much larger and solid lawsuit targets than a small group of people who make a codec pack for torrented anime fansubs or whatever, and MPEG-LA is a bigger and more practiced and eager litigant than Forgent or Unisys. Really, AV1 is [i<]the[/i<] existential threat to MPEG-LA and all its members, who have successfully sought for decades steady revenue streams from owning the rights to lingua franca audio/video formats, which is in effect not being threatened a little bit, but completely and perhaps forever. There's gonna be a throwdown as soon as MS/Google/Mozilla try to put it in browsers or Intel/AMD/Nvidia/ARM release hardware/blobs/drivers to accelerate it, but I'm optimistic about its chances to survive.

      • Meadows
      • 3 years ago

      That may be, but is it four times more compact? You’re not going to help mobile devices that much if you only save 15% space.

      • Chrispy_
      • 3 years ago

      Wow, I was unaware of BPG.

      I looked at most of those comparisons quality is comparable when the BPG is around half the size of the JPG. At the very highest compression levels, the BPG still looks acceptable whilst JPEG is little more than an 8-colour block mess. At those super-compressed filesizes I’ll willingly trade some fine detail to get clean edges and smooth colour gradients!

        • Spunjji
        • 3 years ago

        Not to mention that visual examination indicates it finally fixes JPG’s weird obsession with ruining any significant block of red.

          • meerkt
          • 3 years ago

          JPEG doesn’t inherently ruin reds and other pure colors. Many encoders just use 4:2:0 chrome sampling by default without giving you any control, but in decent ones you can choose 4:4:4.

          Too bad that visual comparison page doesn’t give you control over that, also for BPG.

            • jts888
            • 3 years ago

            Subsampling is part of but not all the story for JPEG color issues.

            The encoding process actually operated on data converted to the Y’C[sub<]B[/sub<]C[sub<]R[/sub<] colorspace, doing independent lossy quantization on each of those channels' discrete cosine transform outputs. Encoders almost uniformly favor the luminance channel heavy over the two chroma channels, which they treat at lower bit rates increasingly as red-headed stepchildren. And while I don't know enough to quantify the real degree of significance, the Y'C[sub<]B[/sub<]C[sub<]R[/sub<] chroma plane does seem to expend a lot of its capacity on blues and magentas, which don't really tend to show up in nature as much as reds-yellows-greens, and our eyes or brains don't really seem to care as much about their subtleties.

            • meerkt
            • 3 years ago

            I haven’t explored in depth low quality JPEG encoding, but with 4:4:4 and high quality settings you can get visually lossless results even when flicking back and forth from the original. So it can be done. While you have to spend more bits the compression ratios are still decent.

            I wonder if future formats will lean toward YCgCo.

            • jts888
            • 3 years ago

            I couldn’t tell you what the future will actually be, but I like the approach that FreeSync 2 is taking, of handling multiple colorspaces internally in source devices and explicitly treating everything on the cable and in the display as just a device-specific RGB space that’s already had all content fully tonemapped for.

            Something like YCgCo might have a place here in lossy display compression formats, since it’s a simpler linear transform of the RGB cube than Yβ€²C[sub<]B[/sub<]C[sub<]R[/sub<] but still yields a reasonable luminance axis to favor and chroma axes to throw away more information from. As far as storage format are concerned, it would be nice if color spaces just became more commonly explicitly defined and handled.

      • meerkt
      • 3 years ago

      BTW, I still don’t understand why JPEG-LS hasn’t become a thing. It’s generally compresses similarly to or slightly better than JPEG2K lossless, but it’s much quicker to encode and decode.

      • synthtel2
      • 3 years ago

      I think [url=http://flif.info/<]FLIF[/url<] is more interesting, myself.

        • meerkt
        • 3 years ago

        FLIF is interesting too, and they say it’s better than BPG lossless, but the web needs lossy more than lossless.

          • synthtel2
          • 3 years ago

          BPG definitely wins on practicality for the average user in 2017, but if bandwidth and storage are going to substantially outpace monitor and sensor resolutions (probable, I think), developments on lossless algos seem more interesting for the future. I’m thinking future as in what should be standard a decade from now, since nothing like this ever changes quickly.

            • meerkt
            • 3 years ago

            I wonder if that’s going to happen anytime soon. In theory it could or should have happened already; since people started amassing photos 15 years ago HDD capacities have increased by 1-2 orders of magnitude. Yet, lossy formats are still the default for most people.

            • synthtel2
            • 3 years ago

            I expect at this point it has more to do with inertia and bandwidth, rather than storage. A terabyte can fit an awful lot of FLIF or FLAC – video and games seem like the main things that can fill it up quickly. Lossless defaults would be a big problem for people on cheesy DSL or with tiny data caps, though. On a related note, having a ton of data to back up can get awkward.

      • meerkt
      • 3 years ago

      How come every comment in this thread is plused?! πŸ™‚

        • helix
        • 3 years ago

        Perhaps the thread is unusually constructive and informative?

        • caconym
        • 3 years ago

        that’s just a compression artifact

      • Growler
      • 3 years ago

      The biggest question is whether the file format acronym is pronounced with a hard or soft B. It could be the second coming of the .gif pronunciation war.

    • Eggrenade
    • 3 years ago

    You’ve misquoted the acronym; RAISR should stand for “Rapid and Accurate Image Super-Resolution,” not “Rapid and Accurate Super Image Resolution.”

      • morphine
      • 3 years ago

      No, we didn’t! It’s correct in the article!

      (Fixed πŸ™‚ )

        • Meadows
        • 3 years ago

        You did fix it, but now it appears with the white background even on the blue site template.

          • Meadows
          • 3 years ago

          I know you don’t work at weekends but come on.

    • Vhalidictes
    • 3 years ago

    Wow, this is so cool! So much so that neither Opera nor Amazon came up with it first… wait…

    • JosiahBradley
    • 3 years ago

    NO NO NO! I have unlimited bandwidth for a reason, I don’t want smaller images with upscaling I want the original. If this starts happening in more places users will simply get used to the terrible new quality standards like when people started watching streaming video versus solid physical media.

      • Ari Atari
      • 3 years ago

      But it’s such a great marketing strategy! Force everyone to use less than great quality images for a while, then bring back the old uncompressed images, call it Coke Classic… err high fidelity viewing, and sell them at a higher cost. People then pay more for nothing!

        • David
        • 3 years ago

        I know you’re joking, but you already have to pay for them to host uploaded images “as is” by way of a Google Drive subscription. Otherwise you get the compressed “HD” version.

          • Ari Atari
          • 3 years ago

          Ah, I did not know that. I don’t happen to use Google+ or any social media. I thought it was a thing like Dropbox, so that is even less appealing if you have to pay to get rid of compression so that you can have compression and you pay to get rid of the compression.

      • lilbuddhaman
      • 3 years ago

      More bandwidth? Haha no! We’ll spend billions on technology to cut bandwidth usage before we spend billions to upgrade infrastructure! Wouldn’t want those little lemmings using that fiber for something else, like home business.

      And just to get conspiracy theorist on you guys… Take this tech plus this one [url<]https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html[/url<] ... and they can start putting subliminal messages into their "optimized" images. Its like the plot from Josie and the Pussycats!

      • Duct Tape Dude
      • 3 years ago

      I’m respectfully handing you a downthumb. Here’s why:

      This technology is obviously not meant for your “unlimited bandwidth,” instead it is for people loading a webpage faster on limited bandwidth. This is the opening line of the article. If they save 75% bandwidth while retaining acceptable image quality, why not? The algorithm looks markedly similar to upscaling done for digital media anyway, but does it smarter.

      You realize that by diminishing quality standards, we now have access to FAR more content? Who wants to exchange RAW/png images to share pictures of their cats? This obviously is not meant for your original family photos, but if I get to browse memes 75% more efficiently, why not?

      As processing power increases, so too does the data compression, decompression, and enhancement. This happens to be a novel enhancement technique (ie: via machine learning) that requires more processing power than standard upscaling, and the results look decent. I really doubt that I’d notice on my phone screen, and the results are superior to any upscaling my phone would do in the first place.

      Lastly, to be pedantic, no one has unlimited [i<]bandwidth[/i<]. You might not have [i<]data caps[/i<], and ISPs might have tons of bandwidth for your individual needs, but internet bandwidth itself is a physically limited resource--particularly over the air.

        • curtisb
        • 3 years ago

        [quote<]while retaining acceptable image quality[/quote<] Define [i<]acceptable[/i<] image quality. I think what we already have is less than acceptable because we're already compromising for both restricted bandwidth and restrictive data caps. The real problem is that we still have areas that are only served by dial-up or really slow DSL. In some locations it's not even that the DSL has to be slow, it's just that the higher tier packages cost too much. EDIT: typo

        • TheJack
        • 3 years ago

        Add to it, that there would most probably be an option to disable this feature, if an image happens to be important. I am all for it.

        • Wirko
        • 3 years ago

        But maybe he actually has unlimited bandwidth … and a data cap. One careless moment and he burn up all his terabytes for the whole month. That’s quite unfortunate.

      • jstern
      • 3 years ago

      Some people don’t have unlimited bandwidth. So any technology that can save bandwidth is a good thing.

      • Ninjitsu
      • 3 years ago

      YEAH! TO HELL WITH THE ADVANTAGES THAT THE OTHER 6 BILLION PEOPLE MIGHT ENJOY! #MURICA

      • p3ngwin
      • 3 years ago

      you probably live in a 1st-world country, compared to many *billions* of people who do not.

      This is for the rest of the world, like the BRIC countries, who do not have the luxury of unlimited data, or even blazing fast hardware, in their hands.

        • Voldenuit
        • 3 years ago

        [quote<]you probably live in a 1st-world country, compared to many *billions* of people who do not.[/quote<] Also, ppl like me who are on T-mobile with half a bar of reception, if we're lucky.

    • DPete27
    • 3 years ago

    Pssh. CSI Miami has had this technology since 2002. What gives!

      • Rikki-Tikki-Tavi
      • 3 years ago

      Came here to say this.

      • nerdrage
      • 3 years ago

      “Enhance!”

        • Waco
        • 3 years ago

        [url<]https://www.youtube.com/watch?v=KiqkclCJsZs[/url<] Also, lol: [url<]https://www.youtube.com/watch?v=Vxq9yj2pVWk&list=RDKiqkclCJsZs[/url<] Dammit: [url<]https://www.youtube.com/watch?v=xbCWYm7B_B4&index=7&list=RDKiqkclCJsZs[/url<]

Pin It on Pinterest

Share This