Nvidia Kepler powers Oak Ridge’s supercomputing Titan

We don’t know for sure whether it will make its way into a graphics card or not, but the biggest variant of Nvidia’s Kepler GPU architecture doesn’t lack for customers. The GK110 chip, which we previewed earlier this year, is making its debut today as the primary flops contributor to Titan, a massive new supercomputer at Oak Ridge National Labs.

Titan consists of 200 cabinet-sized Cray XK7 supercomputers. All told, it includes 18,688 nodes, with each node comprised of a 16-core AMD Opteron processor and an Nvidia Tesla K20 GPU. Nvidia tells us Titan can achieve over 20 petaflops of peak performance, and over 90% of those flops come from its GK110 GPUs.

If those big numbers aren’t impressive enough for you, consider this. Titan is an upgrade from the prior supercomputer, Jaguar, which consumes seven megawatts of power in order to achieve two petaflops of throughput. Titan consumes a little more power, at nine megawatts, in the same physical space, yet it peaks at ten times the flops. That’s a huge claimed increase in power efficiency from one generation to the next.

Titan’s most basic component is a blade for the Cray XK7. Each blade houses four nodes, or four Opteron-plus-Tesla instances, aboard a low-profile, snap-in module. In order to fit into this form factor, the Tesla K20 is mounted on a custom card that’s shorter than a traditional graphics card—and nearly square. Each Tesla card includes 6GB of dedicated GPU memory. You can see four of those cards to the right in the picture above, with the copper heatsinks atop the GK110 GPUs. Across the middle of the blade are the heatsinks for the 16-core Opterons, and on the left are the two Cray Gemini interconnect units that attach the blade to the rest of the system.

Incidentally, the Opteron processors used in the system are dual-chip CPUs based on the Bulldozer microarchitecture. We asked Sumit Gupta, General Manager for Tesla Accelerated Computing at Nvidia, why those CPU were chosen for this project, given the Xeon’s current dominance in the HPC space. Gupta offered an interesting insight into the decision. He told us the contracts for Titan were signed between two and three years ago, and "back then, Bulldozer looked pretty darn good."

Titan is what’s known as an "open science" supercomputer, because researchers from across the nation are free to request time on the system for their projects. Accordingly, Nvidia and Oak Ridge have been working to accelerate some key applications for scientific computing, using both the CUDA and OpenACC APIs. Materials science application WL-LSMS is already purportedly running at a sustained rate of over 10 petaflops. Not surprisingly, Oak Ridge is swimming in proposals from scientists hoping to use Titan, to the tune of triple the time available.

Gupta refused to speculate where Titan might land on supercomputing’s Top 500 list, but he noted that the next revision of the list will be released on November 12, so we’ll have to keep an eye on it.

GK110 GPUs may be shipping to other customers under embargo, but the Tesla K20 hasn’t reached general availability yet. As we learned back in May, supercomputing and HPC clusters could potentially soak up the entire supply of GK110 chips—likely at some very nice prices, we should note—leaving few or none left over for GeForce cards. Nvidia may keep a limited quantity of GK110s in reserve, though, just in case the upcoming Radeon refresh steals away the consumer graphics performance crown.

Comments closed
    • Sam125
    • 7 years ago

    If I were a scientist, I would be so stoked right now. Although, IIRC Oak Ridge is a rather small laboratory and they’re getting to buy some new heavy equipment. I’d be curious as to what some of the larger laboratories like Sandia and are going to be upgrading to.

    Not that any of it matters to me, but supercomputers are super cool to me. Why? I have no idea.

    • jihadjoe
    • 7 years ago

    Maaan that’s a lot of GK110s!
    18000+ of them…

    I’m guessing this is one of the reasons we never got the chip on the desktop. If demand in the HPC segment is enough to basically empty Nvidia’s entire inventory of big Kepler, then there’s absolutely no reason to sell it as GTX780 (not to mention they get much higher margins).

    • MadManOriginal
    • 7 years ago

    Only 6 paragraphs until a purportedly.

    BEST ARTICLE EVAR!

    • BIF
    • 7 years ago

    I love heavy machinery!

    But this should be embarrassing for BOTH Intel and AMD:

    [quote<]...the contracts for Titan were signed between two and three years ago, and "back then, Bulldozer looked pretty darn good." [/quote<] Intel should be embarrassed because their hand-sitting allowed this to happen! AMD should be embarrassed because of the "back then" and "looked" parts. A pox on both their houses, as far as I'm concerned!

    • shank15217
    • 7 years ago

    Yea I bet Nvidia is pretty happy its not AMD southrn islands in those machines.

    • dpaus
    • 7 years ago

    [obligatory] But can it play Crysis? [/obligatory]

      • khands
      • 7 years ago

      More like “How many instances of Crysis could it stream?”

        • Arclight
        • 7 years ago

        I’m gonna go with 2 and they will probably be at under 30 fps.

        • RND8500
        • 7 years ago

        22,343 @ Resolution of 1800 x 1200 @32fps….unless they overclock Titan then they may squeeze out 22,346 instances of Cysis. What I want to know is will they use this on BOINC. lol. That WOULD be cool ! Could you imagine how high it would rank? Jaws would drop lol.

    • Sahrin
    • 7 years ago

    Technically it’s powered by Opterons with Kepler accelerators.

      • OU812
      • 7 years ago

      Technically it’s powered by Kepler accelerators with Opterons relegated to house keeping chores.

      [quote<]Titan can achieve over 20 petaflops of peak performance, and over 90% of those flops come from its GK110 GPUs.[/quote<]

        • Sahrin
        • 7 years ago

        Uh-oh…someone doesn’t understand compute architectures!

          • OU812
          • 7 years ago

          If you think POWERED means “Uses lots of Power but produces lower results” then you are right that AMD should be ranked first since the AMD Opterons only produce 10% of the 20+ petaflops.

          [quote<]Incidentally, the Opteron processors used in the system are dual-chip CPUs based on the Bulldozer microarchitecture. We asked Sumit Gupta, General Manager for Tesla Accelerated Computing at Nvidia, why those CPU were chosen for this project, given the Xeon's current dominance in the HPC space. Gupta offered an interesting insight into the decision. He told us the contracts for Titan were signed between two and three years ago, and "back then, Bulldozer looked pretty darn good."[/quote<] Funny. "back then, Bulldozer looked pretty darn good."

            • Sahrin
            • 7 years ago

            >If you think POWERED means “Uses lots of Power but produces lower results” then you are right that AMD should be ranked first since the AMD Opterons only produce 10% of the 20+ petaflops.

            You don’t understand what I’m saying.

            The Keplers are connected only to the Opterons – they are not directly connected to the system’s backplane. This means that the Opterons are the system’s CPU, and Kepler is used to execute tasks. Your interperetation is like exclaiming that Western Digital just released a new 4TB RAM stick and that the paltry 32GB of system RAM has nothing to do with performance.

            It’s not possible to build a supercomputer out of Keplers – you need to feed them pre-processed data. It’s an Opteron machine with Kepler accelerators. Sorry if that doesn’t gel with your nVidia colored worldview, but that doesn’t make it any less accurate.

            • just brew it!
            • 7 years ago

            Both interpretations are arguably correct. Yes, Opterons are used to implement the front end. But most of the raw computational capability does indeed come from the nVidia chips.

            • Sahrin
            • 7 years ago

            >Both interpretations are arguably correct.

            Which is why I started my post off with “Technically,” and not “FUCK YOU NVIDIA BLOWS.”

        • 5150
        • 7 years ago

        I am superior to you on a username/albumname level.

          • OU812
          • 7 years ago

          and I am younger and newer than you on a username/albumname level.

            • BIF
            • 7 years ago

            Both of you are getting your Signals mixed up. 😉

            • sweatshopking
            • 7 years ago

            I DON’T KNOW WHAT THIS MEANS

            • OU812
            • 7 years ago

            Van Halen album names: 5150 and OU812

            5150 (pronounced “fifty-one-fifty”) is the seventh studio album by American hard rock band Van Halen, released in 1986

            OU812 (pronounced “Oh you ate one too”) is the eighth studio album by American hard rock band Van Halen, released in 1988

      • Silus
      • 7 years ago

      No, technically and literally the Keplers do the vast majority of the work, as is described in this piece of news.

      • Deanjo
      • 7 years ago

      Technically it is powered by the electricity.

        • BIF
        • 7 years ago

        I was gonna say “We have a Winner!”

        Then I thought, “…which is technically powered by either Coal or Oil, with maybe a small bit of a nuclear reaction in the background.”

          • Deanjo
          • 7 years ago

          Actually it’s 100,000,000 million gerbils on wheels.

          • Grigory
          • 7 years ago

          So technically it is powered by the Big Bang. (I dare you to tell me what technically powered the Big Bang. No gods or magic allowed.)

            • sweatshopking
            • 7 years ago

            ALIENS!! DON’T YOU WATCH THE HISTORY CHANNEL?

            • RND8500
            • 7 years ago

            Sorry to burst your bubble but a large array of Kepler accelerators are what powered the big bang ! I call it “The Big Bang via Brute Force Computational Power” or BBvBFCP for short. Kepler’s are currently the only game in town capable of BBvBFCP. Looming in the future will be the next generation in graphics acceleration code named A Bridge Named Sandy* capable of BBBvBFCP or “Beyond the Big Bang via Brute Force Computational Power”

            * Not to be confused with Intel’s CPU code named Sandy bridge.

    • 5150
    • 7 years ago

    Should have named it “Elvira”.

      • derFunkenstein
      • 7 years ago

      wow. Well done.

    • Silus
    • 7 years ago

    Given the announced price for a K20 ($3199), with an order of this magnitude, NVIDIA would cash in almost 60 million dollars!
    Although I’m sure that for such high orders, there’s some sort of discount! Should still be quite a large sum of money!

      • willmore
      • 7 years ago

      It’s a custom design. Maybe that’s the ‘discount’?

        • Airmantharp
        • 7 years ago

        The modules aren’t terribly different looking than MXM modules used in portables, and Nvidia is damn good at making those. Integration is key with designs like these though, as you’d expect that each of these GK110 modules is probably less complicated than a K20 GPGPU.

    • cegras
    • 7 years ago

    By dual chip, is it a MCM?

      • Damage
      • 7 years ago

      Yeah, like a lot of recent Opterons, there are two chips in the big, wide package, which drops into a big, wide socket.

        • Essence
        • 7 years ago

        Just like the AMD Opteron processor were under contract, so were the Nvidia Tesla K20 GPU. There were talks of AMD GPU but nvidia would not allow (Will link later when i find it) it and the other were the software not optimized for AMD GPU`s “currently”, but thats not say in future AMD GPU`s will not be used over the half as good nvidia GPU`s if things keep going as they are e.g. AMD GPU is much faster in Compute and GAMES over ANY nvidia GPU. AMD has nvidia beat on all fronts but the low power mobile stuff and software optimizations but that will soon change also.

        Even the Amazon deal with nvidia was under contract from a year ago. Amazon also wanted AMD. Amazon even attended the recent ARM/AMD meeting on future AMD x64 (arm) CPU`s

          • Airmantharp
          • 7 years ago

          I think you’ve got your stories mixed up- Nvidia is beating the hell out of AMD GPUs with their mid-sized part in gaming, and they beat the hell out of them (and still are!) in GPGPU with their last-gen full-sized part.

          Keep in mind that what’s been branded as the GTX680 is the same level of silicon used in the GTX560. The GK110/K20 is the first we’ve seen of a successor part to the GTX580, and we won’t likely see it in desktops, with as slow as AMD’s full-sized GPUs are in comparison.

          Also, sorry for feeding the troll.

            • Essence
            • 7 years ago

            Have you even seen the GK110/K20 numbers? Dude they get punished in anything not overly optimized for nvidia. What planet are you living on? AMD has nvidia beat on ALL Graphics front, the only BS i ever hear is “Nvidia is beating the hell out of AMD GPUs with their mid-sized part”

            So how come the GK110/K20 is no better? and its the supposedly high end.. lol go be a troll elsewhere. Had the GK110 been any better or superior over AMD, trust me it would be all over “in your face”

            • Airmantharp
            • 7 years ago

            I’m confused- what GK110/K20 numbers?

            They’re GPGPU parts, not graphics cards. Why would you run something not overly optimized for them?

            And yeah, if Nvidia can beat AMD’s full-size part with their mid-range part, then AMD is behind. Why is this hard to understand?

            That’s three questions for you, if you can handle them. Specific answers need only apply.

          • Silus
          • 7 years ago

          Can you tell me where are AMD’s GPUs “much faster” than NVIDIA’s in games ? Oh that’s right, AMD has no such GPU. If anything, in the high-end gaming market (and single GPU) they are mostly tied in performance. It’s win some, lose some for both companies. NVIDIA does have a dual GPU, single card, which AMD lacks and it’s the best performer in games right now, but in single GPU it’s definitely a tie, with NVIDIA having the more power efficient GPUs. As for compute, AMD’s current high-end single GPU is better than NVIDIA’s current high-end single GPU, but NVIDIA’s high-end compute part is GK110, not GK104 and that’s what’s being used in K20 which is vastly superior to anything AMD has for GPGPU.

          Amazon is at whatever meeting by invitation, just as it was in NVIDIA’s meeting for Project Denver a few years ago.
          Also, can you show proof of this “Amazon wanted AMD” ? AMD fanboys tend to blur things as simple as “I want this to be true” with actual reality.

            • Essence
            • 7 years ago

            I have only one thing to say onto you “Cata 12.11 beta”

            Do you have a link for the GK110 being faster in GPGPU?

            There is no point in a card that cannot be produced on a mass scale (ask Intel). A few cards every wafer is just that = Desperation (Check how many have been supplied & still on order)

            AMD`s middle end cards beat nvidia high end card in GPGPU on the market today, you don`t see anybody singing or making a big deal about that and it is a big deal.

            • Silus
            • 7 years ago

            You mean like NVIDIA did just that with Fermi and you and your fanboy friends claimed GPGPU didn’t matter ? Now all of a sudden it’s a “big deal” ?
            Unlike you, I say GPGPU does matter, it’s just that NVIDIA decided that there’s no point in having compute behemoths in the gaming market, when it’s rarely ever used, plus the HPC market is much, much more profitable.

            As for your Cat 12.11 beta thingy, LOL…NVIDIA also boosts performance in their drivers, so what’s your point ?

            As for GK110, everything we know is based on its specs and GK110 trounces everything AMD has based on those specs and is more power efficient, which is also important in HPC and Titan is a good example of that.

            • Essence
            • 7 years ago

            Enough with the BS, since you cant provide any links and rather talk out your A@@ i provided one just to show what a fool you are.. Am still looking for the official nvidia PDF or a link from nvidia that shows what i said.

            [url<]http://vr-zone.com/articles/nvidia-s-top-end-kepler-unveiled-tesla-k20-comes-with-disappointing-specs-performance/17458.html[/url<] Am not even gonna argue with you since clearly you are a fanboy, i can go all day providing you with links that show what i said.. but theres no point as i said with retarded fanboys lol Edit Does logic or facts even compute with your kind? Don`t worry your just going through a phase called denial

            • Airmantharp
            • 7 years ago

            Where do I report you for trolling… seriously.

            Most of us here have run cards (and multiples of cards) from both companies. Some of us have been doing this since before there were only two!

            And some of us appeal to reason.

            • Airmantharp
            • 7 years ago

            AAAnnnddd your article states that the AMD GPU (assume their fastest) is at 4GFlops single-precision, and 1GFlop DP, while the Nvidia card runs 3.5/1.17, respectively.

            And that’s just raw numbers. Nvidia’s overall architecture is more efficient this time (than their last generation and both AMD’s current and previous), which would support their part being faster overall on non-optimized code, even faster when more DP is used, and significantly faster on code that is optimized to the same level for each vendor.

            Trollbegone.

    • chuckula
    • 7 years ago

    AMD’s feelings must be a little hurt over that headline.

      • dpaus
      • 7 years ago

      AMD: the Rodney Dangerfield of CPUs.

    • CampinCarl
    • 7 years ago

    Droooooooooooool.

    • Srsly_Bro
    • 7 years ago

    [quote<]All told, it includes 18,688 nodes, with each node comprised of a 16-core AMD Opteron processor and an Nvidia Tesla K20 GPU.[/quote<] [quote<]Across the middle of the blade are the heatsinks for the 12-core Opterons, and on the left are the two Cray Gemini interconnect units that attach the blade to the rest of the system.[/quote<] Excuse me if I missed something, but is it 12 or 16 core CPUs in Titan?

      • Damage
      • 7 years ago

      Sorry, 16 cores. Will fix!

        • Arclight
        • 7 years ago

        Too bad AMD won’t

      • RND8500
      • 7 years ago

Pin It on Pinterest

Share This