Intel reveals details of its Omni-Path Architecture interconnect

One of the toughest challenges in computing is building large-scale systems like those used in the world of high-performance computing (HPC). Currently, such systems typically communicate between nodes over an interconnect like Ethernet or Infiniband, but Intel intends to replace those technologies with a new interconnect known as Omni-Path Architecture. This week at the Hot Interconnects conference, the firm is unveiling detailed information about how Omni-Path works and how it will be deployed alongside Intel's other products.

Omni-Path is interesting in part because of the intense difficulty of the problem it has to address: allowing the hundreds or thousands of nodes in a supercomputer to communicate with one another at high speed while keeping delays at a minimum. Intel says Omni-Path is meant to scale from "small clusters to the largest supercomputers." To make that happen, the chipmaker has taken on the task of designing nearly every piece of the puzzle needed to establish a new interconnect fabric, from the host adapters to switches, software, and tools. Some of the core technology comes from Intel's acquisition of Aries IP from Cray, and other bits come from its acquisition of True Scale InifiniBand IP from QLogic. Much of the software comes from the existing Open Fabric Alliance. At the end of the day, though, Omni-Path looks to be Intel's own product and very much a rival for other standards—and for products from firms like Mellanox.

Omni-Path has an inside line because it comes from Intel, maker of the Xeon lineup—the dominant CPUs in the data center—and of the Xeon Phi parallel processor. The first Omni-Path products will include discrete host adapters that fit into PCIe slots, but Intel plans to integrate Omni-Path connectivity on to the same package as the Xeon and Xeon Phi—and eventually directly into the processors' silicon. The purported benefits of this integration will be familiar to anyone who's been watching CPUs in the past ten years: lower point-to-point latency thanks to the elimination of a chip-to-chip "hop" in the network, lower power use, a smaller physical footprint, and, of course, lower overall costs. Integration also has the side-effect of making life very difficult for competing interconnect standards that are not integrated into the processor.

At a press preview event during last week's Intel Developer Forum, we got a first look at an Omni-Path host adapter in the form of a PCI Express x16 card. This card has a single port onboard and can support transfer rates up to 100 Gbps. Cards based on this Intel network chip can have one or two ports. The single-port cards typically require 8W of power, with a maximum up to 12W.

Omni-Path supports four "lanes" per connection, and each of those connections has a 25 Gbps transfer rate, for 100 Gbps in aggregate. Those connections can happen over optical links or over copper cables up to three meters in length. Even though it's a next-generation interconnect, Omni-Path supports copper cabling in order to keep costs in check.

Most of Omni-Path's unique magic happens at the lowest layers of the network stack, traditionally known as layers 1 and 2 in the OSI model. Above that, the interconnect will support existing transport and application standards.

The most interesting innovation in Omni-Path may be the insertion of a layer "1.5," known as a link transport layer or LTL, that grew out of work formerly being done at Cray on the Aries interconnect. According to Omni-Path Chief System Architect Phil Murphy, this additional layer breaks packets into 65-bit units known as "flits," and set of 16 flits together with a CRC comprises a packet. Busting packets into smaller units gives Omni-Path the ability to keep transfer latencies low by allowing high-priority packets to pre-empt large, low-priority packets that "we've already started putting on the wire," in Murphy's words. Flits from different packets can be interleaved with one another as needed in order to make sure that critical messages reach their destinations as quickly as possible. Which is very slick.

I'm barely scratching the surface here. If you'd like more info, Intel has released a paper on the OPA fabric, but you have to register at this website in order to download it. I believe IEEE Spectrum will be publishing a paper on Omni-Path, as well.

This interconnect is clearly aimed at a particular mission in a relatively small but lucrative market. Still, the fact that Intel intends to integrate this tech into its CPUs immediately raises intriguing prospects for future uses of this technology—or something very much like it—as part of future system architectures generally, which makes it very much worth watching.

Comments closed
    • ronch
    • 4 years ago

    If AMD intends to play in the server market with Zen they would need some sort of similar interconnect technology. Did they get it with the Seamicro acquisition?

    • lycium
    • 4 years ago

    I’m somewhat horrified to remember that APSTNDP stuff from the OSI model… and now they want to introduce another layer! All of those higher-level ones are so redundant and no one has ever given me a clear explanation of what the “presentation” layer does…

    Anyway, it’s about 10 years too late for me to be moaning about this, and it’s nice to see changes at the lower levels 🙂

      • dmjifn
      • 4 years ago

      “Presentation”? Like on a plate? Man, everyone knows the 7 layers are rice, beans, cheese, lettuce, tomatoes, sour cream, and guac!

      • Ninjitsu
      • 4 years ago

      OSI’s a pretty cool template and reference model, though. I quite like it.

    • Ninjitsu
    • 4 years ago

    I was looking at another site and they had a slide up that reads:

    “2nd generation Intel OPA with Knights Hill”.

    Below that section it says:

    “1st processor/fabric integration on Knights Landing”.

    So is Knights Hill a typo or what?

    • Ninjitsu
    • 4 years ago

    Intel gouging prices like Thunderfial. We need more competition guys, Intel is resting on its laurels.

    /s

      • Pwnstar
      • 4 years ago

      lol at “Thunderfail”.

    • DrCR
    • 4 years ago

    Meh, just Intel trying in vain to compete with KillerNIC.

      • Meadows
      • 4 years ago

      KillerNIC has nothing to do with either this or the market in question.

        • odizzido
        • 4 years ago

        I think he knows that 😛

        • tipoo
        • 4 years ago

        thatsthejoke.png 😛

          • Meadows
          • 4 years ago

          Wasn’t a very good one then.

          • albundy
          • 4 years ago

          tough crowd?

            • Meadows
            • 4 years ago

            “Tough crowd” is an excuse for when your jokes are poor.

      • divide_by_zero
      • 4 years ago

      Ha! Needs more +’s. FTFY.

    • the
    • 4 years ago

    [quote<]Still, the fact that Intel intends to integrate this tech into its CPUs immediately raises intriguing prospects for future uses of this technology—or something very much like it—as part of future system architectures generally, which makes it very much worth watching.[/quote<] Intel interestingly enough didn't strongly define the physical layer of their QPI interconnect that is currently used in-between processors. Enabling a low level, cache coherent path across a cable between chips opens up some interesting possibilities for scalability. The real question for this is how long the cabling could be between nodes before it is physically too long to maintain latencies suitable for coherency. Before it goes fully on-die, I predict they'll be using [url=http://www.intel.com/content/www/us/en/foundry/emib.html<]EMIB[/url<] to bring it on package. Much of this was due to how their silicon photonics process was different than their bulk process. Regardless, this Omnipath Fabric still seems prime to utilize both silicon photonics as well as EMIB in the future considering Intel's public disclosure of bring it into the CPU package.

      • chuckula
      • 4 years ago

      Hrmm… I see the resident anonymous downthumb idiot decided to attack your post.

      Corrected.

      • Kurkotain
      • 4 years ago

      What.

        • tipoo
        • 4 years ago

        long cable get not fast

          • the
          • 4 years ago

          You are my hero.

      • willmore
      • 4 years ago

      The problems:
      1) I’m pretty sure QPI is very well defined–by Intel and for internal use only.
      2) Omni-Path isn’t a memory to memory interface with coherency, it’s a message passing interface more like ethernet.

      You might be right about EMIB, though

        • the
        • 4 years ago

        1a) QPI’s higher levels are rather well defined but the physical layer, while part of the spec, is independent from the higher layers. At some point, Intel was hoping to offer QPI over optical cabling.
        1b) QPI is actually used by a handful of outside companies like Altera for a fast link to FPGAs… oh wait. Companies like HP, Lenovo (IBM), and SGI have a QPI license to build massive coherency fabric. Want a 256 socket server with full cache coherency? [url=https://www.sgi.com/products/servers/uv/uv_3000_30.html<]SGI has one[/url<] and they needed a QPI license from Intel to build it. 2) The physical layer of Omnipath would be useful here. The higher levels of QPI would just tunnel over the lower Omnipath layers. If there is going to be an Omnipath interface directly in every package on a server to connect to other servers which also have the same on package interface, why not? Granted there are limitations like latency which would break QPI, especially with long cable runs, but similar [url=http://www.channelregister.co.uk/2009/11/27/ibm_power7_hpc_server/<]technologies already exist on other platforms.[/url<]

          • Pwnstar
          • 4 years ago

          Altera is not an outside company, though.

            • the
            • 4 years ago

            You missed the joke. Also Altera had a QPI license before Intel bought them.

    • FrankJR
    • 4 years ago

    All the easier to make self aware AI with…

    • cmrcmk
    • 4 years ago

    Since the max throughput of a PCIe v3 x16 link is theoretically 126 Gbps, a second port wouldn’t help for throughput, only redundancy. I wonder if this interconnect is putting pressure on the uncore guys to get PCIe v4 online?

      • willmore
      • 4 years ago

      Not if the next step is to move the link on chip. Then the PCIe from the processor becomes less important as it’s no longer handling this traffic at all.

      These cards are just a stopgap for testing and development. We may never see them in actual production systems–those will wait for the on-chip integration.

    • Bauxite
    • 4 years ago

    Infiniband is surprisingly cheap for the bandwidth (which spills into the grey/2nd hand market nicely) to the point where 10GbE really doesn’t make any sense for a lot of configurations. It is definitely superior to ethernet as well, its closer to pci in terms of latency etc.

    It also has a lot of support and well established backing in the market, intel is going to have to fight pretty hard here on price IMO, they won’t be able to get by just doing their usual (forced) bundling and ‘feature’-pimping.

      • Kurotetsu
      • 4 years ago

      This post intrigued me, so I hit up Ebay to look up the prices of Infiniband parts:

      [url<]http://www.ebay.com/sch/i.html?_sacat=0&_nkw=infiniband&_frs=1[/url<] Adapter cards look dirt cheap, cabling not so much, and switches seem to swing wildly on the price range (DDR switches are fairly cheap, QDR switches not so much, and I imagine FDR-10 and up will get worse). Still nice to see such specialized tech becoming so affordable even though its not terribly useful outside of its niche.

      • crystall
      • 4 years ago

      [quote<]It also has a lot of support and well established backing in the market, intel is going to have to fight pretty hard here on price IMO, they won't be able to get by just doing their usual (forced) bundling and 'feature'-pimping.[/quote<] Yes, they'll have to compete on merit as InfiniBand is stable, cheap and well known technology at this point. They're ticking all the right boxes though including having already provided software support via the ubiquitous OFA API which means that most of the existing RDMA-enabled stacks should run on top of OP without modification.

    • tsk
    • 4 years ago

    Pretty sure I need this for my HTPC.

      • qasdfdsaq
      • 4 years ago

      Pretty sure I need this for my [i<]HPC[/i<]. Wonder if it will be mass-market ready in the next year or two, when our cluster is being replaced.

        • CampinCarl
        • 4 years ago

        Considering that Aurora will be deployed in 2016, I would say that if you can wait ’till 2017 it would almost assuredly be mass market by then. Typically, GA isn’t far behind when the big vendors (Intel, NVidia, AMD, etc.) build a system or two for one of the NLs.

      • chuckula
      • 4 years ago

      So I need at least two of these things to make my tablet stream from youtube right?

Pin It on Pinterest

Share This