AMD’s Epyc 7000-series CPUs revealed

AMD has been in a data-center desert for many years. Abu Dhabi Opterons marked the company’s last serious foray into the market in 2012, and since then, Intel’s market share of all server chips has been at least as high as 99.2%. Long-running murmurs of ARM chips disrupting Intel’s lock on the data center haven’t translated into major threats to Xeons yet. AMD’s own effort to produce ARM cores that were socket-compatible with its x86 CPUs, called Project Skybridge, never produced a shipping product, and the company abandoned it in 2015. The last update to AMD’s data center roadmap promised the high-performance K12 ARM core for this year, too, but we haven’t heard a peep of it since. Despite those various pretenders to the throne, x86 remains the dominant instruction set in the data center, and Xeons are its avatar.

This year might be different. AMD now has a unique arrow in its quiver that other companies don’t: a competitive high-performance x86 architecture. Zen has proven a capable and energy-efficient performer in our tests of AMD’s Ryzen desktop CPUs, and now the company is taking the fight to Intel in the data center with its Epyc CPU lineup. We already covered some details of this potential resurgence in our preview of the Naples platform. Today, AMD is revealing its chip lineups for two-socket and single-socket servers, as well as some projections of those chips’ performance relative to Intel’s Broadwell Xeons.

The starting lineup

To build each Epyc package, AMD uses four eight-core modules connected using its Infinity Fabric interconnect. That gives Epyc CPUs a maximum of 32 cores and 64 threads per socket, eight memory channels to DDR4-2666 ECC RAM per socket, and 128 lanes of PCIe 3.0 in total.

Not every server needs that much horsepower, of course, so AMD has sliced and diced that basic package into a variety of products designed to serve prices ranging from $400 and up to $4000 and up. All Epyc CPUs will offer the same memory and PCIe provisions, and some will offer configurable TDPs for more flexibility in system design.

AMD will also offer a subset of Epyc CPUs designed for one-socket servers only. Just like their two-socket relatives, Epyc single-socket CPUs will offer 128 lanes of PCIe 3.0 and eight memory channels. Single-socket operation is the only restriction for these chips, and they’ll be available for prices from $700 and up to $2000 and up.

Zen dresses up for business

Although the fundamental Zen core in Epyc is basically the same as that found in desktop Ryzen chips, some of its features are more important for server-class workloads than they are for clients.

The first of these is a virtualized APIC, or Advanced Programmable Interrupt Controller. AMD says this feature helps servers running VMs reduce world switch latency (the state change involved when a system switches from executing guest applications to hypervisor operations) by 50% compared to the Bulldozer architecture and its derivatives.

Epyc CPUs will also make extensive use of the AMD Secure Processor, a dedicated microcontroller embedded on the chip. This chip creates a secure environment that can be used to perform useful features like hardware-validated boot, cryptographic key generation, and key management.

The Secure Processor’s key-generation and key-management capabilities will be useful in implementing a feature AMD calls Secure Memory Encryption. Operating systems and hypervisors can request a key from the SP to encrypt sensitive pages, protecting data in flight from being intercepted through attacks on physical memory.

Epyc CPUs will also offer a feature called Secure Encrypted Virtualization that will isolate data owned by hypervisors, virtual machines, and containerized applications from access by other guest environments on a system. With such an arrangement, an attacker in one guest environment wouldn’t be able to read data in memory owned and encrypted by another guest, for example. Epyc CPUs will also offer hardware acceleration for the SHA-1 and SHA-256 hashing algorithms.

 

Potentially Epyc performance

AMD is making some claims about the performance of various Epyc products’ performance today, and the initial outlook is good. However, we do have to take issue with a couple of the choices AMD made on the way to its numbers. After compiling SPECint_rate_base2006 with the -O2 flag in GCC, AMD says observed a 43% delta between its internal numbers for the Xeon E5-2699A v4 and public SPEC numbers for similar systems produced using binaries generated by the Intel C++ compiler. In turn, AMD applied that 43% (or 0.575x) across the board to some publicly-available SPECint_rate_base scores for several two-socket Xeon systems.

It’s certainly fair to say that SPEC results produced with Intel’s compiler might deserve adjustment, but my conversations with other analysts present at the Epyc event suggests that a 43% reduction is optimistic. The -O2 flag for GCC isn’t the most aggressive set of optimizations available from that compiler, and SPEC binaries generated accordingly may not be fully representative of binaries compiled in the real world.

Still, here are AMD’s own numbers for two-socket systems. If we take the company’s assumptions at face value, Epyc would appear to offer large (or even massive) boosts over Broadwell Xeons from top to bottom. The Epyc 7601 dual-socket system handily beats out the pair of 22-core, 44-thread Xeon E5-2699A v4 chips in these benchmarking conditions, and every Epyc chip duo enjoys wide margins of victory over the competition.

We wanted to see how the results might shake out with a different set of assumptions around the Intel SPEC results that AMD started with, though. Friend-of-TR David Kanter suggests that a 20% reduction to public SPEC numbers for Intel CPUs is fairer considering the impacts of Intel’s compiler on the “libquantum” portion of the benchmark and the use of optimizing compilers in general. Accordingly, I rejiggered AMD’s numbers with that in mind. (I’ve applied a 1/0.575x increase to AMD’s own E5-2699A v4 results with a GCC binary to keep things consistent before applying David’s suggested handicap.)

With Intel’s compiler results adjusted this way, two-socket Broadwell systems generally come out slightly behind to moderately behind versus Epyc parts when running SPECint_base_rate2006, especially at the lower end of the market. AMD’s systems still win out, but the differences in performance seem closer to our real-world examinations of Zen and Broadwell performance.

Since code compiled with GCC and ICC coexists in the wild, the truth of Epyc performance likely lies somewhere between these poles. SPECint_rate_base2006 is just one synthetic benchmark, as well, and we’re eager to see real-world testing results with applications like databases, web servers, and so on.

It’s also worth noting that Skylake server parts are coming (although we’re not sure exactly when yet), and those parts will almost certainly compare more favorably to their Epyc counterparts when more details become available. Still, the fact that Epyc merits direct comparisons to Xeon performance is great news. We’ll have more about AMD’s server parts to share soon.

Comments closed
    • shutkajri18
    • 2 years ago

    Cool discussion but do you wanna check these hot models, if yes then go to /hotwalpapers.com\

    • POLAR
    • 2 years ago

    Meanwhile, Intel’s HT..
    [url<]https://danluu.com/cpu-bugs/[/url<]

      • raddude9
      • 2 years ago
    • just brew it!
    • 2 years ago

    So are we any closer to figuring out what causes instability when large parallel compilations are run on Ryzen with the gcc toolchain? It sounds like a potential CPU or platform issue to me, and until it is understood, i’m somewhat reluctant to buy Zen for personal use, and would not recommend it for corporate use.

    (Note: This is coming from someone who currently has an FX-8350 as my primary desktop, and whose personal desktop systems have been exclusively AMD-based since the K6 days. So it’s not like I’m an Intel fanboy or something…)

      • raddude9
      • 2 years ago
        • just brew it!
        • 2 years ago

        Most of the posts in that 9-page thread (even the latest ones) seem to be people just stabbing in the dark and trying random stuff, with mixed results. Is there a specific post where the problem is acknowledged by AMD and a reliable workaround is suggested?

      • credible
      • 2 years ago

      No word of a lie I know very little about programming but after reading your post I googled and found this thread, I found it very interesting and made me feel like I could actually do this stuff and enjoy it…but does this thread have anything to do with the points you are raising because I see no complaints from the posters.

      [url<]https://www.reddit.com/r/Amd/comments/652irm/question_ryzen_benchmarks_focused_on_programming/[/url<]

        • just brew it!
        • 2 years ago

        Talking about this issue:
        [url<]https://www.phoronix.com/forums/forum/hardware/processors-memory/955368-some-ryzen-linux-users-are-facing-issues-with-heavy-compilation-loads[/url<] [url<]https://community.amd.com/thread/215773[/url<] In a nutshell, the gcc toolchain (which is used to build a large part of the modern software ecosystem) experiences random crashes when running on RyZen, and nobody seems to understand why. The crashes do not occur on Intel or older AMD platforms.

          • raddude9
          • 2 years ago
            • just brew it!
            • 2 years ago

            OK, yeah I should have qualified that with “some RyZen systems”. It doesn’t affect everyone; but it seems to be a real issue.

            There are also posts from users claiming that the ASLR workaround didn’t help. Is it possible that these people simply have unstable builds? Yes. But the jury’s still out.

            The fact that some chips apparently don’t handle ASLR correctly under certain conditions is troubling in and of itself. I’m waiting to see what AMD has to say about the issue, and whether there’s a genuine fix (e.g. via microcode update) as opposed to a workaround.

            • raddude9
            • 2 years ago
            • just brew it!
            • 2 years ago

            The fact that the issue only affects some people actually makes me [i<]more[/i<] concerned, not less. Non-deterministic failures like that point to an edge case that isn't being caught by AMD's validation and testing procedures, or a problem with the design of the initial round of motherboards. The former might be fixable with better testing and/or a microcode update; but if it's the latter, then early adopters will likely need to live with the workaround.

    • DPete27
    • 2 years ago

    Where does threadripper fit into this picture? At first I thought epyc WAS threadripper. I’m confused.

      • Waco
      • 2 years ago

      If AMD has any sense, ThreadRipper is 4 dies with 2 cores active per CCX (Epyc castaways, basically).

      Epyc is the name for the Naples 1P/2P server platform.

        • Antimatter
        • 2 years ago

        IDK what configuration AMD is using for Threadripper but since it’s quad channel it is likely 2 dies not 4. All Epyc 7000 CPUs, including the 16 and 8 core variants, support 8 channels therefore use 4 dies.

          • Waco
          • 2 years ago

          There’s no reason they couldn’t have one channel per die active. 🙂

      • ronch
      • 2 years ago

      Epyc used to be called Naples. It’ll have 4 Summit Ridge dies on a single package/socket. A Summit Ridge die has 8 cores, 32 PCIe lanes, 2 64-bit memory channels and so on and so forth. So with 4 dies there, multiply all those figures by… 4 (I think there are exceptions though because Summit Ridge’s connectivity options are nicely and confusingly flexible). Threadripper doesn’t seem to have had a codename before but rumors have been floating around back in March or April that AMD’s real answer to Intel in the HEDT space (i.e. LGA 2011) wasn’t the first wave of Ryzen 7 chips and they are/were preparing a 2-die product with quad channel… the works! It was even rumored to be called Ryzen 9, but apparently AMD’s absolutely brilliant (cough!) marketing team thought better. And so now we have [s<]Grim Rea[/s<] Threadripper! David has awoken!

        • ImSpartacus
        • 2 years ago

        Isn’t the 2-CCX due called Zeppelin and Summit Ridge is the CPU range that uses one of those Zeppelin die (while the others use multiple, etc).

          • ronch
          • 2 years ago

          Oops my bad. Yeah you’re right.

    • deruberhanyok
    • 2 years ago

    AMD, Y U NO LIKE LETTER “I”?

      • ronch
      • 2 years ago

      Yeah Y notyced that too.

      • gmskking
      • 2 years ago

      Threadripper

        • UberGerbil
        • 2 years ago

        They were going to call it Threadrypper, but then realized people tended to pronounce it like dry-pper, which rhymes with diaper.

    • ronch
    • 2 years ago

    Hmm… Judging by the PCIe lanes it looks like all Epyc models listed here will have 4 dies per package. I was hoping they’d use only as many dies as necessary but then again maybe they’ll use scrappy dies on the lower models anyway. Waste not want not.

      • jts888
      • 2 years ago

      It only feels wasteful because most folks don’t have a sense for how little of the cost of CPUs comes from the physical production.

      Manufacturing and assembly of a 4d Epyc is probably only in the high 10s of dollars, so it’s not like AMD is selling them at a loss, and we have no reason to believe their fab contract is constrained enough that they’d have to choose between high and low margin parts.

      Maintaining 8 channel and 128 lane support across the entire product line will be more than worth it for AMD due to the simplified mobo ecosystem it will be able to maintain, especially in comparison to things like LGA 2066 (2 or 4 channel, 16/28/44 lanes, …).

      • Zizy
      • 2 years ago

      Want 2 dies (= half memory and PCIe, limited to 16C), well, grab Threadripper and according board.

      4 dies => tons of memory and storage was the point of Epyc since the beginning. Disruptive IO and bandwidth and stuff, you could see such slides a year ago.
      Even that 8C thingy is interesting for some workloads that require tons of memory or storage, but not much CPU. And lets them use nearly completely broken 1 core per CCX.

      What scrappy dies? AMD will test all dies before they get put in MCM, that’s the main point. Top dies in top parts, salvaged stuff in salvaged parts. No idea what will happen if something goes wrong making MCM, but I guess it shouldn’t happen too often.

    • ronch
    • 2 years ago

    Just a thought, folks. The server market is composed of very conservative IT folks who won’t bet the server farm on a completely new platform with new drivers and such. Early Ryzen buyers had to put up with teething issues but you’d be nuts to to do that with servers. So the question is, how long would it probably take AMD to retake significant market share from Intel? AMD did say that many industry insiders wanted AMD to make a comeback in the server world but I don’t think those guys will be first in line to buy an Epyc server.

    This is all speculation of course. It would be interesting to watch how Epyc does.

      • shank15217
      • 2 years ago

      I don’t think server market is conservative, as long as vendors sell we would happily buy. In fact next purchase cycle might be EPYC for us due to the massive io options AMD offers.

        • Krogoth
        • 2 years ago

        Server/HPC market is typically conservative on massive overhauls. They rather wait until the platform proves itself and most of the “unforseeble” issues are addressed. They do test runs first on a small scale to see the potential of a new platform before committing to it.

        AMD should have a better time this round since they no longer are plagued by the stigma of platforms issues during the K6-K7 era. They prove themselves to provide an dependable enterprise-tier platform back with the K8 (Opteron).

          • ronch
          • 2 years ago

          I mostly agree with everything you said but I just wanna add that although AMD has proven themselves as capable of building server processors, a conservative IT guy will take any new platform with caution. It may be from a company that knows what it’s doing but he also understands how these things go.

            • Redocbew
            • 2 years ago

            Not sure how much the distinction of being “cautious” or “conservative” is really worth here. Every person I’ve ever worked with in technology who was any good had a healthy dose of caution, because we all know making things scream and die is bad, and we’ve all had our ass kicked plenty of times by unexpected problems. That’s not so much a reason to avoid a certain platform as it is a general rule for every day.

            If you hear over and over again your friends and co-workers saying “AUGH! This thing is terrible!”, then that’s a different story, but I don’t think that’s what you mean. Rolling out new machines for an entire data center when they’ve been on the market for a week or so is obviously not a good idea, but anyone who does that you shouldn’t be listening to anyway. 🙂

        • just brew it!
        • 2 years ago

        Depends on who you’re talking about. Big established corporations and government agencies tend to be [i<]very[/i<] conservative. Younger organizations with less red tape who are looking for maximum bang-for-the-buck are willing to take more chances.

          • Waco
          • 2 years ago

          Yeah, it 100% depends on who you have to convince.

          I’ve got purchases that are lined up to be fulfilled by Naples/Epyc if the server platforms turn out as good as they appear to be in previews and leaks…but I’ve spent years building up my ability to buy what [i<]I[/i<] think is appropriate for a given task. For many .gov employees it's not so easy if managers are risk-averse.

          • Anonymous Coward
          • 2 years ago

          I’m waiting with great interest to see what the cloud platforms do. On one hand, they roll out globally, but on the other hand, their use case is exceptionally flexible and could respond fluidly to anything that a new platform turned out to be suitable or unsuitable for.

          I happen to be on AWS, and I think AMD would provide a very fine “m5” family which according to my wishlist would be pretty similar to Broadwell/Haswell-based “m4” but at a much more attractive price per core. If they are doubling up the cores per rack I want to see that savings. However that would call into question the value of the more abstract “t2” instance types so they might seek to avoid diluting their cost per core.

          Sorry, noone else cares. 🙂

    • Prestige Worldwide
    • 2 years ago

    An “Epyc” new CPU for datacenter, high performance workstation, and enthusiasts, branded for your 12 year old COD Xbone360noscopeswag420 disappointment of a son because marketing.

    • derFunkenstein
    • 2 years ago

    So is this Epyc launch an indicator that AMD thinks it’s sorted out all the bugs and rough edges that we, the Ryzen beta-buying public, have found?

      • derFunkenstein
      • 2 years ago

      Hey, minus me all you want, but I paid good money to unknowingly beta test hardware.

        • MOSFET
        • 2 years ago

        (I like you, man, you just seem to be on a negative roll lately. The way you phrased the above, it sounds like you’re blaming the world for your lack of research and/or impulsivity.)

        I just built a Ryzen 5 1600 ESXi box this week. Despite all the warnings online, apparently Ryzen 5 works fine with ESXi. Despite the warnings, apparently Ryzen 5 works fine with 64GB of DDR4-2400. Despite all the negative reviews on Newegg, apparently the Asus Prime X370-Pro works just fine and would get close to a 5-star review from me, if I cared to do such things.

          • derFunkenstein
          • 2 years ago

          Release day reviews made no mention of memory incompatibilities (as in, flat-out refusing to run at anything above SPD and totally ignoring XMP profiles), boards that took an inordinate amount of time to POST (as the B350 Tomahawk did at release), or any of the other weird things I’ve run into. There WERE no warnings online when I bought my system.

          You know who created those warnings? Early adopters like me. I did my very best to document all the weirdness that tech journalists didn’t document, most likely because AMD gave them golden samples and pre-tested configs. Here’s my effort:

          [url<]https://techreport.com/forums/viewtopic.php?f=33&t=119285[/url<] I still use this system, mostly because it's all I've got. Three months on it's vastly improved. It's still not polished to the point Intel's platform is, but it's better.

      • Krogoth
      • 2 years ago

      Pretty much.

      Intel does the same thing with their HEDT-tier platforms going back to Bloomfield.

        • derFunkenstein
        • 2 years ago

        Spoken like someone who never used a Ryzen system.

          • Krogoth
          • 2 years ago

          Ask those early Bloomfield/X58 users who had to deal with stupid memory compatibility issues back in the day.

          Being an early adopter has always been fraught with rough edges and stupid issues.

            • Waco
            • 2 years ago

            Yep. This is nothing new.

          • Convert
          • 2 years ago

          While I’m sure you’re right, and I definitely have doubts these examples are comparable like Krogoth implies, the memory issues with the x58 systems were pretty annoying given how expensive they were.

          I still own a x58 system that forgets half of its system RAM every other reboot. Asus’ attempted fix a year later was to release a bios that takes 15 minutes to post and is unusable in windows if you have a graphics card with more than 512MB of RAM.

          AMD has almost always had **** platforms. I bought AMD all the way up until the CD2 was clearly established. Those AMD platforms all had quirks, which at the time I thought was par for the course. I’ve encountered Intel boards that should have been part of a class action lawsuit (DQ67SW I’m looking at you), but I still find the platform more stable and mature overall.

          As interesting as Zen and Threadripper are, I have no intention of converting until they prove themselves a reliable platform.

            • srg86
            • 2 years ago

            This has also been my experience as well. For years I wanted to support the underdog (and also not use a NetBurst)

            So I used to buy AMD CPUs from a K6 to an Athlon 64 X2 (multiple moves made this a system I used for almost a decade). I also thought the quirks were par for the course for the most part. Since switching to intel, things have been much smoother.

            As for the DQ67SW, I’ve never liked Intel motherboards, they are always quirky, with quirky BIOSs. Platforms are generally rock solid though.

    • timon37
    • 2 years ago

    That “security” processor makes me extremely uncomfortable…
    I guess it’s an extended equivalent to SGX?
    [url<]https://en.wikipedia.org/wiki/Software_Guard_Extensions[/url<]

      • freebird
      • 2 years ago

      Yes and no…

      This article goes over some of the differences…pretty good reading…
      [url<]https://lwn.net/Articles/686808/[/url<]

      • ronch
      • 2 years ago

      It’s interesting how today’s leading edge processors actually have smaller cores hidden deep within them. There’s usually a security processor and a power control unit. Here with Ryzen the security processor is a Cortex A5. I’m not sure, but IIRC AMD also uses the A5 as a power control unit in recent designs. Is AMD using the A5 as a power control unit and as a security processor? And does Ryzen employ 2 A5 cores?

      • maxxcool
      • 2 years ago

      In theory as long as the algorithm is solid and is not easy to collide this is not a bad solution if the key is long enough. it’s all about encrypting the data so skimming types of malware can’t read your stuffs.

      With guest escapes To Hosts being on the rise this is a critical piece of technology for this type of server CPU especially since it’s targeted at heavy virtualization.

      Honestly I can’t wait for this kind of technology to actually make it down to Consumer level Hardware. It will make my job easier eventually.

    • Gadoran
    • 2 years ago

    At the same number of cores AMD is A LOT behind Intel…..this is an huge problem. AMD can only leverage on price so there will be an huge discount of Intel SKUs to the end user. I don’t think the market share situation will change much.
    Unfortunately the MCM solution is a disaster for performance and they are costrained to give away silicon for nothing. No much profitability at the orizon for AMD.

      • Krogoth
      • 2 years ago

      You are completely off the chart here. It is Intel that is going to be feeling the pressure.

      Their fab technology isn’t enough to overcome the issues in making massive monolith dies (yielding and thermal issues). Intel ran into this with their Broadwell-EP and Skylake-X chips. They already made plans to move away from this approach in their future enterprise/HPC designs unfortunately it will take them a couple of years to shift gears.

      AMD has a CPU architecture that can deliver comparable performance unlike the previous Bulldozer/Excavator designs.

      It is telling that almost every major server OEM vendor has jumped onto the Epyc bandwagon.

        • Gadoran
        • 2 years ago

        Indeed, the pressure exist ! and Intel is addessing right now doing discount to be on pair with projected AMD prices.
        About the MCM approach, it is a disaster, far better the monolityc design that ALL the server manufacturers adopt now and will adopt in the near future (Qualcomm too).
        There are any issues in yields and thermals, at least at the die size chosen by Intel and others, but only strong performance advantages.
        MCM is only a regression that has very bad issues under severe server load. After all Intel is an expert in MCM but they avoid it if not constrained.
        Don’t try to transform a tradeoff in a good thing please, it is unfair. Unfortunately there is few of Epyc here, only the price at the end user……with a bad profitability for manufacturer.

        AMD CEO was fair saying that the first bunch of Server cpus will not be stellar and the market has right now a strong presence and hardly the market share will change quickly. All server farms are Xeon based, will take years to change this situation. This strange thing named Epyc is not K8 but only a standard good cpu.

          • Krogoth
          • 2 years ago

          Please remove the blue-tinted shades. They are making you look silly.

            • w76
            • 2 years ago

            I don’t think you give him enough credit. I suspect he’s just short AMD stock (or long Intel, or both), and is shilling on behalf his pocket book. Those are green-tinted shades, but not AMD green. 😉

          • Waco
          • 2 years ago

          There’s no basis in reality for your comments on MCM. Please cite your sources.

          • [+Duracell-]
          • 2 years ago

          What exactly is wrong with MCM? As long as the interconnect between each die is fast and low latency enough, performance should not be bad. This also allows them to harvest bad dies into the lower tier processors easier and save a bit of money.

        • jts888
        • 2 years ago

        One of Intel’s fundamental challenges is that it wants to fend off GPGPUs in HPC by throwing ever-wider FP SIMD in every core, but most customers don’t actually need AVX/AVX2/AVX512/… whatsoever, and the design changes needed to support them (larger dies, wider and more power hungry datapaths) end up being a burden on those chips whether or not they’re used.

        For example, Skylake-E/-X has a whopping 128B-wide datapath from each core through L1D all the way to the large L2, which sounds great until you realize how much area and power wider cache ports really add. But without it, AVX512 (capable of ingesting and digesting whole 64B cache lines every clock) would starve, so they had no choice.

        AMD, who really had no market share to lose, did the smart thing by focusing on int throughput and efficiency while letting Intel struggle to make Xeon E3/E5 into the Master of All Trades. Specializing FP crunching into dedicated processors is definitely the right move for AMD (and Nvidia obviously by default) and could arguably be for Intel as well.

          • Anonymous Coward
          • 2 years ago

          I suspect there is ample room in the server market for an x86 CPU with even less concern for FP throughput than Zen.

            • jts888
            • 2 years ago

            Maybe, but there’s nobody to make it really.

            AMD’s whole gimmick this round is to have a single die be used for everything beefier than a high-end laptop, and Intel will never back down from their stance that AVX is the Next Big Thing.

            Do you remember the bloodbath when they backed away just slightly from Itanium (a.k.a. EPIC architechture chips, heh) by releasing the AMD64/x86-64-compatible Yamhill then Nocona chips? lntel’s fat margins are fed largely by people having unshakable faith that its ISA is solid and always growing in purely positive ways.

          • freebird
          • 2 years ago

          Yeah, my same thoughts on FP processing… AMD has decide to move towards GPGPU for that and not devote Silicon to it… some reviews view this as a deficit, but I doubt much of anyone that has a server farm for do FP processing is do it via a CPU but rather a GPGPU.

          Seems that we have now come FULL circle back to the days prior to the 80486… where the FPU was a separate chip to be installed only if needed. : )

        • stefem
        • 2 years ago

        It’s not the first time I hear that but splitting your die wouldn’t really help much on dissipating the generated heat, the exchange surface remain the same 200mm² * 2 = 400mm²

          • jts888
          • 2 years ago

          Splitting a die and spacing out the halves is equivalent to taking half-sized dies and heatsinks and moving the dies from the edges to the centers of the heatsinks. It’s not a huge thing, but lateral heat transfer in the base does affect how easily heat gets to the pipes/fins/whatever. Of course, fast inter-die links do suck down and put out a big fraction of a Watt each, so it’s clearly not a complete win.

      • ronch
      • 2 years ago

      While single thread performance is still important, reliability, scalability, overall throughput, TCO (which includes energy efficiency), vendor support and platform longevity are the crucial aspects of any server platform.

      • freebird
      • 2 years ago

      I can only assume you own Intel stock/ you are a Intel fanboy/ or Intel paid commentor…

      AMD has a small deficit in per core performance perhaps, but has memory bandwidth/total size advantages over the current Intel lineup.

      The MCM solution shows fine scaling so far for most applications and also allows them to harvest /produce the individuals dies much more cost effectively than one HUGE die, so basically a each ZEN die in the MCM of the Eypic Processor cost the same to manufacture as the die used for Ryzen and they aren’t losing money on any Ryzen processors they sell which go for much much less than any server processor, so YES they will be making money on them…

      EPYC also supports hardware memory encryption and SEV (Secure Encrypted Virtualization)

      Oh yeah, and what UberGerbil said about the IO bandwith with 128 PCIe channels… almost forgot this…from a WCCFTECH article
      [url<]http://wccftech.com/amd-epyc-7000-series-server-processors-launch/[/url<] "A Single AMD EPYC Processor Powers 24 3.2 TB NVMe Drives at Full PCIe x4 Speeds – Has 32 PCIe Lanes Still Left For Use In one of the power demonstrations, AMD used a single AMD EPYC 7601 processor with 128 PCIe lanes to run a total of 24 NVMe drives, each with a capacity of 3.2 TB which totals to 76.8 TB on a full PCIe 3.0 x4 link. The system still had 32 PCIe lanes left for use which is one of the key features of AMD’s EPYC processors. During a 128K random benchmark demo, the system scored 9.1 million (9,178,000) IOPS read, 7.1 million (7,111,000) IOPS write and delivered 53.3 GB/s of storage bandwidth which is impressive feat for the server chips. All of this was achieved on a single socket system which shows that AMD EPYC is even optimized for datacenter tasks where large amounts of data needs to be managed efficiently."

    • ImSpartacus
    • 2 years ago

    [quote<]AMD has sliced and diced that basic package into a variety of products designed to serve prices ranging from under $400 to under $4000.[/quote<] [quote<]Single-socket operation is the only restriction for these chips, and they'll be available for prices under $700 to under $2000.[/quote<] I'm confused. Anandtech and Legit Reviews mentioned those same dollar amounts, but their tables show those as minimum prices, not maximums. [url<]http://www.anandtech.com/show/11551/amds-future-in-servers-new-7000-series-cpus-launched-and-epyc-analysis[/url<] [url<]http://www.legitreviews.com/amd-epyc-7000-series-processor-set-change-game_195601[/url<] And Legit Reviews separately shared a slide with specific pricing for certain CPUs that was consistent with the "greater than" minimums. [url<]https://mobile.twitter.com/LegitReviews/status/877278769270198272[/url<]

      • Jeff Kampman
      • 2 years ago

      Sorry, brain fart.

    • Meadows
    • 2 years ago

    So from now on, every new brand name within AMD will contain the letter “Y” somewhere?

      • Shobai
      • 2 years ago

      We can only hope.

    • brucethemoose
    • 2 years ago

    Think we’ll get an SR-2 like mobo for these suckers?

    A 2.7 all-core turbo sounds pretty good, assuming that 200W TDP limit lets it hold that. Unlocked multis would be even better, if thats even supported… just think, 32 or 64 cores at ~3.5, 8 channel OC’d DDR4.

      • DancinJack
      • 2 years ago

      Maybe for TR…not for EpYc though. These are the “Opterons” of yesteryear. You’re likely wanting ThReAdRiPPeRz.

      edit: BTW, you’re not going to get 64 cores to run at 3.5GHz. Dream all you want, but that won’t happen on today’s silicon.

        • brucethemoose
        • 2 years ago

        Actually, with decent cooling, it should be possible.

        Unlike Intel’s HEDT chips, this is the exact same silicon as Ryzen. The only thing limiting them is the 50W max TDP each chip gets.

        400W (or 800W for 2P) on a giant heatspreader like that is doable with watercooling. Heck, a single 120mm aluminum AIO handles a 350W Fury X, and thats less surface area.

    • Unknown-Error
    • 2 years ago

    Looking at the benchmarks it seem AMD went:

    [quote<][i<] " ****-it! **** ICC! We are going to use our own heavily optimized compiler. " [/i<][/quote<] Nothing wrong with that and since they clearly stated how they derived Xeon scores in their statements, I would not say they lied or cheated. But taking these benchmarks with a mountain of salt is mandatory. The independent reviews (from TR and others) will become even more important than ever when it comes to verifying these claims. Still, the fact that EPYC has some big industry names supporting it is encouraging for AMD and us.

    • ronch
    • 2 years ago

    I once rooted for ARM servers potentially disrupting the server market, perhaps in part because I don’t like Intel monopolizing the market especially with Bulldozer practically sinking AMD’s boat in the server world faster than the iceberg sank the (T)itanic, but now that Zen is such a strong design I just love how AMD is effectively reinstating their claim to the server world and making all those ARM server chip vendors green with envy at how AMD has an x86 license and they don’t. Of course this is good for consumers so long as AMD is competitive. Another Bulldozer from AMD and it’s back to Intel owning the server world. Still, eat your hearts out, ARM server chip vendors!!

      • freebird
      • 2 years ago

      The last Itanic (Itanium) left the ship yard a few months ago, I read sailing off in search of an Iceberg I presume…

      Kept on life-support by HP/Dell and our tax dollars (via the Federal gov’t; apparently the only people stupid enough to buy into it…)

      [url<]http://www.pcworld.com/article/3169623/components-processors/intel-ships-latest-itanium-chip-called-kittson-but-grim-future-looms.html[/url<] "HPE has kept Itanium support active until 2025 mainly due to long-term customer commitments. Many Itanium customers are government agencies, which sign long-term deals with vendors to supply and support servers."

        • ronch
        • 2 years ago

        I reckon the iceberg that sank the Itanic has an AMD64 sticker on it.

        Then about 8 years later Bulldozer came along and bulldozed its maker into the ground.

        Truly a story of Epyc proportions.

        • curtisb
        • 2 years ago

        Dell hasn’t had an Itanium-based server in their lineup for years. I think they dropped them from the lineup either right before or right after Microsoft stopped supporting Windows Server running on Itanium. I believe the last version was Server 2008, maybe Server 2008 R2.

        HPE is the primary customer for OpenVMS and HP/UX systems. We have one such system in our datacenter running OpenVMS. It’ll be at least a few more years before we can retire it. I will give it this…it’s extremely fast when running native code.

    • CampinCarl
    • 2 years ago

    Dear AMD,

    Please provide us with a drop-in replacement for MKL. 2P servers @ 24 cores/2.3GHz each wouldn’t be too awful for compute.

    Also, very excited to see what Cray can gin up with these. Should be impressive.

    • barich
    • 2 years ago

    Totally not a coincidence that AMD has come back with a vengeance shortly after Scott arrived.

      • Chrispy_
      • 2 years ago

      Thanks Scott!

        • aspect
        • 2 years ago

        Fuck you, Scott! – Intel

          • ronch
          • 2 years ago

          Not cool, bro.

    • Krogoth
    • 2 years ago

    It is 2005 again when K8 disrupted Intel’s hold on the enterprise market with the Opteron line-up. It looks like Epyc will end-up doing the same thing.

    It doesn’t seem too far-fetched for Threadripper SKUs to go from $499-$1200.

      • xeridea
      • 2 years ago

      Rumor is 16 core Threadripper is $849

        • Pancake
        • 2 years ago

        I have said that dual channel 64GB max RAM Ryzen is a fail for me.

        Threadripper with 4 channels. Check.

        Affordably priced. Check.

        Single-threaded performance would be noticeably worse than Intel. But enough to care? Maybe not. Can’t say I have a use for 16 cores but at that sort of price it would fit into the “just because” category…

      • Klimax
      • 2 years ago

      Maybe. Only real world will determine whether or not AMD got there.

      • ronch
      • 2 years ago

      I think the first 64-bit Opteron models came out in 2003.

    • Bumper
    • 2 years ago

    I’ve read that Epyc and MI25 will be able to connect using infinity fabric. Any information given on that?

      • jts888
      • 2 years ago

      Zeppelin can multiplex IF traffic over its IO lanes, including those wired to PCIe slots in principle.
      There’s not even widespread rumors that Vega will support the same, but honestly I’d be surprised if it didn’t, given the quasi-APU functionality it would enable.

        • Bumper
        • 2 years ago

        Ah yes. That’s a cool functionality, I hope the software is there for it. I was thinking along the lines of what nvidia does with Nvlink to link gpus.

        On another topic, whatever happened to the freedom fabric that AMD acquired with Seamicro? I wonder if that’ll pop up again in their exascale quest.

        And btw Epyc looks very impressive. I hope it has a big impact on Intel’s 99% monopoly.

        • freebird
        • 2 years ago

        I imagine they would enable it…

        Don’t forget: a big talking point of Vega was the HBCC (High Bandwidth Cache Controller) with a virtual address space of 512TB! I think Vega and Navi with the HBCC and Epyc’s IO bandwidth (along with 7nm, perhaps) will lead to some amazing things we haven’t even dreamed possible yet…

        but only time and some smart programmers will tell us in the future.

        Imagine an AI Server farm full of Epyc servers filled with Vega/Navi GPGPUs leveraging the HBCC to all “talk” together… maybe Skynet isn’t too far off.

        [url<]https://www.google.com/search?q=skynet+meme&tbm=isch&imgil=udFUtN7Dc65BCM%253A%253BY1ZKf23ZTRsI6M%253Bhttps%25253A%25252F%25252Fimgflip.com%25252Fi%25252Fut0dv&source=iu&pf=m&fir=udFUtN7Dc65BCM%253A%252CY1ZKf23ZTRsI6M%252C_&usg=__uiRL-4vT5op6UDP8-hXg48IGPtg%3D&biw=1920&bih=950&ved=0ahUKEwj0g--krs_UAhVE7oMKHYo6A2oQyjcINQ&ei=Qp9KWfTaG8TcjwSK9YzQBg#imgrc=udFUtN7Dc65BCM:[/url<] Ok, i'm gonna start digging my bunker now... 😀

          • jts888
          • 2 years ago

          PCIe already deals in the host’s (or VM host’s, potentially) physical address space, so a large address space for Vega isn’t in and of itself telling.

          What Zeppelin’s IF architecture allows is multiple processor groups, Zen CCXs or otherwise, to talk with each other in the true unique physical address space (bypassing IOMMU protections as well as bottlenecks and increased processing latency) in a cache coherent messaging protocol that supports various memory ordering/consistency models as well.

          If Vega supports the same type of bypass, the GPU bits would be able to talk with Ryzen/Threadripper/Epyc CCXs and memory controllers as easily as Epyc CCXs in different sockets talk to each other, which is in turn nearly as fast as CCXs on different die in the same socket talk to each other, which is again not expected to be much slower than two CCXs on the same die talking to each other. Raven Ridge is expected to connect its single CCX to its onboard GPU via IF, so none of this is that much of a stretch.

    • ikeke1
    • 2 years ago

    [url<]https://www.servethehome.com/amd-epyc-7601-dual-socket-early-power-consumption-observations/[/url<] STH has some preliminary power consumption numbers. Platform: Supermciro 2U Ultra Server with NVMe Support CPU: 2x AMD EPYC 7601 (32 cores/ 64 threads each) RAM: 512GB using 16x 32GB DDR4-2400 ECC RDIMM (eight per CPU) Boot SSD: Intel DC S3520 480GB NVMe SSD(s): Intel DC P3600 800GB Network Adapter: Mellanox ConnectX-3 Pro EN 40GbE (2x QSFP+ 1m DACs) Idle - ~100W Our 60% enterprise static load - ~350W GROMACS AVX2_256 across 128 threads - 483W Just let that last number sink in.....slowly.

      • jts888
      • 2 years ago

      8W per fully running core [i<]at the wall[/i<] is just nuts...

        • derFunkenstein
        • 2 years ago

        Yeah, that didn’t strike me as being bad at all. Unless the power supply gets listed (which it’s not in the article), I just kind of assume the system is only drawing around 80-85% of that figure. So figure ~400W to power [s<]32[/s<] 64 cores, .5 TB of RAM, and two relatively power-hungry SSDs. No complaints here.

          • jts888
          • 2 years ago

          I imagine that particular machine could be induced to draw a bit more power (something that pushed huge memory buffers around, plus heavy network IO, plus enabling the encryption on the DDR4 controllers), but honestly these machines won’t be bought by people trying to murder them with AVX crunching anyway.

          Folks running file servers, web farms, memcached, and similar should be quite happy I suspect.

          • ikeke1
          • 2 years ago

          *to power 64 cores*

            • derFunkenstein
            • 2 years ago

            Duh, my bad. Forgot it was a 2S system.

          • freebird
          • 2 years ago

          Don’t forget the power consumption for the Network Adapter:

          Power consumption:
          Power consumption (typical): 8.8 W typical (both ports active)
          Power consumption (maximum): 9.4 W maximum for passive cables only,
          13.4 W maximum for active optical modules

          😀

        • psuedonymous
        • 2 years ago

        [url=https://www.servethehome.com/supermicro-x10sdv-7tp8f-high-end-xeon-d-platform/<]For comparison[/url<], the Xeon D-1587 runs at a 91.8W system power for 16 cores at 2.3GHz turbo (vs. the Epyc 7601's 3.2GHz) for 5.7W per core. Xeon-D is Intel's most efficient server processor line.

      • Chrispy_
      • 2 years ago

      That last number didn’t sink in [i<]slowly[/i<]; It was more like a number-howitzer fired it straight through Intel's corporate brain, gory mess and all. Seriously, that's a total platform draw of [b<]<4W per thread for a full AVX2 load[/b<]. For me as an IT manager running a VMWare estate that has no compute requirement, the result is of little consequence, but our visualization and modelling department will need horse tranquilizers just to calm down from this news when I break it to them tomorrow. Does anyone know where I can buy cheap horse tranquilizers? The alternative is a cricket bat and I landed myself in deep trouble with HR last time...

        • jts888
        • 2 years ago

        How competitive does this result end up being in FLOPS/Watt vs. Broadwell and Skylake?

        It’s nice to know Epyc’s not super power hungry at its worst, but I wasn’t even really hoping for it to be an AVX juggernaut.

          • ptsant
          • 2 years ago

          I don’t think EPYC is aimed at HPC. Would probably lose in FLOPS, but win in most other server metrics.

            • Waco
            • 2 years ago

            HPC is dominated by the highest FLOPs/byte/watt – and it appears that Zen is targeted nearly perfectly for a nice ratio.

            • ptsant
            • 2 years ago

            Isn’t the AVX512 going to dominate HPC?

            • Waco
            • 2 years ago

            Depends on the type of HPC workload.

        • chuckula
        • 2 years ago

        I’d be careful with the horse tranquilizers.

        And if AVX is really important to you, I’d be careful in looking at one number without context before making a purchase.

        Like if 4 watts per thread in AVX2 sounds great, how about 2.6 watts per thread to power 64 cores in AVX-512.

        [url<]https://www.servethehome.com/supermicro-sys-5038k-i-es1-intel-xeon-phi-x200-developer-workstation-review/[/url<]

          • Waco
          • 2 years ago

          I think the per-thread throughput difference between a Phi and a Zen chip for many workloads is pretty staggering.

            • Klimax
            • 2 years ago

            Not in the original reply, thus he made valid reply.

            • Waco
            • 2 years ago

            I’m not even sure what this means. I was adding context to the watts/core discussion.

          • abiprithtr
          • 2 years ago

          Wow !! That Zeon Phi is some serious [b<][u<]S[/u<][/b<]uper [b<][u<]H[/u<][/b<]igh [b<][u<]I[/u<][/b<]ntel [b<][u<]T[/u<][/b<]hreading, and runs seriously [b<][u<]S[/u<][/b<]luggish and [b<][u<]L[/u<][/b<]eisurely to [b<][u<]O[/u<][/b<]perate the [b<][u<]W[/u<][/b<]orkloads in the gcc compile benchmarks. No wonder it just has to sip energy.

            • Spunjji
            • 2 years ago

            I lol’d 😀

        • Klimax
        • 2 years ago

        And performance? Otherwise it would be trivial to break such comparison by mass of Atom cores. (Incidentally like Xeon Phi…)

        • Haserath
        • 2 years ago

        At what performance comparitively? Zen doesn’t use full 256-bit AVX in hardware like Intel.

        • coolflame57
        • 2 years ago

        picture or it didn’t happen

    • smilingcrow
    • 2 years ago

    Came across this at Anandtech which shows 16C 2P starting in the range $6-800:

    EPYC 7351 16 / 32 2.4 ? 2.9 155W/170W >$1100
    EPYC 7301 16 / 32 2.2 ? 2.7 155W/170W >$800
    EPYC 7281 16 / 32 2.1 ? 2.7 155W/170W >$600

    This for single socket which shows 16C in the range $700-1,000:

    EPYC 7401P 24 / 48 2.0 2.8 3.0 155W/170W >$1000
    EPYC 7351P 16 / 32 2.4 ? 2.9 155W/170W >$700

    That seems cheap for HEDT let alone Server so maybe TR will be 16C for under $1K?

      • smilingcrow
      • 2 years ago

      More pricing here:
      [url<]https://www.theregister.co.uk/2017/06/20/amd_epyc_launch/[/url<] and [url<]https://twitter.com/LegitReviews/status/877278769270198272[/url<] EPYC 7401P 24 / 48 2.0 2.8 3.0 155W/170W $1075 EPYC 7351P 16 / 32 2.4 ? 2.9 155W/170W $750 The 7401P giving all 24 cores at 2.8 boost clock for $1,075 sounds interesting.

    • Rza79
    • 2 years ago

    Epyc 7551: 1345
    Epyc 7601: 1840

    That doesn’t seem right considering that the only difference between the two is a 200Mhz difference. The Epyc 7551 results seem too low compared to the 7451.

      • Jeff Kampman
      • 2 years ago

      Those are the numbers that AMD provided; I double-checked.

        • dpaus
        • 2 years ago

        Different memory speed?

    • AMDisDEC
    • 2 years ago

    Lisa, Lisa, Lisa!
    You’re doing it again.

    Lisa Su has very likely killed that failed ARM Server project to instead focus exclusively on x86. I don’t even like the fact they used the ARM Cortex as the embedded security processor. Since they are signed up with RISC-V, they’ll probably replace it in the future with a proprietary design.
    Lisa Su is the real, Wonder Woman.

      • Pancake
      • 2 years ago

      Dude, she’s married.

        • AMDisDEC
        • 2 years ago

        What’s your point?

          • chuckula
          • 2 years ago

          SO YOU’RE SAYING THERE’S A CHANCE!

            • AMDisDEC
            • 2 years ago

            méi cuò, Shì, juéduìd!!!!

            LISA!!!

      • Welch
      • 2 years ago

      I’d imagine the ARM processor would help with security because it isn’t even the same architecture as the CPU itself. Any sort of attempted attack would have to first compromise the ARM to gain access. But in order to do that has to go through (mostly) an OS designed for x86 as well as x86 hardware. Short of dissolving the layers away with acid like that one brilliant SOB who compromised Intel TPM modules… I just don’t know how it could be done by Joe Schmo hacker.

        • NoOne ButMe
        • 2 years ago

        Intel hardware security okay… software for makes fail

    • chuckula
    • 2 years ago

    [quote<]AMD is making some claims about the performance of various Epyc products' performance today, and the initial outlook is good. However, we do have to take issue with a couple of the choices AMD made on the way to its numbers. After compiling SPECint_rate_base2006 with the -O2 flag in GCC, AMD says observed a 43% delta between its internal numbers for the Xeon E5-2699A v4 and public SPEC numbers for similar systems produced using binaries generated by the Intel C++ compiler. In turn, AMD applied that 43% (or 0.575x) across the board to some publicly-available SPECint_rate_base scores for several two-socket Xeon systems. It's certainly fair to say that SPEC results produced with Intel's compiler might deserve adjustment, but my conversations with other analysts present at the Epyc event suggests that a 43% reduction is optimistic. The -O2 flag for GCC isn't the most aggressive set of optimizations available from that compiler, and SPEC binaries generated accordingly may not be fully representative of binaries compiled in the real world.[/quote<] Whoa boy. As a favor to AMD I'm just going to pretend they didn't bother making any performance claims at all and I'll wait for server reviews.

    • DancinJack
    • 2 years ago

    [quote<]To build each Epyc package, AMD uses four eight-core modules connected using its Infinity Fabric interconnect.[/quote<] So, how are they building the 7251? Four modules with only two cores activated for each? Yowza.

      • chuckula
      • 2 years ago

      It’s possible to do. It’s probably a niche product for doing some type of storage system that doesn’t need too much CPU but likes PCIe lanes.

        • blahsaysblah
        • 2 years ago

        So the 8 memory channels for all chips is also accurate?

          • chuckula
          • 2 years ago

          If all of the chips are there, each one has a dual-channel memory controller so it should be accurate. There’s a lot of unused silicon but AMD can bin it that way if they want to.

          • DancinJack
          • 2 years ago

          That’s what the chart says. 8-ch RAM and 128 PCIe lanes for all.

            • dodozoid
            • 2 years ago

            Well, apparently, its not always cores the server guys crave…

            • jts888
            • 2 years ago

            As crazy as it sounds, some people are better off throwing $10k at 1TB RAM and $500 at only 8 cores since they’re getting gouged horribly by per-core licensed software that gobbles down memory like it was going out of style. (kdb/Q, I’m looking at you)

      • NTMBK
      • 2 years ago

      Gives them something to do with chips that have 3 defective cores in a single CCX.

        • ikeke1
        • 2 years ago

        According to rumours AMD has ~80% 8c/16t yields on Zen.

        [url<]http://www.guru3d.com/news-story/amd-ryzen-14nm-wafer-yields-pass-80-threadripper-cpus-on-track.html[/url<]

      • Anonymous Coward
      • 2 years ago

      I wonder why the clock speeds are so low with only one active core per CCX.

      Lots of L3 and memory bandwidth on tap, anyway.

        • Spunjji
        • 2 years ago

        Probably a coincidence of factors such as:

        1) They can use really, /really/ awful dies to hit those guaranteed clock speeds,
        2) Their customers aren’t asking for more,
        3) They /really/ don’t want to sell more of these than they have to because they’ll be wasting lots of good dies.

    • chuckula
    • 2 years ago

    It’s great that they at least gave price ranges but for a product that has nominally launched today you think they could have been just a tad more precise.

      • maxxcool
      • 2 years ago

      No surprises here 2k$, for 32 cores, 1k$ for 16 cores, 600+$ for 8 cores that are retail units that are ‘server rated’. edit (1 core WS)

        • DancinJack
        • 2 years ago

        None of these are “server rated” Ryzen. These are all four dies in one package.

          • maxxcool
          • 2 years ago

          Yup i am aware there are 4 dies on a mcp. i want to see the tops sanded off and a die shot because i bet 20$ paypal that these are retail chips that passed server certifications and have unfused features.

    • chuckula
    • 2 years ago

    I expect you to come back with some Threadrippers that Wasson snuck out from the labs.

    I just don’t want you to go into detail about [i<]how[/i<] he snuck them out. TMI man. TMI.

      • Duct Tape Dude
      • 2 years ago

      With sufficient thrust, buffaloes fly just fine.

        • Beahmont
        • 2 years ago

        Yes, but under that model of physics it’s the landing that’s the problem.

      • CuttinHobo
      • 2 years ago

      Christopher Walken’s character in Pulp Fiction.

      You’re welcome. 😀

      • Welch
      • 2 years ago

      He’d be damned if any slopes gonna put their greasy yellow hands on his boy’s birthright, so he hid it, in the one place he knew he could hide something: his shoulder bag.

    • UberGerbil
    • 2 years ago

    Oddly, it may be the 128 PCIe channels that make the most difference for this market.

      • DragonDaddyBear
      • 2 years ago

      I’m not a server person. How so?

        • cygnus1
        • 2 years ago

        NVMe storage and 100gb network connections

        • curtisb
        • 2 years ago

        Lots of 40Gb or even 100Gb NICs, specifically cards with multiple ports. A single 100-Gb port is 12.5GB/s per direction (without overhead). You can easily fit four SFP-sized ports on a single card, making the requirement 50-GByte/s per direction, or 100GB/s in total.

        Also very high-speed PCIe-based NVMe drives.

        You can never have enough network ports when you’re using your servers for massive amounts of virtualization. You take how many connections you need and at least double it for redundancy. My latest host servers each have 10 x 10Gb ports, and we might be considered mid-sized by business standards…about 250 or so employees (faculty and staff) with roughly 2500 or so active customers (students).

          • cmrcmk
          • 2 years ago

          Holy crap. We only run 2x 10GbE on our hosts (32 cores, 768 GB RAM) and they’re way underutilized. Either your servers are far more chatty than mine or you have redundancy for days.

            • curtisb
            • 2 years ago

            2 x 10Gbps links for the host server data traffic
            2 x 10Gbps links for VM data traffic
            2 x 10Gbps links for iSCSI for the OS volumes
            2 x 10Gbps links for iSCSI for data volumes
            2 x 10Gbps links for the heartbeat (the faster the heartbeat link, the faster Live Migration occurs)

            Each set is configured using an MLAG setup, except for the iSCSI links which use MPIO. One link in each group goes to one switch stack, and the other link goes to another switch stack. The switch stacks have 160Gbps aggregate stacking bandwidth (2 x QSFP+ direct attach cables), and 320Gpbs aggregate bandwidth between the stacks (4 x QSFP+ direct attach cables).

            • cmrcmk
            • 2 years ago

            Are you driving UCS blades?

            • curtisb
            • 2 years ago

            Nope. The hosts are Dell PowerEdge R730’s with 16 cores/32 threads and 384GB of RAM each. The previous iteration was PowerEdge R720’s with the same configuration, except that the host data connections were 2 x 1Gbps. We don’t run our databases on those hosts. Those are on a dedicated physical SQL cluster.

            Having that many links isn’t necessarily about the bandwidth either. It’s about reducing traffic contention, which translates to reduced latency. Your iSCSI traffic should be dedicated anyway.

        • Waco
        • 2 years ago

        Lots of high-speed connections for networking + lots of high-speed storage controllers.

      • stdRaichu
      • 2 years ago

      Supermicro have posted up basic specs of a single-socket EPYC mobo with 72 lanes of PCIe 3.0 connectivity as well as 16 SATA ports and an integrated SAS controller, the [url=https://www.supermicro.nl/products/nfo/AMD_SP3.cfm?pg=MOBO<]H11SSL*[/url<]. That's a pretty incredible amount of IO for a 1P box.

Pin It on Pinterest

Share This