Personal computing discussed

Moderators: Flying Fox, morphine

 
DancinJack
Grand Gerbil Poohbah
Posts: 3180
Joined: Sat Nov 25, 2006 3:21 pm
Location: Kansas

Re: Skylake X: Bye Bye Rings, Hello Mesh

Wed Jul 12, 2017 11:32 pm

It's a brand new platform Krogoth... Saying their platform is stable because it was 8 years ago with completely different chipsets, CPUs, and manufacturers makes no sense. But OK.

And if we're going to compare this to K8/Operton days, Intel isn't in "serious trouble." In 2006, Opteron/AMD's heyday, AMD had roughly 22 percent of the server marketshare. So, Intel might not make near as much money as they have been the past decade, but they aren't in "serious trouble" as you put it.

Like I said previously, AMD did a really good job with these chips and they will chip away at Intel over the next few years, but it'll still be an uphill battle.
i7 6700K - Z170 - 16GiB DDR4 - GTX 1080 - 512GB SSD - 256GB SSD - 500GB SSD - 3TB HDD- 27" IPS G-sync - Win10 Pro x64 - Ubuntu/Mint x64 :: 2015 13" rMBP Sierra :: Canon EOS 80D/Sony RX100
 
Krogoth
Silver subscriber
Gerbil Elder
Posts: 5353
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime
Contact:

Re: Skylake X: Bye Bye Rings, Hello Mesh

Wed Jul 12, 2017 11:56 pm

DancinJack wrote:
It's a brand new platform Krogoth... Saying their platform is stable because it was 8 years ago with completely different chipsets, CPUs, and manufacturers makes no sense. But OK.

And if we're going to compare this to K8/Operton days, Intel isn't in "serious trouble." In 2006, Opteron/AMD's heyday, AMD had roughly 22 percent of the server marketshare. So, Intel might not make near as much money as they have been the past decade, but they aren't in "serious trouble" as you put it.

Like I said previously, AMD did a really good job with these chips and they will chip away at Intel over the next few years, but it'll still be an uphill battle.


Skylake-X/EP and Socket 2066/3674 are also brand new platforms. AMD doesn't have the sigma that had back in the K6-K7 era. What is going kill Intel is on the production side. They are having hell of a time to producing high-core count chips (seen with lack of availability with Skylake-X) since it is a PITA to produce "good" massive monolith dies. The 14nm process hasn't changed this reality. AMD has a much easier time producing high-core count chips because of MCM approach. AMD can easily engage in the price war while retaining decent margins on HPC/Enterprise market.

Intel isn't to have a viable counter for at least another three years or more (barring issues with 10nm process and beyond).

It is going to be hard to conceive PHB-types to suggest a budget for 20K-30K server "32-core Intel system" versus 10K for a "32-core AMD server" and they have comparable performance to each other and roughly the same track record as far as reliability is concerned.
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC3-12800, Sapphire RX Vega 64, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1658
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Skylake X: Bye Bye Rings, Hello Mesh

Thu Jul 13, 2017 10:41 am

DancinJack wrote:
Kougar wrote:
Mr Bill wrote:
Pretty good server article over at Anandtech intel-skylake-ep-vs-amd-epyc-7000-cpu-battle-of-the-decade


Promising results for EPYC too. AMD's is offering more cores at lower prices and still winning a lot of the benches. :)


Yeah, unfortunately for AMD though, the server space isn't quite as forgiving as the desktop space. (I couldn't think of a better word than forgiving) I doubt AMD will see a ton of market share this year for Epyc (I still hate the name), but over the next few years we should see a good uptick for them. Intel still obviously has a big advantage of being the incumbent in a lot of cases, but AMD has done a really good job with those chips. I hope they can get the platform to a place as stable as Intel (for the most part) quickly.


AMD is definitely going to lose the high-end, based not only on the benchmarks but on the new/weird caching caused by the MCM design.

However, I'm not sure how relevant that loss will be, 4-8 socket servers are on the way out, and the ones that exist are definitely used for VMWare. This is where Epyc will win big - it has sufficient PCIe channels to be used as the core of a giant VM host server, and that's a huge market.
 
Krogoth
Silver subscriber
Gerbil Elder
Posts: 5353
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime
Contact:

Re: Skylake X: Bye Bye Rings, Hello Mesh

Thu Jul 13, 2017 8:27 pm

Vhalidictes wrote:
DancinJack wrote:
Kougar wrote:

Promising results for EPYC too. AMD's is offering more cores at lower prices and still winning a lot of the benches. :)


Yeah, unfortunately for AMD though, the server space isn't quite as forgiving as the desktop space. (I couldn't think of a better word than forgiving) I doubt AMD will see a ton of market share this year for Epyc (I still hate the name), but over the next few years we should see a good uptick for them. Intel still obviously has a big advantage of being the incumbent in a lot of cases, but AMD has done a really good job with those chips. I hope they can get the platform to a place as stable as Intel (for the most part) quickly.


AMD is definitely going to lose the high-end, based not only on the benchmarks but on the new/weird caching caused by the MCM design.

However, I'm not sure how relevant that loss will be, 4-8 socket servers are on the way out, and the ones that exist are definitely used for VMWare. This is where Epyc will win big - it has sufficient PCIe channels to be used as the core of a giant VM host server, and that's a huge market.


MCM doesn't introduce anything new as far caching is concerned. It has the same problems cache issues found in 4S+ systems. Current Eypc chips are really just "4S CPUs" built on the same CPU packing. Ring and Mesh-bug monolith approach that Intel taken has its own set of issues. It just comes down to design/performance trade-offs. Intel gambled too much on their fabrication tech to overcome the issues on the production/yields side.
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC3-12800, Sapphire RX Vega 64, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
 
the
Gold subscriber
Gerbil Elite
Posts: 851
Joined: Tue Jun 29, 2010 2:26 am

Re: Skylake X: Bye Bye Rings, Hello Mesh

Thu Jul 13, 2017 11:16 pm

Krogoth wrote:
Skylake-X/EP and Socket 2066/3674 are also brand new platforms. AMD doesn't have the sigma that had back in the K6-K7 era. What is going kill Intel is on the production side. They are having hell of a time to producing high-core count chips (seen with lack of availability with Skylake-X) since it is a PITA to produce "good" massive monolith dies. The 14nm process hasn't changed this reality. AMD has a much easier time producing high-core count chips because of MCM approach. AMD can easily engage in the price war while retaining decent margins on HPC/Enterprise market.

Intel isn't to have a viable counter for at least another three years or more (barring issues with 10nm process and beyond).


Intel does have a plan and it'll arrive sooner than your three year estimate: EMIB. Intel's first part using EMIB is shipping, though that was originally an Altera FPGA design. Combine Intel's new mesh on-die network with EMIB multi-die scaling, and Intel becomes power/thermal limited for how much they wish to scale the design.

AMD currently uses traditional wire bonding for MCM with EPYC but I see them moving over to interposers soon enough. Once AMD migrates IO to its own die, it'll be very similar to Intel's EMIB strategy at a high level. The low level implementation is just going to be very different.

nVidia recently released a paper regarding multi-die GPUs. Their strategy appears to mimic AMD's with using interposers.

The only other big data center player which hasn't announced plans to side step large die issues is IBM. They has had no problem using traditional MCM at the high end.
Dual Opteron 6376, 96 GB DDR3, Asus KGPE-D16, GTX 970
Mac Pro Dual Xeon E5645, 48 GB DDR3, GTX 770
Core i7 3930K@4.2 Ghz, 32 GB DDR3, GA-X79-UP5-Wifi
Core i7 2600K@4.4 Ghz, 16 GB DDR3, GTX 970, GA-X68XP-UD4
 
Kougar
Silver subscriber
Minister of Gerbil Affairs
Posts: 2029
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: Skylake X: Bye Bye Rings, Hello Mesh

Fri Jul 14, 2017 2:59 pm

the wrote:
The only other big data center player which hasn't announced plans to side step large die issues is IBM. They has had no problem using traditional MCM at the high end.


Sure, except those are ceramic substrate packages something around a quarter inch thick, not the thin laminated PCB Intel and AMD use. Intel already has MCM issues when it shrank Skylake's package thickness a little too far.
 
Mr Bill
Gold subscriber
Graphmaster Gerbil
Posts: 1455
Joined: Mon Jan 21, 2002 7:00 pm
Location: Colorado Western Slope
Contact:

Re: Skylake X: Bye Bye Rings, Hello Mesh

Fri Jul 14, 2017 3:19 pm

Kougar wrote:
the wrote:
The only other big data center player which hasn't announced plans to side step large die issues is IBM. They has had no problem using traditional MCM at the high end.


Sure, except those are ceramic substrate packages something around a quarter inch thick, not the thin laminated PCB Intel and AMD use. Intel already has MCM issues when it shrank Skylake's package thickness a little too far.
Does MCM stand for Multi Channel Mesh?
Kougar wrote:
NTMBK wrote:
chuckula wrote:
The most interesting part is that this isn't Intel's first mesh. That distinction goes to Knights Landing where up to 72 cores are connected together with a mesh. So if KNL hits 72 cores, it looks like there is room to scale this interconnect.


KNL has cores organised in "tiles" of 2 cores, meaning it only has 36 nodes on the mesh.


Oh, wow. I didn't know that. A few things make a lot more sense all of a sudden.

DragonDaddyBear wrote:
Any smart gerbils have an idea on how this compares to AMD's Infinity Fabric?


AMD's Infinity Fabric is based on PCI Express. So that tells us it's a point-to-point design, and isn't really comparable nor remotely as scalable as a mesh or even Intel's old ring topologies. Infinity Fabric is intended more for communication across sockets rather than across cores.
So, Intel has a mesh in which each "node" has channeling, caching, and routing capacity? But AMD's mesh more resembles a bunch of data channels between nodes? But if the nodes are CPU cores, don't the nodes then also have caching and routing capacity too?
X6 1100T BE | Gigabyte GA-990FXA-UD3 AM3+ | XFX HD 7870 | 16 GB DDR3 | Samsung 830/850 Pro SSD's | Logitech cherry MX-brown G710+ | Logitech G303 Daedalus Apex mouse | SeaSonic SS-660XP 80+ Pt | BenQ 24' 1900x1200 IPS | APC Back-UPS NS-1350 | WinXP64 Pro
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1658
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Skylake X: Bye Bye Rings, Hello Mesh

Fri Jul 14, 2017 3:31 pm

MCM = Multi-Chip Module
 
Mr Bill
Gold subscriber
Graphmaster Gerbil
Posts: 1455
Joined: Mon Jan 21, 2002 7:00 pm
Location: Colorado Western Slope
Contact:

Re: Skylake X: Bye Bye Rings, Hello Mesh

Fri Jul 14, 2017 5:56 pm

Vhalidictes wrote:
MCM = Multi-Chip Module
Thanks! When it came up in the discussion, I thought it must be a reference to mesh or fabric and that was confusing.
X6 1100T BE | Gigabyte GA-990FXA-UD3 AM3+ | XFX HD 7870 | 16 GB DDR3 | Samsung 830/850 Pro SSD's | Logitech cherry MX-brown G710+ | Logitech G303 Daedalus Apex mouse | SeaSonic SS-660XP 80+ Pt | BenQ 24' 1900x1200 IPS | APC Back-UPS NS-1350 | WinXP64 Pro
 
Kougar
Silver subscriber
Minister of Gerbil Affairs
Posts: 2029
Joined: Tue Dec 02, 2008 2:12 am
Location: Texas

Re: Skylake X: Bye Bye Rings, Hello Mesh

Wed Jul 19, 2017 1:27 pm

Mr Bill wrote:
So, Intel has a mesh in which each "node" has channeling, caching, and routing capacity? But AMD's mesh more resembles a bunch of data channels between nodes? But if the nodes are CPU cores, don't the nodes then also have caching and routing capacity too?


AMD's Infinity fabric is less of an intra-core mesh and more of an inter-core mesh. Maybe I'm splitting meaningless hairs, but I see it as a intra-core communications network as nothing inside the CCX uses it to talk elsewhere within the CCX.

The L2 in Ryzen is private so that data doesn't get shared across cores, all the cores in each complex have direct access to the L3 instead. I don't believe the Infinity Fabric even comes into play unless cores on a different CCX want to pull data from another CCX's L3. So Intel's mesh topology is a fair bit different than how AMD designed its chips. It might be more accurate to say each CCX is in a mesh network, or each CCX is a node in a mesh.

Mr Bill wrote:
Vhalidictes wrote:
MCM = Multi-Chip Module
Thanks! When it came up in the discussion, I thought it must be a reference to mesh or fabric and that was confusing.


Half of the topics in these forums goes over my head too :lol: I may not even be fully accurate, I enjoy reading deep dives but that doesn't mean I catch the important connections or even understand the material.
 
Vhalidictes
Gold subscriber
Gerbil Jedi
Posts: 1658
Joined: Fri Jan 07, 2005 2:32 pm
Location: Paragon City, RI

Re: Skylake X: Bye Bye Rings, Hello Mesh

Wed Jul 19, 2017 1:32 pm

Kougar, I think you've got an accurate description there; despite your protests you seem to have a handle on the Ryzen design :)

The way I'd put it is this: AMD is essentially putting a 4-socket system on a chip (MCM module). This is particularly apt because from what I've gathered you don't really need a motherboard chipset with Ryzen - almost all features have at least limited support right from the CPU module.

Once the APU version appears they should add a sound codec - at that point you really wouldn't need much of a mainboard at all.
 
chuckula
Gold subscriber
Gerbil Jedi
Topic Author
Posts: 1569
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Skylake X: Bye Bye Rings, Hello Mesh

Wed Jul 19, 2017 1:38 pm

Vhalidictes wrote:
Kougar, I think you've got an accurate description there; despite your protests you seem to have a handle on the Ryzen design :)

The way I'd put it is this: AMD is essentially putting a 4-socket system on a chip (MCM module).


That's actually more true than you know if you read their Epyc design guidelines. Basically the operating systems are being tuned to act as if an Epyc server is either a 4 socket server (for one Epyc package) or eight socket server (for two Epyc packages) even though phyiscally they are in two sockets. The tuning comes through specifying that each hunk of RyZen silicon is in a different NUMA domain. Frankly they could go even further and up the number 16 NUMA domains for the largest Epyc systems since each CCX on each die is really its own NUMA domain.
4770K @ 4.7 GHz; 32GB DDR3-2133; GTX-1080; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.

Who is online

Users browsing this forum: No registered users and 6 guests