The POWER7 Processor

[Shining Arcanine]

Recently, I learned about the POWER7 processor and it is a computational monster:

http://arstechnica.com/hardware/news/20 ... istors.ars
http://en.wikipedia.org/wiki/POWER7

To summarize things:

• It is an 8-issue RISC chip
• It has 8 cores
• It supports up to 4 threads per core
• It has 4MB L3 cache per core, which is not shared
• It fits 1.2 billion transistors into a 567mm^2 die on a 45nm process
• It has a 200 watt TDP
• It is capable of up to 264.96 gigaflops per chip

Unfortunately, it does not seem to be available in workstations. Does anyone know if IBM's deal with Levono keeps them from producing workstations?

By the way, it would be interesting to see benchmarks comparing the POWER7's performance to that of Gulftown, especially since I think it would validate my thinking regarding RISC versus CISC at the high-end. The differences in TDP probably could be ignored considering that the transistor counts are the same and the difference in process technologies is probably more responsible for the higher TDP of POWER7 than anything inherent to the RISC versus CISC debate.

[SNM]

It's a big superscalar processor. That "up to" is important; if you can't shove enough floating-point ops at it the processor won't come anywhere near its peak performance.

[Glorious]

Unfortunately, it does not seem to be available in workstations. Does anyone know if IBM's deal with Levono keeps them from producing workstations?

I dobut that has anything to do with it. It's not being offered as workstation because there no market for such a thing. The non-x86 workstation is dying if it isn't already completely dead.

By the way, it would be interesting to see benchmarks comparing the POWER7's performance to that of Gulftown, especially since I think it would validate my thinking regarding RISC versus CISC at the high-end.

...the only meaningful performance measurement of multi-cored, SMTing monstrosities such as these is how well they handle a massive multi-threaded workload. Such a measurement tells you little, if anything, about the relative merits of the respective ISAs.

If you were to measure single-threaded performance I would expect Gulftown to win, but that's completely missing the point with an architecture like POWER7.

The differences in TDP probably could be ignored considering that the transistor counts are the same and the difference in process technologies is probably more responsible for the higher TDP of POWER7 than anything inherent to the RISC versus CISC debate

Bad assumption. Companies don't count transistors the same way and not all transistors are the same. Same with TDP.

[ekul]

The only cross-platform benchmark is SPEC. It has its problems but it does give a good estimate.

in the speed test:
A 12 core gulftown system got 42.1 SPECint and 48.1 SPECfp
A 16 core power 780 system got 44.0 SPECint and 71.5 SPECfp

in the throughput test:

A 12 core gulftown system got 356 SPECint and 248 SPECfp
A 16 core power 780 system got 586 SPECint and 531 SPECfp

As you can see when it comes to throughput intel can't touch IBM, especially considering a 16 core server is the smallest POWER7 system IBM makes while 2 sockets is really as big as nehalem gets (nehalem-ex changes that but really big versions that that aren't ready yet)

POWER7 and Nehalem don't play on the same level. POWER7 is for huge UNIX servers that handle crushing loads. Nehalem, while intel's best-ever server design, is for small 1 and 2 socket servers.


POWER7 is a very clever design. It uses eDRAM for cache which is slightly slower but much more dense. Quad channel RAM gives it enough bandwidth to keep 32 threads fed. It has a decimal floating point unit so it can do financial calculations much faster since you can't use binary floating point for money. If only IBM had bought Sun to give it a decent operating system. I hate AIX so much. Linux on POWER7 is supposed to be pretty good though.

[Shining Arcanine]

What stops IBM from trying to replace x86 with POWER7 like Intel tried with Itanium? They actually have a good processor design while Intel did not. Could it be that they do not want to sell it at the margins that would make it competitive, even though the volumes would probably more than compensate for it?

[just brew it!]

What stops IBM from trying to replace x86 with POWER7 like Intel tried with Itanium?

Maybe Intel's experience with Itanium?

The POWER architecture was historically used in desktop systems (Apple). Apple ditched it for x86, because in the desktop and low- to mid-range server markets it is very difficult to compete with the economies of scale and established hardware/software ecosystem of x86.

POWER architecture will continue to be a niche player, in markets where x86 has historically not been dominant. (They're also used quite a bit in high-performance embedded systems, for example...)

[Shining Arcanine]

What stops IBM from trying to replace x86 with POWER7 like Intel tried with Itanium?

Maybe Intel's experience with Itanium?

The POWER architecture was historically used in desktop systems (Apple). Apple ditched it for x86, because in the desktop and low- to mid-range server markets it is very difficult to compete with the economies of scale and established hardware/software ecosystem of x86.

POWER architecture will continue to be a niche player, in markets where x86 has historically not been dominant. (They're also used quite a bit in high-performance embedded systems, for example...)

I think that Intel's experience with Itanium was not because Intel tried to replace x86, but because Intel tried to replace x86 with something inferior.

I just read Wikipedia's article on the subject of the PowerPC's history:

http://en.wikipedia.org/wiki/PowerPC#History

It seems that the POWER architecture was an attempt to counter Wintel. Unfortunately, IBM did not have the engineering resources to keep up with Intel and it fell behind, which is when Apple dropped it for Intel. IBM has since managed to leapfrog Intel with the POWER7, but it is still behind in process technologies in comparison to Intel. POWER7 on a 32nm architecture would be interesting to see, especially if they could use the die shrink as an opportunity to lower core voltages.

[SNM]

It seems that the POWER architecture was an attempt to counter Wintel. Unfortunately, IBM did not have the engineering resources to keep up with Intel and it fell behind, which is when Apple dropped it for Intel. IBM has since managed to leapfrog Intel with the POWER7, but it is still behind in process technologies in comparison to Intel. POWER7 on a 32nm architecture would be interesting to see, especially if they could use the die shrink as an opportunity to lower core voltages.

What the heck makes you think IBM has leapfrogged Intel? Power7 can, under very specific workloads, outperform Nehalem. But the only workloads it wins under are the sort that are starting to be run on graphics cards! Otherwise x86 processors have all the advantages in power consumption, legacy software, speculative execution...

[tfp]

Also the G5 that IBM made for Apple was pretty much IBM's version of the P4. Long pipeline, not a lot of work done per-clock, and made a lot of heat. Really it’s fortunate that the process tech has kept up. Even though IBM has stopped making desktop chips they were able to keep the high clock speeds on the server side and make the chip wider (and adding SMT) over the iterations allowing the chips to have a higher clock and still get more work done. The 200W per CPU thermal envelope has really helped in that department vs. what is reasonable on a desktop.

Frankly if the PPC manufactures had a company with a reasonable amount of volume and were supplying a company worth working with, unlike Apple, PPC might still have been used in desktops today. However it just really wasn't worth while financially to innovate on the PPC + Altivec front when Apple is the only real customer of PPC desktop chips. If the general OS vendors at the time wouldn't have dropped support which it had on Windows, OS/2, and Solaris for a shot time it might have fared better overall.

PPC lives on in IBM POWER CPUs, consoles like Wii, PS3, and Xbox 360, as well as embedded CPUs from companies like Freescale.

It should be noted that another major reason for IBM not taking over the desktop world with POWER7, other than cost, heat, size, ect, is that they don't make anything other than servers. They sold all of the desktop and laptop stuff because of the low margins, if I’m remembering correctly.

What the heck makes you think IBM has leapfrogged Intel? Power7 can, under very specific workloads, outperform Nehalem. But the only workloads it wins under are the sort that are starting to be run on graphics cards! Otherwise x86 processors have all the advantages in power consumption, legacy software, speculative execution...

Your points make more sense from a desktop mindset than from a server prespective. Legacy software? Power is backwards compatible with itself so it's really not an issue for anyone upgrading or adding systems. Same goes for HP moving people from PA-RISC to Itanium1 to Itanium2 ect. The same would apply to Sparc as well. If intel was really wanted x86 to compete in the very high end servers that Power and Itanium play in Intel would be upping the power usage and termal requirements of thier x86 chips to truely compete in that type of computing as well as adding other system level enhancements. (TDP 200W vs 95W or 115W or whatever) A quick google also has showen that IBM has had speculative execution on it's POWER chips and has been slowly increasing the amount of speculation over time. So I'm not sure that you are correct on all of your assumptions.

I found this link to an IBM page to be kind of funny.

http://www.ibm.com/developerworks/wikis ... nvironment

Issue:

MPI_Allreduce could give wrong results (for 64 Bit MPI applications on IBM Power 7 Servers)

Speculative instruction execution on IBM Power 7 Servers, together with an MPI_Allreduce algorithm that is missing an instruction sync, may result in some data in shared memory being accessed by one process before is completely stored by a second process. The result can be an incorrect result for the MPI_Allreduce.

Servers prior to Power 7 do not use the level of speculative execution that exposes this problem.

[SNM]

What the heck makes you think IBM has leapfrogged Intel? Power7 can, under very specific workloads, outperform Nehalem. But the only workloads it wins under are the sort that are starting to be run on graphics cards! Otherwise x86 processors have all the advantages in power consumption, legacy software, speculative execution...

Your points make more sense from a desktop mindset than from a server prespective. Legacy software? Power is backwards compatible with itself so it's really not an issue for anyone upgrading or adding systems. Same goes for HP moving people from PA-RISC to Itanium1 to Itanium2 ect. The same would apply to Sparc as well. If intel was really wanted x86 to compete in the very high end servers that Power and Itanium play in Intel would be upping the power usage and termal requirements of thier x86 chips to truely compete in that type of computing as well as adding other system level enhancements. (TDP 200W vs 95W or 115W or whatever) A quick google also has showen that IBM has had speculative execution on it's POWER chips and has been slowly increasing the amount of speculation over time. So I'm not sure that you are correct on all of your assumptions.

Well, he was discussing IBM leapfrogging Intel in an attempt to combat the Wintel juggernaut, so the desktop discussion seems pretty apt.
Not to mention that, in point of fact, x86 DOES compete in the high-end server market; check out the Top 500 list.

[bitvector]

What the heck makes you think IBM has leapfrogged Intel? Power7 can, under very specific workloads, outperform Nehalem. But the only workloads it wins under are the sort that are starting to be run on graphics cards! Otherwise x86 processors have all the advantages in power consumption, legacy software, speculative execution...

Your points make more sense from a desktop mindset than from a server prespective.

Agreed; historically, POWER has had some more advanced ISA features for the "giant, expensive server" segment of the market that have been lacking in x86 land -- e.g. more advanced virtualization / logical partitioning features or exotic, business-oriented features like hardware decimal floating point. I was working for IBM when POWER6 was released, and they had a release party where they were talking about their view of the market and their plans -- they focused mostly on displacing Itanium and SPARC in the high-end proprietary Unix (enterprise) market. IBM has always been whole-heartedly in that area of the market.

Not to mention that, in point of fact, x86 DOES compete in the high-end server market; check out the Top 500 list.

Terminology is always nebulous, but I wouldn't really consider that the "high-end server market" per se. As you probably know, the Top500 list represents a very odd and tiny niche segment for high-performance scientific computing. They basically want as many FLOPS as the power grid can handle, exotic networking interconnects and little else. I mean people at Sandia spend a bunch of time ripping the seats and floormats out of the Linux kernel just to eek out a bit more raw performance. That might change at some point, but when I think of "high-end servers" where POWER competes, I think of enterprise commercial Unix servers. Basically businesses buying expensive machines with (traditionally) Solaris, AIX or HP-UX to run random business software, not MPI jobs. And of course, "enterprise servers" is an amorphous market segment which does bleed into x86 land at various points and also all the way up into IBM's even higher-cost "z series" offerings -- where whatever computation you're running is so important that the same instruction stream is concurrently executed by two independent processors and checked to make sure the results match.

But x86 tends to dominate in any market where "cheap and in volume" is the right strategy, like filling a data center with fungible servers to reply to web requests. And the HPC community has found it works for them too, but the non-performance-critical enterprise market is still buying enough big expensive lower-volume machines for now to give POWER a raison d'etre. But now that Linux is increasingly acceptable in that area, and the greater x86 platform is adding "big boy" features like IO virtualization, it'll probably start to impinge on that market too.

[SNM]

True, but there was another thread around here recently where somebody pointed out that the chief business advantages of SPARC/Itanium/PPC are not in the chips themselves but in the proprietary nature of who can supply them.
I'm not trying to say that the Power7 is a bad processor; it indeed looks like a really nice one; I'm just saying that it has no business in any space currently occupied by x86 and that its chief strengths look to be largely shared with, say, Fermi.

[Shining Arcanine]

But x86 tends to dominate in any market where "cheap and in volume" is the right strategy, like filling a data center with fungible servers to reply to web requests. And the HPC community has found it works for them too, but the non-performance-critical enterprise market is still buying enough big expensive lower-volume machines for now to give POWER a raison d'etre. But now that Linux is increasingly acceptable in that area, and the greater x86 platform is adding "big boy" features like IO virtualization, it'll probably start to impinge on that market too.

I am not certain that x86 is doing quite that well. Look at mobile phones which, outnumber conventional computers by at least an order of magnitude and you will not find x86 anywhere. Game consoles lack x86 too, as they are all POWER based. I am not so certain that "cheap and in volume" is x86's market segment. Instead, it looks as if the space between the two, "profitable, and in volume" is where x86 is doing well. As far as I have read on technical news sites, Intel has not really been happy with their Atom processors because the margins are so low, but the reason it is there is to keep ARM from starting to scale up and threaten its more profitable sales. With Atom, it seems that Intel is trying to lose those sales to itself than to another company.

[morphine]

I am not certain that x86 is doing quite that well.

Given that close to 100% of personal computers (anything above a smartphone, basically) all use x86, I think it's doing pretty well.

If you want to say that x86's usage might start seeing some decline in the long run though, then I agree.

Look at mobile phones which, outnumber conventional computers by at least an order of magnitude and you will not find x86 anywhere.

And a mobile phone is not a personal computer.

Game consoles lack x86 too, as they are all POWER based.

Same as above.

[Shining Arcanine]

I am not certain that x86 is doing quite that well.

Given that close to 100% of personal computers (anything above a smartphone, basically) all use x86, I think it's doing pretty well.

If you want to say that x86's usage might start seeing some decline in the long run though, then I agree.

Look at mobile phones which, outnumber conventional computers by at least an order of magnitude and you will not find x86 anywhere.

And a mobile phone is not a personal computer.

Game consoles lack x86 too, as they are all POWER based.

Same as above.

Embedded devices are low-end computers and my point is that x86 is not present there.

[Glorious]

Embedded devices are low-end computers and my point is that x86 is not present there.

...and it never was, as people keep trying to tell you. Same with consoles, excepting the original Xbox, of course.

[Shining Arcanine]

It looks like Intel is planning to release 10-core Westmere-EX processors for 4+ socket servers:

http://www.pcworld.com/article/199346/i ... cores.html

That leaves Itanium no place in Intel's product line. Perhaps this is a sign that they plan to discontinue Itanium development and produce larger TDP design x86 processors to compete with POWER7.

[just brew it!]

I doubt that competing with POWER7 is on their radar; even if it is, it is almost certainly not the top priority. In the x86 server CPU space, they are far more concerned with staying a step ahead of AMD.

[tfp]

It looks like Intel is planning to release 10-core Westmere-EX processors for 4+ socket servers:

http://www.pcworld.com/article/199346/i ... cores.html

That leaves Itanium no place in Intel's product line. Perhaps this is a sign that they plan to discontinue Itanium development and produce larger TDP design x86 processors to compete with POWER7.

Seriously? Why would you think this leaves Itanium with no place in Intel's product line? Last time I checked x86 didn't run on the OSes that HP sells with Itanium, HP is pretty much the only real seller right now of Itanium systems. Also just like everyone that talks about how hard it is to migrate off of x86 to something else that applies for mainframes like POWER, Z or Itanium to x86.

Itanium might die someday but this is not going to end the product.

[Glorious]

Seriously? Why would you think this leaves Itanium with no place in Intel's product line?

Because he doesn't have any idea what he's talking about and he ignore-lists those who do.

Last time I checked x86 didn't run on the OSes that HP sells with Itanium, HP is pretty much the only real seller right now of Itanium systems.

Exactly.

HP-UX and OpenVMS run natively on Itanium now. HP made the deliberate choice to port those OSes to Itanium despite obviously having the option of porting them to x86. Those OSes aren't dead. They are alive and well and HP is doing its best to keep them that way. In fact, OpenVMS 8.4 released today. It includes, among other things, support for HP's hypervisor and other such goodies intended to try and further dissuade HP's customers from going the x86 emulation route.

Since HP and Intel jointly developed the Itanium specifically for this market (it was the replacement for the HP-owned DEC Alpha and HP' s homegrown PA-RISC), I'm sure that Intel is legally precluded from unilaterally terminating Itanium development & support. There is no real technical place for it, for sure, but that hasn't been the point of the architecture for quite some time now, at least since Madison.

Shining Arcanine, as usual, completely misses the entire point of the platform: to secure HP's server and support sales. This is not a high-volume market, to be sure, but it's an incredibly lucrative one. If you worry about the sustainability of the entire endeavor I'd consider SPARC to be the canary in the mine. I'm sure it'll go before Itanium will. Wait for it to be announced as officially dead before you start carving the tombstone for Itanium.

Also just like everyone that talks about how hard it is to migrate off of x86 to something else that applies for mainframes like POWER, Z or Itanium to x86.

It's a lot of effort for absolutely no real benefit, unless you are running specific types of systems. Of course, if you're running those types of systems you were never on x86 to begin with. You needed an industrial-strength robust and flexible OS and X86 didn't have anything like that until relatively recently. So if you were seriously into straight business enterprise you were on IBM mainframes and it was never about compute performance to begin with. If you were less serious or had other considerations as well it was about numerous other proprietary enterprise systems.In general, it was about robustness, flexibility/scalability and transactional throughput. Bitvector went over all this already, I'm essentially just restating what he said in different terms.

Ignoring the inherent difficulties of porting between the architectures, there's a severe institutional aversion to x86. I'm the youngest, by far, in my area and everyone still treats x86 like a toy. There are a few who are interested in linux, and there are a few who are interested in virtualization. But that all happens on the periphery. The core of what we do remains entirely on enterprise-level systems. We have no intention of doing anything else and every attempt to move us away from it (of which there are plenty) is strenuously resisted.

The crazy thing is that Shining Arcanine has switched to predicting the demise of x86 to now predicting its ascendancy.

[bitvector]

Look at mobile phones which, outnumber conventional computers by at least an order of magnitude and you will not find x86 anywhere.

And a mobile phone is not a personal computer.

And to expand on this -- SA, you're missing that the entire mobile domain had one big fat factor above and beyond "cheap and in volume": power consumption. Power really is the premiere concern for battery-based systems. Mobile and embedded domain people were dealing with power consumption issues decades before the "data center computing" crowd really even had it on their top 5 list. And while Intel+x86 is making strides in that area, they're still playing catch-up.

And I'm confused about what your point is about POWER and x86 again.

[Crayon Shin Chan]

I've had an IBM Intellistation POWER 275 for a year now. It's the second last POWER workstation IBM ever made - the next step up was the POWER5 Intellistation POWER 285, and that really was the end of the line. When you use it, it's obvious why - AIX just didn't have a desktop community.

FIrst off, nothing runs on AIX. It wasn't until only recently that Firefox 3 was compiled for AIX, and it crashes every time I exit it. There simply is no community. AFAIK the POWER 275 was only sold for CATIA, and the GXT6500P was the last graphics card that IBM ever made - by the time the POWER 285 came out, it was already old hat. There isn't even a build of Blender for AIX - most of the time there is no real way to use the 3D capabilities of the GXT6500P. Actually, most of the time there is little chance to use the 2D graphics capabilities too, because it's a bitch to get KDE 3.4 running - specifically, you have to install KDE 3.0 on the AIX Linux Toolbox, and then upgrade each rpm package individually to KDE 3.4, and even worse to get a relatively recent version of GNOME. There is no package management for AIX - rpm is used alongside with AIX's native installp, but it doesn't resolve dependencies and doesn't download anything for you, because there is no repository. CDE is what most people supposedly use, but it's so ugly and labyrinthine that I avoid it like the plague. There is also nothing really graphical that's worth running on AIX - again, except for CATIA, which I suspect was the only reason the POWER 275 and 285 were ever sold.

In conclusion, there was never a desktop push for the RS/6000. There was a case for it back when MATLAB R13 and Mathematica 6.0 ran on it, but it petered out 7 years ago. I really think all that it's good for is a server, or for running SAP. And definitely not a file server - SCSI disks are expensive, and the RS/6000 series uses a weird 80pin SCA SCSI to a gold fingered PCB board converter which you can never find apart from buying lots of old 9GB RS/6000 disks.

Oh yes, totally forgot to mention LInux. Run Linux on the RS/6000, and you lose most of the point of buying a RS/6000 workstation. You can only use a GXT135P at best under Linux (a Matrox G400/450), and only recently GXT4/6x00P support was added but it won't go past framebuffer support. In effect, it becomes just a server, where frankly AIX is a better choice because of LPARs and WPARs.

POWER isn't going away anytime soon, despite rumours that POWER7 will be the last POWER CPU to be made. Remember, IBM at one point could afford to develop System Z, P and the AS/400 architectures at the same time and didn't actually lose money. It's got really big bucks, because people are willing to pay that much - and not just because it is fast (the POWER7 is the only RISCky architecture left that can compete with x86 on performance). I doubt that the Itanium can compete with the POWER7 on performance, but that never really mattered for most customers. SGI made a few minor tweaks to its R10000 CPU and still sold those for 8 years. HP did the same with the PA-8000. Only IBM is coming out with real architectural improvements in the RISC world.

[Shining Arcanine]

What sort of programs would you run that need high end graphics on a RISC workstation?

[Shining Arcanine]

Look at mobile phones which, outnumber conventional computers by at least an order of magnitude and you will not find x86 anywhere.

And a mobile phone is not a personal computer.

And to expand on this -- SA, you're missing that the entire mobile domain had one big fat factor above and beyond "cheap and in volume": power consumption. Power really is the premiere concern for battery-based systems. Mobile and embedded domain people were dealing with power consumption issues decades before the "data center computing" crowd really even had it on their top 5 list. And while Intel+x86 is making strides in that area, they're still playing catch-up.

And I'm confused about what your point is about POWER and x86 again.

I am confused too. The point of this thread was to ask why POWER7 processors are not available in workstations.

[Crayon Shin Chan]

The same reason why there are no newer graphics cards than the GXT6500P that run in a RS/6000.

[derFunkenstein]

The same reason why there are no newer graphics cards than the GXT6500P that run in a RS/6000.

Maybe you ought to just come out and say whatever that reason is, because I have no idea how old that hardware is without Googling it.

My theory is: nobody wants them because they don't run the software people need on high-end workstations.

[just brew it!]

I am confused too. The point of this thread was to ask why POWER7 processors are not available in workstations.

And I think we answered that. POWER was used in workstations historically. So were 680x0, MIPS Rx000, and a bunch of other CPU architectures. That didn't work out; for a variety of reasons x86 rules the roost in workstation land, and this is unlikely to change any time soon. Having a superior architecture does not guarantee market dominance (this applies to software as well, just look at MS DOS and Windows vs. all the other OSes which have challenged them over the years).

[Shining Arcanine]

The same reason why there are no newer graphics cards than the GXT6500P that run in a RS/6000.

Maybe you ought to just come out and say whatever that reason is, because I have no idea how old that hardware is without Googling it.

My theory is: nobody wants them because they don't run the software people need on high-end workstations.

It seems that they were introduced in February 2002 and discontinued on January 2, 2009:

http://en.wikipedia.org/wiki/IBM_Intell ... _POWER_265
http://www-03.ibm.com/systems/intellist ... index.html

There is not much information available on the actual specifications of the card, with the exception of IBM's statement that it provides "leading-edge OpenGL performance".

[just brew it!]

There is not much information available on the actual specifications of the card, with the exception of IBM's statement that it provides "leading-edge OpenGL performance".

Yeah, leading edge compared to other cards which were available in 2002...

[Shining Arcanine]

There is not much information available on the actual specifications of the card, with the exception of IBM's statement that it provides "leading-edge OpenGL performance".

Yeah, leading edge compared to other cards which were available in 2002...

Maybe IBM invented the flux capacitor and started selling chips from 2010 in 2002 by importing them from the future.

Anyway, I think every graphics card manufacturer's page claims leading edge performance, even the really old ones that are for parts that are obsolete. I wonder what versions of OpenGL are supported by the card. That would probably be a better indication of its performance.