TR Forums

Hey all, I'm in the process of sourcing parts for a build that will include an i7-5960x, Asrock x99 Taichi, and 128GB kit of GSkill Ripjaw V 3000MHz RAM.

(Just to head that one off at the pass - I do fluid sim work. Yes, I very much do need 128GB of RAM )


Initially I just assumed 128GB would be fine, as the motherboard specs listed it as a maximum. But on closer inspection, Intel ARK states that Haswell-E chips only officially support 64GB.

There are a few examples around the net that people have had it working fine with 128, but further to that, some vague suggestions that while Windows will show the motherboard reporting 128GB, the CPU will only be limited to addressing 64GB of it.

The reason given for this is a line in the Intel datasheet for the 5960x which reads:
"The processor supports up to 46 bits of physical address space and 48 bits of virtual address space."
- where supposedly the 46 bits physical address space works out to a 65,536MB limit.

But then on the more recent i7-6950x datasheet, a CPU which officially supports 128GB, it reads:
"The processor supports up to 46 bits of physical address space and 48 bits of virtual address space."
- which strikes me as remarkably similar.

Can anyone shed any light on this? One person absently suggested that so long as the RAM used was single-rank or dual-rank, not quad-rank, it would work fine... but they failed to elaborate beyond that statement. I have no idea how RAM ranks affect anything... would this be a factor?


All discussions of the topic I've dug up so far have so much "just enough knowledge to be dangerous" noise, that I can't work out what's going on. Ideally I'd like to work out the answer before I break the seals on the boxed 5960x

Why are you not using Xeons for this?

It's 64GB. It was literally the first link when I googled this. 6950X will get you 128.

http://ark.intel.com/compare/82930,94456

I'm hoping to run an overclock, as I need all the performance I can get, so I thought I'd try out a non-Xeon approach.

Yes, the second line I wrote was "But on closer inspection, Intel ARK states that Haswell-E chips only officially support 64GB."
- I'm well aware that's the official line on the matter, (and as such asking any Intel support channel will get me nowhere)... but the reality is clearly not as cut-and-dried. I'm trying to understand whether there is some technical reason why the limit is 64, or if it's simply down to that being the arbitrary figure they officially support. As I said, some people have had it work with 128 - I'm trying to understand the technicalities, so I can work out whether their cases were just reporting a larger RAM pool than they could actually use, or if the chip really can access it just fine.

I sure wouldn't count on it working. My guess is that Intel intentionally disabled the address bit that enables 128GB support, as a product segmentation move.

Something else to consider... you're probably not going to be able to run the memory as fast if you cram the maximum amount in there. Would your application benefit more from a larger amount of slower memory, or a smaller amount of faster memory?

Yeah, I figured I might not get the full 3000 out of the RAM... I certainly could do with more-and-slower... just figured I'd see how much I could nudge it towards more-and-quicker.

Bah... I figured the good deal I got on the 5960x would be a good way to get an extra 2 cores for similar money over the 6850k... but it seems like there's a real chance it won't actually work with this amount of RAM.
I've certainly come across a few different cases where people have had 128GB installed with a 5960x, and Windows has reported the full amount... do you reckon it's possible for the full amount to be visible to Windows even while the CPU can't address all of it?

I think it's a motherboard BIOS thing.

I had to do something similar for a bank of i7-5830 nodes a while back. Asrock were the only board vendor that I could find that stated 128GB compatibility - So I bought 8 X99 Extreme4 boards and 1TB of RAM. The only hiccups I had were that the RAM required the latest board firmware before it all worked.

If Windows sees it and the system boots and runs clean maybe it is just a matter of Intel not validating operation with 128GB. In which case you may be OK.

Yeah, I'm hoping it's just an Intel validation thing, and the capability is there.

Thanks Chrispy, good to know someone's actually tried it. I've definitely got an Asrock board that states support for it, and I'll ensure I've got the latest bios installed just in case. Fingers crossed it works out then!

Guess I'll know soon enough, and well, if I can't get it to work, the chip will just have to go back on eBay!

More likely because 16GB unbuffered DDR4 modules came out after the cpu launch, it has enough hardware bits to address it.

For a well confirmed earlier example: socket 1366 desktop models list a maximum of 24GB (6x4) but are widely known to work with 48GB (6x8) unbuffered DDR3. For a counterpoint 16GB unbuffered DDR3 modules are known to not be supported by 115x cpus, though they were a very late update near the end of life of the standard.

On a related note, asrock is the best vendor choice when attempting to do things "outside of spec" or "not officially supported".

46-bit address space is certainly not the limit. It amounts to 64 terabytes. Maybe terawords. Or, expressed in a more sinister way, tebibytes.

Bauxite wrote:

More likely because 16GB unbuffered DDR4 modules came out after the cpu launch, it has enough hardware bits to address it.

For a well confirmed earlier example: socket 1366 desktop models list a maximum of 24GB (6x4) but are widely known to work with 48GB (6x8) unbuffered DDR3. For a counterpoint 16GB unbuffered DDR3 modules are known to not be supported by 115x cpus, though they were a very late update near the end of life of the standard.

The 16 GB DDR3 unbuffered DIMMs were always part of the spec, Intel just implemented it wrong on Nehalem through Haswell. Broadwell added unofficial support for 16 GB unbuffered DIMMs. Oddly, the Silvermont Atoms work with 16 GB DDR3 DIMMs. AMD's chips were never validated with 16 GB unbuffered DDR3 DIMMs but they reportedly work just fine.

Similarly, DDR4 memory permits up to 512 GB LRDIMMs in the spec no one is shipping them yet. I think that the largest unbuffered DDR4 DIMM according to spec is 64 GB but again, no one is shipping them. There is plenty of room to grow as the successor isn't expected until 2020ish.

Heh, yeah, I was just giving some thought to what 46-bits of addressable memory would really mean. I worked it out at 8 terabytes though... (64 terabits) - my maths may well be wrong though

Well, certainly seems like it's worth a try anyway.

@Bauxite
Yeah, I tried Asrock a few years ago after getting fed up with Asus and MSI issues... I've never felt the need to look elsewhere since - they seem to have the least bugs and quirks, the least annoyingly-laid-out bios pages, the clearest specs pages and manuals, and the better thought out feature sets.
Strange that they started as a spin-off of Asus' budget brand.

Wirko wrote:

46-bit address space is certainly not the limit. It amounts to 64 terabytes. Maybe terawords. Or, expressed in a more sinister way, tebibytes.

That is indeed the current limit for x86 systems (64 terabytes). SGI (now part of HPE) makes some systems capable of reaching that capacity in a single NUMA system. Those large systems also hit the limit of how many CPU cores can be in a system.

Sky Lake-EP is extending both of those limits when it ships later this year.

If it were me, doing professional engineering work with 128GB of RAM i'd definitely get ECC RAM which requires a Xeon. I know you said you need overclocking, but the peace of mind of knowing that my final results are accurate and didn't have bits flipped during the calculation would be worth it to me. The more RAM you have, the higher the probability that one of those bits can be flipped erroneously.

I'd get a xeon too (and did w/ 8x32 reg ecc, same $/GB as 16 sticks at the time) because the venn diagram for "engineering", "simulation", "loves big pools of ram" and "heavily threaded" is a strong overlap.

Not sure what you're paying for a 5960x, but at its usual store price range you can easily find a 16/18/20/22 core xeon on fleabay of various providence. My production 22 core was the same as the stupidly overpriced 10 core i7. If you carefully select ES/QS they can be a lot cheaper, and conveniently asrock is the best choice for non-production cpus as well.

just brew it! wrote:

If Windows sees it and the system boots and runs clean maybe it is just a matter of Intel not validating operation with 128GB. In which case you may be OK.

The main issue isn't that it won't work at all. It probably will for some value of stability.

The issue is that Overclocking with that much RAM is probably a no-go. Or even running the RAM at a fairly high speed.

EDIT: A similar Xeon might be cheaper, as well.

This dude seems to have gotten it to work.

You should be ok with 128GB, I have an X99 box we just upgraded to 128GB...

edit: crap, I think it's Broadwell-E in that box.

the wrote:

Wirko wrote:

46-bit address space is certainly not the limit. It amounts to 64 terabytes. Maybe terawords. Or, expressed in a more sinister way, tebibytes.

That is indeed the current limit for x86 systems (64 terabytes). SGI (now part of HPE) makes some systems capable of reaching that capacity in a single NUMA system. Those large systems also hit the limit of how many CPU cores can be in a system.

Sky Lake-EP is extending both of those limits when it ships later this year.

You're right, sir. My point (poorly worded) was that CPU address space is not the limiting factor even if the platform was/is limited to 64GB RAM.

Takeshi7 wrote:

If it were me, doing professional engineering work with 128GB of RAM i'd definitely get ECC RAM which requires a Xeon. I know you said you need overclocking, but the peace of mind of knowing that my final results are accurate and didn't have bits flipped during the calculation would be worth it to me. The more RAM you have, the higher the probability that one of those bits can be flipped erroneously.

Bauxite wrote:

I'd get a xeon too (and did w/ 8x32 reg ecc, same $/GB as 16 sticks at the time) because the venn diagram for "engineering", "simulation", "loves big pools of ram" and "heavily threaded" is a strong overlap.

Not sure what you're paying for a 5960x, but at its usual store price range you can easily find a 16/18/20/22 core xeon on fleabay of various providence. My production 22 core was the same as the stupidly overpriced 10 core i7. If you carefully select ES/QS they can be a lot cheaper, and conveniently asrock is the best choice for non-production cpus as well.

It's professional visual-effects work... so nothing of vital importance. The most I lose is a few re-simmed frames if a sim crashes outright. Speed is the biggest deal, and VFX software is a pretty mixed bag of multi-threaded and single-threaded processes, so a speedy 8-core is a decent trade-off. I have my (small) rack of Xeons for the fully threaded heavy-lifting like rendering anyway. This build is the day-to-day workhorse.

As another reference, here is an old AnandTech article about it:
http://www.anandtech.com/show/9188/gski ... t-ddr42800
To be more precise, it is 8Gbit DDR4.

If I may add some fuel to the fire, I would do the following:

1) Take it as gospel that Intel knows exactly how much ram the processors they make are capable of using, even if the limit is an artificial one that Intel introduced as a way of getting potential buyers to pick a Xeon over an i7.

2) See if the application you normally use is GPU accelerated (I'm willing to bet money that it does) and then read this article:

https://www.pugetsystems.com/labs/hpc/N ... -wait-775/

The guy that wrote it had a Ph.D. in Theoretical Chemistry and has done more than a few reviews of hardware for Puget using Linux+CUDA running fluid dynamics and particle simulation workloads.

If you're familiar with NAMD, read this article and pay attention to this sentence:

https://www.pugetsystems.com/labs/hpc/N ... 0-GPU-815/

Running NAMD with GPU acceleration can increase performance by a factor of 8-10 over CPU alone! This is enough performance to facilitate moderate sized MD simulations to be run in a reasonable amount of time on a single node workstation.

Now, are you sure you're going about this the right way?

sophisticles wrote:

1) Take it as gospel that Intel knows exactly how much ram the processors they make are capable of using, even if the limit is an artificial one that Intel introduced as a way of getting potential buyers to pick a Xeon over an i7.

Well, aside from the people saying they've actually gotten it to work. At this point I'm going with "enabled in the silicon but not officially validated by Intel". Which isn't to say that they couldn't still disable it on future steppings (since it isn't officially supported).

sophisticles wrote:

2) See if the application you normally use is GPU accelerated (I'm willing to bet money that it does)

I would think that if there was a GPU accelerated version, OP would've already looked into that angle. He knows what software he's using, and would presumably be aware of its capabilities. OP also said: "VFX software is a pretty mixed bag of multi-threaded and single-threaded processes". Single-threaded isn't going to benefit at all from GPU acceleration.

just brew it! wrote:

I would think that if there was a GPU accelerated version, OP would've already looked into that angle. He knows what software he's using, and would presumably be aware of its capabilities. OP also said: "VFX software is a pretty mixed bag of multi-threaded and single-threaded processes". Single-threaded isn't going to benefit at all from GPU acceleration.

In his opening post he said he does "sim" work, which I took to mean Simulations, which are very well threaded workloads; later on he changed it to semi-pro visual effects work, but even if it's a 50/50 mix of single threaded and multi-threaded workloads, he may still see some good speed gains with some gpu loving.

I'm not aware of any GPU fluid solvers that would be able to handle the kind of job that generates 100+ GB of RAM usage, because VRAM is so much more constrained. The reason you need to keep that whole 100 GB in RAM is that what happens to 1 voxel is dependent on the 26 voxels that surround it, and what happens to *those* voxels is dependent on etc, etc ... consider a 1024x1024x1024 cube of voxels that each contain float information about heat, velocity (* 3 axes), density, fuel, smoke, fire, etc.

You definitely need a good 3D card for getting your scene set up, but once you click simulate, it's all gonna be CPU out of necessity.

Disclaimer: I haven't done any professional VFX fluid-sim work in about six years, so my viewpoint might be outdated.

Yeah, pretty much got it. There are certainly occasionaly workloads that benefit from me switching OpenCL on, but the largest simulations just can't fit inside the available VRAM.

I've got that side of things covered with a newly acquired 1080 ti - that 11GB will be able to handle some pretty significant smoke/fire sim, but water sim is my main focus, and that is a lot more complex, hybrid particle/grid solve system that isn't even clearly faster when it *can* fit on a GPU.

Actually, even just my day-to-day non-sim workflow has recently started hitting up against the 32GB I have in my current machine constantly. It seems like everything I do lately requires processing a hundred-million polygon mesh.

Highly tuned, specialised simulation applications may well benefit from GPU acceleration as a rule, but for a varied VFX workload, GPU acceleration is just too specialised and inflexible to be useful to the point where I focus primarily on it. GPU acceleration giving an 8-10x speed increase is very much "under ideal circumstances"... and well, in VFX, nothing is ever under ideal circumstances I certainly use it whenever it's useful to, but often it takes more effort to tune, tweak and optimise it enough that it even runs, than you save in time running it. If I had to run the same basic simulation, unchanged, a thousand times, it'd be worth taking the time, but I typically have to build a new bespoke setup for every handful of shots I work on.


Anyway, thanks all for your thoughts and help on the matter. Always appreciated. I should have the build up and running in the next few days, so I'll post here to say how it went. Fingers crossed I haven't misjudged anything too badly!

Well, as an update to all this - I put it all together, and everything works great!

System with everything running stock worked perfectly. The x99 Taichi already seemed to have the latest bios, and the 128GB RAM worked out of the box.

I tried loading the XMP profile for 3000MHz, 1.35V, 125 BLCK, and 14-14-14-34 timings... and well, it didn't get far, as expected
I've since settled on a conservative middle-ground - 2666MHz, 1.2V, 100 BCLK, and the same 14-14-14-34 timings, and it seems to be rock-solid so far.

The 5960x handles being run at 4.2GHz with 1.15V core, which seems like a respectable enough middle-ground overclock. Not sure I particularly want to push it any further... kicks out too much heat for my case to successfully exhaust

Good for you that it works.

Any benchmarks of the upgrade compared to your old system? Show us some dramatic diference!

Hehe, well, for gaming, it's basically identical performance *with* the overclock (Over the lowly, but remarkably resilient 3770k I had before, which ran at an easy 4.4 OC for the past 4 years). I guess that *might* improve a little over time as more games make use of DX12/Vulkan threading features, and everyone starts fully exploiting Ryzen's 8 cores.

For VFX, well, it's basically twice as fast for the threaded stuff, and I can at last return to a state of disregarding RAM footprint - 128GB feels like moving into a spacious new house, and thinking of wonderful new ways to fill it up with clutter

GrimDanfango wrote:

Well, as an update to all this - I put it all together, and everything works great!

System with everything running stock worked perfectly. The x99 Taichi already seemed to have the latest bios, and the 128GB RAM worked out of the box.

I tried loading the XMP profile for 3000MHz, 1.35V, 125 BLCK, and 14-14-14-34 timings... and well, it didn't get far, as expected
I've since settled on a conservative middle-ground - 2666MHz, 1.2V, 100 BCLK, and the same 14-14-14-34 timings, and it seems to be rock-solid so far.

The 5960x handles being run at 4.2GHz with 1.15V core, which seems like a respectable enough middle-ground overclock. Not sure I particularly want to push it any further... kicks out too much heat for my case to successfully exhaust

Hi GrimDanfango,

I'm doing similar VFX with sims etc. I already have 5960x and was considering upgrading my cpu to another that could address 128gb. Now after reading your post I'm having a change of heart. I was wondering if you can confirm that the machine is actually addressing all 128gb?