Personal computing discussed

Moderators: renee, Flying Fox, Thresher

 
bandannaman
Gerbil
Topic Author
Posts: 10
Joined: Fri Dec 17, 2004 12:55 pm

Will a high-RAM system run the RAM at full rated speed?

Sat Mar 29, 2014 7:32 pm

OK experts, I'm hoping y'all can help me out here.

My next personal build (whenever that is) is going to be a virtualization monster. I'm a software engineer and I frequently deal with complicated multi-VM deployments, so having a system where I can go crazy with Vagrant, Puppet, LXC, or whatever and get it working or failing fast is attractive to me. Also because e-peen.

A 64 GB quad-channel system like the ASUS X79 Deluxe (TR review http://techreport.com/review/25310/asus-x79-deluxe-motherboard-reviewed, ASUS https://www.asus.com/Motherboards/X79DELUXE/) or the "workstation" variant (https://www.asus.com/Motherboards/P9X79E_WS/) would be a candidate if I were buying today. Max non-overclocked RAM speed is 1866.

Mighty indeed would be 256 GB (registered RAM) in their Xeon Dual-Proc: https://www.asus.com/Motherboards/Z9PED8_WS/. Max RAM speed is also 1866 (and nobody overclocks in that space).

What I'm wondering is, in either case, whether that 1866 is still achievable with every memory slot filled with the max supported RAM. I can't find the information even from the information available in their QVLs and mobo manuals.

I remember feeling burned a few years ago when I tricked out my new system with expensive high-end RAM filling both channels (this was a big deal at the time), but it turned the command rate down from 1T to 2T -- and my Prime95 scores tanked. ISTR it also being harder to make stable without lowering the speed. I basically shouldn't have spent the extra money on the RAM -- lower-spec RAM would have been fine.

So -- what's the story? Is it worth buying high-dollar RAM (from the QVL) in hopes of hitting those rated speeds? If not, at what point does the "hit" start to take hold? Is it more about number of slots filled, number of channels used, total RAM size, ...?
----
Bandannaman
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Will a high-RAM system run the RAM at full rated speed?

Sat Mar 29, 2014 7:44 pm

I don't have a conclusive answer for you, but registered DIMMs in general tend to be much more tolerant of having all channels stuffed. OTOH this comes at the expense of lower baseline performance (higher latency).
Nostalgia isn't what it used to be.
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: Will a high-RAM system run the RAM at full rated speed?

Sat Mar 29, 2014 11:56 pm

IIRC 1866 is only possible with one DIMM per channel on X79 (Four DIMMs total).

Beyond that the memory controller is supposed to drop back to 1600.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
ronch
Graphmaster Gerbil
Posts: 1142
Joined: Mon Apr 06, 2009 7:55 am

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 1:29 am

With AMD FX my board's manual says putting two DIMMs per channel will drop speeds down to 1600, from 1866. So yeah, same as Intel. These DIMMs we have today have very tight timings.
NEC V20 > AMD Am386DX-40 > AMD Am486DX2-66 > Intel Pentium-200 > Cyrix 6x86MX-PR233 > AMD K6-2/450 > AMD Athlon 800 > Intel Pentium 4 2.8C > AMD Athlon 64 X2 4800 > AMD Phenom II X3 720 > AMD FX-8350 > RYZEN?
 
ozymandias
Gerbil XP
Posts: 468
Joined: Mon Nov 22, 2004 9:50 am

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 2:12 am

It could, but would be very difficult above 1866. Owning a 64GB system myself, I can tell that there are several difficulties in having such a system.

On the other hand, the massive amount of bandwith generated by quad channel kind of compensates for the slower MHz.
 
Voldenuit
Minister of Gerbil Affairs
Posts: 2888
Joined: Sat Sep 03, 2005 11:10 pm

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 2:25 am

ozymandias wrote:
It could, but would be very difficult above 1866. Owning a 64GB system myself, I can tell that there are several difficulties in having such a system.

On the other hand, the massive amount of bandwith generated by quad channel kind of compensates for the slower MHz.


What about going to 2T? The latency hit would be pretty bad to some memory-intensive tasks, wouldn't it? Any advice for the OP to restore 1T command rate to RAM?
Wind, Sand and Stars.
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 8:10 am

Ryu Connor wrote:
IIRC 1866 is only possible with one DIMM per channel on X79 (Four DIMMs total).

Beyond that the memory controller is supposed to drop back to 1600.

I wonder if that is also the case for the 2nd board (with the registered DIMMs)...

ozymandias wrote:
On the other hand, the massive amount of bandwith generated by quad channel kind of compensates for the slower MHz.

It can't completely mask the longer latencies though, for workloads that have highly random access patterns.

Voldenuit wrote:
What about going to 2T? The latency hit would be pretty bad to some memory-intensive tasks, wouldn't it? Any advice for the OP to restore 1T command rate to RAM?

Referring specifically to the board with registered DIMMs again - Is 1T even *possible* with registered DIMMs?
Nostalgia isn't what it used to be.
 
Yugiyurigyu
Gerbil
Posts: 15
Joined: Fri Sep 23, 2011 7:59 am

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 9:16 am

Here's what it is for supermicro motherboards (page 4, "Socket R"). Looks like if you do up to 1 dimm per channel, no more than dual-ranks per dimm, then you can get full speed.

http://www.supermicro.com/support/resou ... config.pdf


I'm pretty sure it's a function of the processor (ever since Nehalem). The old Nehalem datasheets from intel used to have similar tables, but I can't seem to find them on the new datasheets (for Ivy Bridge EP).

Ivy Bridge EP datasheets, if you're interest.
http://www.intel.com/content/dam/www/pu ... -vol-1.pdf
http://www.intel.com/content/dam/www/pu ... -vol-2.pdf

Nehalem datasheet. page 127, table 3-2 is what you'd be looking for.
http://www.intel.com/content/dam/www/pu ... asheet.pdf


Also, bigger dimms (32gb+ tend to be "load-reduced"). No idea how that affects things, but it might.
 
f0d
Gerbil XP
Posts: 422
Joined: Sun Jan 23, 2011 3:07 pm
Location: austrALIEN

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 9:43 am

it can be done but it all depends on how good your cpu and the memory controller on it is

as others have said it is "supposed" to drop to 1600 when putting 2dimms per channel but if you have a cpu that has a good memory controller on it i think you should be fine with 1866@64g

memory overclocking is like cpu overclocking though you might get lucky and get a cpu that can handle 1866 or even 2133 and higher but you might even get one that wont go any higher than 1600 so its all to do with the luck of the draw

you could try and put more voltage into the memory controller and make the memory controller more stable at higher memory speeds but some people dont like doing that so i am hesitant to recommend it


if it was me trying to run high speed memory with 2 dimms per channel i would get 1 speed memory higher than i want to run (so 2133 instead of 1866 if i want to run at 1866) and i would try that see how it goes and **maybe** bump up the memory controller voltage on the cpu if it doesnt like the higher speed (depends on how brave you are)


either way i dont think the performance difference from 1600 - 1866 is worth the hassle in most cases and if the memory does only run at 1600 i dont think it would be all much much difference performance wise (reviews of memory intensive programs that i have seen doesnt show much difference between 1600 and even 2133)

edit: this only applies to x79 and the memory controller on i7 cpu's as i have one and have screwed around with it alot (i have had memory overclocked to 2600 and oc'd the cpu to 5ghz) as far as xeons go they are a different thing altogether and i dont think you will be able to run anything other than officially supported speeds
Last edited by f0d on Sun Mar 30, 2014 11:35 am, edited 1 time in total.
Image
 
just brew it!
Administrator
Posts: 54500
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 10:05 am

f0d wrote:
memory overclocking is like cpu overclocking though you might get lucky and get a cpu that can handle 1866 or even 2133 and higher but you might even get one that wont go any higher than 1600 so its all to do with the luck of the draw

Yup, this sort of thing has been true for a number of years. My old M3A78-CM (DDR2 Socket AM2+) with timings set to "Auto" would drop the memory bus to 667 whenever all DIMM slots were filled. Manually bumping it back to 800 didn't seem to cause any problems though...

f0d wrote:
either way i dont think the performance difference from 1600 - 1866 is worth the hassle in most cases and if the memory does only run at 1600 i dont think it would be all much much difference performance wise (reviews of memory intensive programs that i have seen doesnt show much difference between 1600 and even 2133)

Well, going from 1600 to 1866 is only a 17% bump in raw speed, effectively even less if you need to loosen up some of the timings to get there. Then factor in the fact that memory performance is only one piece of the puzzle when dealing with real-world application performance. At the end of the day you're probably looking at an effect that is down in the low single digits in terms of percentages. IOW, enough to be measurable on synthetic benchmarks, but not enough to be perceptible by the user.
Nostalgia isn't what it used to be.
 
drsauced
Gerbil Jedi
Posts: 1543
Joined: Mon Apr 21, 2003 1:38 pm
Location: Here!

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 10:37 am

I tend to err on the side of reliability with VM Monsters, which means e-peen will suffer. But, you know, after a bigger e-peen machine gets tired, you, uh, keep chugging along. Ahem.

At a certain point, as JBI pointed out, there will be other considerations for the whole system. I'm finding RAM capacity doesn't have as much to do with performance as plain old storage capacity. With a lot of VM's, VHD and VHDx storage becomes an issue, and if you're going to SAN the VHDs, then network bandwidth becomes an issue. I'm leaning towards local/RAID storage for VHD in the next round of our VM Monster upgrade. In 2020, when we'll have budget.
Calm seas never made a skilled mariner. But, sadly I'm an A's fan.
 
Corrado
Minister of Gerbil Affairs
Posts: 2574
Joined: Sun Feb 17, 2002 7:00 pm

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 11:07 am

drsauced wrote:
I tend to err on the side of reliability with VM Monsters, which means e-peen will suffer. But, you know, after a bigger e-peen machine gets tired, you, uh, keep chugging along. Ahem.

At a certain point, as JBI pointed out, there will be other considerations for the whole system. I'm finding RAM capacity doesn't have as much to do with performance as plain old storage capacity. With a lot of VM's, VHD and VHDx storage becomes an issue, and if you're going to SAN the VHDs, then network bandwidth becomes an issue. I'm leaning towards local/RAID storage for VHD in the next round of our VM Monster upgrade. In 2020, when we'll have budget.


If you're doing nfs or 1gbit iSCSI, sure. If you're doing FC, 10gbit FCoE, or 10gbiit iSCSI, or hell, even LACP with 2 1gbit iSCSI provided its a dedicated iSCSI vlan, bandwidth is not much of a concern unless you're running mass disk IO on a 5 drive sata pool/aggregate/raid group. We have 600+ VMS on a 10k sas pool with about 150 900gb drives on 8gbit FC. We don't see more than 2-3gbit peak and that's during virus definition pushes or during patching. You're likely to be far more bound by your storage backend than by the interconnects.
 
Ryu Connor
Global Moderator
Posts: 4369
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: Will a high-RAM system run the RAM at full rated speed?

Sun Mar 30, 2014 3:42 pm

just brew it! wrote:
Ryu Connor wrote:
IIRC 1866 is only possible with one DIMM per channel on X79 (Four DIMMs total).

Beyond that the memory controller is supposed to drop back to 1600.

I wonder if that is also the case for the 2nd board (with the registered DIMMs)...


http://www.anandtech.com/show/7479/serv ... s-memory/2

According to AnandTech IVB-E can handle up to 2 DIMMs per channel at 1866 if it is registered memory or load reduced memory.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
bandannaman
Gerbil
Topic Author
Posts: 10
Joined: Fri Dec 17, 2004 12:55 pm

Re: Will a high-RAM system run the RAM at full rated speed?

Tue Apr 01, 2014 5:43 pm

Ryu Connor wrote:
http://www.anandtech.com/show/7479/serv ... s-memory/2

According to AnandTech IVB-E can handle up to 2 DIMMs per channel at 1866 if it is registered memory or load reduced memory.


Great link, not sure how I missed that one.

drsauced wrote:
I tend to err on the side of reliability with VM Monsters, which means e-peen will suffer. But, you know, after a bigger e-peen machine gets tired, you, uh, keep chugging along. Ahem.


Indeed. Slow and steady, but utterly reliable to perform on command!

drsauced wrote:
At a certain point, as JBI pointed out, there will be other considerations for the whole system. I'm finding RAM capacity doesn't have as much to do with performance as plain old storage capacity. With a lot of VM's, VHD and VHDx storage becomes an issue, and if you're going to SAN the VHDs, then network bandwidth becomes an issue. I'm leaning towards local/RAID storage for VHD in the next round of our VM Monster upgrade. In 2020, when we'll have budget.


I don't expect to use a SAN with this system, although who knows what the tech tradeoffs will be by the time I pull the trigger. The plan is just to try to get as much internal bandwidth as possible to the disk subsystem(s) and rely on RAM to cache/accelerate.

f0d wrote:
either way i dont think the performance difference from 1600 - 1866 is worth the hassle in most cases and if the memory does only run at 1600 i dont think it would be all much much difference performance wise (reviews of memory intensive programs that i have seen doesnt show much difference between 1600 and even 2133)

edit: this only applies to x79 and the memory controller on i7 cpu's as i have one and have screwed around with it alot (i have had memory overclocked to 2600 and oc'd the cpu to 5ghz) as far as xeons go they are a different thing altogether and i dont think you will be able to run anything other than officially supported speeds


Right -- I don't plan to overclock this rig at all. The main thing I'm wondering is about whether the money to get 1866 (or whatever the "max" non-OC number is for a mobo) would be flat-out wasted if the system infrastructure wouldn't even try to use the extra speed.

Yugiyurigyu wrote:
Here's what it is for supermicro motherboards (page 4, "Socket R"). Looks like if you do up to 1 dimm per channel, no more than dual-ranks per dimm, then you can get full speed.

http://www.supermicro.com/support/resou ... config.pdf


Great doc -- I was getting confused looking a the QVLs for the high-end motherboard since they kept showing the "ranks" for the DIMMs -- something I've never heard of, and something I assume consumer-grade RAM never has to deal with (?) since it's a Xeon thing.

The fact that it's processor-dependent and not really chipset- or motherboard- dependent is also interesting -- I thought it might have been a motherboard quality thing. That simplifies it down quite a bit.

Great answers everyone, I very much appreciate the info and links. Thanks!
----

Bandannaman
 
Deanjo
Graphmaster Gerbil
Posts: 1212
Joined: Tue Mar 03, 2009 11:31 am

Re: Will a high-RAM system run the RAM at full rated speed?

Tue Apr 01, 2014 5:56 pm

ronch wrote:
With AMD FX my board's manual says putting two DIMMs per channel will drop speeds down to 1600, from 1866. So yeah, same as Intel. These DIMMs we have today have very tight timings.


Is that with the system in your profile ronch? If it is, I'm surprised. My 990FX Sabertooth has no issues running all four banks at 2133.

http://valid.x86.fr/ret86h

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On