Dude, where's my 4GB?

Don't see a specific place for your hardware question? This is the forum for you!

Moderators: mac_h8r1, Nemesis

Postposted on Mon Jul 16, 2007 2:31 pm

OK, makes sense. Not much, but it makes sense.

Still SLIGHTLY disingenious, as PAE is usually considered as a method of getting more than a 4 GiB address space, but...
bhtooefr
Darth Gerbil
Silver subscriber
 
 
Posts: 7754
Joined: Mon Feb 16, 2004 11:20 am
Location: Holy crap, a city! (Newark, OH)

Postposted on Mon Jul 16, 2007 3:00 pm

sounds like PAE is a prerequisite for NX/XD, so it's only supported enough that it can use other processor features dependant upon it.
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Mon Jul 16, 2007 3:02 pm

Ok people, assuming I got all of this, here's a sort of compiled mini-guide. Please correct anything I have wrong:
----

Summing it up:

- You can't get much more than 3GB out of 32-bit Windows XP, period. Addressable amount will vary (3 / 3.3 / 3.5GB) depending on the devices you have, mainly your graphics card.
    Why? above 3GB is necessary for device mapping, for example, drivers accessing your graphics card RAM as if it were main memory.
- PAE does *not* help in the former case.
    Why? Enabling PAE is only good for enabling the NX/XD (DEP) functionality when using modern CPUs, because even if it's enabled, 32-bit Windows will not map stuff above the 4GB boundary, thus falling in the previous situation.
- The "memory remap" option in the BIOS does *not* help you in the former situation. The 32-bit addressable space problem remains with 32-bit WinXP. It helps in other situation (see below).

How can I use more than 3GB of RAM then?

- Your motherboard must support more than 3GB (well duh)
- Your motherboadd must support memory remapping (most modern boards do, look for "Memory remap above 4G" or something similar).
    Why? The 3-4GB area is for devices. This "splits" the RAM in two pieces below 3GB and above 4 or so GB, respectively. Don't worry, the PC knows what it's doing.
- Your operating system must be 64-bit OR 32-bit Windows Server, or Linux with proper PAE support.
    What? In more realistic/practical terms, think WinXP 64, Vista 64, some 64-bit variation of any Linux distro, etc.

BIG NOTE: If you have a 64-bit operating system but your program is 32-bit only, it will still be only able to address 4GB. That means that, i.e., even if you're running XP64 with 8GB of RAM, Supreme Commander will only be able to use, at maximum, 4 of those 8 gigabytes.

-- This "guide" created with information from, among others: bitvector (mainly) / UberGerbil / derFunkenstein / just brew it! / Convert
Last edited by morphine on Mon Jul 16, 2007 3:33 pm, edited 2 times in total.
morphine
Gerbil Khan
Silver subscriber
 
 
Posts: 9952
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Postposted on Mon Jul 16, 2007 3:07 pm

Good job morphine! My eyes glazed over after the first page. 8)
JJCDAD
Gerbil Jedi
 
Posts: 1867
Joined: Fri Sep 17, 2004 3:11 pm
Location: Is this heaven? No, it's Iowa.

Postposted on Mon Jul 16, 2007 3:26 pm

morphine wrote:Why? This enables remapping all devices' memory addresses to "above" your RAM for 64-bit OSes.

Actually it moves part of your physical RAM addresses above the device addresses since that's easier.
Last edited by bitvector on Mon Jul 16, 2007 3:28 pm, edited 1 time in total.
bitvector
Grand Gerbil Poohbah
 
Posts: 3234
Joined: Wed Jun 22, 2005 4:39 pm
Location: Mountain View, CA

Postposted on Mon Jul 16, 2007 3:28 pm

I think Supreme Commander is not the application you want to use in that illustration because it was built with the LARGE_ADDRESS_AWARE

or maybe it's a great example because it's a 32-bit app that can use all 4GB as user space should it be available. Now i'm not sure. :/
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Mon Jul 16, 2007 3:29 pm

bitvector wrote:
morphine wrote:Why? This enables remapping all devices' memory addresses to "above" your RAM for 64-bit OSes.

Actually it moves part of your physical RAM addresses above the device addresses since that's easier.

Right you are sir, post corrected.
morphine
Gerbil Khan
Silver subscriber
 
 
Posts: 9952
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Postposted on Mon Jul 16, 2007 3:29 pm

morphine, you deserve this

Image

Damage or some mod has to make your guide a sticky. :D

derFunkenstein wrote:I think Supreme Commander is not the application you want to use in that illustration because it was built with the LARGE_ADDRESS_AWARE

or maybe it's a great example because it's a 32-bit app that can use all 4GB as user space should it be available. Now i'm not sure. :/


32-bit applications will only use up to a maxmium of 4GB sanning OS and other hardware limitations. The LARGE-ADDRESS_AWARE tag is simply telling Windows (32/64bit verisons) that the application can use beyond the default memory limit of 2GB per process.
Last edited by Krogoth on Mon Jul 16, 2007 3:33 pm, edited 1 time in total.
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC-12800, EVGA 660Ti, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
Krogoth
Maximum Gerbil
Silver subscriber
 
 
Posts: 4402
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime

Postposted on Mon Jul 16, 2007 3:32 pm

Krogoth wrote:morphine, you deserve this

**beer image**

Damage or some mod has to make your guide a sticky. :D

Thanks ;), but honestly that should go to the forementioned people in my post. JBI, for example, wouldn't refuse such a gift :)
morphine
Gerbil Khan
Silver subscriber
 
 
Posts: 9952
Joined: Fri Dec 27, 2002 8:51 pm
Location: Portugal (that's next to Spain)

Postposted on Mon Jul 16, 2007 6:12 pm

I recall reading about and dealing with this issue more than ten years ago with NT4.

It's always amusing to see how this industry tends to ignore a problem until it becomes widespread.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3521
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Postposted on Mon Jul 16, 2007 8:34 pm

Ryu Connor wrote:It's always amusing to see how this industry tends to ignore a problem until it becomes widespread.

Indeed, I can't wait to see what happens in 15-30 years when "OMG How do I get my machine to address more than 16 exbibytes?!" posts start appearing. :roll:
Desktop: FX-8350 | 32 GB | XFX Radeon 6950 | Windows 7 x64
Laptop: i7 740QM | 12 GB | Mobility Radeon 5850 | Windows 8.1.1.1.1 x64
SuperSpy
Gerbil Jedi
Gold subscriber
 
 
Posts: 1563
Joined: Thu Sep 12, 2002 9:34 pm
Location: TR Forums

Postposted on Mon Jul 16, 2007 8:56 pm

SuperSpy wrote:Indeed, I can't wait to see what happens in 15-30 years when "OMG How do I get my machine to address more than 16 exbibytes?!" posts start appearing.


Somewhat sooner unless AMD64/Intel64 continue to push their address space.

Presently they only support 40bits physical (1 tebibye).
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3521
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Postposted on Mon Jul 16, 2007 9:22 pm

Morphine, NICE explanation. Simple and to-the-point. Yay!


Well, I'm ready for the jump to 128-bit or maybe we'll just go to 256-bit next...

And the answer to the inevitable questions, "Who will ever need that much addressibility?"

We'll need it for folding, of course; or to carry DNA codes in core. Not just on my laptop, no. I want it available in my PDA Phone, too! :)

64-Bit makes so much possible already, I think we haven't really even scratched the surface, not even with our imaginations.
BIF
Gerbil Jedi
Gold subscriber
 
 
Posts: 1545
Joined: Tue May 25, 2004 7:41 pm

Postposted on Mon Jul 16, 2007 10:35 pm

Ryu Connor wrote:
SuperSpy wrote:Indeed, I can't wait to see what happens in 15-30 years when "OMG How do I get my machine to address more than 16 exbibytes?!" posts start appearing.


Somewhat sooner unless AMD64/Intel64 continue to push their address space.

Presently they only support 40bits physical (1 tebibye).


x86-64 architect supports up to 2^64 bytes. The problem lies with the memory controllers. IIRC, the memory controller on the K8 supports up to 48-bit for memory addressing. I am not sure what is the story on Intel front. It is safe to say that server-grade stuff has a higher celling.

I seriously doubt that a desktop form factor using semiconductors will ever be able to reach 1TB limit for 40-bit memory addressing.

Edit: after some furthering reading. It seems that K8 is indeed limited to 40bit memory addressing due to EV6 bus.

It is 36-bit for most of the Intel front.
Ivy Bridge i5-3570K@4.0Ghz, Gigabyte Z77X-UD3H, 2x4GiB of PC-12800, EVGA 660Ti, Corsair CX-600 and Fractal Refined R4 (W). Kentsfield Q6600@3Ghz, HD 4850 2x2GiB PC2-6400, Gigabyte EP45-DS4P, OCZ Modstream 700W, and PC-7B.
Krogoth
Maximum Gerbil
Silver subscriber
 
 
Posts: 4402
Joined: Tue Apr 15, 2003 3:20 pm
Location: somewhere on Core Prime

Postposted on Tue Jul 17, 2007 5:51 am

However, the memory address issues aren't an architectural flaw, and I highly doubt that any of the motherboards that the current AMD64/EM64T chips ride on would be able to support more than a terabyte of RAM even if there were the address space to support it. (I checked, and at least some of the EM64T chips are 40-bit...)

So, it's a non-issue. When it becomes an issue, add the address lines, and you're done.
bhtooefr
Darth Gerbil
Silver subscriber
 
 
Posts: 7754
Joined: Mon Feb 16, 2004 11:20 am
Location: Holy crap, a city! (Newark, OH)

Postposted on Tue Jul 17, 2007 8:38 am

Krogoth wrote:x86-64 architect supports up to 2^64 bits.

Fixed. You were giving it 8x the address space than it really has.
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Tue Jul 17, 2007 9:10 am

derFunkenstein wrote:
Krogoth wrote:x86-64 architect supports up to 2^64 bits.

Fixed. You were giving it 8x the address space than it really has.

No, he had it right with bytes. x86 is byte addressed. Each address refers to a byte, and each is 64 bits, which gives you 2^64 possible bytes.

If x86 was bit addressed, 32-bits would only give you 512MB of address space. Bit addressable architectures are rare: on his webpage, Andrew Morton notes that it was a pain to port gcc to the TI34010 due to this quirk.
bitvector
Grand Gerbil Poohbah
 
Posts: 3234
Joined: Wed Jun 22, 2005 4:39 pm
Location: Mountain View, CA

Postposted on Tue Jul 17, 2007 9:13 am

If the 6502 (and MANY other 8-bit word, 16-bit address processors) were bit-addressed, it'd have been an 8 kiB address space, not 64.
bhtooefr
Darth Gerbil
Silver subscriber
 
 
Posts: 7754
Joined: Mon Feb 16, 2004 11:20 am
Location: Holy crap, a city! (Newark, OH)

Postposted on Tue Jul 17, 2007 9:34 am

bitvector wrote:No, he had it right with bytes. x86 is byte addressed. Each address refers to a byte, and each is 64 bits, which gives you 2^64 possible bytes.

If x86 was bit addressed, 32-bits would only give you 512MB of address space. Bit addressable architectures are rare: on his webpage, Andrew Morton notes that it was a pain to port gcc to the TI34010 due to this quirk.

gah. Oops.

So basically an address is a 32- or 64-bit integer that links to the specific point where a byte of data is located?
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Tue Jul 17, 2007 9:41 am

Or any other. In the case of AMD64 processors, it's 52-bit*.

* For physical addresses. Virtual addresses go out to 64-bit.

As I understand, a physical address in the AMD 52-bit system, in hexadecimal representation, would range up to 0xF FFFF FFFF FFFF.

The current processors, however, AS I UNDERSTAND, would only allow you to go to 0x0 00FF FFFF FFFF. Addresses above that, AGAIN AS I UNDERSTAND, are invalid.

The simplest explanation I found of all of this, anyway, was in an old Apple ][e's manual.

Imagine an array of mailboxes, 16x16. Each row and column is lettered 0 through F. Each mailbox can hold eight bits - that is, one byte. (Which is really convenient if your processor can handle at least an 8-bit word.)

So, memory address... 0x9B... referred to the mailbox in row 9 (that is, the 10th row), column B (that is, the 12th column.) Or vice versa, it doesn't really matter. So, you would load the contents of 0x9B into a register, and now that would be what you're working on. (FWIW, I'm not using the nomenclature for hexadecimal that they used back then, which would have been $9B.)

Now, imagine that there's a 16x16 array... of these arrays of mailboxes. Call each array of mailbox a page, and the main array your address space, and you've got a memory model. So, I can obtain the contents of... 0x009B, and I've got that same memory location. 0x9B9B, however, is quite obviously a different location.
bhtooefr
Darth Gerbil
Silver subscriber
 
 
Posts: 7754
Joined: Mon Feb 16, 2004 11:20 am
Location: Holy crap, a city! (Newark, OH)

Postposted on Tue Jul 17, 2007 12:05 pm

derFunkenstein wrote:So basically an address is a 32- or 64-bit integer that links to the specific point where a byte of data is located?

Yes, you can think of it as an offset in memory. Address n refers to the nth byte from 0 (or bit or word or page, depending on what the unit of addressing is).
bitvector
Grand Gerbil Poohbah
 
Posts: 3234
Joined: Wed Jun 22, 2005 4:39 pm
Location: Mountain View, CA

Postposted on Thu Jul 19, 2007 10:07 am

It was a struggle - mostly because my computer spontaneously rebooted mid-upgrade and totally errored out after that, so I had to start over (I think it was caused because I didn't uninstall the 32-bit version of Daemon Tools) - but I did manage to get upgraded to Vista Home Premium 64-bit last night. It totally proved bitvector's first post right:

1.) It still only showed 3007MB of memory without memory remapping
2.) Enabling memory remapping now shows the full 3GB
3.) It made me re-activate because I ended up having to do a clean install - fortunately I did back up everything important to another drive before attempting the upgrade.
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Thu Jul 19, 2007 12:44 pm

Thank you all for your insight.... I know you guys are sick of thinking about it but I get confused and would like to have it correct in my old head.

From a different site:
...........
http://anandtech.com/showdoc.aspx?i=3044&p=3
Quote:
Final Thoughts

Among all of our tests, there are three identifiable patterns, all relating to some difference between Vista and XP. Those are:

1. Vista is using more address space than XP in all situations
2. The amount of address space used with Vista seems to be related to the amount of video memory on our video card
3. XP on the other hand does not fluctuate at all, the address space usage is the same no matter what card we use.


From the thread quoted:
So, it does appear from their testing that video memory for 3D apps eats up a lot more application address space than XP, and that it is variable with the amount of video ram you have on your video card.

This is not necessarily all bad, just make sure you have lots of RAM when running Vista, especially for 3D apps!


So now I'm confused...What is the relationship between the [Virtual?] address space and the hardware ram referenced in the quote?
Also, (at a guess) isn't XP utilizing the address space for the PCIe buss while Vista is using it to map the gfx card memory?


I know you guys are not responsible for other sites and their comments but it appears that you people have a handle on the ram issue and so that is why I'm asking.
TIA for any help that can be offered
OD
oldDummy
Gerbil XP
Gold subscriber
 
 
Posts: 315
Joined: Sun Apr 11, 2004 9:19 am
Location: right coast

Postposted on Thu Jul 19, 2007 1:19 pm

I think it's because of the way that video RAM is virtualized (or supposed to be, at least) in Vista - more address space is required in order to account for VRAM virtualization that doesn't exist at all in XP.
I do not understand what I do. For what I want to do, I do not do. But what I hate, I do.
derFunkenstein
Gerbil God
Gold subscriber
 
 
Posts: 21286
Joined: Fri Feb 21, 2003 9:13 pm
Location: WHAT?

Postposted on Thu Jul 19, 2007 1:35 pm

oldDummy wrote:So now I'm confused...What is the relationship between the [Virtual?] address space and the hardware ram referenced in the quote?


The reference quote isn't so perfect.

Vista uses more virtual address space per application than XP. Anecdotally this seems to be related to the DX10 feature of creating virtual memory for GPUs.

This means that some applications/games that under XP got real close to the 2GB virtual address application limit, might actually cross the 2GB limit in Vista.

This means that more applications need to be Large Address Aware. More RAM isn't necessarily gonna fix anything, as there is often a large disconnect between the amount of actual physical RAM in use and the amount of virtual memory in use.

The first Anandtech article shows just that, which makes makes said blurb all the stranger.

oldDummy wrote:Also, (at a guess) isn't XP utilizing the address space for the PCIe buss while Vista is using it to map the gfx card memory?


Don't get address range assigned for expansion confused with virtual address range assigned for GPU virtual memory confused.

Course that's easier said than done.
"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"
Ryu Connor
Global Moderator
Gold subscriber
 
 
Posts: 3521
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA

Postposted on Thu Jul 19, 2007 2:47 pm

oldDummy wrote:
So now I'm confused...

The reference quote isn't so perfect.

Well, at least that explains why it confused me.
:o

Thanks for your time and energy,
OD
oldDummy
Gerbil XP
Gold subscriber
 
 
Posts: 315
Joined: Sun Apr 11, 2004 9:19 am
Location: right coast

Postposted on Mon Jul 23, 2007 10:08 am

I like this article: Ask Dan: What's with the 3Gb memory barrier?

PAE is useless since most drivers don't support it.

supreme commander can only use 2gb on windows 64, unless using boot.ini stuff, or patching the game to be large memory aware. (there will be an official patch/expansion, and there is an unofficial one.)
l33t-g4m3r
Gerbil Jedi
Silver subscriber
 
 
Posts: 1973
Joined: Mon Dec 29, 2003 2:54 am

Postposted on Tue Aug 14, 2007 6:48 am

Not directly related but a good read:

http://anandtech.com/systems/showdoc.aspx?i=3060

OD
oldDummy
Gerbil XP
Gold subscriber
 
 
Posts: 315
Joined: Sun Apr 11, 2004 9:19 am
Location: right coast

Postposted on Tue Aug 14, 2007 12:23 pm

Actually, that's being discussed in this thread. :)

Although, I haven't seen that link in there...
bhtooefr
Darth Gerbil
Silver subscriber
 
 
Posts: 7754
Joined: Mon Feb 16, 2004 11:20 am
Location: Holy crap, a city! (Newark, OH)

Postposted on Tue Aug 14, 2007 12:43 pm

bhtooefr wrote:Actually, that's being discussed in this thread. :)

Although, I haven't seen that link in there...


Sure it is. I posted it yesterday afternoon right after Anand put the page up.
snowdog
Gerbil First Class
 
Posts: 111
Joined: Sat Jan 04, 2003 4:11 pm
Location: Ottawa, Canada

PreviousNext

Return to General Hardware

Who is online

Users browsing this forum: betrayal, edwpang and 7 guests