TR Forums

First let me stipulate:

I know that RAM is vastly faster than any SSD.
I know that an SSD is no replacement for RAM.
I get it. Thanks.

But we have swap files because sometimes we need them. And faster is better. So here's my question --

Will an NVMe SSD drive be noticeably faster than a SATA SSD drive in a scenario where we need to make use of a large (say, 16 GB) swap file?

A couple of examples. Right now on my iMac I have 24 GB of physical memory with about 10GB of swap used (along with a fair bit compressed). This is basically because I have a bunch of apps open across 9 desktops. I keep them open because it's easier for me to keep them open than to quit and restart. I just want the switching between them to be as fast as possible. So, for example, when I move over to an instance of R that's been sitting there for a few hours doing nothing (but taking up 3 GB of RAM), I'd like the switch (as data is loaded from swap) to be as instantaneous as possible.

Meanwhile, over on my 2990wx Linux system, I have 64 processes running at various stages of work. I can (and have) written programs to move data that won't be used for a while to disk. But it's annoying to have to do that -- I'd rather let the OS do it. In a scenario like that, will NVMe be faster than SATA for swap? Or is there some other bottleneck that I'm not aware of?

Why go with flash at all? Get Optane 16GB. Pretty cheap, way faster and works as a storage device in an M2 slot regardless of whether you have AMD or Intel platform.

Igor_Kavinski wrote:

Why go with flash at all? Get Optane 16GB. Pretty cheap, way faster and works as a storage device in an M2 slot regardless of whether you have AMD or Intel platform.

So I guess that inspires another question -- is the difference between Optane and NVMe noticeable when used as swap in the scenarios I describe?

edit --

a point of frustration is that when I google on these questions, all I find are comparisons to RAM, and the conclusion "RAM is faster." grr! I know RAM is faster, Internet! But RAM is more expensive, which is why we have tiered storage. So the question is, is there a noticeable benefit to Optane vs NVMe SSD vs SATA SSD when used as swap. The question is not "is RAM faster", because obviously it is. (I'm not griping at anybody here -- I'm griping at the Internet)

https://www.reddit.com/r/intel/comments ... page_file/
https://hardforum.com/threads/16gb-opta ... g.1959689/

Those two folks aren't displeased with their Optane purchases. However, the 16GB Optane has a pretty low sequential write speed. If you want something better, you might have to go for the 58GB version which is quite a bit more pricey. An NVMe SSD will deliver blazing fast write speeds but the Optane would still beat it in latency. Also, I couldn't find an NVMe SSD for $25. There's this:

https://www.amazon.com/Transcend-TS128G ... 07CXC32T2/

But it's using SLC caching so once that runs out, write speeds will become abysmal. Optane memory doesn't have the gradual slowdown with time issue when it comes to writing, AFAIK. It will deliver consistent performance till it is exhausted by the writes, which shouldn't be much in your use case since you are leaving the applications running instead of launching them from scratch everytime.

Igor_Kavinski wrote:

https://www.reddit.com/r/intel/comments/b1j8v6/using_optane_as_a_windows_page_file/
https://hardforum.com/threads/16gb-opta ... g.1959689/

Those two folks aren't displeased with their Optane purchases. However, the 16GB Optane has a pretty low sequential write speed. If you want something better, you might have to go for the 58GB version which is quite a bit more pricey. An NVMe SSD will deliver blazing fast write speeds but the Optane would still beat it in latency. Also, I couldn't find an NVMe SSD for $25. There's this:

https://www.amazon.com/Transcend-TS128G ... 07CXC32T2/

But it's using SLC caching so once that runs out, write speeds will become abysmal. Optane memory doesn't have the gradual slowdown with time issue when it comes to writing, AFAIK. It will deliver consistent performance till it is exhausted by the writes, which shouldn't be much in your use case since you are leaving the applications running instead of launching them from scratch everytime.

Thanks for the links! I will investigate...

You want max bandwidth of reasonable depth 4K streaming reads/writes. NVMe will rock in that scenario versus SATA, in general.

Yup, what Waco said. Swap activity is basically shoving 4K pages around as fast as possible. NVMe should be much faster.

That said, how much of a perceived difference it makes in practice will depend on how frequently you're actually hitting the swap. Could be barely noticeable or could be substantial.

16GB Optane endurance 182.5 TBW: https://ark.intel.com/content/www/us/en ... point.html

250GB Samsung 970 EVO Plus SSD endurance 150 TBW: https://www.anandtech.com/show/13761/th ... ssd-review

If your mobo has two M2 slots, you could get two Optane sticks and put them in RAID 0 and still save some money while getting better endurance.

https://www.tweaktown.com/reviews/8234/ ... ndex6.html

Notice how NVMe SSD RAID is nowhere close to Optane RAID?

Compare the 4K scores for 2 x Optane with 2 x 950 Pro: https://www.tweaktown.com/reviews/8234/ ... ndex5.html

Then there's the issue of write amplification due to garbage collection. Meaning the NVMe SSD will lose endurance when it is idle and its controller is doing housekeeping tasks. Optane doesn't have this issue so you get pretty close to the rated endurance unlike the SSD.

https://www.tomshardware.com/reviews/in ... 031-3.html

Q: Optane does not need the UNMAP command. Does that mean there is no internal garbage collection? If I continuously rewrite to an Optane drive, can I expect any performance degradation? Does the concept of write amplification apply on Optane? If so, under what circumstances?

A: To answer your first two questions: Intel Optane is a write-in-place media, therefore it incurs no performance penalties from garbage collection or its associated write amplification. We have defenestrated garbage collection!

I would expect NVMe to kill it vs SATA on latency alone, as noted. Bulk transfer is ~3 times faster.

Optane is interesting, as Igor noted... if the swaps really are mostly 4K r/w, optane might open a can of whoop-arse on NVMe, where it can use its major IOPS advantage.

roncat wrote:

Optane is interesting, as Igor noted... if the swaps really are mostly 4K r/w, optane might open a can of whoop-arse on NVMe, where it can use its major IOPS advantage.

Well, if you want a data point, the cumulative I/O stats for the swap device on my Linux desktop indicate that the average I/O request size has been exactly 4K. That (combined with the fact that the VM page size is 4K so it "just makes sense") is enough evidence for me.

My next build will put the swap on NVMe.

Wow, guys -- it's crazy that even though this site is in a zombie-like state, I can still find better answers to questions here than anything google can cough up. Maybe I suck at the google, but I'm inclined to say that you're all awesome. Very helpful posts!

just brew it! wrote:

My next build will put the swap on NVMe.

Anything stopping you from doing that in your current build?

Igor_Kavinski wrote:

just brew it! wrote:

My next build will put the swap on NVMe.

Anything stopping you from doing that in your current build?

Lack of an NVMe slot?

Yes, it's a pretty old system at this point. And with 32GB of RAM, the swap really doesn't get used enough to make it worth retrofitting a PCIe-to-NVMe adapter.

blastdoor wrote:

Wow, guys -- it's crazy that even though this site is in a zombie-like state, I can still find better answers to questions here than anything google can cough up. Maybe I suck at the google, but I'm inclined to say that you're all awesome. Very helpful posts!

Yeah, almost everyone here is awesome, a seasoned veteran in some area of the computer tech realm. There's a reason I for one keep checking in ever so often, with no plans of ever ceasing to do so, and I suspect that may be the case for others as well.

16GB Optane for just $21.45: https://smile.amazon.com/Intel-Optane-M ... 07CGYTXQK/

16GB hardly seems worth it though. You'd be much better off just adding another 16GB of RAM. (Yes I know that would cost more, but we're still talking only $50 or so.)

That Optane may be the best thing one can get for $21 but the write speed is very poor. Swapping 1 GB to disk would take almost ocho (ouch) seconds. Even 8 GB of additional RAM should help the overall performance more, and it currently costs around $28 on Amazon (not for Mac indeed, and it's only PC4-17000).

just brew it! wrote:

roncat wrote:

Optane is interesting, as Igor noted... if the swaps really are mostly 4K r/w, optane might open a can of whoop-arse on NVMe, where it can use its major IOPS advantage.

Well, if you want a data point, the cumulative I/O stats for the swap device on my Linux desktop indicate that the average I/O request size has been exactly 4K. That (combined with the fact that the VM page size is 4K so it "just makes sense") is enough evidence for me.

Based on what you and other guys said, the 4K size seems to be the case most of the time, or all of the time ... but it looks like poorly designed virtual memory management in OSes. Just about every storage media reaches much higher transfer rates with larger blocks. Why doesn't the VM manager somehow coalesce several small I/O requests into larger ones, both on writes and on reads? Is swap performance simply irrelevant because once you let your system do the swapping, you've given up any and all performance anyway?

Wirko wrote:

Based on what you and other guys said, the 4K size seems to be the case most of the time, or all of the time ... but it looks like poorly designed virtual memory management in OSes. Just about every storage media reaches much higher transfer rates with larger blocks. Why doesn't the VM manager somehow coalesce several small I/O requests into larger ones, both on writes and on reads? Is swap performance simply irrelevant because once you let your system do the swapping, you've given up any and all performance anyway?

"It's complicated!"

I was thinking about this a bit too. The OS manages memory in 4KB chunks, since that's normally the native harwdare VM page size. Larger (4MB) VM pages are supported on x86, but this would potentially waste large amounts of RAM because there are many cases where kernel data structures (e.g. I/O buffers) need to be aligned to a page boundary.

Seems to me you might be able to improve things at least on the writing (swap out to the page file) side by also swapping out any adjacent pages whose time of last use is similar. But... user processes operate on virtual addresses, while DMA is done on physical addresses. Pages which are contiguous from an application perspective (and which are therefore likely to have similar "last used" times) aren't necessarily contiguous in physical RAM. So this also requires hardware scatter/gather support by the underlying swap device (which isn't a given).

For reading (swapping pages back in), it's inherently random since it is done on-demand, triggered at the instant when the CPU attempts to touch a page which isn't physically present in RAM. Attempting to "pre-fetch" subsequent pages before they are touched is going to result in a lot of pages needlessly getting read back in that will never be used, wasting a lot of physical RAM and creating more VM pressure.

It all comes back to what people have been repeating over and over again - if at all possible, you really want to avoid heavy reliance on swap.

Wirko wrote:

... but it looks like poorly designed virtual memory management in OSes. Just about every storage media reaches much higher transfer rates with larger blocks. Why doesn't the VM manager somehow coalesce several small I/O requests into larger ones, both on writes and on reads? Is swap performance simply irrelevant because once you let your system do the swapping, you've given up any and all performance anyway?

There's actually a very good reason for this (historically). The swap "filesystem", if you want to call it that, needs to be functional under *extreme* memory pressure. Coalescing writes could be done when there is plenty of free memory but not if there's no memory to allocate to actually coalesce the pages. Under extreme pressure even allocating single pages can be difficult.

That said, someone could go through and write an aggregation layer that coalesced blocks, but it would come at a latency hit and require a bunch of static structures to be allocated that would always be eating RAM. Granted, it wouldn't have to be tons, but the benefit (slightly faster paging when you're already a few orders of magnitude slower than RAM) is likely not worth the added complexity or chance of actually running out of memory.

Makes sense.

Makes even less sense than I thought before. I'm referring to the Intel's push to market 8GB of RAM in notebooks as 24GB of memory because Optane is counted in. "24GB" Ideapads can still be found on Amazon, as well as "20GB" HPs.

https://www.extremetech.com/computing/271543-pc-oems-are-selling-laptops-with-optane-cache-drives-and-claiming-its-memory

Wirko wrote:

Makes even less sense than I thought before. I'm referring to the Intel's push to market 8GB of RAM in notebooks as 24GB of memory because Optane is counted in. "24GB" Ideapads can still be found on Amazon, as well as "20GB" HPs.

https://www.extremetech.com/computing/271543-pc-oems-are-selling-laptops-with-optane-cache-drives-and-claiming-its-memory

It lets laptop makers advertise a "24GB" laptop that has lower production costs than a real 24GB laptop. Marketing sleaze at its finest.