Page 1 of 8

Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 2:06 pm
by ColeLT1
I woke up to a text from a friend, but nothing that is hard evidence yet, to reports of a large intel bug, not sure if FUD, rumors, or some substance to these "reports," maybe rowhammer related?

https://www.reddit.com/r/sysadmin/comme ... _incoming/
http://www.game-debate.com/news/24293/r ... erformance
https://news.ycombinator.com/item?id=16046636

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 2:24 pm
by chuckula
The apocalypse is CONFIRMED.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 4:49 pm
by Ninjitsu
Just came to check on this, a post on the RPS forums seems to suggest it can't be fixed in firmware, and will lead to a 5-30% perf hit.
https://forum.rockpapershotgun.com/t/ba ... ners/28522

Cites: https://www.theregister.co.uk/2018/01/0 ... sign_flaw/

A fundamental design flaw in Intel's processor chips has forced a significant redesign of the Linux and Windows kernels to defang the chip-level security bug.

Programmers are scrambling to overhaul the open-source Linux kernel's virtual memory system. Meanwhile, Microsoft is expected to publicly introduce the necessary changes to its Windows operating system in this month's Patch Tuesday: these changes were seeded to beta testers running fast-ring Windows Insider builds in November and December.

Crucially, these updates to both Linux and Windows will incur a performance hit on Intel products. The effects are still being benchmarked, however we're looking at a ballpark figure of five to 30 per cent slow down, depending on the task and the processor model. More recent Intel chips have features – specifically, PCID – to reduce the performance hit.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 5:09 pm
by derFunkenstein
Ninjitsu wrote:
Cites: https://www.theregister.co.uk/2018/01/0 ... sign_flaw/

More recent Intel chips have features – specifically, PCID – to reduce the performance hit.

How far back do we have to go to find CPUs that don't have process-context identifiers, which the quoted article says will reduce the performance hit? And what's the different levels of support for the feature? Sandy Bridge apparently has it, but it's not listed on Nehalem (which aside from performance and efficiency reasons, you is already compromised and patched and is old anyway).

So if all the CPUs made in the last going-on-seven years will not feel the full effect of the performance penalty, then all the prognosticators on Reddit predicting big EPYC sales are going to look silly.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 5:46 pm
by Ryu Connor
https://lwn.net/Articles/738997/

Dave Hansen wrote:
tl;dr:

KAISER makes it harder to defeat KASLR, but makes syscalls and interrupts slower. These patches are based on work from a team at Graz University of Technology posted here[1]. The major addition is support for Intel PCIDs which builds on top of Andy Lutomorski's PCID work merged for 4.14. PCIDs make KAISER's overhead very reasonable for a wide variety of use cases.


Dave Hansen wrote:
Process
Context IDentifiers (PCIDs) are used to to[sic] ensure that the TLB is not flushed when switching between page tables, which makes syscalls roughly 2x faster than without it.


Dave Hansen wrote:
To give folks an idea what the performance impact is like, I took the following test and ran it single-threaded:

https://github.com/antonblanchard/will- ... s/lseek1.c

It's a pretty quick syscall so this shows how much KAISER slows down syscalls (and how much PCIDs help). The units here are lseeks/second:

no kaiser: 5.2M
kaiser+ pcid: 3.0M
kaiser+nopcid: 2.2M

"nopcid" is literally with the "nopcid" command-line option which turns PCIDs off entirely.


IMO, for servers this patch is valuable. For desktop users, it can be thought of more as an additional defense in depth against malware (of the rootkit variety) attacks. Depending on how worried you are about that particular threat, you could choose to cut KAISER off.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 7:41 pm
by DancinJack
Meh, I'll wait for some more solid info before digging too deep. I won't be too upset if I lose 5 percent performance on average though. It does suck this made it into so many CPUs and is just now being found, but I doubt there is a big backlash from it either.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 7:59 pm
by just brew it!
How big a deal this is for my employer will depend on what the underlying vulnerability is. Really wish there was more info available so we could gauge how screwed (or not) we are, and plan accordingly. At least it should only affect a subset of our workloads, since we're (usually) not CPU-bound.

For me personally it's basically a non-issue; my home desktop and server run AMD CPUs (which apparently are unaffected), and I probably wouldn't notice the performance hit on my Intel-based laptop since I don't tend to use it for CPU-intensive stuff anyway.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 8:05 pm
by CampinCarl
Just as a note, via the Linux Kernel Mailing List there may be perf hits to AMD systems on the official patch (4.14.11, I believe) unless this patch actually made it into the baseline for 4.14.11 (forgive me, I don't typically try to read the patch tree for the linux kernel). You'll apparently be able to avoid it if you set
pti=off
or
nopti
in your grub config.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 8:19 pm
by Ryu Connor
just brew it! wrote:
How big a deal this is for my employer will depend on what the underlying vulnerability is. Really wish there was more info available so we could gauge how screwed (or not) we are, and plan accordingly. At least it should only affect a subset of our workloads, since we're (usually) not CPU-bound.

For me personally it's basically a non-issue; my home desktop and server run AMD CPUs (which apparently are unaffected), and I probably wouldn't notice the performance hit on my Intel-based laptop since I don't tend to use it for CPU-intensive stuff anyway.


We've apparently been living with this bug in the Intel MMU for quite some time.

https://gruss.cc/files/kaiser.pdf

It is my interpretation that these are nation state or organized crime level attacks. I can see why hosts of virtualization platforms would want these leaks to bypass KASLR closed. I see less reason for the average end users to be concerned by it.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:04 pm
by just brew it!
Ryu Connor wrote:
It is my interpretation that these are nation state or organized crime level attacks. I can see why hosts of virtualization platforms would want these leaks to bypass KASLR closed. I see less reason for the average end users to be concerned by it.

That's my interpretation as well, at least from a security standpoint. The problem is, if the patch is enabled by default (as it should be), then everyone running an affected Intel CPU pays a performance price unless the workaround is manually disabled.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:13 pm
by chuckula
For some context about OMG IT'S THE END OF THE WORLD... some synthetic Linux benchmarks showed some hits although the real-world tests like a kernel compile run and x264/FFMPEG encoding were basically unchanged:

https://www.phoronix.com/scan.php?page= ... 6pti&num=1

As for the world of games.... zero change (the post bug-fix even wins by tiny margins in some tests): https://www.phoronix.com/scan.php?page= ... ming-Tests

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:28 pm
by just brew it!
chuckula wrote:
For some context about OMG IT'S THE END OF THE WORLD... some synthetic Linux benchmarks showed some hits although the real-world tests like a kernel compile run and x264/FFMPEG encoding were basically unchanged:

https://www.phoronix.com/scan.php?page= ... 6pti&num=1

As for the world of games.... zero change (the post bug-fix even wins by tiny margins in some tests): https://www.phoronix.com/scan.php?page= ... ming-Tests

I think you must have looked at the wrong chart or something; compile test shows a ~15% hit.

As expected, computationally intensive tasks like video encoding and gaming (which spend the vast majority of their time running user code, not making system calls) are largely unaffected.

So (to paint things with a broad brush), I/O intensive workloads with fast I/O devices will likely take a noticeable hit due to the additional syscall overhead; computationally heavy workloads and workloads which are limited by I/O device performance will probably take a minimal hit.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:36 pm
by chuckula
just brew it! wrote:
chuckula wrote:
For some context about OMG IT'S THE END OF THE WORLD... some synthetic Linux benchmarks showed some hits although the real-world tests like a kernel compile run and x264/FFMPEG encoding were basically unchanged:

https://www.phoronix.com/scan.php?page= ... 6pti&num=1

As for the world of games.... zero change (the post bug-fix even wins by tiny margins in some tests): https://www.phoronix.com/scan.php?page= ... ming-Tests

I think you must have looked at the wrong chart or something; compile test shows a ~15% hit.



No, I'm right. I was talking about the timed linux kernel compilation test. "Compilebench" is yet another synthetic benchmark.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:42 pm
by just brew it!
chuckula wrote:
just brew it! wrote:
chuckula wrote:
For some context about OMG IT'S THE END OF THE WORLD... some synthetic Linux benchmarks showed some hits although the real-world tests like a kernel compile run and x264/FFMPEG encoding were basically unchanged:

https://www.phoronix.com/scan.php?page= ... 6pti&num=1

As for the world of games.... zero change (the post bug-fix even wins by tiny margins in some tests): https://www.phoronix.com/scan.php?page= ... ming-Tests

I think you must have looked at the wrong chart or something; compile test shows a ~15% hit.


No, I'm right. I was talking about the timed linux kernel compilation test. "Compilebench" is yet another synthetic benchmark.

Ahh, yes; I see that now. Mea culpa. I got confused by the fact that they put that test in between the two video encoding tests.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 10:54 pm
by mudcore
Are we trying to downplay this because it doesn't seem like it will impact the typical user that cares about gaming benchmarks or? Seems like this could be pretty serious for things such as web servers. Those PostgreSQL numbers are also not exactly heart warming.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 11:00 pm
by DancinJack
mudcore wrote:
Are we trying to downplay this because it doesn't seem like it will impact the typical user that cares about gaming benchmarks or? Seems like this could be pretty serious for things such as web servers. Those PostgreSQL numbers are also not exactly heart warming.


Isn't that what this thread has been saying?

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 11:04 pm
by just brew it!
Yeah, this looks like it will cause some heartburn for people with specific use cases. Based on what we know so far, most of the affected use cases are likely to be on the enterprise and/or service provider side. Basically, anything that spends a lot of time making system calls and isn't already I/O-limited.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 11:09 pm
by mudcore
DancinJack wrote:
mudcore wrote:
Are we trying to downplay this because it doesn't seem like it will impact the typical user that cares about gaming benchmarks or? Seems like this could be pretty serious for things such as web servers. Those PostgreSQL numbers are also not exactly heart warming.


Isn't that what this thread has been saying?


Sorry, I guess I'm mainly taking issue with chuckula's phrasing and presentation of "typical" workloads. No its not the end of the world but this bug looks like it could have a rather significant performance impact on the majority of the web and database servers of the world. That's kinda huge... right?

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 11:21 pm
by just brew it!
mudcore wrote:
Sorry, I guess I'm mainly taking issue with chuckula's phrasing and presentation of "typical" workloads. No its not the end of the world but this bug looks like it could have a rather significant performance impact on the majority of the web and database servers of the world. That's kinda huge... right?

Server performance under load is going to drop, and/or power consumption will rise since CPUs will be spending less time idling. No, it won't be as visible to the average user as (say) a 20% framerate reduction in your favorite game, but it will have indirect effects since it will increase the cost of doing business for anyone who relies on servers using the affected CPUs. This cost increase will ultimately get passed on to the consumer somehow.

Re: Intel Processor bug incoming?

Posted: Tue Jan 02, 2018 11:26 pm
by Kougar
It's worth mentioning Intel's CEO dumped all of the shares he was legally allowed to sell. https://www.fool.com/investing/2017/12/ ... stock.aspx

Ninjitsu wrote:


If the speculative code execution is the problem, then disabling it outright will permanently affect performance. And AMD wouldn't be affected because Ryzen doesn't perform speculative code execution to boost general performance.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 12:46 am
by Bauxite
Wonder what kind of dumb buzzword name this one will get, since nobody can resist being a tool anymore.

Since it seems to be very much a part of VM risks, I predict something trying to be catchy like cloudwalker or thunderstorm or whatever to reference the "running-on-someone-elses-computers" business. The I/O thing is a bigger worry to me but thats not sexy and flashy so nope.

Also, lol good luck finding epyc in stock now unless you're one of the big boys.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 3:04 am
by adampk17
I wonder how long until the potentially ginormous class-action lawsuit is started? I don't see how it couldn't be.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 3:43 am
by just brew it!
Kougar wrote:
If the speculative code execution is the problem, then disabling it outright will permanently affect performance.

This would be a huge performance hit. Possibly first-gen Atom level of performance hit. Ain't gonna happen.

Kougar wrote:
And AMD wouldn't be affected because Ryzen doesn't perform speculative code execution to boost general performance.

Not sure where you get the idea that Ryzen doesn't do speculative execution.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 4:30 am
by jihadjoe
Do you guys think a proper fix to the silicon will make it to Ice Lake?

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 4:33 am
by adampk17
jihadjoe wrote:
Do you guys think a proper fix to the silicon will make it to Ice Lake?


My uneducated guess is that Intel would not release a new processor that still had this errata.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 4:49 am
by fyo
Note that the Phoronix benchmarks are not "clean" and include a bunch of updates, not just the Intel PTI bug fix. Theoretically, we could have all sorts of performance improvements that are masking some of the effects of the bug fix. It's likely that any such effect is very small, but we just don't know and benchmarking has a way of hitting corner cases.

Additionally, the two platforms they compare are very, very different, despite both being new-ish Intel. The slower platform sports a HDD, the newer an SSD. The faster disk should result in a greater performance hit, since the syscalls are more likely to bottleneck performance. However, the newer platform supports PCID, which apparently mitigates the performance hit somewhat.

The database benchmarks are particularly scary, with the slow HDD system taking a 19% hit compared to a 13% hit for the SSD system. However, Phoronix chose to run pgbench in "buffer scaling" mode, meaning that the disk isn't *as* important. Ideally, this is giving us an idea of the positive effect PCID can have on the impact of the bug.

The faster system (SSD + PCID) takes a MASSIVE hit of over 50% on multithreaded file system performance. The HDD on the slower system apparently completely bottlenecks performance resulting in zero performance impact.

What we need are benchmarks with the SAME kernel, just with the PTI on and off, respectively. And on some more hardware, particularly AMD and pre-PCID Intel with SSD.

As for "real users" being affected, I'm not a gamer and I have a number of systems that are IO limited in one way or another. Thankfully, my file server is on an ARM system, so it should be unaffected, but I do have a couple database-bottlenecked systems and an Intel-based, ISP-provided crappy box that is also IO bound in a myriad of ways (running Intel + Windows). If the latter is hit with a 20% performance hit, it will quite frankly become unusable.

SSD systems could also experience a significant hit when swapping, which would also hurt me in some rare situations and might (finally) push me to upgrade some memory.

EDIT: Reading up on Intel PCID, it appears that all i6xxx processors support it. Providing it is actually enabled in the Phoronix tests, that makes the benchmark results very odd, to say the least. They don't make it easier by using two different versions of Ubuntu, of course (newer version for the older system).

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 5:21 am
by the
adampk17 wrote:
jihadjoe wrote:
Do you guys think a proper fix to the silicon will make it to Ice Lake?


My uneducated guess is that Intel would not release a new processor that still had this errata.


I would presume so as well but CES is around the corner with mobile Cannon Lake. These chips are arguably already a year late and too close to launch to implement a fix. If Intel were to commit to releasing no new chips with this flaw, it'd pretty much kill Cannon Lake entirely. Intel's first generation of 10 nm product would be dead as many other 10 nm designs were moved backward to 14 nm or post poned into the second generation of 10 nm.

A not quiet parallel example would be the TSX errata where Intel shipped some with it but it was disabled by default. Basically developers who needed it for development could turn it on. Only Haswell-EP and EX chips got a new stepping with it fixed as consumers has to essentially wait for Sky Lake to guarantee systems with it enabled and functioning. Some but not all consumer Broadwell chips had the bug but all Broadwell-E/EP/EX chips had the fix.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 5:33 am
by the
fyo wrote:
Note that the Phoronix benchmarks are not "clean" and include a bunch of updates, not just the Intel PTI bug fix. Theoretically, we could have all sorts of performance improvements that are masking some of the effects of the bug fix. It's likely that any such effect is very small, but we just don't know and benchmarking has a way of hitting corner cases.


I will give Phoronix credit as this point this out themselves in their articles. I suspect we'll be seeing something more comprehensive coming in a few days.

fyo wrote:
As for "real users" being affected, I'm not a gamer and I have a number of systems that are IO limited in one way or another. Thankfully, my file server is on an ARM system, so it should be unaffected, but I do have a couple database-bottlenecked systems and an Intel-based, ISP-provided crappy box that is also IO bound in a myriad of ways (running Intel + Windows). If the latter is hit with a 20% performance hit, it will quite frankly become unusable.

SSD systems could also experience a significant hit when swapping, which would also hurt me in some rare situations and might (finally) push me to upgrade some memory.


Servers are particularly vulnerable as you point out to their need for continuous IO under certain workloads. They'll also see the biggest performance hit.

I also think that people are underestimating the potential to invoke this exploit: if a high CPU or IO load is necessary to exploit this, simply include some small high load function to kick off before attempting the exploit itself. I will agree that consumer systems rarely fall under the types of load that welcome this exploit but there is no reason why malware simply couldn't recreate the scenario where it could take advantage of it.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 6:15 am
by bhtooefr
In the sense that it's being used here, "speculative execution" really means "branch prediction". (Nowadays it's often used to mean "execute both sides of the branch", but it used to just mean branch prediction 20 years ago, and I don't think anyone's actually doing it in the modern sense yet, outside of compiled code for VLIW/EPIC CPUs that use static scheduling.)

That means that every AMD x86 CPU from the K5 on does it.

However, AMD is claiming that their CPUs will prevent page fault-causing memory accesses from lower to higher privileges during speculative execution, and therefore that this exploit does not apply to them. This implies that Intel's CPUs are successfully making these memory accesses - now, because it's branch prediction, eventually something will trip, find out that it was unprivileged, and rewind things before it gets committed to the "live" copy of memory, but in the mean time, malicious code had a chance to see what was in privileged memory, and establish and use a sidechannel (timing at the very least has been shown to potentially be viable) to get that data out.

Re: Intel Processor bug incoming?

Posted: Wed Jan 03, 2018 6:46 am
by modulusshift
I have no idea how far back this bug affects, but I've done a bit of research into when important features for reducing KAISER's impact were introduced. The two key features seem to be PCID, which was introduced in Westmere, and INVPCID, which streamlines PCID's implementation, introduced in Haswell. So Ivy Bridge should take a noticeable hit from Haswell, and Westmere should go from merely old to nigh unusable in any sys call heavy workloads. Haswell and onwards should be impacted pretty similarly.