Incorporate KAISER patches to Chrome OS kernels |
||||||||||
Issue descriptionIt's a cheap way to prevent kernel infoleaks.
Showing comments 21 - 120
of 120
Older ›
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/59a697a04dda6f6b7b68cf5160df6d544c4f1166 commit 59a697a04dda6f6b7b68cf5160df6d544c4f1166 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 01:25:41 2017 CHROMIUM: kaiser: paranoid_entry pass cr3 need to paranoid_exit Neel Natu points out that paranoid_entry() was wrong to assume that an entry that did not need swapgs would not need SWITCH_KERNEL_CR3: paranoid_entry (used for debug breakpoint, int3, double fault or MCE; though I think it's only the MCE case that is cause for concern here) can break in at an awkward time, between cr3 switch and swapgs, but its handling always needs kernel gs and kernel cr3. Easy to fix in itself, but paranoid_entry() also needs to convey to paranoid_exit() (and my reading of macro idtentry says paranoid_entry and paranoid_exit are always paired) how to restore the prior state. The swapgs state is already conveyed by %ebx (0 or 1), so extend that also to convey when SWITCH_USER_CR3 will be needed (2 or 3). (Yes, I'd much prefer that 0 meant no swapgs, whereas it's the other way round: and a convention shared with error_entry() and error_exit(), which I don't want to touch. Perhaps I should have inverted the bit for switch cr3 too, but did not.) paranoid_exit() would be straightforward, except for TRACE_IRQS: it did TRACE_IRQS_IRETQ when doing swapgs, but TRACE_IRQS_IRETQ_DEBUG when not: which is it supposed to use when SWITCH_USER_CR3 is split apart from that? As best as I can determine, commit 5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep") missed the swapgs case, and should have used TRACE_IRQS_IRETQ_DEBUG there too (the discrepancy has nothing to do with the liberal use of _NO_STACK and _UNSAFE_STACK hereabouts: TRACE_IRQS_OFF_DEBUG has just been used in all cases); discrepancy lovingly preserved across several paranoid_exit() cleanups, but I'm now removing it. Neel further indicates that to use SWITCH_USER_CR3_NO_STACK there in paranoid_exit() is now not only unnecessary but unsafe: might corrupt syscall entry's unsafe_stack_register_backup of %rax. Just use SWITCH_USER_CR3: and delete SWITCH_USER_CR3_NO_STACK altogether, before we make the mistake of using it again. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> hughd adds: this commit fixes an issue in the Kaiser-without-PCIDs part of the series, and ought to be moved earlier, if you decided to make a release of Kaiser-without-PCIDs. The patch to paranoid_exit needed reworking, because unlike in later releases, here paranoid_exit handles userspace cases. Conflicts: arch/x86/entry/entry_64.S Change-Id: Ic1d75cf0657f7943a2fe86cb2f05a40db0a3068e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757214 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/59a697a04dda6f6b7b68cf5160df6d544c4f1166/arch/x86/kernel/entry_64.S [modify] https://crrev.com/59a697a04dda6f6b7b68cf5160df6d544c4f1166/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f511e2d272a6aa65ec4e0422dd2041864e254dc6 commit f511e2d272a6aa65ec4e0422dd2041864e254dc6 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 01:25:40 2017 CHROMIUM: kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user Mostly this commit is just unshouting X86_CR3_PCID_KERN_VAR and X86_CR3_PCID_USER_VAR: we usually name variables in lower-case. But why does x86_cr3_pcid_noflush need to be __aligned(PAGE_SIZE)? Ah, it's a leftover from when kaiser_add_user_map() once complained about mapping the same page twice. Make it __read_mostly instead. (I'm a little uneasy about all the unrelated data which shares its page getting user-mapped too, but that was so before, and not a big deal: though we call it user-mapped, it's not mapped with _PAGE_USER.) And there is a little change around the two calls to do_nmi(). Previously they set the NOFLUSH bit (if PCID supported) when forcing to kernel context before do_nmi(); now they also have the NOFLUSH bit set (if PCID supported) when restoring context after: nothing done in do_nmi() should require a TLB to be flushed here. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) Change-Id: Iccab6e460ad9b2cc160e555180727020357f4335 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757213 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/f511e2d272a6aa65ec4e0422dd2041864e254dc6/arch/x86/kernel/entry_64.S [modify] https://crrev.com/f511e2d272a6aa65ec4e0422dd2041864e254dc6/arch/x86/mm/kaiser.c [modify] https://crrev.com/f511e2d272a6aa65ec4e0422dd2041864e254dc6/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f7421b6d61bcffc88c92ed87c1023edb6d6fbff1 commit f7421b6d61bcffc88c92ed87c1023edb6d6fbff1 Author: Jorge Lucangeli Obes <jorgelo@chromium.org> Date: Thu Nov 09 01:25:42 2017 CHROMIUM: kaiser: Fix kernel_stack. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ib3f14eb79a6dbc1f128e4607f4fb2d8d81e78ea5 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757215 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/f7421b6d61bcffc88c92ed87c1023edb6d6fbff1/arch/x86/kernel/cpu/common.c
,
Nov 9 2017
,
Nov 9 2017
This bug requires manual review: M63 has already been promoted to the beta branch, so this requires manual review Please contact the milestone owner if you have questions. Owners: cmasso@(Android), cmasso@(iOS), gkihumba@(ChromeOS), govind@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ac9f5f8a35dd01b9655e98f8c2452a3588b29564 commit ac9f5f8a35dd01b9655e98f8c2452a3588b29564 Author: Richard Fellner <richard.fellner@student.tugraz.at> Date: Thu Nov 09 06:21:21 2017 BACKPORT: KAISER: Kernel Address Isolation This patch introduces our implementation of KAISER (Kernel Address Isolation to have Side-channels Efficiently Removed), a kernel isolation technique to close hardware side channels on kernel address information. More information about the patch can be found on: https://github.com/IAIK/KAISER From: Richard Fellner <richard.fellner@student.tugraz.at> From: Daniel Gruss <daniel.gruss@iaik.tugraz.at> Subject: [RFC, PATCH] x86_64: KAISER - do not map kernel in user mode Date: Thu, 4 May 2017 14:26:50 +0200 Link: http://marc.info/?l=linux-kernel&m=149390087310405&w=2 Kaiser-4.10-SHA1: c4b1831d44c6144d3762ccc72f0c4e71a0c713e5 To: <linux-kernel@vger.kernel.org> To: <kernel-hardening@lists.openwall.com> Cc: <clementine.maurice@iaik.tugraz.at> Cc: <moritz.lipp@iaik.tugraz.at> Cc: Michael Schwarz <michael.schwarz@iaik.tugraz.at> Cc: Richard Fellner <richard.fellner@student.tugraz.at> Cc: Ingo Molnar <mingo@kernel.org> Cc: <kirill.shutemov@linux.intel.com> Cc: <anders.fogh@gdata-adan.de> After several recent works [1,2,3] KASLR on x86_64 was basically considered dead by many researchers. We have been working on an efficient but effective fix for this problem and found that not mapping the kernel space when running in user mode is the solution to this problem [4] (the corresponding paper [5] will be presented at ESSoS17). With this RFC patch we allow anybody to configure their kernel with the flag CONFIG_KAISER to add our defense mechanism. If there are any questions we would love to answer them. We also appreciate any comments! Cheers, Daniel (+ the KAISER team from Graz University of Technology) [1] http://www.ieee-security.org/TC/SP2013/papers/4977a191.pdf [2] https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-Using-Undocumented-CPU-Behaviour-To-See-Into-Kernel-Mode-And-Break-KASLR-In-The-Process.pdf [3] https://www.blackhat.com/docs/us-16/materials/us-16-Jang-Breaking-Kernel-Address-Space-Layout-Randomization-KASLR-With-Intel-TSX.pdf [4] https://github.com/IAIK/KAISER [5] https://gruss.cc/files/kaiser.pdf BUG=chromium:781913 TEST=Build, deploy on soraka. Conflicts: arch/x86/entry/entry_64.S arch/x86/entry/entry_64_compat.S arch/x86/mm/Makefile kernel/fork.c Change-Id: I40217b72c391002580c92b300ddad4798a9de800 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755859 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/kernel/fork.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/security/Kconfig [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/mm/Makefile [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/include/linux/percpu-defs.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/init/main.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/mm/pgtable.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/hw_irq.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/kernel/process.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/processor.h [add] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/mm/pageattr.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/kernel/head_64.S [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/entry/entry_64.S [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/kernel/irqinit.c [modify] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/kernel/espfix_64.c [add] https://crrev.com/ac9f5f8a35dd01b9655e98f8c2452a3588b29564/arch/x86/mm/kaiser.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/fa47f3f7ddff5c2d99ae1657221168b4fabf6435 commit fa47f3f7ddff5c2d99ae1657221168b4fabf6435 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:22 2017 CHROMIUM: kaiser: merged update Merged fixes and cleanups, rebased to 4.4.89 tree (no 5-level paging). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/kernel/cpu/common.c arch/x86/mm/kaiser.c kernel/fork.c Change-Id: I82adde2cb43c1fe5792a97bc16fb973053c9f8c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755860 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/entry/entry_64.S [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/mm/pgtable.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/kernel/fork.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/kernel/tracepoint.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/mm/pageattr.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/security/Kconfig [add] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/include/linux/kaiser.h [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/kernel/espfix_64.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/mm/kaiser.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/kernel/ldt.c [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/fa47f3f7ddff5c2d99ae1657221168b4fabf6435/arch/x86/kernel/head_64.S
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ac21472e1451bffa3235086919ff037e00e9b65f commit ac21472e1451bffa3235086919ff037e00e9b65f Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:23 2017 CHROMIUM: kaiser: do not set _PAGE_NX on pgd_none native_pgd_clear() uses native_set_pgd(), so native_set_pgd() must avoid setting the _PAGE_NX bit on an otherwise pgd_none() entry: usually that just generated a warning on exit, but sometimes more mysterious and damaging failures (our production machines could not complete booting). The original fix to this just avoided adding _PAGE_NX to an empty entry; but eventually more problems surfaced with kexec, and EFI mapping expected to be a problem too. So now instead change native_set_pgd() to update shadow only if _PAGE_USER: A few places (kernel/machine_kexec_64.c, platform/efi/efi_64.c for sure) use set_pgd() to set up a temporary internal virtual address space, with physical pages remapped at what Kaiser regards as userspace addresses: Kaiser then assumes a shadow pgd follows, which it will try to corrupt. This appears to be responsible for the recent kexec and kdump failures; though it's unclear how those did not manifest as a problem before. Ah, the shadow pgd will only be assumed to "follow" if the requested pgd is on an even-numbered page: so I suppose it was going wrong 50% of the time all along. What we need is a flag to set_pgd(), to tell it we're dealing with userspace. Er, isn't that what the pgd's _PAGE_USER bit is saying? Add a test for that. But we cannot do the same for pgd_clear() (which may be called to clear corrupted entries - set aside the question of "corrupt in which pgd?" until later), so there just rely on pgd_clear() not being called in the problematic cases - with a WARN_ON_ONCE() which should fire half the time if it is. But this is getting too big for an inline function: move it into arch/x86/mm/kaiser.c (which then demands a boot/compressed mod); and de-void and de-space native_get_shadow/normal_pgd() while here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/kaslr.c (not in this tree) Change-Id: Id1259efe7a01a4fc601be6b0f9c5fd55239300c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755861 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/ac21472e1451bffa3235086919ff037e00e9b65f/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/ac21472e1451bffa3235086919ff037e00e9b65f/arch/x86/mm/kaiser.c [modify] https://crrev.com/ac21472e1451bffa3235086919ff037e00e9b65f/arch/x86/boot/compressed/misc.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8e1e9d530e2bd7cd89f2721219254a045e8cbd48 commit 8e1e9d530e2bd7cd89f2721219254a045e8cbd48 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:25 2017 CHROMIUM: kaiser: stack map PAGE_SIZE at THREAD_SIZE-PAGE_SIZE Kaiser only needs to map one page of the stack; and kernel/fork.c did not build on powerpc (no __PAGE_KERNEL). It's all cleaner if linux/kaiser.h provides kaiser_map_thread_stack() and kaiser_unmap_thread_stack() wrappers around asm/kaiser.h's kaiser_add_mapping() and kaiser_remove_mapping(). And use linux/kaiser.h in init/main.c to avoid the #ifdefs there. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ib0d66c132a6ff873c83a0684392b4ddf06de6a10 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755862 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/8e1e9d530e2bd7cd89f2721219254a045e8cbd48/init/main.c [modify] https://crrev.com/8e1e9d530e2bd7cd89f2721219254a045e8cbd48/kernel/fork.c [modify] https://crrev.com/8e1e9d530e2bd7cd89f2721219254a045e8cbd48/include/linux/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c6c8d8ca877952a5f1f89df8f47627e79cc305e9 commit c6c8d8ca877952a5f1f89df8f47627e79cc305e9 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:26 2017 CHROMIUM: kaiser: fix build and FIXME in alloc_ldt_struct() Include linux/kaiser.h instead of asm/kaiser.h to build ldt.c without CONFIG_KAISER. kaiser_add_mapping() does already return an error code, so fix the FIXME. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I9f64fd683d5750a927dd08574327a1bbf0929a11 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755863 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/c6c8d8ca877952a5f1f89df8f47627e79cc305e9/arch/x86/kernel/ldt.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/01f235baf809019a00527d661cc6cb2041fbad5b commit 01f235baf809019a00527d661cc6cb2041fbad5b Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:27 2017 CHROMIUM: kaiser: KAISER depends on SMP It is absurd that KAISER should depend on SMP, but apparently nobody has tried a UP build before: which breaks on implicit declaration of function 'per_cpu_offset' in arch/x86/mm/kaiser.c. Now, you would expect that to be trivially fixed up; but looking at the System.map when that block is #ifdef'ed out of kaiser_init(), I see that in a UP build __per_cpu_user_mapped_end is precisely at __per_cpu_user_mapped_start, and the items carefully gathered into that section for user-mapping on SMP, dispersed elsewhere on UP. So, some other kind of section assignment will be needed on UP, but implementing that is not a priority: just make KAISER depend on SMP for now. Also inserted a blank line before the option, tidied up the brief Kconfig help message, and added an "If unsure, Y". BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Idd72fa2a008d117cf3ee7de05a8d54dbc248fb9e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755864 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/01f235baf809019a00527d661cc6cb2041fbad5b/security/Kconfig
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/43a094a2f624732b0eefa66b4333e4712b806dcd commit 43a094a2f624732b0eefa66b4333e4712b806dcd Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:28 2017 CHROMIUM: kaiser: fix regs to do_nmi() ifndef CONFIG_KAISER pjt has observed that nmi's second (nmi_from_kernel) call to do_nmi() adjusted the %rdi regs arg, rightly when CONFIG_KAISER, but wrongly when not CONFIG_KAISER. Although the minimal change is to add an #ifdef CONFIG_KAISER around the addq line, that looks cluttered, and I prefer how the first call to do_nmi() handled it: prepare args in %rdi and %rsi before getting into the CONFIG_KAISER block, since it does not touch them at all. And while we're here, place the "#ifdef CONFIG_KAISER" that follows each, to enclose the "Unconditionally restore CR3" comment: matching how the "Unconditionally use kernel CR3" comment above is enclosed. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Icb8e9a1809e48ce73395a9747b440e6ceae057d0 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755865 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/43a094a2f624732b0eefa66b4333e4712b806dcd/arch/x86/entry/entry_64.S
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8637113958847c7cdb2765e1143b8adf0bd9e339 commit 8637113958847c7cdb2765e1143b8adf0bd9e339 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:30 2017 CHROMIUM: kaiser: fix perf crashes Avoid perf crashes: place debug_store in the user-mapped per-cpu area instead of allocating, and use page allocator plus kaiser_add_mapping() to keep the BTS and PEBS buffers user-mapped (that is, present in the user mapping, though visible only to kernel and hardware). The PEBS fixup buffer does not need this treatment. The need for a user-mapped struct debug_store showed up before doing any conscious perf testing: in a couple of kernel paging oopses on Westmere, implicating the debug_store offset of the per-cpu area. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/events/intel/ds.c (not in this tree) arch/x86/kernel/cpu/perf_event_intel_ds.c (patched instead of that) Change-Id: I0c931d8a84f5f5a13f93a8a9fd4bdb89c7ab19aa Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755866 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/8637113958847c7cdb2765e1143b8adf0bd9e339/arch/x86/kernel/cpu/perf_event_intel_ds.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/7f4856b454700da9907315b57a1ce24011e1c18c commit 7f4856b454700da9907315b57a1ce24011e1c18c Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:31 2017 CHROMIUM: kaiser: ENOMEM if kaiser_pagetable_walk() NULL kaiser_add_user_map() took no notice when kaiser_pagetable_walk() failed. And avoid its might_sleep() when atomic (though atomic at present unused). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia8a7e3e07f23feb351f22ffd67c7bfac5630d019 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755867 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/7f4856b454700da9907315b57a1ce24011e1c18c/arch/x86/mm/kaiser.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/1bc83b3acbaebea13803ded40578a0c12c4f7100 commit 1bc83b3acbaebea13803ded40578a0c12c4f7100 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:32 2017 CHROMIUM: kaiser: tidied up asm/kaiser.h somewhat Mainly deleting a surfeit of blank lines, and reflowing header comment. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I408260c74e143d058adce4675131c844f2dac9f3 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755868 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/1bc83b3acbaebea13803ded40578a0c12c4f7100/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2a10c1a7f53f7235ed0c3017afec8d9bbc1f4f43 commit 2a10c1a7f53f7235ed0c3017afec8d9bbc1f4f43 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:33 2017 CHROMIUM: kaiser: tidied up kaiser_add/remove_mapping slightly Yes, unmap_pud_range_nofree()'s declaration ought to be in a header file really, but I'm not sure we want to use it anyway: so for now just declare it inside kaiser_remove_mapping(). And there doesn't seem to be such a thing as unmap_p4d_range(), even in a 5-level paging tree. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I38dae645f7502f7d5ba0fecc257831f87b4279f9 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755869 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/2a10c1a7f53f7235ed0c3017afec8d9bbc1f4f43/arch/x86/mm/kaiser.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/80b8008a5e74f402d72201c409752bbabbd06992 commit 80b8008a5e74f402d72201c409752bbabbd06992 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:35 2017 CHROMIUM: kaiser: kaiser_remove_mapping() move along the pgd When removing the bogus comment from kaiser_remove_mapping(), I really ought to have checked the extent of its bogosity: as Neel points out, there is nothing to stop unmap_pud_range_nofree() from continuing beyond the end of a pud (and starting in the wrong position on the next). Fix kaiser_remove_mapping() to constrain the extent and advance pgd pointer correctly: use pgd_addr_end() macro as used throughout base mm (but don't assume page-rounded start and size in this case). But this bug was very unlikely to trigger in this backport: since any buddy allocation is contained within a single pud extent, and we are not using vmapped stacks (and are only mapping one page of stack anyway): the only way to hit this bug here would be when freeing a large modified ldt. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I2a98a4764bb931e206ebfbf9a5bf3c1a4deb87fe Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755870 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/80b8008a5e74f402d72201c409752bbabbd06992/arch/x86/mm/kaiser.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0188de2dded589e3150d5b5938adb295ba9375ca commit 0188de2dded589e3150d5b5938adb295ba9375ca Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:36 2017 CHROMIUM: kaiser: align addition to x86/mm/Makefile Use tab not space so they line up properly, kaslr.o also. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/Makefile Change-Id: I8db22a668201a4bf1024208c6aa38e08d725ffc8 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755871 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/0188de2dded589e3150d5b5938adb295ba9375ca/arch/x86/mm/Makefile
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/cd2b86f7196eed81957ea4f47c04701ad7b74f7b commit cd2b86f7196eed81957ea4f47c04701ad7b74f7b Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:37 2017 CHROMIUM: kaiser: cleanups while trying for gold link While trying to get our gold link to work, four cleanups: matched the gdt_page declaration to its definition; in fiddling unsuccessfully with PERCPU_INPUT(), lined up backslashes; lined up the backslashes according to convention in percpu-defs.h; deleted the unused irq_stack_pointer addition to irq_stack_union. Sad to report that aligning backslashes does not appear to help gold align to 8192: but while these did not help, they are worth keeping. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Iaed06bb95203cd34eef9f3fdfa7eda8a421ad8ca Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755872 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/cd2b86f7196eed81957ea4f47c04701ad7b74f7b/include/linux/percpu-defs.h [modify] https://crrev.com/cd2b86f7196eed81957ea4f47c04701ad7b74f7b/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/cd2b86f7196eed81957ea4f47c04701ad7b74f7b/arch/x86/include/asm/processor.h [modify] https://crrev.com/cd2b86f7196eed81957ea4f47c04701ad7b74f7b/arch/x86/include/asm/desc.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/88bf87b7c68a66ed89584a7426d1be20843cdcec commit 88bf87b7c68a66ed89584a7426d1be20843cdcec Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:39 2017 CHROMIUM: kaiser: name that 0x1000 KAISER_SHADOW_PGD_OFFSET There's a 0x1000 in various places, which looks better with a name. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ie64ada4e5f572404efc9b7e2854d5d87fd9ef78e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755913 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/88bf87b7c68a66ed89584a7426d1be20843cdcec/arch/x86/entry/entry_64.S [modify] https://crrev.com/88bf87b7c68a66ed89584a7426d1be20843cdcec/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/cf1d433fafa19b280791ce5ba00cac64ab18e4f6 commit cf1d433fafa19b280791ce5ba00cac64ab18e4f6 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:40 2017 CHROMIUM: kaiser: delete KAISER_REAL_SWITCH option We fail to see what CONFIG_KAISER_REAL_SWITCH is for: it seems to be left over from early development, and now just obscures tricky parts of the code. Delete it before adding PCIDs, or nokaiser boot option. (Or if there is some good reason to keep the option, then it needs a help text - and a "depends on KAISER", so that all those without KAISER are not asked the question.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I27469aa929149bb9af3f8f5de0f4423e4f9ed8ce Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755914 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/cf1d433fafa19b280791ce5ba00cac64ab18e4f6/arch/x86/entry/entry_64.S [modify] https://crrev.com/cf1d433fafa19b280791ce5ba00cac64ab18e4f6/security/Kconfig [modify] https://crrev.com/cf1d433fafa19b280791ce5ba00cac64ab18e4f6/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a7374df72e4b1945d04bc5cb9829c27c1d98b52a commit a7374df72e4b1945d04bc5cb9829c27c1d98b52a Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:41 2017 CHROMIUM: kaiser: vmstat show NR_KAISERTABLE as nr_overhead The kaiser update made an interesting choice, never to free any shadow page tables. Contention on global spinlock was worrying, particularly with it held across page table scans when freeing. Something had to be done: I was going to add refcounting; but simply never to free them is an appealing choice, minimizing contention without complicating the code (the more a page table is found already, the less the spinlock is used). But leaking pages in this way is also a worry: can we get away with it? At the very least, we need a count to show how bad it actually gets: in principle, one might end up wasting about 1/256 of memory that way (1/512 for when direct-mapped pages have to be user-mapped, plus 1/512 for when they are user-mapped from the vmalloc area on another occasion (but we don't have vmalloc'ed stacks, so only large ldts are vmalloc'ed). Add per-cpu stat NR_KAISERTABLE: including 256 at startup for the shared pgd entries, and 1 for each intermediate page table added thereafter for user-mapping - but leave out the 1 per mm, for its shadow pgd, because that distracts from the monotonic increase. Shown in /proc/vmstat as nr_overhead (0 if kaiser not enabled). In practice, it doesn't look so bad so far: more like 1/12000 after nine hours of gtests below; and movable pageblock segregation should tend to cluster the kaiser tables into a subset of the address space (if not, they will be bad for compaction too). But production may tell a different story: keep an eye on this number, and bring back lighter freeing if it gets out of control (maybe a shrinker). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Id10448e0350f4ed7ab40b374c866225d12724887 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755915 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/a7374df72e4b1945d04bc5cb9829c27c1d98b52a/arch/x86/mm/kaiser.c [modify] https://crrev.com/a7374df72e4b1945d04bc5cb9829c27c1d98b52a/mm/vmstat.c [modify] https://crrev.com/a7374df72e4b1945d04bc5cb9829c27c1d98b52a/include/linux/mmzone.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310 commit dc1e80b0a539d7066f1f30f70c9e96f1d74e8310 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:42 2017 CHROMIUM: kaiser: enhanced by kernel and user PCIDs Merged performance improvements to Kaiser, using distinct kernel and user Process Context Identifiers to minimize the TLB flushing. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I17de75d3c84b878015e7f6a885bebe8d770b9e73 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755916 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/entry/entry_64.S [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/include/asm/cpufeature.h [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/mm/kaiser.c [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/kvm/x86.c [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/include/uapi/asm/processor-flags.h [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/mm/tlb.c [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/dc1e80b0a539d7066f1f30f70c9e96f1d74e8310/arch/x86/kernel/cpu/common.c
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00 commit ad3609e37688c670f0a86dedbb3f8f3bf8fbca00 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:44 2017 CHROMIUM: kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user We have many machines (Westmere, Sandybridge, Ivybridge) supporting PCID but not INVPCID: on these load_new_mm_cr3() simply crashed. Flushing user context inside load_new_mm_cr3() without the use of invpcid is difficult: momentarily switch from kernel to user context and back to do so? I'm not sure whether that can be safely done at all, and would risk polluting user context with kernel internals, and kernel context with stale user externals. Instead, follow the hint in the comment that was there: change X86_CR3_PCID_USER_VAR to be a per-cpu variable, then load_new_mm_cr3() can leave a note in it, for SWITCH_USER_CR3 on return to userspace to flush user context TLB, instead of default X86_CR3_PCID_USER_NOFLUSH. Which works well enough that there's no need to do it this way only when invpcid is unsupported: it's a good alternative to invpcid here. But there's a couple of inlines in asm/tlbflush.h that need to do the same trick, so it's best to localize all this per-cpu business in mm/kaiser.c: moving that part of the initialization from setup_pcid() to kaiser_setup_pcid(); with kaiser_flush_tlb_on_return_to_user() the function for noting an X86_CR3_PCID_USER_FLUSH. And let's keep a KAISER_SHADOW_PGD_OFFSET in there, to avoid the extra OR on exit. I did try to make the feature tests in asm/tlbflush.h more consistent with each other: there seem to be far too many ways of performing such tests, and I don't have a good grasp of their differences. At first I converted them all to be static_cpu_has(): but that proved to be a mistake, as the comment in __native_flush_tlb_single() hints; so then I reversed and made them all this_cpu_has(). Probably all gratuitous change, but that's the way it's working at present. I am slightly bothered by the way non-per-cpu X86_CR3_PCID_KERN_VAR gets re-initialized by each cpu (before and after these changes): no problem when (as usual) all cpus on a machine have the same features, but in principle incorrect. However, my experiment to per-cpu-ify that one did not end well... BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia29c40d3255f6fa2178b87287b025af775dc21fd Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755917 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00/arch/x86/mm/tlb.c [modify] https://crrev.com/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00/arch/x86/mm/kaiser.c [modify] https://crrev.com/ad3609e37688c670f0a86dedbb3f8f3bf8fbca00/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7 commit 50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:45 2017 CHROMIUM: kaiser: PCID 0 for kernel and 128 for user Why was 4 chosen for kernel PCID and 6 for user PCID? No good reason in a backport where PCIDs are only used for Kaiser. If we continue with those, then we shall need to add Andy Lutomirski's 4.13 commit 6c690ee1039b ("x86/mm: Split read_cr3() into read_cr3_pa() and __read_cr3()"), which deals with the problem of read_cr3() callers finding stray bits in the cr3 that they expected to be page-aligned; and for hibernation, his 4.14 commit f34902c5c6c0 ("x86/hibernate/64: Mask off CR3's PCID bits in the saved CR3"). But if 0 is used for kernel PCID, then there's no need to add in those commits - whenever the kernel looks, it sees 0 in the lower bits; and 0 for kernel seems an obvious choice. And I naughtily propose 128 for user PCID. Because there's a place in _SWITCH_TO_USER_CR3 where it takes note of the need for TLB FLUSH, but needs to reset that to NOFLUSH for the next occasion. Currently it does so with a "movb $(0x80)" into the high byte of the per-cpu quadword, but that will cause a machine without PCID support to crash. Now, if %al just happened to have 0x80 in it at that point, on a machine with PCID support, but 0 on a machine without PCID support... (That will go badly wrong once the pgd can be at a physical address above 2^56, but even with 5-level paging, physical goes up to 2^52.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I4b3a3ddd80c00e4ff25a4104db7339a9ad6b0566 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755918 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7/arch/x86/mm/tlb.c [modify] https://crrev.com/50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9932194a686277480c94d04044c445719d14f043 commit 9932194a686277480c94d04044c445719d14f043 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:46 2017 CHROMIUM: kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user Mostly this commit is just unshouting X86_CR3_PCID_KERN_VAR and X86_CR3_PCID_USER_VAR: we usually name variables in lower-case. But why does x86_cr3_pcid_noflush need to be __aligned(PAGE_SIZE)? Ah, it's a leftover from when kaiser_add_user_map() once complained about mapping the same page twice. Make it __read_mostly instead. (I'm a little uneasy about all the unrelated data which shares its page getting user-mapped too, but that was so before, and not a big deal: though we call it user-mapped, it's not mapped with _PAGE_USER.) And there is a little change around the two calls to do_nmi(). Previously they set the NOFLUSH bit (if PCID supported) when forcing to kernel context before do_nmi(); now they also have the NOFLUSH bit set (if PCID supported) when restoring context after: nothing done in do_nmi() should require a TLB to be flushed here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I148587672c36207d7f781fd67b74897dd783f097 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755919 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/9932194a686277480c94d04044c445719d14f043/arch/x86/entry/entry_64.S [modify] https://crrev.com/9932194a686277480c94d04044c445719d14f043/arch/x86/mm/kaiser.c [modify] https://crrev.com/9932194a686277480c94d04044c445719d14f043/arch/x86/include/asm/kaiser.h
,
Nov 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4a42e979a5cf9cd7cec7bba4fa3858516388599b commit 4a42e979a5cf9cd7cec7bba4fa3858516388599b Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 09 06:21:48 2017 CHROMIUM: kaiser: paranoid_entry pass cr3 need to paranoid_exit Neel Natu points out that paranoid_entry() was wrong to assume that an entry that did not need swapgs would not need SWITCH_KERNEL_CR3: paranoid_entry (used for debug breakpoint, int3, double fault or MCE; though I think it's only the MCE case that is cause for concern here) can break in at an awkward time, between cr3 switch and swapgs, but its handling always needs kernel gs and kernel cr3. Easy to fix in itself, but paranoid_entry() also needs to convey to paranoid_exit() (and my reading of macro idtentry says paranoid_entry and paranoid_exit are always paired) how to restore the prior state. The swapgs state is already conveyed by %ebx (0 or 1), so extend that also to convey when SWITCH_USER_CR3 will be needed (2 or 3). (Yes, I'd much prefer that 0 meant no swapgs, whereas it's the other way round: and a convention shared with error_entry() and error_exit(), which I don't want to touch. Perhaps I should have inverted the bit for switch cr3 too, but did not.) paranoid_exit() would be straightforward, except for TRACE_IRQS: it did TRACE_IRQS_IRETQ when doing swapgs, but TRACE_IRQS_IRETQ_DEBUG when not: which is it supposed to use when SWITCH_USER_CR3 is split apart from that? As best as I can determine, commit 5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep") missed the swapgs case, and should have used TRACE_IRQS_IRETQ_DEBUG there too (the discrepancy has nothing to do with the liberal use of _NO_STACK and _UNSAFE_STACK hereabouts: TRACE_IRQS_OFF_DEBUG has just been used in all cases); discrepancy lovingly preserved across several paranoid_exit() cleanups, but I'm now removing it. Neel further indicates that to use SWITCH_USER_CR3_NO_STACK there in paranoid_exit() is now not only unnecessary but unsafe: might corrupt syscall entry's unsafe_stack_register_backup of %rax. Just use SWITCH_USER_CR3: and delete SWITCH_USER_CR3_NO_STACK altogether, before we make the mistake of using it again. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> hughd adds: this commit fixes an issue in the Kaiser-without-PCIDs part of the series, and ought to be moved earlier, if you decided to make a release of Kaiser-without-PCIDs. Change-Id: I25cc60d92f270239c7cde1aacb5e04432c4c742d Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755920 Reviewed-by: Guenter Roeck <groeck@chromium.org> [modify] https://crrev.com/4a42e979a5cf9cd7cec7bba4fa3858516388599b/arch/x86/entry/entry_64.S [modify] https://crrev.com/4a42e979a5cf9cd7cec7bba4fa3858516388599b/arch/x86/include/asm/kaiser.h
,
Nov 13 2017
Grace, this is the second part of the KAISER patches, popping it back to the top of your inbox. They landed late Wednesday night last week on ToT, so I would expect we can safely attempt to land this maybe Tuesday or Wednesday. Let me know what you think.
,
Nov 14 2017
If you can merge them Tue they'll make it into our beta push this week. Thanks
,
Nov 14 2017
I will try to land them tonight.
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460 commit 0de0d1ab2816ecfae3ea73404a7c6c9adba7d460 Author: Richard Fellner <richard.fellner@student.tugraz.at> Date: Tue Nov 14 21:43:51 2017 BACKPORT: KAISER: Kernel Address Isolation This patch introduces our implementation of KAISER (Kernel Address Isolation to have Side-channels Efficiently Removed), a kernel isolation technique to close hardware side channels on kernel address information. More information about the patch can be found on: https://github.com/IAIK/KAISER From: Richard Fellner <richard.fellner@student.tugraz.at> From: Daniel Gruss <daniel.gruss@iaik.tugraz.at> Subject: [RFC, PATCH] x86_64: KAISER - do not map kernel in user mode Date: Thu, 4 May 2017 14:26:50 +0200 Link: http://marc.info/?l=linux-kernel&m=149390087310405&w=2 Kaiser-4.10-SHA1: c4b1831d44c6144d3762ccc72f0c4e71a0c713e5 To: <linux-kernel@vger.kernel.org> To: <kernel-hardening@lists.openwall.com> Cc: <clementine.maurice@iaik.tugraz.at> Cc: <moritz.lipp@iaik.tugraz.at> Cc: Michael Schwarz <michael.schwarz@iaik.tugraz.at> Cc: Richard Fellner <richard.fellner@student.tugraz.at> Cc: Ingo Molnar <mingo@kernel.org> Cc: <kirill.shutemov@linux.intel.com> Cc: <anders.fogh@gdata-adan.de> After several recent works [1,2,3] KASLR on x86_64 was basically considered dead by many researchers. We have been working on an efficient but effective fix for this problem and found that not mapping the kernel space when running in user mode is the solution to this problem [4] (the corresponding paper [5] will be presented at ESSoS17). With this RFC patch we allow anybody to configure their kernel with the flag CONFIG_KAISER to add our defense mechanism. If there are any questions we would love to answer them. We also appreciate any comments! Cheers, Daniel (+ the KAISER team from Graz University of Technology) [1] http://www.ieee-security.org/TC/SP2013/papers/4977a191.pdf [2] https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-Using-Undocumented-CPU-Behaviour-To-See-Into-Kernel-Mode-And-Break-KASLR-In-The-Process.pdf [3] https://www.blackhat.com/docs/us-16/materials/us-16-Jang-Breaking-Kernel-Address-Space-Layout-Randomization-KASLR-With-Intel-TSX.pdf [4] https://github.com/IAIK/KAISER [5] https://gruss.cc/files/kaiser.pdf BUG=chromium:781913 TEST=Build, deploy on soraka. Conflicts: arch/x86/entry/entry_64.S arch/x86/entry/entry_64_compat.S arch/x86/mm/Makefile kernel/fork.c Change-Id: I40217b72c391002580c92b300ddad4798a9de800 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755859 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ac9f5f8a35dd01b9655e98f8c2452a3588b29564) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765025 [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/kernel/fork.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/security/Kconfig [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/mm/Makefile [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/include/linux/percpu-defs.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/init/main.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/mm/pgtable.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/hw_irq.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/kernel/process.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/processor.h [add] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/mm/pageattr.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/kernel/head_64.S [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/entry/entry_64.S [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/kernel/irqinit.c [modify] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/kernel/espfix_64.c [add] https://crrev.com/0de0d1ab2816ecfae3ea73404a7c6c9adba7d460/arch/x86/mm/kaiser.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f5610cae69a1659f4e6ee5735173e32af8dc61ad commit f5610cae69a1659f4e6ee5735173e32af8dc61ad Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:30 2017 CHROMIUM: kaiser: merged update Merged fixes and cleanups, rebased to 4.4.89 tree (no 5-level paging). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/kernel/cpu/common.c arch/x86/mm/kaiser.c kernel/fork.c Change-Id: I82adde2cb43c1fe5792a97bc16fb973053c9f8c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755860 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit fa47f3f7ddff5c2d99ae1657221168b4fabf6435) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765026 [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/entry/entry_64.S [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/mm/pgtable.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/kernel/fork.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/kernel/tracepoint.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/mm/pageattr.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/security/Kconfig [add] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/include/linux/kaiser.h [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/kernel/espfix_64.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/mm/kaiser.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/kernel/ldt.c [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/f5610cae69a1659f4e6ee5735173e32af8dc61ad/arch/x86/kernel/head_64.S
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/16263245d7fda9ab582899ebdc7c726fbb6a0925 commit 16263245d7fda9ab582899ebdc7c726fbb6a0925 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:35 2017 CHROMIUM: kaiser: do not set _PAGE_NX on pgd_none native_pgd_clear() uses native_set_pgd(), so native_set_pgd() must avoid setting the _PAGE_NX bit on an otherwise pgd_none() entry: usually that just generated a warning on exit, but sometimes more mysterious and damaging failures (our production machines could not complete booting). The original fix to this just avoided adding _PAGE_NX to an empty entry; but eventually more problems surfaced with kexec, and EFI mapping expected to be a problem too. So now instead change native_set_pgd() to update shadow only if _PAGE_USER: A few places (kernel/machine_kexec_64.c, platform/efi/efi_64.c for sure) use set_pgd() to set up a temporary internal virtual address space, with physical pages remapped at what Kaiser regards as userspace addresses: Kaiser then assumes a shadow pgd follows, which it will try to corrupt. This appears to be responsible for the recent kexec and kdump failures; though it's unclear how those did not manifest as a problem before. Ah, the shadow pgd will only be assumed to "follow" if the requested pgd is on an even-numbered page: so I suppose it was going wrong 50% of the time all along. What we need is a flag to set_pgd(), to tell it we're dealing with userspace. Er, isn't that what the pgd's _PAGE_USER bit is saying? Add a test for that. But we cannot do the same for pgd_clear() (which may be called to clear corrupted entries - set aside the question of "corrupt in which pgd?" until later), so there just rely on pgd_clear() not being called in the problematic cases - with a WARN_ON_ONCE() which should fire half the time if it is. But this is getting too big for an inline function: move it into arch/x86/mm/kaiser.c (which then demands a boot/compressed mod); and de-void and de-space native_get_shadow/normal_pgd() while here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/kaslr.c (not in this tree) Change-Id: Id1259efe7a01a4fc601be6b0f9c5fd55239300c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755861 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ac21472e1451bffa3235086919ff037e00e9b65f) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765067 [modify] https://crrev.com/16263245d7fda9ab582899ebdc7c726fbb6a0925/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/16263245d7fda9ab582899ebdc7c726fbb6a0925/arch/x86/mm/kaiser.c [modify] https://crrev.com/16263245d7fda9ab582899ebdc7c726fbb6a0925/arch/x86/boot/compressed/misc.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/148228e9653bd395479b0b3a312f0de830c84228 commit 148228e9653bd395479b0b3a312f0de830c84228 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:40 2017 CHROMIUM: kaiser: stack map PAGE_SIZE at THREAD_SIZE-PAGE_SIZE Kaiser only needs to map one page of the stack; and kernel/fork.c did not build on powerpc (no __PAGE_KERNEL). It's all cleaner if linux/kaiser.h provides kaiser_map_thread_stack() and kaiser_unmap_thread_stack() wrappers around asm/kaiser.h's kaiser_add_mapping() and kaiser_remove_mapping(). And use linux/kaiser.h in init/main.c to avoid the #ifdefs there. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ib0d66c132a6ff873c83a0684392b4ddf06de6a10 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755862 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 8e1e9d530e2bd7cd89f2721219254a045e8cbd48) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765068 [modify] https://crrev.com/148228e9653bd395479b0b3a312f0de830c84228/init/main.c [modify] https://crrev.com/148228e9653bd395479b0b3a312f0de830c84228/kernel/fork.c [modify] https://crrev.com/148228e9653bd395479b0b3a312f0de830c84228/include/linux/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e50c532800ea9e9ad666bb225cfffc32039e78da commit e50c532800ea9e9ad666bb225cfffc32039e78da Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:44 2017 CHROMIUM: kaiser: fix build and FIXME in alloc_ldt_struct() Include linux/kaiser.h instead of asm/kaiser.h to build ldt.c without CONFIG_KAISER. kaiser_add_mapping() does already return an error code, so fix the FIXME. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I9f64fd683d5750a927dd08574327a1bbf0929a11 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755863 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit c6c8d8ca877952a5f1f89df8f47627e79cc305e9) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765069 [modify] https://crrev.com/e50c532800ea9e9ad666bb225cfffc32039e78da/arch/x86/kernel/ldt.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a8eeea1e7f5535e16617738eec3550e63e7add33 commit a8eeea1e7f5535e16617738eec3550e63e7add33 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:48 2017 CHROMIUM: kaiser: KAISER depends on SMP It is absurd that KAISER should depend on SMP, but apparently nobody has tried a UP build before: which breaks on implicit declaration of function 'per_cpu_offset' in arch/x86/mm/kaiser.c. Now, you would expect that to be trivially fixed up; but looking at the System.map when that block is #ifdef'ed out of kaiser_init(), I see that in a UP build __per_cpu_user_mapped_end is precisely at __per_cpu_user_mapped_start, and the items carefully gathered into that section for user-mapping on SMP, dispersed elsewhere on UP. So, some other kind of section assignment will be needed on UP, but implementing that is not a priority: just make KAISER depend on SMP for now. Also inserted a blank line before the option, tidied up the brief Kconfig help message, and added an "If unsure, Y". BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Idd72fa2a008d117cf3ee7de05a8d54dbc248fb9e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755864 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 01f235baf809019a00527d661cc6cb2041fbad5b) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765070 [modify] https://crrev.com/a8eeea1e7f5535e16617738eec3550e63e7add33/security/Kconfig
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4228abf65ea7ab11b02bf66606c39ef632825071 commit 4228abf65ea7ab11b02bf66606c39ef632825071 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:52 2017 CHROMIUM: kaiser: fix regs to do_nmi() ifndef CONFIG_KAISER pjt has observed that nmi's second (nmi_from_kernel) call to do_nmi() adjusted the %rdi regs arg, rightly when CONFIG_KAISER, but wrongly when not CONFIG_KAISER. Although the minimal change is to add an #ifdef CONFIG_KAISER around the addq line, that looks cluttered, and I prefer how the first call to do_nmi() handled it: prepare args in %rdi and %rsi before getting into the CONFIG_KAISER block, since it does not touch them at all. And while we're here, place the "#ifdef CONFIG_KAISER" that follows each, to enclose the "Unconditionally restore CR3" comment: matching how the "Unconditionally use kernel CR3" comment above is enclosed. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Icb8e9a1809e48ce73395a9747b440e6ceae057d0 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755865 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 43a094a2f624732b0eefa66b4333e4712b806dcd) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765071 [modify] https://crrev.com/4228abf65ea7ab11b02bf66606c39ef632825071/arch/x86/entry/entry_64.S
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/43c4e91d837cf9a26a7d4b35da239327b5a3b32d commit 43c4e91d837cf9a26a7d4b35da239327b5a3b32d Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:47:56 2017 CHROMIUM: kaiser: fix perf crashes Avoid perf crashes: place debug_store in the user-mapped per-cpu area instead of allocating, and use page allocator plus kaiser_add_mapping() to keep the BTS and PEBS buffers user-mapped (that is, present in the user mapping, though visible only to kernel and hardware). The PEBS fixup buffer does not need this treatment. The need for a user-mapped struct debug_store showed up before doing any conscious perf testing: in a couple of kernel paging oopses on Westmere, implicating the debug_store offset of the per-cpu area. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/events/intel/ds.c (not in this tree) arch/x86/kernel/cpu/perf_event_intel_ds.c (patched instead of that) Change-Id: I0c931d8a84f5f5a13f93a8a9fd4bdb89c7ab19aa Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755866 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 8637113958847c7cdb2765e1143b8adf0bd9e339) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765072 [modify] https://crrev.com/43c4e91d837cf9a26a7d4b35da239327b5a3b32d/arch/x86/kernel/cpu/perf_event_intel_ds.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ffb3189bbc558a8da4261f419c796a6e8ae952eb commit ffb3189bbc558a8da4261f419c796a6e8ae952eb Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:01 2017 CHROMIUM: kaiser: ENOMEM if kaiser_pagetable_walk() NULL kaiser_add_user_map() took no notice when kaiser_pagetable_walk() failed. And avoid its might_sleep() when atomic (though atomic at present unused). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia8a7e3e07f23feb351f22ffd67c7bfac5630d019 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755867 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 7f4856b454700da9907315b57a1ce24011e1c18c) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765073 [modify] https://crrev.com/ffb3189bbc558a8da4261f419c796a6e8ae952eb/arch/x86/mm/kaiser.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/65fa519c5341ebd6e8082f69c1c604dc513b5843 commit 65fa519c5341ebd6e8082f69c1c604dc513b5843 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:05 2017 CHROMIUM: kaiser: tidied up asm/kaiser.h somewhat Mainly deleting a surfeit of blank lines, and reflowing header comment. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I408260c74e143d058adce4675131c844f2dac9f3 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755868 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 1bc83b3acbaebea13803ded40578a0c12c4f7100) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765074 [modify] https://crrev.com/65fa519c5341ebd6e8082f69c1c604dc513b5843/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e03f93eea0e4637a55231e12fa0e7c94d079a6e8 commit e03f93eea0e4637a55231e12fa0e7c94d079a6e8 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:09 2017 CHROMIUM: kaiser: tidied up kaiser_add/remove_mapping slightly Yes, unmap_pud_range_nofree()'s declaration ought to be in a header file really, but I'm not sure we want to use it anyway: so for now just declare it inside kaiser_remove_mapping(). And there doesn't seem to be such a thing as unmap_p4d_range(), even in a 5-level paging tree. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I38dae645f7502f7d5ba0fecc257831f87b4279f9 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755869 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 2a10c1a7f53f7235ed0c3017afec8d9bbc1f4f43) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765075 [modify] https://crrev.com/e03f93eea0e4637a55231e12fa0e7c94d079a6e8/arch/x86/mm/kaiser.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ec984a36935529e879f0e9e883b5563ccab1a675 commit ec984a36935529e879f0e9e883b5563ccab1a675 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:13 2017 CHROMIUM: kaiser: kaiser_remove_mapping() move along the pgd When removing the bogus comment from kaiser_remove_mapping(), I really ought to have checked the extent of its bogosity: as Neel points out, there is nothing to stop unmap_pud_range_nofree() from continuing beyond the end of a pud (and starting in the wrong position on the next). Fix kaiser_remove_mapping() to constrain the extent and advance pgd pointer correctly: use pgd_addr_end() macro as used throughout base mm (but don't assume page-rounded start and size in this case). But this bug was very unlikely to trigger in this backport: since any buddy allocation is contained within a single pud extent, and we are not using vmapped stacks (and are only mapping one page of stack anyway): the only way to hit this bug here would be when freeing a large modified ldt. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I2a98a4764bb931e206ebfbf9a5bf3c1a4deb87fe Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755870 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 80b8008a5e74f402d72201c409752bbabbd06992) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765076 [modify] https://crrev.com/ec984a36935529e879f0e9e883b5563ccab1a675/arch/x86/mm/kaiser.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/261fe7a14e556c63612502bf7120ed6adb7d9c6f commit 261fe7a14e556c63612502bf7120ed6adb7d9c6f Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:18 2017 CHROMIUM: kaiser: align addition to x86/mm/Makefile Use tab not space so they line up properly, kaslr.o also. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/Makefile Change-Id: I8db22a668201a4bf1024208c6aa38e08d725ffc8 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755871 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 0188de2dded589e3150d5b5938adb295ba9375ca) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765077 [modify] https://crrev.com/261fe7a14e556c63612502bf7120ed6adb7d9c6f/arch/x86/mm/Makefile
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae commit c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:22 2017 CHROMIUM: kaiser: cleanups while trying for gold link While trying to get our gold link to work, four cleanups: matched the gdt_page declaration to its definition; in fiddling unsuccessfully with PERCPU_INPUT(), lined up backslashes; lined up the backslashes according to convention in percpu-defs.h; deleted the unused irq_stack_pointer addition to irq_stack_union. Sad to report that aligning backslashes does not appear to help gold align to 8192: but while these did not help, they are worth keeping. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Iaed06bb95203cd34eef9f3fdfa7eda8a421ad8ca Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755872 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit cd2b86f7196eed81957ea4f47c04701ad7b74f7b) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765078 [modify] https://crrev.com/c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae/include/linux/percpu-defs.h [modify] https://crrev.com/c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae/arch/x86/include/asm/processor.h [modify] https://crrev.com/c3f4b4bd455b97017cb04fc6ec5a4c1b130edcae/arch/x86/include/asm/desc.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/678beb53c4f3f950aac46303fe9a55a1e868fce8 commit 678beb53c4f3f950aac46303fe9a55a1e868fce8 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:27 2017 CHROMIUM: kaiser: name that 0x1000 KAISER_SHADOW_PGD_OFFSET There's a 0x1000 in various places, which looks better with a name. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ie64ada4e5f572404efc9b7e2854d5d87fd9ef78e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755913 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 88bf87b7c68a66ed89584a7426d1be20843cdcec) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765079 [modify] https://crrev.com/678beb53c4f3f950aac46303fe9a55a1e868fce8/arch/x86/entry/entry_64.S [modify] https://crrev.com/678beb53c4f3f950aac46303fe9a55a1e868fce8/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/458177124a2756a6ea51de47cbbcb85f3d7c90fd commit 458177124a2756a6ea51de47cbbcb85f3d7c90fd Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:30 2017 CHROMIUM: kaiser: delete KAISER_REAL_SWITCH option We fail to see what CONFIG_KAISER_REAL_SWITCH is for: it seems to be left over from early development, and now just obscures tricky parts of the code. Delete it before adding PCIDs, or nokaiser boot option. (Or if there is some good reason to keep the option, then it needs a help text - and a "depends on KAISER", so that all those without KAISER are not asked the question.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I27469aa929149bb9af3f8f5de0f4423e4f9ed8ce Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755914 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit cf1d433fafa19b280791ce5ba00cac64ab18e4f6) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765080 [modify] https://crrev.com/458177124a2756a6ea51de47cbbcb85f3d7c90fd/arch/x86/entry/entry_64.S [modify] https://crrev.com/458177124a2756a6ea51de47cbbcb85f3d7c90fd/security/Kconfig [modify] https://crrev.com/458177124a2756a6ea51de47cbbcb85f3d7c90fd/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/cb4ab525a33d66172fdd77b2607cffa4c620e022 commit cb4ab525a33d66172fdd77b2607cffa4c620e022 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:34 2017 CHROMIUM: kaiser: vmstat show NR_KAISERTABLE as nr_overhead The kaiser update made an interesting choice, never to free any shadow page tables. Contention on global spinlock was worrying, particularly with it held across page table scans when freeing. Something had to be done: I was going to add refcounting; but simply never to free them is an appealing choice, minimizing contention without complicating the code (the more a page table is found already, the less the spinlock is used). But leaking pages in this way is also a worry: can we get away with it? At the very least, we need a count to show how bad it actually gets: in principle, one might end up wasting about 1/256 of memory that way (1/512 for when direct-mapped pages have to be user-mapped, plus 1/512 for when they are user-mapped from the vmalloc area on another occasion (but we don't have vmalloc'ed stacks, so only large ldts are vmalloc'ed). Add per-cpu stat NR_KAISERTABLE: including 256 at startup for the shared pgd entries, and 1 for each intermediate page table added thereafter for user-mapping - but leave out the 1 per mm, for its shadow pgd, because that distracts from the monotonic increase. Shown in /proc/vmstat as nr_overhead (0 if kaiser not enabled). In practice, it doesn't look so bad so far: more like 1/12000 after nine hours of gtests below; and movable pageblock segregation should tend to cluster the kaiser tables into a subset of the address space (if not, they will be bad for compaction too). But production may tell a different story: keep an eye on this number, and bring back lighter freeing if it gets out of control (maybe a shrinker). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Id10448e0350f4ed7ab40b374c866225d12724887 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755915 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit a7374df72e4b1945d04bc5cb9829c27c1d98b52a) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765081 [modify] https://crrev.com/cb4ab525a33d66172fdd77b2607cffa4c620e022/arch/x86/mm/kaiser.c [modify] https://crrev.com/cb4ab525a33d66172fdd77b2607cffa4c620e022/mm/vmstat.c [modify] https://crrev.com/cb4ab525a33d66172fdd77b2607cffa4c620e022/include/linux/mmzone.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e9826b4dc5e14e9b34722347c3f70752385875f2 commit e9826b4dc5e14e9b34722347c3f70752385875f2 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:38 2017 CHROMIUM: kaiser: enhanced by kernel and user PCIDs Merged performance improvements to Kaiser, using distinct kernel and user Process Context Identifiers to minimize the TLB flushing. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I17de75d3c84b878015e7f6a885bebe8d770b9e73 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755916 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit dc1e80b0a539d7066f1f30f70c9e96f1d74e8310) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765082 [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/entry/entry_64.S [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/include/asm/cpufeature.h [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/mm/kaiser.c [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/kvm/x86.c [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/include/uapi/asm/processor-flags.h [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/mm/tlb.c [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/e9826b4dc5e14e9b34722347c3f70752385875f2/arch/x86/kernel/cpu/common.c
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9 commit f0a38c615f89ac4aa22908179e2f4c6d1a047ec9 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:42 2017 CHROMIUM: kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user We have many machines (Westmere, Sandybridge, Ivybridge) supporting PCID but not INVPCID: on these load_new_mm_cr3() simply crashed. Flushing user context inside load_new_mm_cr3() without the use of invpcid is difficult: momentarily switch from kernel to user context and back to do so? I'm not sure whether that can be safely done at all, and would risk polluting user context with kernel internals, and kernel context with stale user externals. Instead, follow the hint in the comment that was there: change X86_CR3_PCID_USER_VAR to be a per-cpu variable, then load_new_mm_cr3() can leave a note in it, for SWITCH_USER_CR3 on return to userspace to flush user context TLB, instead of default X86_CR3_PCID_USER_NOFLUSH. Which works well enough that there's no need to do it this way only when invpcid is unsupported: it's a good alternative to invpcid here. But there's a couple of inlines in asm/tlbflush.h that need to do the same trick, so it's best to localize all this per-cpu business in mm/kaiser.c: moving that part of the initialization from setup_pcid() to kaiser_setup_pcid(); with kaiser_flush_tlb_on_return_to_user() the function for noting an X86_CR3_PCID_USER_FLUSH. And let's keep a KAISER_SHADOW_PGD_OFFSET in there, to avoid the extra OR on exit. I did try to make the feature tests in asm/tlbflush.h more consistent with each other: there seem to be far too many ways of performing such tests, and I don't have a good grasp of their differences. At first I converted them all to be static_cpu_has(): but that proved to be a mistake, as the comment in __native_flush_tlb_single() hints; so then I reversed and made them all this_cpu_has(). Probably all gratuitous change, but that's the way it's working at present. I am slightly bothered by the way non-per-cpu X86_CR3_PCID_KERN_VAR gets re-initialized by each cpu (before and after these changes): no problem when (as usual) all cpus on a machine have the same features, but in principle incorrect. However, my experiment to per-cpu-ify that one did not end well... BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia29c40d3255f6fa2178b87287b025af775dc21fd Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755917 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ad3609e37688c670f0a86dedbb3f8f3bf8fbca00) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765083 [modify] https://crrev.com/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9/arch/x86/mm/tlb.c [modify] https://crrev.com/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9/arch/x86/mm/kaiser.c [modify] https://crrev.com/f0a38c615f89ac4aa22908179e2f4c6d1a047ec9/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2a4d323b6b9933e9546f6803379b972a270d797c commit 2a4d323b6b9933e9546f6803379b972a270d797c Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:47 2017 CHROMIUM: kaiser: PCID 0 for kernel and 128 for user Why was 4 chosen for kernel PCID and 6 for user PCID? No good reason in a backport where PCIDs are only used for Kaiser. If we continue with those, then we shall need to add Andy Lutomirski's 4.13 commit 6c690ee1039b ("x86/mm: Split read_cr3() into read_cr3_pa() and __read_cr3()"), which deals with the problem of read_cr3() callers finding stray bits in the cr3 that they expected to be page-aligned; and for hibernation, his 4.14 commit f34902c5c6c0 ("x86/hibernate/64: Mask off CR3's PCID bits in the saved CR3"). But if 0 is used for kernel PCID, then there's no need to add in those commits - whenever the kernel looks, it sees 0 in the lower bits; and 0 for kernel seems an obvious choice. And I naughtily propose 128 for user PCID. Because there's a place in _SWITCH_TO_USER_CR3 where it takes note of the need for TLB FLUSH, but needs to reset that to NOFLUSH for the next occasion. Currently it does so with a "movb $(0x80)" into the high byte of the per-cpu quadword, but that will cause a machine without PCID support to crash. Now, if %al just happened to have 0x80 in it at that point, on a machine with PCID support, but 0 on a machine without PCID support... (That will go badly wrong once the pgd can be at a physical address above 2^56, but even with 5-level paging, physical goes up to 2^52.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I4b3a3ddd80c00e4ff25a4104db7339a9ad6b0566 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755918 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765084 [modify] https://crrev.com/2a4d323b6b9933e9546f6803379b972a270d797c/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/2a4d323b6b9933e9546f6803379b972a270d797c/arch/x86/mm/tlb.c [modify] https://crrev.com/2a4d323b6b9933e9546f6803379b972a270d797c/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f08748f16a5cc9314128aba09d033613f8faf88e commit f08748f16a5cc9314128aba09d033613f8faf88e Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:50 2017 CHROMIUM: kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user Mostly this commit is just unshouting X86_CR3_PCID_KERN_VAR and X86_CR3_PCID_USER_VAR: we usually name variables in lower-case. But why does x86_cr3_pcid_noflush need to be __aligned(PAGE_SIZE)? Ah, it's a leftover from when kaiser_add_user_map() once complained about mapping the same page twice. Make it __read_mostly instead. (I'm a little uneasy about all the unrelated data which shares its page getting user-mapped too, but that was so before, and not a big deal: though we call it user-mapped, it's not mapped with _PAGE_USER.) And there is a little change around the two calls to do_nmi(). Previously they set the NOFLUSH bit (if PCID supported) when forcing to kernel context before do_nmi(); now they also have the NOFLUSH bit set (if PCID supported) when restoring context after: nothing done in do_nmi() should require a TLB to be flushed here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I148587672c36207d7f781fd67b74897dd783f097 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755919 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 9932194a686277480c94d04044c445719d14f043) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765085 [modify] https://crrev.com/f08748f16a5cc9314128aba09d033613f8faf88e/arch/x86/entry/entry_64.S [modify] https://crrev.com/f08748f16a5cc9314128aba09d033613f8faf88e/arch/x86/mm/kaiser.c [modify] https://crrev.com/f08748f16a5cc9314128aba09d033613f8faf88e/arch/x86/include/asm/kaiser.h
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/894358a0af93890cf277df74d58b461d1d60b587 commit 894358a0af93890cf277df74d58b461d1d60b587 Author: Hugh Dickins <hughd@google.com> Date: Tue Nov 14 21:48:55 2017 CHROMIUM: kaiser: paranoid_entry pass cr3 need to paranoid_exit Neel Natu points out that paranoid_entry() was wrong to assume that an entry that did not need swapgs would not need SWITCH_KERNEL_CR3: paranoid_entry (used for debug breakpoint, int3, double fault or MCE; though I think it's only the MCE case that is cause for concern here) can break in at an awkward time, between cr3 switch and swapgs, but its handling always needs kernel gs and kernel cr3. Easy to fix in itself, but paranoid_entry() also needs to convey to paranoid_exit() (and my reading of macro idtentry says paranoid_entry and paranoid_exit are always paired) how to restore the prior state. The swapgs state is already conveyed by %ebx (0 or 1), so extend that also to convey when SWITCH_USER_CR3 will be needed (2 or 3). (Yes, I'd much prefer that 0 meant no swapgs, whereas it's the other way round: and a convention shared with error_entry() and error_exit(), which I don't want to touch. Perhaps I should have inverted the bit for switch cr3 too, but did not.) paranoid_exit() would be straightforward, except for TRACE_IRQS: it did TRACE_IRQS_IRETQ when doing swapgs, but TRACE_IRQS_IRETQ_DEBUG when not: which is it supposed to use when SWITCH_USER_CR3 is split apart from that? As best as I can determine, commit 5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep") missed the swapgs case, and should have used TRACE_IRQS_IRETQ_DEBUG there too (the discrepancy has nothing to do with the liberal use of _NO_STACK and _UNSAFE_STACK hereabouts: TRACE_IRQS_OFF_DEBUG has just been used in all cases); discrepancy lovingly preserved across several paranoid_exit() cleanups, but I'm now removing it. Neel further indicates that to use SWITCH_USER_CR3_NO_STACK there in paranoid_exit() is now not only unnecessary but unsafe: might corrupt syscall entry's unsafe_stack_register_backup of %rax. Just use SWITCH_USER_CR3: and delete SWITCH_USER_CR3_NO_STACK altogether, before we make the mistake of using it again. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> hughd adds: this commit fixes an issue in the Kaiser-without-PCIDs part of the series, and ought to be moved earlier, if you decided to make a release of Kaiser-without-PCIDs. Change-Id: I25cc60d92f270239c7cde1aacb5e04432c4c742d Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755920 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 4a42e979a5cf9cd7cec7bba4fa3858516388599b) Signed-off-by: Aditya Kali <adityakali@google.com> Reviewed-on: https://chromium-review.googlesource.com/765086 [modify] https://crrev.com/894358a0af93890cf277df74d58b461d1d60b587/arch/x86/entry/entry_64.S [modify] https://crrev.com/894358a0af93890cf277df74d58b461d1d60b587/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445 commit 2f44f3b9c300f154cb7a43ff4c28a4a0629e6445 Author: Richard Fellner <richard.fellner@student.tugraz.at> Date: Wed Nov 15 02:44:11 2017 BACKPORT: KAISER: Kernel Address Isolation This patch introduces our implementation of KAISER (Kernel Address Isolation to have Side-channels Efficiently Removed), a kernel isolation technique to close hardware side channels on kernel address information. More information about the patch can be found on: https://github.com/IAIK/KAISER From: Richard Fellner <richard.fellner@student.tugraz.at> From: Daniel Gruss <daniel.gruss@iaik.tugraz.at> Subject: [RFC, PATCH] x86_64: KAISER - do not map kernel in user mode Date: Thu, 4 May 2017 14:26:50 +0200 Link: http://marc.info/?l=linux-kernel&m=149390087310405&w=2 Kaiser-4.10-SHA1: c4b1831d44c6144d3762ccc72f0c4e71a0c713e5 To: <linux-kernel@vger.kernel.org> To: <kernel-hardening@lists.openwall.com> Cc: <clementine.maurice@iaik.tugraz.at> Cc: <moritz.lipp@iaik.tugraz.at> Cc: Michael Schwarz <michael.schwarz@iaik.tugraz.at> Cc: Richard Fellner <richard.fellner@student.tugraz.at> Cc: Ingo Molnar <mingo@kernel.org> Cc: <kirill.shutemov@linux.intel.com> Cc: <anders.fogh@gdata-adan.de> After several recent works [1,2,3] KASLR on x86_64 was basically considered dead by many researchers. We have been working on an efficient but effective fix for this problem and found that not mapping the kernel space when running in user mode is the solution to this problem [4] (the corresponding paper [5] will be presented at ESSoS17). With this RFC patch we allow anybody to configure their kernel with the flag CONFIG_KAISER to add our defense mechanism. If there are any questions we would love to answer them. We also appreciate any comments! Cheers, Daniel (+ the KAISER team from Graz University of Technology) [1] http://www.ieee-security.org/TC/SP2013/papers/4977a191.pdf [2] https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-Using-Undocumented-CPU-Behaviour-To-See-Into-Kernel-Mode-And-Break-KASLR-In-The-Process.pdf [3] https://www.blackhat.com/docs/us-16/materials/us-16-Jang-Breaking-Kernel-Address-Space-Layout-Randomization-KASLR-With-Intel-TSX.pdf [4] https://github.com/IAIK/KAISER [5] https://gruss.cc/files/kaiser.pdf BUG=chromium:781913 TEST=Build, deploy on caroline. Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) arch/x86/entry/entry_64_compat.S (not in this tree) arch/x86/ia32/ia32entry.S (patched instead of that) arch/x86/include/asm/hw_irq.h arch/x86/include/asm/pgtable_types.h arch/x86/include/asm/processor.h arch/x86/kernel/irqinit.c arch/x86/kernel/process.c arch/x86/mm/Makefile arch/x86/mm/pgtable.c init/main.c Change-Id: Ie80c6f856fe77056e12eab8a6c49d10531003cde Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757193 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770007 [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/kernel/fork.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/security/Kconfig [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/mm/Makefile [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/process_64.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/init/main.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/mm/pgtable.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/hw_irq.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/process.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/ia32/ia32entry.S [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/processor.h [add] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/mm/pageattr.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/include/linux/percpu-defs.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/head_64.S [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/irqinit.c [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/entry_64.S [modify] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/kernel/espfix_64.c [add] https://crrev.com/2f44f3b9c300f154cb7a43ff4c28a4a0629e6445/arch/x86/mm/kaiser.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7 commit a6bec3ddec2a0bb8b157ea427bea50e13936bdd7 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:44:24 2017 CHROMIUM: kaiser: merged update Merged fixes and cleanups, rebased to 3.18.72 tree (no 5-level paging). BUG=chromium:781913 TEST=Build, deploy on caroline. hughd adds: there's a "DEFAULT_FRAME 0" here which I suspect is wrong, but was taking too long to work out what's right: deferred until later with "/* ???? */"; but then I was thrilled to read 4.2 commit 131484c8da97 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") and simply set it aside as not worth bothering about. Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) arch/x86/include/asm/pgtable_types.h arch/x86/kernel/cpu/common.c arch/x86/kernel/ldt.c arch/x86/mm/kaiser.c arch/x86/mm/pgtable.c kernel/fork.c Change-Id: Ib46534697cd9b6006e5b2cadcc76c479ba41159c Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757194 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770008 [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/mm/pgtable.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/kernel/fork.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/kernel/tracepoint.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/kernel/entry_64.S [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/mm/pageattr.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/security/Kconfig [add] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/include/linux/kaiser.h [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/kernel/espfix_64.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/mm/kaiser.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/kernel/ldt.c [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/a6bec3ddec2a0bb8b157ea427bea50e13936bdd7/arch/x86/kernel/head_64.S
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e068cdbee474be75d1f43f3801d142979b15f562 commit e068cdbee474be75d1f43f3801d142979b15f562 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:44:29 2017 CHROMIUM: kaiser: do not set _PAGE_NX on pgd_none native_pgd_clear() uses native_set_pgd(), so native_set_pgd() must avoid setting the _PAGE_NX bit on an otherwise pgd_none() entry: usually that just generated a warning on exit, but sometimes more mysterious and damaging failures (our production machines could not complete booting). The original fix to this just avoided adding _PAGE_NX to an empty entry; but eventually more problems surfaced with kexec, and EFI mapping expected to be a problem too. So now instead change native_set_pgd() to update shadow only if _PAGE_USER: A few places (kernel/machine_kexec_64.c, platform/efi/efi_64.c for sure) use set_pgd() to set up a temporary internal virtual address space, with physical pages remapped at what Kaiser regards as userspace addresses: Kaiser then assumes a shadow pgd follows, which it will try to corrupt. This appears to be responsible for the recent kexec and kdump failures; though it's unclear how those did not manifest as a problem before. Ah, the shadow pgd will only be assumed to "follow" if the requested pgd is on an even-numbered page: so I suppose it was going wrong 50% of the time all along. What we need is a flag to set_pgd(), to tell it we're dealing with userspace. Er, isn't that what the pgd's _PAGE_USER bit is saying? Add a test for that. But we cannot do the same for pgd_clear() (which may be called to clear corrupted entries - set aside the question of "corrupt in which pgd?" until later), so there just rely on pgd_clear() not being called in the problematic cases - with a WARN_ON_ONCE() which should fire half the time if it is. But this is getting too big for an inline function: move it into arch/x86/mm/kaiser.c (which then demands a boot/compressed mod); and de-void and de-space native_get_shadow/normal_pgd() while here. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/boot/compressed/misc.h arch/x86/mm/kaslr.c (not in this tree) Change-Id: Ia33254afd3845287e4518d65097c24b8422643ec Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757195 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770009 [modify] https://crrev.com/e068cdbee474be75d1f43f3801d142979b15f562/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/e068cdbee474be75d1f43f3801d142979b15f562/arch/x86/mm/kaiser.c [modify] https://crrev.com/e068cdbee474be75d1f43f3801d142979b15f562/arch/x86/boot/compressed/misc.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/dd4a9f2a53337e3d70c8cabcfc8f705d5a800331 commit dd4a9f2a53337e3d70c8cabcfc8f705d5a800331 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:44:42 2017 CHROMIUM: kaiser: stack map PAGE_SIZE at THREAD_SIZE-PAGE_SIZE Kaiser only needs to map one page of the stack; and kernel/fork.c did not build on powerpc (no __PAGE_KERNEL). It's all cleaner if linux/kaiser.h provides kaiser_map_thread_stack() and kaiser_unmap_thread_stack() wrappers around asm/kaiser.h's kaiser_add_mapping() and kaiser_remove_mapping(). And use linux/kaiser.h in init/main.c to avoid the #ifdefs there. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: init/main.c Change-Id: I78174dfd06ecae6d9447e3038bc1d84275aedf39 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757196 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770010 [modify] https://crrev.com/dd4a9f2a53337e3d70c8cabcfc8f705d5a800331/init/main.c [modify] https://crrev.com/dd4a9f2a53337e3d70c8cabcfc8f705d5a800331/kernel/fork.c [modify] https://crrev.com/dd4a9f2a53337e3d70c8cabcfc8f705d5a800331/include/linux/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/247b8dd27193fb337abc5d84c9c5de3c7f183c60 commit 247b8dd27193fb337abc5d84c9c5de3c7f183c60 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:44:54 2017 CHROMIUM: kaiser: fix build and FIXME in alloc_ldt_struct() Include linux/kaiser.h instead of asm/kaiser.h to build ldt.c without CONFIG_KAISER. kaiser_add_mapping() does already return an error code, so fix the FIXME. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ib0cdbbd63d336f3750b0799496685d1519993cc4 Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757197 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770011 [modify] https://crrev.com/247b8dd27193fb337abc5d84c9c5de3c7f183c60/arch/x86/kernel/ldt.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/581567c2089eb7614065e88e8e51d17c37eaa7a1 commit 581567c2089eb7614065e88e8e51d17c37eaa7a1 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:44:59 2017 CHROMIUM: kaiser: KAISER depends on SMP It is absurd that KAISER should depend on SMP, but apparently nobody has tried a UP build before: which breaks on implicit declaration of function 'per_cpu_offset' in arch/x86/mm/kaiser.c. Now, you would expect that to be trivially fixed up; but looking at the System.map when that block is #ifdef'ed out of kaiser_init(), I see that in a UP build __per_cpu_user_mapped_end is precisely at __per_cpu_user_mapped_start, and the items carefully gathered into that section for user-mapping on SMP, dispersed elsewhere on UP. So, some other kind of section assignment will be needed on UP, but implementing that is not a priority: just make KAISER depend on SMP for now. Also inserted a blank line before the option, tidied up the brief Kconfig help message, and added an "If unsure, Y". BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ida6186330c5072e0b5853be9adf19556deecca2e Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757198 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770012 [modify] https://crrev.com/581567c2089eb7614065e88e8e51d17c37eaa7a1/security/Kconfig
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f3a1de61b336e94002ad6c4df8f1560f3a7197fa commit f3a1de61b336e94002ad6c4df8f1560f3a7197fa Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:04 2017 CHROMIUM: kaiser: fix regs to do_nmi() ifndef CONFIG_KAISER pjt has observed that nmi's second (nmi_from_kernel) call to do_nmi() adjusted the %rdi regs arg, rightly when CONFIG_KAISER, but wrongly when not CONFIG_KAISER. Although the minimal change is to add an #ifdef CONFIG_KAISER around the addq line, that looks cluttered, and I prefer how the first call to do_nmi() handled it: prepare args in %rdi and %rsi before getting into the CONFIG_KAISER block, since it does not touch them at all. And while we're here, place the "#ifdef CONFIG_KAISER" that follows each, to enclose the "Unconditionally restore CR3" comment: matching how the "Unconditionally use kernel CR3" comment above is enclosed. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) Change-Id: I5b58c574b799f67987f09f55dfed06058c19aaf0 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757199 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770013 [modify] https://crrev.com/f3a1de61b336e94002ad6c4df8f1560f3a7197fa/arch/x86/kernel/entry_64.S
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3ab26875701f6cee0e1cafe5a6b48fc1a92e2120 commit 3ab26875701f6cee0e1cafe5a6b48fc1a92e2120 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:18 2017 CHROMIUM: kaiser: fix perf crashes Avoid perf crashes: place debug_store in the user-mapped per-cpu area instead of allocating, and use page allocator plus kaiser_add_mapping() to keep the BTS and PEBS buffers user-mapped (that is, present in the user mapping, though visible only to kernel and hardware). The PEBS fixup buffer does not need this treatment. The need for a user-mapped struct debug_store showed up before doing any conscious perf testing: in a couple of kernel paging oopses on Westmere, implicating the debug_store offset of the per-cpu area. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/events/intel/ds.c (not in this tree) arch/x86/kernel/cpu/perf_event_intel_ds.c (patched instead of that) Change-Id: I5d4df356715e333749ff940bc92eb6640a3a1f68 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757200 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770014 [modify] https://crrev.com/3ab26875701f6cee0e1cafe5a6b48fc1a92e2120/arch/x86/kernel/cpu/perf_event_intel_ds.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/10e7e003eca6b4a84de9f9bc509e852fa5ba47cb commit 10e7e003eca6b4a84de9f9bc509e852fa5ba47cb Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:32 2017 CHROMIUM: kaiser: ENOMEM if kaiser_pagetable_walk() NULL kaiser_add_user_map() took no notice when kaiser_pagetable_walk() failed. And avoid its might_sleep() when atomic (though atomic at present unused). BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/kaiser.c Change-Id: Ie7d97a10cd0cbf4c7c8525854020c49df4cbc249 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757201 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770015 [modify] https://crrev.com/10e7e003eca6b4a84de9f9bc509e852fa5ba47cb/arch/x86/mm/kaiser.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/57009ddbd5121711c1440bbd6f2a9d7a0f16312b commit 57009ddbd5121711c1440bbd6f2a9d7a0f16312b Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:36 2017 CHROMIUM: kaiser: tidied up asm/kaiser.h somewhat Mainly deleting a surfeit of blank lines, and reflowing header comment. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: If83307355f4f2aa401e6e3ef1520f41511572aaf Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757202 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770016 [modify] https://crrev.com/57009ddbd5121711c1440bbd6f2a9d7a0f16312b/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/66d67ec81a3c3ddcf0703ac30ae4917a646226d8 commit 66d67ec81a3c3ddcf0703ac30ae4917a646226d8 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:41 2017 CHROMIUM: kaiser: tidied up kaiser_add/remove_mapping slightly Yes, unmap_pud_range_nofree()'s declaration ought to be in a header file really, but I'm not sure we want to use it anyway: so for now just declare it inside kaiser_remove_mapping(). And there doesn't seem to be such a thing as unmap_p4d_range(), even in a 5-level paging tree. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: I6cacea1f740dd80451dae1ba4fae21a13be3343d Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757203 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770017 [modify] https://crrev.com/66d67ec81a3c3ddcf0703ac30ae4917a646226d8/arch/x86/mm/kaiser.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/34b4a154e3b20e32b932d7bfc709ec8ea14971d1 commit 34b4a154e3b20e32b932d7bfc709ec8ea14971d1 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:45:48 2017 CHROMIUM: kaiser: kaiser_remove_mapping() move along the pgd When removing the bogus comment from kaiser_remove_mapping(), I really ought to have checked the extent of its bogosity: as Neel points out, there is nothing to stop unmap_pud_range_nofree() from continuing beyond the end of a pud (and starting in the wrong position on the next). Fix kaiser_remove_mapping() to constrain the extent and advance pgd pointer correctly: use pgd_addr_end() macro as used throughout base mm (but don't assume page-rounded start and size in this case). But this bug was very unlikely to trigger in this backport: since any buddy allocation is contained within a single pud extent, and we are not using vmapped stacks (and are only mapping one page of stack anyway): the only way to hit this bug here would be when freeing a large modified ldt. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: I5e80f3da47a979c65e70fbc638381b6a0db905de Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757204 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770018 [modify] https://crrev.com/34b4a154e3b20e32b932d7bfc709ec8ea14971d1/arch/x86/mm/kaiser.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/3a5e732fc5edda9133acaa1d37c2825d995b63d6 commit 3a5e732fc5edda9133acaa1d37c2825d995b63d6 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:49:52 2017 CHROMIUM: kaiser: align addition to x86/mm/Makefile Use tab not space so they line up properly, kaslr.o also. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/Makefile Change-Id: I99b05abd7b2cc4992a97985afebc0df2e8e57109 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757205 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770019 [modify] https://crrev.com/3a5e732fc5edda9133acaa1d37c2825d995b63d6/arch/x86/mm/Makefile
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/fcdaed0426085c845be58c58cab2f74590106b68 commit fcdaed0426085c845be58c58cab2f74590106b68 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:49:58 2017 CHROMIUM: kaiser: cleanups while trying for gold link While trying to get our gold link to work, four cleanups: matched the gdt_page declaration to its definition; in fiddling unsuccessfully with PERCPU_INPUT(), lined up backslashes; lined up the backslashes according to convention in percpu-defs.h; deleted the unused irq_stack_pointer addition to irq_stack_union. Sad to report that aligning backslashes does not appear to help gold align to 8192: but while these did not help, they are worth keeping. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: I8ef6cf29360b5430da75e6b580b4538668f022bb Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757206 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770020 [modify] https://crrev.com/fcdaed0426085c845be58c58cab2f74590106b68/include/linux/percpu-defs.h [modify] https://crrev.com/fcdaed0426085c845be58c58cab2f74590106b68/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/fcdaed0426085c845be58c58cab2f74590106b68/arch/x86/include/asm/processor.h [modify] https://crrev.com/fcdaed0426085c845be58c58cab2f74590106b68/arch/x86/include/asm/desc.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c90a48cdc05dd0a877a5ec02d1818e6a15e00613 commit c90a48cdc05dd0a877a5ec02d1818e6a15e00613 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:03 2017 CHROMIUM: kaiser: name that 0x1000 KAISER_SHADOW_PGD_OFFSET There's a 0x1000 in various places, which looks better with a name. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) Change-Id: I431f09588a9b11a2114b9192f78889f52fea52ed Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757207 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770021 [modify] https://crrev.com/c90a48cdc05dd0a877a5ec02d1818e6a15e00613/arch/x86/kernel/entry_64.S [modify] https://crrev.com/c90a48cdc05dd0a877a5ec02d1818e6a15e00613/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0a06bacf5a56c3bae6f5e1ec3df46f2a795ebca1 commit 0a06bacf5a56c3bae6f5e1ec3df46f2a795ebca1 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:08 2017 CHROMIUM: kaiser: delete KAISER_REAL_SWITCH option We fail to see what CONFIG_KAISER_REAL_SWITCH is for: it seems to be left over from early development, and now just obscures tricky parts of the code. Delete it before adding PCIDs, or nokaiser boot option. (Or if there is some good reason to keep the option, then it needs a help text - and a "depends on KAISER", so that all those without KAISER are not asked the question.) BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) Change-Id: I94570de669a5721030a1ab994cb961fbfc93b4c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757208 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770022 [modify] https://crrev.com/0a06bacf5a56c3bae6f5e1ec3df46f2a795ebca1/arch/x86/kernel/entry_64.S [modify] https://crrev.com/0a06bacf5a56c3bae6f5e1ec3df46f2a795ebca1/security/Kconfig [modify] https://crrev.com/0a06bacf5a56c3bae6f5e1ec3df46f2a795ebca1/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/af2e7f309d70df478f0c39ec14cadb16060b0bf0 commit af2e7f309d70df478f0c39ec14cadb16060b0bf0 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:13 2017 CHROMIUM: kaiser: vmstat show NR_KAISERTABLE as nr_overhead The kaiser update made an interesting choice, never to free any shadow page tables. Contention on global spinlock was worrying, particularly with it held across page table scans when freeing. Something had to be done: I was going to add refcounting; but simply never to free them is an appealing choice, minimizing contention without complicating the code (the more a page table is found already, the less the spinlock is used). But leaking pages in this way is also a worry: can we get away with it? At the very least, we need a count to show how bad it actually gets: in principle, one might end up wasting about 1/256 of memory that way (1/512 for when direct-mapped pages have to be user-mapped, plus 1/512 for when they are user-mapped from the vmalloc area on another occasion (but we don't have vmalloc'ed stacks, so only large ldts are vmalloc'ed). Add per-cpu stat NR_KAISERTABLE: including 256 at startup for the shared pgd entries, and 1 for each intermediate page table added thereafter for user-mapping - but leave out the 1 per mm, for its shadow pgd, because that distracts from the monotonic increase. Shown in /proc/vmstat as nr_overhead (0 if kaiser not enabled). In practice, it doesn't look so bad so far: more like 1/12000 after nine hours of gtests below; and movable pageblock segregation should tend to cluster the kaiser tables into a subset of the address space (if not, they will be bad for compaction too). But production may tell a different story: keep an eye on this number, and bring back lighter freeing if it gets out of control (maybe a shrinker). BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ia11496fce27fe374809fd925a86f527911ea4885 Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757209 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770023 [modify] https://crrev.com/af2e7f309d70df478f0c39ec14cadb16060b0bf0/arch/x86/mm/kaiser.c [modify] https://crrev.com/af2e7f309d70df478f0c39ec14cadb16060b0bf0/mm/vmstat.c [modify] https://crrev.com/af2e7f309d70df478f0c39ec14cadb16060b0bf0/include/linux/mmzone.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/dcc04cd359681afac6164049ec7990253ff5bcd6 commit dcc04cd359681afac6164049ec7990253ff5bcd6 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:18 2017 CHROMIUM: kaiser: enhanced by kernel and user PCIDs Merged performance improvements to Kaiser, using distinct kernel and user Process Context Identifiers to minimize the TLB flushing. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) arch/x86/entry/entry_64_compat.S (not in this tree) arch/x86/ia32/ia32entry.S (patched instead of that) arch/x86/include/asm/pgtable_types.h arch/x86/mm/tlb.c Change-Id: Iea3c3390a2977b2a0e2dcca1d470a3d9dd1f093a Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757210 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770024 [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/include/asm/cpufeature.h [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/mm/kaiser.c [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/kvm/x86.c [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/include/uapi/asm/processor-flags.h [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/kernel/entry_64.S [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/mm/tlb.c [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/ia32/ia32entry.S [modify] https://crrev.com/dcc04cd359681afac6164049ec7990253ff5bcd6/arch/x86/kernel/cpu/common.c
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce commit 5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:23 2017 CHROMIUM: kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user We have many machines (Westmere, Sandybridge, Ivybridge) supporting PCID but not INVPCID: on these load_new_mm_cr3() simply crashed. Flushing user context inside load_new_mm_cr3() without the use of invpcid is difficult: momentarily switch from kernel to user context and back to do so? I'm not sure whether that can be safely done at all, and would risk polluting user context with kernel internals, and kernel context with stale user externals. Instead, follow the hint in the comment that was there: change X86_CR3_PCID_USER_VAR to be a per-cpu variable, then load_new_mm_cr3() can leave a note in it, for SWITCH_USER_CR3 on return to userspace to flush user context TLB, instead of default X86_CR3_PCID_USER_NOFLUSH. Which works well enough that there's no need to do it this way only when invpcid is unsupported: it's a good alternative to invpcid here. But there's a couple of inlines in asm/tlbflush.h that need to do the same trick, so it's best to localize all this per-cpu business in mm/kaiser.c: moving that part of the initialization from setup_pcid() to kaiser_setup_pcid(); with kaiser_flush_tlb_on_return_to_user() the function for noting an X86_CR3_PCID_USER_FLUSH. And let's keep a KAISER_SHADOW_PGD_OFFSET in there, to avoid the extra OR on exit. I did try to make the feature tests in asm/tlbflush.h more consistent with each other: there seem to be far too many ways of performing such tests, and I don't have a good grasp of their differences. At first I converted them all to be static_cpu_has(): but that proved to be a mistake, as the comment in __native_flush_tlb_single() hints; so then I reversed and made them all this_cpu_has(). Probably all gratuitous change, but that's the way it's working at present. I am slightly bothered by the way non-per-cpu X86_CR3_PCID_KERN_VAR gets re-initialized by each cpu (before and after these changes): no problem when (as usual) all cpus on a machine have the same features, but in principle incorrect. However, my experiment to per-cpu-ify that one did not end well... BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ic269da92352cbda636f1f499faf2ec24bf525783 Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757211 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770025 [modify] https://crrev.com/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce/arch/x86/mm/tlb.c [modify] https://crrev.com/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce/arch/x86/mm/kaiser.c [modify] https://crrev.com/5bf168bb9f9084d1648ae46a0d6dc3f41f7c86ce/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c6c734dfa0efbfa4f01a734e524397dac103f687 commit c6c734dfa0efbfa4f01a734e524397dac103f687 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:28 2017 CHROMIUM: kaiser: PCID 0 for kernel and 128 for user Why was 4 chosen for kernel PCID and 6 for user PCID? No good reason in a backport where PCIDs are only used for Kaiser. If we continue with those, then we shall need to add Andy Lutomirski's 4.13 commit 6c690ee1039b ("x86/mm: Split read_cr3() into read_cr3_pa() and __read_cr3()"), which deals with the problem of read_cr3() callers finding stray bits in the cr3 that they expected to be page-aligned; and for hibernation, his 4.14 commit f34902c5c6c0 ("x86/hibernate/64: Mask off CR3's PCID bits in the saved CR3"). But if 0 is used for kernel PCID, then there's no need to add in those commits - whenever the kernel looks, it sees 0 in the lower bits; and 0 for kernel seems an obvious choice. And I naughtily propose 128 for user PCID. Because there's a place in _SWITCH_TO_USER_CR3 where it takes note of the need for TLB FLUSH, but needs to reset that to NOFLUSH for the next occasion. Currently it does so with a "movb $(0x80)" into the high byte of the per-cpu quadword, but that will cause a machine without PCID support to crash. Now, if %al just happened to have 0x80 in it at that point, on a machine with PCID support, but 0 on a machine without PCID support... (That will go badly wrong once the pgd can be at a physical address above 2^56, but even with 5-level paging, physical goes up to 2^52.) BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ie7be90a0c3a9671aed4570df143bce363e318ef2 Originally-from: Dave Hansen <dave.hansen@intel.com> Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757212 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770026 [modify] https://crrev.com/c6c734dfa0efbfa4f01a734e524397dac103f687/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/c6c734dfa0efbfa4f01a734e524397dac103f687/arch/x86/mm/tlb.c [modify] https://crrev.com/c6c734dfa0efbfa4f01a734e524397dac103f687/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ecf1fda03c0fc17d9ca43b134564ba1f67777656 commit ecf1fda03c0fc17d9ca43b134564ba1f67777656 Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:33 2017 CHROMIUM: kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user Mostly this commit is just unshouting X86_CR3_PCID_KERN_VAR and X86_CR3_PCID_USER_VAR: we usually name variables in lower-case. But why does x86_cr3_pcid_noflush need to be __aligned(PAGE_SIZE)? Ah, it's a leftover from when kaiser_add_user_map() once complained about mapping the same page twice. Make it __read_mostly instead. (I'm a little uneasy about all the unrelated data which shares its page getting user-mapped too, but that was so before, and not a big deal: though we call it user-mapped, it's not mapped with _PAGE_USER.) And there is a little change around the two calls to do_nmi(). Previously they set the NOFLUSH bit (if PCID supported) when forcing to kernel context before do_nmi(); now they also have the NOFLUSH bit set (if PCID supported) when restoring context after: nothing done in do_nmi() should require a TLB to be flushed here. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/entry/entry_64.S (not in this tree) arch/x86/kernel/entry_64.S (patched instead of that) Change-Id: Iccab6e460ad9b2cc160e555180727020357f4335 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757213 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770027 [modify] https://crrev.com/ecf1fda03c0fc17d9ca43b134564ba1f67777656/arch/x86/kernel/entry_64.S [modify] https://crrev.com/ecf1fda03c0fc17d9ca43b134564ba1f67777656/arch/x86/mm/kaiser.c [modify] https://crrev.com/ecf1fda03c0fc17d9ca43b134564ba1f67777656/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/93eb9efbae8c6a69e303177a2baa8808f9bf71fa commit 93eb9efbae8c6a69e303177a2baa8808f9bf71fa Author: Hugh Dickins <hughd@google.com> Date: Wed Nov 15 02:50:38 2017 CHROMIUM: kaiser: paranoid_entry pass cr3 need to paranoid_exit Neel Natu points out that paranoid_entry() was wrong to assume that an entry that did not need swapgs would not need SWITCH_KERNEL_CR3: paranoid_entry (used for debug breakpoint, int3, double fault or MCE; though I think it's only the MCE case that is cause for concern here) can break in at an awkward time, between cr3 switch and swapgs, but its handling always needs kernel gs and kernel cr3. Easy to fix in itself, but paranoid_entry() also needs to convey to paranoid_exit() (and my reading of macro idtentry says paranoid_entry and paranoid_exit are always paired) how to restore the prior state. The swapgs state is already conveyed by %ebx (0 or 1), so extend that also to convey when SWITCH_USER_CR3 will be needed (2 or 3). (Yes, I'd much prefer that 0 meant no swapgs, whereas it's the other way round: and a convention shared with error_entry() and error_exit(), which I don't want to touch. Perhaps I should have inverted the bit for switch cr3 too, but did not.) paranoid_exit() would be straightforward, except for TRACE_IRQS: it did TRACE_IRQS_IRETQ when doing swapgs, but TRACE_IRQS_IRETQ_DEBUG when not: which is it supposed to use when SWITCH_USER_CR3 is split apart from that? As best as I can determine, commit 5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep") missed the swapgs case, and should have used TRACE_IRQS_IRETQ_DEBUG there too (the discrepancy has nothing to do with the liberal use of _NO_STACK and _UNSAFE_STACK hereabouts: TRACE_IRQS_OFF_DEBUG has just been used in all cases); discrepancy lovingly preserved across several paranoid_exit() cleanups, but I'm now removing it. Neel further indicates that to use SWITCH_USER_CR3_NO_STACK there in paranoid_exit() is now not only unnecessary but unsafe: might corrupt syscall entry's unsafe_stack_register_backup of %rax. Just use SWITCH_USER_CR3: and delete SWITCH_USER_CR3_NO_STACK altogether, before we make the mistake of using it again. BUG=chromium:781913 TEST=Build, deploy on caroline. Originally-from: Dave Hansen <dave.hansen@intel.com> hughd adds: this commit fixes an issue in the Kaiser-without-PCIDs part of the series, and ought to be moved earlier, if you decided to make a release of Kaiser-without-PCIDs. The patch to paranoid_exit needed reworking, because unlike in later releases, here paranoid_exit handles userspace cases. Conflicts: arch/x86/entry/entry_64.S Change-Id: Ic1d75cf0657f7943a2fe86cb2f05a40db0a3068e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757214 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770028 [modify] https://crrev.com/93eb9efbae8c6a69e303177a2baa8808f9bf71fa/arch/x86/kernel/entry_64.S [modify] https://crrev.com/93eb9efbae8c6a69e303177a2baa8808f9bf71fa/arch/x86/include/asm/kaiser.h
,
Nov 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/de9f7f8596fa9f4fbbce75ada1845fac9a250306 commit de9f7f8596fa9f4fbbce75ada1845fac9a250306 Author: Jorge Lucangeli Obes <jorgelo@chromium.org> Date: Wed Nov 15 02:50:43 2017 CHROMIUM: kaiser: Fix kernel_stack. BUG=chromium:781913 TEST=Build, deploy on caroline. Change-Id: Ib3f14eb79a6dbc1f128e4607f4fb2d8d81e78ea5 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/757215 Reviewed-by: Guenter Roeck <groeck@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/770029 [modify] https://crrev.com/de9f7f8596fa9f4fbbce75ada1845fac9a250306/arch/x86/kernel/cpu/common.c
,
Nov 15 2017
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c96c0c74b581668c549c5ea9bda9959ffb8967c9 commit c96c0c74b581668c549c5ea9bda9959ffb8967c9 Author: Richard Fellner <richard.fellner@student.tugraz.at> Date: Thu Nov 16 19:40:02 2017 BACKPORT: KAISER: Kernel Address Isolation This patch introduces our implementation of KAISER (Kernel Address Isolation to have Side-channels Efficiently Removed), a kernel isolation technique to close hardware side channels on kernel address information. More information about the patch can be found on: https://github.com/IAIK/KAISER From: Richard Fellner <richard.fellner@student.tugraz.at> From: Daniel Gruss <daniel.gruss@iaik.tugraz.at> Subject: [RFC, PATCH] x86_64: KAISER - do not map kernel in user mode Date: Thu, 4 May 2017 14:26:50 +0200 Link: http://marc.info/?l=linux-kernel&m=149390087310405&w=2 Kaiser-4.10-SHA1: c4b1831d44c6144d3762ccc72f0c4e71a0c713e5 To: <linux-kernel@vger.kernel.org> To: <kernel-hardening@lists.openwall.com> Cc: <clementine.maurice@iaik.tugraz.at> Cc: <moritz.lipp@iaik.tugraz.at> Cc: Michael Schwarz <michael.schwarz@iaik.tugraz.at> Cc: Richard Fellner <richard.fellner@student.tugraz.at> Cc: Ingo Molnar <mingo@kernel.org> Cc: <kirill.shutemov@linux.intel.com> Cc: <anders.fogh@gdata-adan.de> After several recent works [1,2,3] KASLR on x86_64 was basically considered dead by many researchers. We have been working on an efficient but effective fix for this problem and found that not mapping the kernel space when running in user mode is the solution to this problem [4] (the corresponding paper [5] will be presented at ESSoS17). With this RFC patch we allow anybody to configure their kernel with the flag CONFIG_KAISER to add our defense mechanism. If there are any questions we would love to answer them. We also appreciate any comments! Cheers, Daniel (+ the KAISER team from Graz University of Technology) [1] http://www.ieee-security.org/TC/SP2013/papers/4977a191.pdf [2] https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-Using-Undocumented-CPU-Behaviour-To-See-Into-Kernel-Mode-And-Break-KASLR-In-The-Process.pdf [3] https://www.blackhat.com/docs/us-16/materials/us-16-Jang-Breaking-Kernel-Address-Space-Layout-Randomization-KASLR-With-Intel-TSX.pdf [4] https://github.com/IAIK/KAISER [5] https://gruss.cc/files/kaiser.pdf BUG=chromium:781913 TEST=Build, deploy on soraka. Conflicts: arch/x86/entry/entry_64.S arch/x86/entry/entry_64_compat.S arch/x86/mm/Makefile kernel/fork.c Change-Id: I40217b72c391002580c92b300ddad4798a9de800 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755859 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ac9f5f8a35dd01b9655e98f8c2452a3588b29564) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/764882 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> Tested-by: Marcelo Bocchi <marcelo.bocchi@gmail.com> [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/kernel/fork.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/security/Kconfig [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/mm/Makefile [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/include/linux/percpu-defs.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/init/main.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/mm/pgtable.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/hw_irq.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/kernel/process.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/processor.h [add] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/mm/pageattr.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/kernel/head_64.S [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/entry/entry_64.S [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/kernel/irqinit.c [modify] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/kernel/espfix_64.c [add] https://crrev.com/c96c0c74b581668c549c5ea9bda9959ffb8967c9/arch/x86/mm/kaiser.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/c1b520daa741f21dc5ed3981aa253edc591499e5 commit c1b520daa741f21dc5ed3981aa253edc591499e5 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:06 2017 CHROMIUM: kaiser: merged update Merged fixes and cleanups, rebased to 4.4.89 tree (no 5-level paging). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/kernel/cpu/common.c arch/x86/mm/kaiser.c kernel/fork.c Change-Id: I82adde2cb43c1fe5792a97bc16fb973053c9f8c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755860 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit fa47f3f7ddff5c2d99ae1657221168b4fabf6435) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/764883 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/entry/entry_64.S [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/mm/pgtable.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/kernel/fork.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/include/asm/pgtable.h [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/kernel/tracepoint.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/mm/pageattr.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/security/Kconfig [add] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/include/linux/kaiser.h [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/kernel/espfix_64.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/mm/kaiser.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/kernel/ldt.c [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/c1b520daa741f21dc5ed3981aa253edc591499e5/arch/x86/kernel/head_64.S
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/db8a00fe2b0c6aacb69d23ace2c84a32b7bc3dd1 commit db8a00fe2b0c6aacb69d23ace2c84a32b7bc3dd1 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:10 2017 CHROMIUM: kaiser: do not set _PAGE_NX on pgd_none native_pgd_clear() uses native_set_pgd(), so native_set_pgd() must avoid setting the _PAGE_NX bit on an otherwise pgd_none() entry: usually that just generated a warning on exit, but sometimes more mysterious and damaging failures (our production machines could not complete booting). The original fix to this just avoided adding _PAGE_NX to an empty entry; but eventually more problems surfaced with kexec, and EFI mapping expected to be a problem too. So now instead change native_set_pgd() to update shadow only if _PAGE_USER: A few places (kernel/machine_kexec_64.c, platform/efi/efi_64.c for sure) use set_pgd() to set up a temporary internal virtual address space, with physical pages remapped at what Kaiser regards as userspace addresses: Kaiser then assumes a shadow pgd follows, which it will try to corrupt. This appears to be responsible for the recent kexec and kdump failures; though it's unclear how those did not manifest as a problem before. Ah, the shadow pgd will only be assumed to "follow" if the requested pgd is on an even-numbered page: so I suppose it was going wrong 50% of the time all along. What we need is a flag to set_pgd(), to tell it we're dealing with userspace. Er, isn't that what the pgd's _PAGE_USER bit is saying? Add a test for that. But we cannot do the same for pgd_clear() (which may be called to clear corrupted entries - set aside the question of "corrupt in which pgd?" until later), so there just rely on pgd_clear() not being called in the problematic cases - with a WARN_ON_ONCE() which should fire half the time if it is. But this is getting too big for an inline function: move it into arch/x86/mm/kaiser.c (which then demands a boot/compressed mod); and de-void and de-space native_get_shadow/normal_pgd() while here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/kaslr.c (not in this tree) Change-Id: Id1259efe7a01a4fc601be6b0f9c5fd55239300c1 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755861 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ac21472e1451bffa3235086919ff037e00e9b65f) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/764884 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/db8a00fe2b0c6aacb69d23ace2c84a32b7bc3dd1/arch/x86/include/asm/pgtable_64.h [modify] https://crrev.com/db8a00fe2b0c6aacb69d23ace2c84a32b7bc3dd1/arch/x86/mm/kaiser.c [modify] https://crrev.com/db8a00fe2b0c6aacb69d23ace2c84a32b7bc3dd1/arch/x86/boot/compressed/misc.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8bb60e4424bf84f25ac76513216fcb36b145c0b6 commit 8bb60e4424bf84f25ac76513216fcb36b145c0b6 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:14 2017 CHROMIUM: kaiser: stack map PAGE_SIZE at THREAD_SIZE-PAGE_SIZE Kaiser only needs to map one page of the stack; and kernel/fork.c did not build on powerpc (no __PAGE_KERNEL). It's all cleaner if linux/kaiser.h provides kaiser_map_thread_stack() and kaiser_unmap_thread_stack() wrappers around asm/kaiser.h's kaiser_add_mapping() and kaiser_remove_mapping(). And use linux/kaiser.h in init/main.c to avoid the #ifdefs there. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ib0d66c132a6ff873c83a0684392b4ddf06de6a10 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755862 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 8e1e9d530e2bd7cd89f2721219254a045e8cbd48) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/764885 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/8bb60e4424bf84f25ac76513216fcb36b145c0b6/init/main.c [modify] https://crrev.com/8bb60e4424bf84f25ac76513216fcb36b145c0b6/kernel/fork.c [modify] https://crrev.com/8bb60e4424bf84f25ac76513216fcb36b145c0b6/include/linux/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/54338306259aa08e65924b96ade27a8123979eed commit 54338306259aa08e65924b96ade27a8123979eed Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:18 2017 CHROMIUM: kaiser: fix build and FIXME in alloc_ldt_struct() Include linux/kaiser.h instead of asm/kaiser.h to build ldt.c without CONFIG_KAISER. kaiser_add_mapping() does already return an error code, so fix the FIXME. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I9f64fd683d5750a927dd08574327a1bbf0929a11 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755863 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit c6c8d8ca877952a5f1f89df8f47627e79cc305e9) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/764886 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/54338306259aa08e65924b96ade27a8123979eed/arch/x86/kernel/ldt.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/710de4949c6fbe33fe770667618a9dff680187da commit 710de4949c6fbe33fe770667618a9dff680187da Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:22 2017 CHROMIUM: kaiser: KAISER depends on SMP It is absurd that KAISER should depend on SMP, but apparently nobody has tried a UP build before: which breaks on implicit declaration of function 'per_cpu_offset' in arch/x86/mm/kaiser.c. Now, you would expect that to be trivially fixed up; but looking at the System.map when that block is #ifdef'ed out of kaiser_init(), I see that in a UP build __per_cpu_user_mapped_end is precisely at __per_cpu_user_mapped_start, and the items carefully gathered into that section for user-mapping on SMP, dispersed elsewhere on UP. So, some other kind of section assignment will be needed on UP, but implementing that is not a priority: just make KAISER depend on SMP for now. Also inserted a blank line before the option, tidied up the brief Kconfig help message, and added an "If unsure, Y". BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Idd72fa2a008d117cf3ee7de05a8d54dbc248fb9e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755864 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 01f235baf809019a00527d661cc6cb2041fbad5b) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765027 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/710de4949c6fbe33fe770667618a9dff680187da/security/Kconfig
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/08d3668ddad92cb8dfcf73f347ddec81bbf41f4f commit 08d3668ddad92cb8dfcf73f347ddec81bbf41f4f Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:26 2017 CHROMIUM: kaiser: fix regs to do_nmi() ifndef CONFIG_KAISER pjt has observed that nmi's second (nmi_from_kernel) call to do_nmi() adjusted the %rdi regs arg, rightly when CONFIG_KAISER, but wrongly when not CONFIG_KAISER. Although the minimal change is to add an #ifdef CONFIG_KAISER around the addq line, that looks cluttered, and I prefer how the first call to do_nmi() handled it: prepare args in %rdi and %rsi before getting into the CONFIG_KAISER block, since it does not touch them at all. And while we're here, place the "#ifdef CONFIG_KAISER" that follows each, to enclose the "Unconditionally restore CR3" comment: matching how the "Unconditionally use kernel CR3" comment above is enclosed. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Icb8e9a1809e48ce73395a9747b440e6ceae057d0 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755865 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 43a094a2f624732b0eefa66b4333e4712b806dcd) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765028 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/08d3668ddad92cb8dfcf73f347ddec81bbf41f4f/arch/x86/entry/entry_64.S
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/9e71ba6cc1ea5788666f8eafbd0eb0664819afd7 commit 9e71ba6cc1ea5788666f8eafbd0eb0664819afd7 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:30 2017 CHROMIUM: kaiser: fix perf crashes Avoid perf crashes: place debug_store in the user-mapped per-cpu area instead of allocating, and use page allocator plus kaiser_add_mapping() to keep the BTS and PEBS buffers user-mapped (that is, present in the user mapping, though visible only to kernel and hardware). The PEBS fixup buffer does not need this treatment. The need for a user-mapped struct debug_store showed up before doing any conscious perf testing: in a couple of kernel paging oopses on Westmere, implicating the debug_store offset of the per-cpu area. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/events/intel/ds.c (not in this tree) arch/x86/kernel/cpu/perf_event_intel_ds.c (patched instead of that) Change-Id: I0c931d8a84f5f5a13f93a8a9fd4bdb89c7ab19aa Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755866 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 8637113958847c7cdb2765e1143b8adf0bd9e339) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765029 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/9e71ba6cc1ea5788666f8eafbd0eb0664819afd7/arch/x86/kernel/cpu/perf_event_intel_ds.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/fa53f53e58eeddaf79f9aa57384605d54bba772b commit fa53f53e58eeddaf79f9aa57384605d54bba772b Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:34 2017 CHROMIUM: kaiser: ENOMEM if kaiser_pagetable_walk() NULL kaiser_add_user_map() took no notice when kaiser_pagetable_walk() failed. And avoid its might_sleep() when atomic (though atomic at present unused). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia8a7e3e07f23feb351f22ffd67c7bfac5630d019 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755867 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 7f4856b454700da9907315b57a1ce24011e1c18c) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765030 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/fa53f53e58eeddaf79f9aa57384605d54bba772b/arch/x86/mm/kaiser.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/314ee773508bc29e15d0eec73934a7b6510ce166 commit 314ee773508bc29e15d0eec73934a7b6510ce166 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:40:38 2017 CHROMIUM: kaiser: tidied up asm/kaiser.h somewhat Mainly deleting a surfeit of blank lines, and reflowing header comment. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I408260c74e143d058adce4675131c844f2dac9f3 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755868 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 1bc83b3acbaebea13803ded40578a0c12c4f7100) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765031 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/314ee773508bc29e15d0eec73934a7b6510ce166/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/182df36efd3a74b91e98ff3678e972c4fd7a27bb commit 182df36efd3a74b91e98ff3678e972c4fd7a27bb Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:39 2017 CHROMIUM: kaiser: tidied up kaiser_add/remove_mapping slightly Yes, unmap_pud_range_nofree()'s declaration ought to be in a header file really, but I'm not sure we want to use it anyway: so for now just declare it inside kaiser_remove_mapping(). And there doesn't seem to be such a thing as unmap_p4d_range(), even in a 5-level paging tree. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I38dae645f7502f7d5ba0fecc257831f87b4279f9 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755869 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 2a10c1a7f53f7235ed0c3017afec8d9bbc1f4f43) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765032 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/182df36efd3a74b91e98ff3678e972c4fd7a27bb/arch/x86/mm/kaiser.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/df15708bda97142ab4ef7387390cd82ddab994f6 commit df15708bda97142ab4ef7387390cd82ddab994f6 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:42 2017 CHROMIUM: kaiser: kaiser_remove_mapping() move along the pgd When removing the bogus comment from kaiser_remove_mapping(), I really ought to have checked the extent of its bogosity: as Neel points out, there is nothing to stop unmap_pud_range_nofree() from continuing beyond the end of a pud (and starting in the wrong position on the next). Fix kaiser_remove_mapping() to constrain the extent and advance pgd pointer correctly: use pgd_addr_end() macro as used throughout base mm (but don't assume page-rounded start and size in this case). But this bug was very unlikely to trigger in this backport: since any buddy allocation is contained within a single pud extent, and we are not using vmapped stacks (and are only mapping one page of stack anyway): the only way to hit this bug here would be when freeing a large modified ldt. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I2a98a4764bb931e206ebfbf9a5bf3c1a4deb87fe Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755870 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 80b8008a5e74f402d72201c409752bbabbd06992) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765033 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/df15708bda97142ab4ef7387390cd82ddab994f6/arch/x86/mm/kaiser.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/f835a1d6627cd67881d6bf4d1b2948aa77182799 commit f835a1d6627cd67881d6bf4d1b2948aa77182799 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:46 2017 CHROMIUM: kaiser: align addition to x86/mm/Makefile Use tab not space so they line up properly, kaslr.o also. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Conflicts: arch/x86/mm/Makefile Change-Id: I8db22a668201a4bf1024208c6aa38e08d725ffc8 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755871 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 0188de2dded589e3150d5b5938adb295ba9375ca) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765034 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/f835a1d6627cd67881d6bf4d1b2948aa77182799/arch/x86/mm/Makefile
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c commit 337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:50 2017 CHROMIUM: kaiser: cleanups while trying for gold link While trying to get our gold link to work, four cleanups: matched the gdt_page declaration to its definition; in fiddling unsuccessfully with PERCPU_INPUT(), lined up backslashes; lined up the backslashes according to convention in percpu-defs.h; deleted the unused irq_stack_pointer addition to irq_stack_union. Sad to report that aligning backslashes does not appear to help gold align to 8192: but while these did not help, they are worth keeping. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Iaed06bb95203cd34eef9f3fdfa7eda8a421ad8ca Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755872 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit cd2b86f7196eed81957ea4f47c04701ad7b74f7b) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765035 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c/include/linux/percpu-defs.h [modify] https://crrev.com/337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c/include/asm-generic/vmlinux.lds.h [modify] https://crrev.com/337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c/arch/x86/include/asm/processor.h [modify] https://crrev.com/337e9340cd1e7ee9659c24a5f1e2b59f573a7d0c/arch/x86/include/asm/desc.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/a855104838368439dbf0987a8d05addaaf0913c4 commit a855104838368439dbf0987a8d05addaaf0913c4 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:54 2017 CHROMIUM: kaiser: name that 0x1000 KAISER_SHADOW_PGD_OFFSET There's a 0x1000 in various places, which looks better with a name. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ie64ada4e5f572404efc9b7e2854d5d87fd9ef78e Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755913 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 88bf87b7c68a66ed89584a7426d1be20843cdcec) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765036 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/a855104838368439dbf0987a8d05addaaf0913c4/arch/x86/entry/entry_64.S [modify] https://crrev.com/a855104838368439dbf0987a8d05addaaf0913c4/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/8e378d1b4a7ca8c8efcbb5517c7a3702cded299e commit 8e378d1b4a7ca8c8efcbb5517c7a3702cded299e Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:41:58 2017 CHROMIUM: kaiser: delete KAISER_REAL_SWITCH option We fail to see what CONFIG_KAISER_REAL_SWITCH is for: it seems to be left over from early development, and now just obscures tricky parts of the code. Delete it before adding PCIDs, or nokaiser boot option. (Or if there is some good reason to keep the option, then it needs a help text - and a "depends on KAISER", so that all those without KAISER are not asked the question.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I27469aa929149bb9af3f8f5de0f4423e4f9ed8ce Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755914 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit cf1d433fafa19b280791ce5ba00cac64ab18e4f6) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765037 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/8e378d1b4a7ca8c8efcbb5517c7a3702cded299e/arch/x86/entry/entry_64.S [modify] https://crrev.com/8e378d1b4a7ca8c8efcbb5517c7a3702cded299e/security/Kconfig [modify] https://crrev.com/8e378d1b4a7ca8c8efcbb5517c7a3702cded299e/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6d7682610bdf6119427498953d82ef75f9a90018 commit 6d7682610bdf6119427498953d82ef75f9a90018 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:01 2017 CHROMIUM: kaiser: vmstat show NR_KAISERTABLE as nr_overhead The kaiser update made an interesting choice, never to free any shadow page tables. Contention on global spinlock was worrying, particularly with it held across page table scans when freeing. Something had to be done: I was going to add refcounting; but simply never to free them is an appealing choice, minimizing contention without complicating the code (the more a page table is found already, the less the spinlock is used). But leaking pages in this way is also a worry: can we get away with it? At the very least, we need a count to show how bad it actually gets: in principle, one might end up wasting about 1/256 of memory that way (1/512 for when direct-mapped pages have to be user-mapped, plus 1/512 for when they are user-mapped from the vmalloc area on another occasion (but we don't have vmalloc'ed stacks, so only large ldts are vmalloc'ed). Add per-cpu stat NR_KAISERTABLE: including 256 at startup for the shared pgd entries, and 1 for each intermediate page table added thereafter for user-mapping - but leave out the 1 per mm, for its shadow pgd, because that distracts from the monotonic increase. Shown in /proc/vmstat as nr_overhead (0 if kaiser not enabled). In practice, it doesn't look so bad so far: more like 1/12000 after nine hours of gtests below; and movable pageblock segregation should tend to cluster the kaiser tables into a subset of the address space (if not, they will be bad for compaction too). But production may tell a different story: keep an eye on this number, and bring back lighter freeing if it gets out of control (maybe a shrinker). BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Id10448e0350f4ed7ab40b374c866225d12724887 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755915 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit a7374df72e4b1945d04bc5cb9829c27c1d98b52a) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765038 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/6d7682610bdf6119427498953d82ef75f9a90018/arch/x86/mm/kaiser.c [modify] https://crrev.com/6d7682610bdf6119427498953d82ef75f9a90018/mm/vmstat.c [modify] https://crrev.com/6d7682610bdf6119427498953d82ef75f9a90018/include/linux/mmzone.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6d7f11fe95185375b3d1e9335bae4485af3ef02e commit 6d7f11fe95185375b3d1e9335bae4485af3ef02e Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:05 2017 CHROMIUM: kaiser: enhanced by kernel and user PCIDs Merged performance improvements to Kaiser, using distinct kernel and user Process Context Identifiers to minimize the TLB flushing. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I17de75d3c84b878015e7f6a885bebe8d770b9e73 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755916 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit dc1e80b0a539d7066f1f30f70c9e96f1d74e8310) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765039 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/entry/entry_64.S [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/include/asm/cpufeature.h [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/mm/kaiser.c [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/entry/entry_64_compat.S [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/kvm/x86.c [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/include/asm/kaiser.h [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/include/uapi/asm/processor-flags.h [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/mm/tlb.c [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/6d7f11fe95185375b3d1e9335bae4485af3ef02e/arch/x86/kernel/cpu/common.c
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/28e58d2c436b68465ad19af8e862308832894967 commit 28e58d2c436b68465ad19af8e862308832894967 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:09 2017 CHROMIUM: kaiser: load_new_mm_cr3() let SWITCH_USER_CR3 flush user We have many machines (Westmere, Sandybridge, Ivybridge) supporting PCID but not INVPCID: on these load_new_mm_cr3() simply crashed. Flushing user context inside load_new_mm_cr3() without the use of invpcid is difficult: momentarily switch from kernel to user context and back to do so? I'm not sure whether that can be safely done at all, and would risk polluting user context with kernel internals, and kernel context with stale user externals. Instead, follow the hint in the comment that was there: change X86_CR3_PCID_USER_VAR to be a per-cpu variable, then load_new_mm_cr3() can leave a note in it, for SWITCH_USER_CR3 on return to userspace to flush user context TLB, instead of default X86_CR3_PCID_USER_NOFLUSH. Which works well enough that there's no need to do it this way only when invpcid is unsupported: it's a good alternative to invpcid here. But there's a couple of inlines in asm/tlbflush.h that need to do the same trick, so it's best to localize all this per-cpu business in mm/kaiser.c: moving that part of the initialization from setup_pcid() to kaiser_setup_pcid(); with kaiser_flush_tlb_on_return_to_user() the function for noting an X86_CR3_PCID_USER_FLUSH. And let's keep a KAISER_SHADOW_PGD_OFFSET in there, to avoid the extra OR on exit. I did try to make the feature tests in asm/tlbflush.h more consistent with each other: there seem to be far too many ways of performing such tests, and I don't have a good grasp of their differences. At first I converted them all to be static_cpu_has(): but that proved to be a mistake, as the comment in __native_flush_tlb_single() hints; so then I reversed and made them all this_cpu_has(). Probably all gratuitous change, but that's the way it's working at present. I am slightly bothered by the way non-per-cpu X86_CR3_PCID_KERN_VAR gets re-initialized by each cpu (before and after these changes): no problem when (as usual) all cpus on a machine have the same features, but in principle incorrect. However, my experiment to per-cpu-ify that one did not end well... BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: Ia29c40d3255f6fa2178b87287b025af775dc21fd Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755917 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit ad3609e37688c670f0a86dedbb3f8f3bf8fbca00) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765040 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/28e58d2c436b68465ad19af8e862308832894967/arch/x86/include/asm/tlbflush.h [modify] https://crrev.com/28e58d2c436b68465ad19af8e862308832894967/arch/x86/mm/tlb.c [modify] https://crrev.com/28e58d2c436b68465ad19af8e862308832894967/arch/x86/kernel/cpu/common.c [modify] https://crrev.com/28e58d2c436b68465ad19af8e862308832894967/arch/x86/mm/kaiser.c [modify] https://crrev.com/28e58d2c436b68465ad19af8e862308832894967/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/4ea4abe2397d6ea6516efca832a10a7a8ccb0d57 commit 4ea4abe2397d6ea6516efca832a10a7a8ccb0d57 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:13 2017 CHROMIUM: kaiser: PCID 0 for kernel and 128 for user Why was 4 chosen for kernel PCID and 6 for user PCID? No good reason in a backport where PCIDs are only used for Kaiser. If we continue with those, then we shall need to add Andy Lutomirski's 4.13 commit 6c690ee1039b ("x86/mm: Split read_cr3() into read_cr3_pa() and __read_cr3()"), which deals with the problem of read_cr3() callers finding stray bits in the cr3 that they expected to be page-aligned; and for hibernation, his 4.14 commit f34902c5c6c0 ("x86/hibernate/64: Mask off CR3's PCID bits in the saved CR3"). But if 0 is used for kernel PCID, then there's no need to add in those commits - whenever the kernel looks, it sees 0 in the lower bits; and 0 for kernel seems an obvious choice. And I naughtily propose 128 for user PCID. Because there's a place in _SWITCH_TO_USER_CR3 where it takes note of the need for TLB FLUSH, but needs to reset that to NOFLUSH for the next occasion. Currently it does so with a "movb $(0x80)" into the high byte of the per-cpu quadword, but that will cause a machine without PCID support to crash. Now, if %al just happened to have 0x80 in it at that point, on a machine with PCID support, but 0 on a machine without PCID support... (That will go badly wrong once the pgd can be at a physical address above 2^56, but even with 5-level paging, physical goes up to 2^52.) BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I4b3a3ddd80c00e4ff25a4104db7339a9ad6b0566 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755918 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 50f6c87fe9ac9b2f7f91c9cf67af00e7706d08a7) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765041 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/4ea4abe2397d6ea6516efca832a10a7a8ccb0d57/arch/x86/include/asm/pgtable_types.h [modify] https://crrev.com/4ea4abe2397d6ea6516efca832a10a7a8ccb0d57/arch/x86/mm/tlb.c [modify] https://crrev.com/4ea4abe2397d6ea6516efca832a10a7a8ccb0d57/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/750c6146dd5eac35b31e084dee3a19fff0deb040 commit 750c6146dd5eac35b31e084dee3a19fff0deb040 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:17 2017 CHROMIUM: kaiser: x86_cr3_pcid_noflush and x86_cr3_pcid_user Mostly this commit is just unshouting X86_CR3_PCID_KERN_VAR and X86_CR3_PCID_USER_VAR: we usually name variables in lower-case. But why does x86_cr3_pcid_noflush need to be __aligned(PAGE_SIZE)? Ah, it's a leftover from when kaiser_add_user_map() once complained about mapping the same page twice. Make it __read_mostly instead. (I'm a little uneasy about all the unrelated data which shares its page getting user-mapped too, but that was so before, and not a big deal: though we call it user-mapped, it's not mapped with _PAGE_USER.) And there is a little change around the two calls to do_nmi(). Previously they set the NOFLUSH bit (if PCID supported) when forcing to kernel context before do_nmi(); now they also have the NOFLUSH bit set (if PCID supported) when restoring context after: nothing done in do_nmi() should require a TLB to be flushed here. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> Change-Id: I148587672c36207d7f781fd67b74897dd783f097 Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755919 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 9932194a686277480c94d04044c445719d14f043) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765042 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/750c6146dd5eac35b31e084dee3a19fff0deb040/arch/x86/entry/entry_64.S [modify] https://crrev.com/750c6146dd5eac35b31e084dee3a19fff0deb040/arch/x86/mm/kaiser.c [modify] https://crrev.com/750c6146dd5eac35b31e084dee3a19fff0deb040/arch/x86/include/asm/kaiser.h
,
Nov 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/35b73d1fb74c40cde3eb4b5f841861b3eb2b0007 commit 35b73d1fb74c40cde3eb4b5f841861b3eb2b0007 Author: Hugh Dickins <hughd@google.com> Date: Thu Nov 16 19:42:20 2017 CHROMIUM: kaiser: paranoid_entry pass cr3 need to paranoid_exit Neel Natu points out that paranoid_entry() was wrong to assume that an entry that did not need swapgs would not need SWITCH_KERNEL_CR3: paranoid_entry (used for debug breakpoint, int3, double fault or MCE; though I think it's only the MCE case that is cause for concern here) can break in at an awkward time, between cr3 switch and swapgs, but its handling always needs kernel gs and kernel cr3. Easy to fix in itself, but paranoid_entry() also needs to convey to paranoid_exit() (and my reading of macro idtentry says paranoid_entry and paranoid_exit are always paired) how to restore the prior state. The swapgs state is already conveyed by %ebx (0 or 1), so extend that also to convey when SWITCH_USER_CR3 will be needed (2 or 3). (Yes, I'd much prefer that 0 meant no swapgs, whereas it's the other way round: and a convention shared with error_entry() and error_exit(), which I don't want to touch. Perhaps I should have inverted the bit for switch cr3 too, but did not.) paranoid_exit() would be straightforward, except for TRACE_IRQS: it did TRACE_IRQS_IRETQ when doing swapgs, but TRACE_IRQS_IRETQ_DEBUG when not: which is it supposed to use when SWITCH_USER_CR3 is split apart from that? As best as I can determine, commit 5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep") missed the swapgs case, and should have used TRACE_IRQS_IRETQ_DEBUG there too (the discrepancy has nothing to do with the liberal use of _NO_STACK and _UNSAFE_STACK hereabouts: TRACE_IRQS_OFF_DEBUG has just been used in all cases); discrepancy lovingly preserved across several paranoid_exit() cleanups, but I'm now removing it. Neel further indicates that to use SWITCH_USER_CR3_NO_STACK there in paranoid_exit() is now not only unnecessary but unsafe: might corrupt syscall entry's unsafe_stack_register_backup of %rax. Just use SWITCH_USER_CR3: and delete SWITCH_USER_CR3_NO_STACK altogether, before we make the mistake of using it again. BUG=chromium:781913 TEST=Build, deploy on soraka. Originally-from: Dave Hansen <dave.hansen@intel.com> hughd adds: this commit fixes an issue in the Kaiser-without-PCIDs part of the series, and ought to be moved earlier, if you decided to make a release of Kaiser-without-PCIDs. Change-Id: I25cc60d92f270239c7cde1aacb5e04432c4c742d Signed-off-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/755920 Reviewed-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit 4a42e979a5cf9cd7cec7bba4fa3858516388599b) Signed-off-by: Daniel Wang <wonderfly@google.com> Reviewed-on: https://chromium-review.googlesource.com/765043 Reviewed-by: Bernie Thompson <bhthompson@chromium.org> [modify] https://crrev.com/35b73d1fb74c40cde3eb4b5f841861b3eb2b0007/arch/x86/entry/entry_64.S [modify] https://crrev.com/35b73d1fb74c40cde3eb4b5f841861b3eb2b0007/arch/x86/include/asm/kaiser.h
,
Dec 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/daf6e9e4739adde66f024a540e2379e657fd5c8b commit daf6e9e4739adde66f024a540e2379e657fd5c8b Author: Jan Beulich <JBeulich@suse.com> Date: Wed Dec 20 00:57:56 2017 BACKPORT: x86/ldt: Fix small LDT allocation for Xen While the following commit: 37868fe113 ("x86/ldt: Make modify_ldt synchronous") added a nice comment explaining that Xen needs page-aligned whole page chunks for guest descriptor tables, it then nevertheless used kzalloc() on the small size path. As I'm unaware of guarantees for kmalloc(PAGE_SIZE, ) to return page-aligned memory blocks, I believe this needs to be switched back to __get_free_page() (or better get_zeroed_page()). BUG=chromium:781913 TEST=Build, boot caroline Change-Id: I6c402e7ee63487d40944bc081e827f0c804b4b37 Signed-off-by: Jan Beulich <jbeulich@suse.com> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Andy Lutomirski <luto@kernel.org> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Brian Gerst <brgerst@gmail.com> Cc: David Vrabel <david.vrabel@citrix.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: http://lkml.kernel.org/r/55E735D6020000780009F1E6@prv-mh.provo.novell.com Signed-off-by: Ingo Molnar <mingo@kernel.org> [needed for kaiser support; backport: 2nd half of patch does not apply] Signed-off-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit f454b478861325f067fd58ba7ee9f1b5c4a9d6a0) Reviewed-on: https://chromium-review.googlesource.com/834048 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Jorge Lucangeli Obes <jorgelo@chromium.org> Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org> [modify] https://crrev.com/daf6e9e4739adde66f024a540e2379e657fd5c8b/arch/x86/kernel/ldt.c
,
Jan 31 2018
Gonna mark as fixed and open new bugs for 3.14 and 3.8.
Showing comments 21 - 120
of 120
Older ›
|
||||||||||
►
Sign in to add a comment |
||||||||||
