linux/arch/x86/lib
Linus Torvalds 94a855111e - Add the call depth tracking mitigation for Retbleed which has
been long in the making. It is a lighterweight software-only fix for
 Skylake-based cores where enabling IBRS is a big hammer and causes a
 significant performance impact.
 
 What it basically does is, it aligns all kernel functions to 16 bytes
 boundary and adds a 16-byte padding before the function, objtool
 collects all functions' locations and when the mitigation gets applied,
 it patches a call accounting thunk which is used to track the call depth
 of the stack at any time.
 
 When that call depth reaches a magical, microarchitecture-specific value
 for the Return Stack Buffer, the code stuffs that RSB and avoids its
 underflow which could otherwise lead to the Intel variant of Retbleed.
 
 This software-only solution brings a lot of the lost performance back,
 as benchmarks suggest:
 
   https://lore.kernel.org/all/20220915111039.092790446@infradead.org/
 
 That page above also contains a lot more detailed explanation of the
 whole mechanism
 
 - Implement a new control flow integrity scheme called FineIBT which is
 based on the software kCFI implementation and uses hardware IBT support
 where present to annotate and track indirect branches using a hash to
 validate them
 
 - Other misc fixes and cleanups
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmOZp5EACgkQEsHwGGHe
 VUrZFxAAvi/+8L0IYSK4mKJvixGbTFjxN/Swo2JVOfs34LqGUT6JaBc+VUMwZxdb
 VMTFIZ3ttkKEodjhxGI7oGev6V8UfhI37SmO2lYKXpQVjXXnMlv/M+Vw3teE38CN
 gopi+xtGnT1IeWQ3tc/Tv18pleJ0mh5HKWiW+9KoqgXj0wgF9x4eRYDz1TDCDA/A
 iaBzs56j8m/FSykZHnrWZ/MvjKNPdGlfJASUCPeTM2dcrXQGJ93+X2hJctzDte0y
 Nuiw6Y0htfFBE7xoJn+sqm5Okr+McoUM18/CCprbgSKYk18iMYm3ZtAi6FUQZS1A
 ua4wQCf49loGp15PO61AS5d3OBf5D3q/WihQRbCaJvTVgPp9sWYnWwtcVUuhMllh
 ZQtBU9REcVJ/22bH09Q9CjBW0VpKpXHveqQdqRDViLJ6v/iI6EFGmD24SW/VxyRd
 73k9MBGrL/dOf1SbEzdsnvcSB3LGzp0Om8o/KzJWOomrVKjBCJy16bwTEsCZEJmP
 i406m92GPXeaN1GhTko7vmF0GnkEdJs1GVCZPluCAxxbhHukyxHnrjlQjI4vC80n
 Ylc0B3Kvitw7LGJsPqu+/jfNHADC/zhx1qz/30wb5cFmFbN1aRdp3pm8JYUkn+l/
 zri2Y6+O89gvE/9/xUhMohzHsWUO7xITiBavewKeTP9GSWybWUs=
 =cRy1
 -----END PGP SIGNATURE-----

Merge tag 'x86_core_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 core updates from Borislav Petkov:

 - Add the call depth tracking mitigation for Retbleed which has been
   long in the making. It is a lighterweight software-only fix for
   Skylake-based cores where enabling IBRS is a big hammer and causes a
   significant performance impact.

   What it basically does is, it aligns all kernel functions to 16 bytes
   boundary and adds a 16-byte padding before the function, objtool
   collects all functions' locations and when the mitigation gets
   applied, it patches a call accounting thunk which is used to track
   the call depth of the stack at any time.

   When that call depth reaches a magical, microarchitecture-specific
   value for the Return Stack Buffer, the code stuffs that RSB and
   avoids its underflow which could otherwise lead to the Intel variant
   of Retbleed.

   This software-only solution brings a lot of the lost performance
   back, as benchmarks suggest:

       https://lore.kernel.org/all/20220915111039.092790446@infradead.org/

   That page above also contains a lot more detailed explanation of the
   whole mechanism

 - Implement a new control flow integrity scheme called FineIBT which is
   based on the software kCFI implementation and uses hardware IBT
   support where present to annotate and track indirect branches using a
   hash to validate them

 - Other misc fixes and cleanups

* tag 'x86_core_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (80 commits)
  x86/paravirt: Use common macro for creating simple asm paravirt functions
  x86/paravirt: Remove clobber bitmask from .parainstructions
  x86/debug: Include percpu.h in debugreg.h to get DECLARE_PER_CPU() et al
  x86/cpufeatures: Move X86_FEATURE_CALL_DEPTH from bit 18 to bit 19 of word 11, to leave space for WIP X86_FEATURE_SGX_EDECCSSA bit
  x86/Kconfig: Enable kernel IBT by default
  x86,pm: Force out-of-line memcpy()
  objtool: Fix weak hole vs prefix symbol
  objtool: Optimize elf_dirty_reloc_sym()
  x86/cfi: Add boot time hash randomization
  x86/cfi: Boot time selection of CFI scheme
  x86/ibt: Implement FineIBT
  objtool: Add --cfi to generate the .cfi_sites section
  x86: Add prefix symbols for function padding
  objtool: Add option to generate prefix symbols
  objtool: Avoid O(bloody terrible) behaviour -- an ode to libelf
  objtool: Slice up elf_create_section_symbol()
  kallsyms: Revert "Take callthunks into account"
  x86: Unconfuse CONFIG_ and X86_FEATURE_ namespaces
  x86/retpoline: Fix crash printing warning
  x86/paravirt: Fix a !PARAVIRT build warning
  ...
2022-12-14 15:03:00 -08:00
..
.gitignore .gitignore: add SPDX License Identifier 2020-03-25 11:50:48 +01:00
atomic64_32.c
atomic64_386_32.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
atomic64_cx8_32.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
cache-smp.c smp: Remove smp_call_function() and on_each_cpu() return values 2019-06-23 14:26:26 +02:00
checksum_32.S x86/checksum_32: Remove .fixup usage 2021-12-11 09:09:49 +01:00
clear_page_64.S x86/clear_user: Make it faster 2022-08-18 12:36:42 +02:00
cmdline.c treewide: Use fallthrough pseudo-keyword 2020-08-23 17:36:59 -05:00
cmpxchg8b_emu.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
cmpxchg16b_emu.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
copy_mc_64.S x86/copy_mc_64: Remove .fixup usage 2021-12-11 09:09:46 +01:00
copy_mc.c x86, libnvdimm/test: Remove COPY_MC_TEST 2020-10-26 18:08:35 +01:00
copy_page_64.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
copy_user_64.S x86/uaccess: Don't jump between functions 2022-04-19 21:58:53 +02:00
cpu.c x86/lib/cpu: Address missing prototypes warning 2019-08-08 08:25:53 +02:00
csum-copy_64.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
csum-partial_64.c uml/x86: use x86 load_unaligned_zeropad() 2022-01-30 21:26:39 -05:00
csum-wrappers_64.c net: unexport csum_and_copy_{from,to}_user 2022-04-29 14:37:59 -07:00
delay.c x86/delay: Fix the wrong asm constraint in delay_loop() 2022-04-05 21:21:57 +02:00
error-inject.c x86/error_inject: Align function properly 2022-10-17 16:40:59 +02:00
getuser.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
hweight.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
inat.c x86/insn: Add a __ignore_sync_check__ marker 2021-03-15 11:00:57 +01:00
insn-eval.c x86/32: Remove lazy GS macros 2022-04-14 14:09:43 +02:00
insn.c x86/insn: Use get_unaligned() instead of memcpy() 2021-10-06 11:56:37 +02:00
iomap_copy_64.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
iomem.c x86: kmsan: handle open-coded assembly in lib/iomem.c 2022-10-03 14:03:24 -07:00
kaslr.c x86/kaslr: Fix build warning in KASLR code in boot stub 2022-04-11 09:41:12 +02:00
Makefile x86/mem: Move memmove to out of line assembler 2022-11-01 15:44:07 -07:00
memcpy_32.c x86/mem: Move memmove to out of line assembler 2022-11-01 15:44:07 -07:00
memcpy_64.S x86: Add types to indirectly called assembly functions 2022-09-26 10:13:15 -07:00
memmove_32.S x86/mem: Move memmove to out of line assembler 2022-11-01 15:44:07 -07:00
memmove_64.S x86: Use return-thunk in asm code 2022-06-27 10:33:58 +02:00
memset_64.S x86: clean up symbol aliasing 2022-02-22 16:21:34 +00:00
misc.c
msr-reg-export.c
msr-reg.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
msr-smp.c x86/msr: Fix wr/rdmsr_safe_regs_on_cpu() prototypes 2021-03-22 21:37:03 +01:00
msr.c x86/msr: Make locally used functions static 2021-04-08 11:57:40 +02:00
pc-conf-reg.c x86: Add support for 0x22/0x23 port I/O configuration space 2021-08-10 23:31:43 +02:00
putuser.S x86/putuser: Provide room for padding 2022-10-17 16:41:10 +02:00
retpoline.S x86/calldepth: Add ret/call counting for debug 2022-10-17 16:41:16 +02:00
string_32.c lib/string: Move helper functions out of string.c 2021-09-25 08:20:49 -07:00
strstr_32.c
usercopy_32.c x86/usercopy: Remove .fixup usage 2021-12-11 09:09:50 +01:00
usercopy_64.c x86/clear_user: Make it faster 2022-08-18 12:36:42 +02:00
usercopy.c x86/uaccess: instrument copy_from_user_nmi() 2022-11-08 15:57:24 -08:00
x86-opcode-map.txt x86/insn: Add AVX512-FP16 instructions to the x86 instruction decoder 2022-01-23 20:38:01 +01:00