linux/arch/arm64/kernel
Mark Rutland a76521d160 arm64: Avoid cpus_have_const_cap() for ARM64_{SVE,SME,SME2,FA64}
In system_supports_{sve,sme,sme2,fa64}() we use cpus_have_const_cap() to
check for the relevant cpucaps, but this is only necessary so that
sve_setup() and sme_setup() can run prior to alternatives being patched,
and otherwise alternative_has_cap_*() would be preferable.

For historical reasons, cpus_have_const_cap() is more complicated than
it needs to be. Before cpucaps are finalized, it will perform a bitmap
test of the system_cpucaps bitmap, and once cpucaps are finalized it
will use an alternative branch. This used to be necessary to handle some
race conditions in the window between cpucap detection and the
subsequent patching of alternatives and static branches, where different
branches could be out-of-sync with one another (or w.r.t. alternative
sequences). Now that we use alternative branches instead of static
branches, these are all patched atomically w.r.t. one another, and there
are only a handful of cases that need special care in the window between
cpucap detection and alternative patching.

Due to the above, it would be nice to remove cpus_have_const_cap(), and
migrate callers over to alternative_has_cap_*(), cpus_have_final_cap(),
or cpus_have_cap() depending on when their requirements. This will
remove redundant instructions and improve code generation, and will make
it easier to determine how each callsite will behave before, during, and
after alternative patching.

All of system_supports_{sve,sme,sme2,fa64}() will return false prior to
system cpucaps being detected. In the window between system cpucaps being
detected and patching alternatives, we need system_supports_sve() and
system_supports_sme() to run to initialize SVE and SME properties, but
all other users of system_supports_{sve,sme,sme2,fa64}() don't depend on
the relevant cpucap becoming true until alternatives are patched:

* No KVM code runs until after alternatives are patched, and so this can
  safely use cpus_have_final_cap() or alternative_has_cap_*().

* The cpuid_cpu_online() callback in arch/arm64/kernel/cpuinfo.c is
  registered later from cpuinfo_regs_init() as a device_initcall, and so
  this can safely use cpus_have_final_cap() or alternative_has_cap_*().

* The entry, signal, and ptrace code isn't reachable until userspace has
  run, and so this can safely use cpus_have_final_cap() or
  alternative_has_cap_*().

* Currently perf_reg_validate() will un-reserve the PERF_REG_ARM64_VG
  pseudo-register before alternatives are patched, and before
  sve_setup() has run. If a sampling event is created early enough, this
  would allow perf_ext_reg_value() to sample (the as-yet uninitialized)
  thread_struct::vl[] prior to alternatives being patched.

  It would be preferable to defer this until alternatives are patched,
  and this can safely use alternative_has_cap_*().

* The context-switch code will run during this window as part of
  stop_machine() used during alternatives_patch_all(), and potentially
  for other work if other kernel threads are created early. No threads
  require the use of SVE/SME/SME2/FA64 prior to alternatives being
  patched, and it would be preferable for the related context-switch
  logic to take effect after alternatives are patched so that ths is
  guaranteed to see a consistent system-wide state (e.g. anything
  initialized by sve_setup() and sme_setup().

  This can safely ues alternative_has_cap_*().

This patch replaces the use of cpus_have_const_cap() with
alternative_has_cap_unlikely(), which will avoid generating code to test
the system_cpucaps bitmap and should be better for all subsequent calls
at runtime. The sve_setup() and sme_setup() functions are modified to
use cpus_have_cap() directly so that they can observe the cpucaps being
set prior to alternatives being patched.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Mark Brown <broonie@kernel.org>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2023-10-16 14:17:06 +01:00
..
pi arm64: unwind: add asynchronous unwind tables to kernel and modules 2022-11-09 18:06:35 +00:00
probes arm64: kprobes: Drop ID map text from kprobes blacklist 2023-02-07 18:25:10 +00:00
vdso arm64: vdso: remove two .altinstructions related symbols 2023-07-27 12:32:43 +01:00
vdso32 kbuild: Drop -Wdeclaration-after-statement 2023-06-26 11:14:19 +02:00
.gitignore
acpi_numa.c arm64: numa: Don't check node against MAX_NUMNODES 2022-07-19 19:10:28 +01:00
acpi_parking_protocol.c treewide: Drop function_nocfi 2022-09-26 10:13:14 -07:00
acpi.c arm64: acpi: Fix possible memory leak of ffh_ctxt 2023-02-24 14:21:49 +00:00
alternative.c Merge branches 'for-next/kpti', 'for-next/missing-proto-warn', 'for-next/iss2-decode', 'for-next/kselftest', 'for-next/misc', 'for-next/feat_mops', 'for-next/module-alloc', 'for-next/sysreg', 'for-next/cpucap', 'for-next/acpi', 'for-next/kdump', 'for-next/acpi-doc', 'for-next/doc' and 'for-next/tpidr2-fix', remote-tracking branch 'arm64/for-next/perf' into for-next/core 2023-06-23 18:32:20 +01:00
armv8_deprecated.c sysctl: Add size to register_sysctl 2023-08-15 15:26:17 -07:00
asm-offsets.c arm64: ftrace: fix build error with CONFIG_FUNCTION_GRAPH_TRACER=n 2023-07-05 09:46:19 -04:00
cacheinfo.c cacheinfo: Add arm64 early level initializer implementation 2023-04-13 09:32:33 +01:00
compat_alignment.c arm64: compat: Work around uninitialized variable warning 2023-04-05 17:51:47 +01:00
cpu_errata.c arm64: Fixup user features at boot time 2023-10-16 12:57:52 +01:00
cpu_ops.c
cpu-reset.S arm64: kernel: remove SHF_WRITE|SHF_EXECINSTR from .idmap.text 2023-05-02 12:42:22 +01:00
cpufeature.c arm64: Avoid cpus_have_const_cap() for ARM64_HAS_PAN 2023-10-16 14:17:04 +01:00
cpuidle.c arm64: Explicitly include correct DT includes 2023-07-31 14:45:38 +01:00
cpuinfo.c arm64: add HWCAP for FEAT_HBC (hinted conditional branches) 2023-08-04 17:32:13 +01:00
crash_core.c arm64: move PAC masks to <asm/pointer_auth.h> 2023-04-13 12:27:11 +01:00
crash_dump.c vmcore: convert copy_oldmem_page() to take an iov_iter 2022-04-29 14:37:59 -07:00
debug-monitors.c arm64: kgdb: Set PSTATE.SS to 1 to re-enable single-step 2023-04-14 13:39:47 +01:00
efi-header.S arm64: efi: Enable BTI codegen and add PE/COFF annotation 2023-04-20 15:43:45 +02:00
efi-rt-wrapper.S arm64: efi: Avoid workqueue to check whether EFI runtime is live 2023-01-16 15:27:31 +01:00
efi.c arm64: Avoid cpus_have_const_cap() for ARM64_HAS_BTI 2023-10-16 14:17:04 +01:00
elfcore.c arm64: mte: Avoid the racy walk of the vma list during core dump 2023-01-05 15:12:12 +00:00
entry-common.c arm64: syscall: unmask DAIF earlier for SVCs 2023-08-11 12:23:48 +01:00
entry-fpsimd.S arm64/sme: Implement context switching for ZT0 2023-01-20 12:23:06 +00:00
entry-ftrace.S arm64: ftrace: Enable HAVE_FUNCTION_GRAPH_RETVAL 2023-06-20 18:38:37 -04:00
entry.S arm64: sdei: abort running SDEI handlers during crash 2023-08-04 17:35:33 +01:00
fpsimd.c arm64: Avoid cpus_have_const_cap() for ARM64_{SVE,SME,SME2,FA64} 2023-10-16 14:17:06 +01:00
ftrace.c arm64: module: mandate MODULE_PLTS 2023-06-06 17:39:05 +01:00
head.S arm64/mm: Directly use ID_AA64MMFR2_EL1_VARange_MASK 2023-07-27 11:11:44 +01:00
hibernate-asm.S arm64: kexec: install a copy of the linear-map 2021-10-01 13:31:00 +01:00
hibernate.c arm64: hibernate: remove WARN_ON in save_processor_state 2023-06-21 13:33:49 +01:00
hw_breakpoint.c hw_breakpoint: fix single-stepping when using bpf_overflow_handler 2023-08-18 17:04:09 +01:00
hyp-stub.S ARM64: 2023-07-03 15:32:22 -07:00
idle.c arch/idle: Change arch_cpu_idle() behavior: always exit with IRQs disabled 2023-01-13 11:48:15 +01:00
idreg-override.c arm64/sysreg: Fix broken strncpy() -> strscpy() conversion 2023-09-06 19:22:54 +01:00
image-vars.h efi/zboot: arm64: Inject kernel code size symbol into the zboot payload 2023-04-26 18:01:41 +02:00
image.h
io.c
irq.c Merge branch 'for-next/stacks' into for-next/core 2022-12-06 11:26:40 +00:00
jump_label.c jump_label: make initial NOP patching the special case 2022-06-24 09:48:55 +02:00
kaslr.c Merge branch for-next/module-alloc into kvmarm/next 2023-06-15 13:04:15 +00:00
kexec_image.c arm64: Fix dangling references to Documentation/arm64 2023-06-21 08:53:31 -06:00
kgdb.c arm64: kgdb: Set PSTATE.SS to 1 to re-enable single-step 2023-04-14 13:39:47 +01:00
kuser32.S arm64: Update Documentation/arm references 2023-06-12 06:33:48 -06:00
machine_kexec_file.c arm64/kexec: Fix missing extra range for crashkres_low. 2022-09-01 11:50:00 +01:00
machine_kexec.c Merge branch 'for-next/misc' into for-next/core 2023-04-20 11:22:09 +01:00
Makefile - Arnd Bergmann has fixed a bunch of -Wmissing-prototypes in 2023-06-28 10:59:38 -07:00
module-plts.c arm64: module: Use module_init_layout_section() to spot init sections 2023-08-03 13:42:02 -07:00
module.c arm64: module: rework module VA range selection 2023-06-06 17:39:06 +01:00
mte.c arm64: mte: simplify swap tag restoration logic 2023-08-18 10:12:02 -07:00
paravirt.c arm64: paravirt: remove conduit check in has_pv_steal_clock 2022-11-09 18:11:56 +00:00
patch-scs.c arm64: Apply dynamic shadow call stack patching in two passes 2023-01-26 17:54:15 +00:00
patching.c arm64: patching: Add aarch64_insn_write_literal_u64() 2023-01-24 11:49:43 +00:00
pci.c arm64: Explicitly include correct DT includes 2023-07-31 14:45:38 +01:00
perf_callchain.c arm64: use XPACLRI to strip PAC 2023-04-13 12:27:11 +01:00
perf_regs.c perf: arm64: Add SVE vector granule register to user regs 2022-09-22 15:06:02 +01:00
pointer_auth.c
process.c arm64: Avoid cpus_have_const_cap() for ARM64_SSBS 2023-10-16 14:17:05 +01:00
proton-pack.c arm64: delete dead code in this_cpu_set_vectors() 2023-04-20 10:44:54 +01:00
psci.c treewide: Drop function_nocfi 2022-09-26 10:13:14 -07:00
ptrace.c arm64 updates for 6.6 2023-08-28 17:34:54 -07:00
reloc_test_core.c ARM64: reloc_test: add __init/__exit annotations to module init/exit funcs 2022-09-29 18:04:25 +01:00
reloc_test_syms.S
relocate_kernel.S arm64: kexec: load from kimage prior to clobbering 2022-05-17 14:25:35 +01:00
return_address.c arm64: Make return_address() use arch_stack_walk() 2021-12-10 14:06:04 +00:00
sdei.c arm64: sdei: abort running SDEI handlers during crash 2023-08-04 17:35:33 +01:00
setup.c arm64: kaslr: split kaslr/module initialization 2023-06-06 17:39:05 +01:00
signal32.c x86/shstk: Add user control-protection fault handler 2023-08-02 15:01:50 -07:00
signal.c Add x86 shadow stack support 2023-08-31 12:20:12 -07:00
sigreturn32.S arm64: compat: Move sigreturn32.S to .rodata section 2022-07-01 12:05:45 +01:00
sleep.S arm64: kernel: remove SHF_WRITE|SHF_EXECINSTR from .idmap.text 2023-05-02 12:42:22 +01:00
smccc-call.S
smp_spin_table.c treewide: Drop function_nocfi 2022-09-26 10:13:14 -07:00
smp.c arm64: Rework setup_cpu_features() 2023-10-16 12:57:51 +01:00
stacktrace.c arm64: use XPACLRI to strip PAC 2023-04-13 12:27:11 +01:00
suspend.c arm64: Avoid cpus_have_const_cap() for ARM64_HAS_DIT 2023-10-16 14:17:04 +01:00
sys32.c
sys_compat.c arm64: compat: Do not treat syscall number as ESR_ELx for a bad syscall 2022-04-29 19:26:27 +01:00
sys.c
syscall.c arm64: syscall: unmask DAIF earlier for SVCs 2023-08-11 12:23:48 +01:00
time.c arm64: Make profile_pc() use arch_stack_walk() 2021-12-10 14:06:04 +00:00
topology.c RISC-V Patches for the 6.1 Merge Window, Part 1 2022-10-09 13:24:01 -07:00
trace-events-emulation.h
traps.c - Yosry Ahmed brought back some cgroup v1 stats in OOM logs. 2023-06-28 10:28:11 -07:00
vdso32-wrap.S
vdso-wrap.S
vdso.c arm64: Avoid cpus_have_const_cap() for ARM64_HAS_BTI 2023-10-16 14:17:04 +01:00
vmlinux.lds.S arm64: lds: move .got section out of .text 2023-05-02 13:12:45 +01:00
watchdog_hld.c arm64: enable perf events based hard lockup detector 2023-06-09 17:44:22 -07:00