iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
linux/arch/arm64/kernel/entry-fpsimd.S

101 lines
1.9 KiB
ArmAsm
Raw Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 234 Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is distributed in the hope that it will be useful but without any warranty without even the implied warranty of merchantability or fitness for a particular purpose see the gnu general public license for more details you should have received a copy of the gnu general public license along with this program if not see http www gnu org licenses extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 503 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Alexios Zavras <alexios.zavras@intel.com> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Enrico Weigelt <info@metux.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190602204653.811534538@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-06-03 07:44:50 +02:00
/* SPDX-License-Identifier: GPL-2.0-only */
arm64: Floating point and SIMD This patch adds support for FP/ASIMD register bank saving and restoring during context switch and FP exception handling to generate SIGFPE. There are 32 128-bit registers and the context switching is currently done non-lazily. Benchmarks on real hardware are required before implementing lazy FP state saving/restoring. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
2012-03-05 11:49:32 +00:00
/*
* FP/SIMD state saving and restoring
*
* Copyright (C) 2012 ARM Ltd.
* Author: Catalin Marinas <catalin.marinas@arm.com>
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
arm64: move FP-SIMD save/restore code to a macro In order to be able to reuse the save-restore code in KVM, move it to a pair of macros, similar to what the 32bit code does. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-11-12 13:24:27 +00:00
#include <asm/fpsimdmacros.h>
arm64: Floating point and SIMD This patch adds support for FP/ASIMD register bank saving and restoring during context switch and FP exception handling to generate SIGFPE. There are 32 128-bit registers and the context switching is currently done non-lazily. Benchmarks on real hardware are required before implementing lazy FP state saving/restoring. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
2012-03-05 11:49:32 +00:00
/*
* Save the FP registers.
*
* x0 - pointer to struct fpsimd_state
*/
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_START(fpsimd_save_state)
arm64: move FP-SIMD save/restore code to a macro In order to be able to reuse the save-restore code in KVM, move it to a pair of macros, similar to what the 32bit code does. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-11-12 13:24:27 +00:00
fpsimd_save x0, 8
arm64: Floating point and SIMD This patch adds support for FP/ASIMD register bank saving and restoring during context switch and FP exception handling to generate SIGFPE. There are 32 128-bit registers and the context switching is currently done non-lazily. Benchmarks on real hardware are required before implementing lazy FP state saving/restoring. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
2012-03-05 11:49:32 +00:00
ret
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_END(fpsimd_save_state)
arm64: Floating point and SIMD This patch adds support for FP/ASIMD register bank saving and restoring during context switch and FP exception handling to generate SIGFPE. There are 32 128-bit registers and the context switching is currently done non-lazily. Benchmarks on real hardware are required before implementing lazy FP state saving/restoring. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
2012-03-05 11:49:32 +00:00
/*
* Load the FP registers.
*
* x0 - pointer to struct fpsimd_state
*/
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_START(fpsimd_load_state)
arm64: move FP-SIMD save/restore code to a macro In order to be able to reuse the save-restore code in KVM, move it to a pair of macros, similar to what the 32bit code does. Signed-off-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2012-11-12 13:24:27 +00:00
fpsimd_restore x0, 8
arm64: Floating point and SIMD This patch adds support for FP/ASIMD register bank saving and restoring during context switch and FP exception handling to generate SIGFPE. There are 32 128-bit registers and the context switching is currently done non-lazily. Benchmarks on real hardware are required before implementing lazy FP state saving/restoring. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Tony Lindgren <tony@atomide.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Nicolas Pitre <nico@linaro.org> Acked-by: Olof Johansson <olof@lixom.net> Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
2012-03-05 11:49:32 +00:00
ret
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_END(fpsimd_load_state)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
#ifdef CONFIG_ARM64_SVE
arm64/sve: Implement a helper to flush SVE registers Introduce a new helper that will zero all SVE registers but the first 128-bits of each vector. This will be used by subsequent patches to avoid costly store/maipulate/reload sequences in places like do_sve_acc(). Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-6-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:52 +01:00
arm64/sve: Add some comments for sve_save/load_state() The use of macros for the actual function bodies means legibility is always going to be a bit of a challenge, especially while we can't rely on SVE support in the toolchain, but this helps a little. Signed-off-by: Mark Brown <broonie@kernel.org> Link: https://lore.kernel.org/r/20210812201143.35578-1-broonie@kernel.org Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2021-08-12 21:11:43 +01:00
/*
* Save the SVE state
*
* x0 - pointer to buffer for state
* x1 - pointer to storage for FPSR
*/
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_START(sve_save_state)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
sve_save 0, x1, 2
ret
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_END(sve_save_state)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
arm64/sve: Add some comments for sve_save/load_state() The use of macros for the actual function bodies means legibility is always going to be a bit of a challenge, especially while we can't rely on SVE support in the toolchain, but this helps a little. Signed-off-by: Mark Brown <broonie@kernel.org> Link: https://lore.kernel.org/r/20210812201143.35578-1-broonie@kernel.org Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2021-08-12 21:11:43 +01:00
/*
* Load the SVE state
*
* x0 - pointer to buffer for state
* x1 - pointer to storage for FPSR
* x2 - VQ-1
*/
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_START(sve_load_state)
arm64/sve: Write ZCR_EL1 on context switch only if changed Writes to ZCR_EL1 are self-synchronising, and so may be expensive in typical implementations. This patch adopts the approach used for costly system register writes elsewhere in the kernel: the system register write is suppressed if it would not change the stored value. Since the common case will be that of switching between tasks that use the same vector length as one another, prediction hit rates on the conditional branch should be reasonably good, with lower expected amortised cost than the unconditional execution of a heavyweight self-synchronising instruction. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2018-05-14 18:51:09 +01:00
sve_load 0, x1, x2, 3, x4
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
ret
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_END(sve_load_state)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_START(sve_get_vl)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
_sve_rdvl 0, 1
ret
arm64: kernel: Convert to modern annotations for assembly functions In an effort to clarify and simplify the annotation of assembly functions in the kernel new macros have been introduced. These replace ENTRY and ENDPROC and also add a new annotation for static functions which previously had no ENTRY equivalent. Update the annotations in the core kernel code to the new macros. Signed-off-by: Mark Brown <broonie@kernel.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Link: https://lore.kernel.org/r/20200501115430.37315-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-01 12:54:29 +01:00
SYM_FUNC_END(sve_get_vl)
arm64/sve: Implement a helper to flush SVE registers Introduce a new helper that will zero all SVE registers but the first 128-bits of each vector. This will be used by subsequent patches to avoid costly store/maipulate/reload sequences in places like do_sve_acc(). Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-6-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:52 +01:00
arm64/sve: Rework SVE access trap to convert state in registers When we enable SVE usage in userspace after taking a SVE access trap we need to ensure that the portions of the register state that are not shared with the FPSIMD registers are zeroed. Currently we do this by forcing the FPSIMD registers to be saved to the task struct and converting them there. This is wasteful in the common case where the task state is loaded into the registers and we will immediately return to userspace since we can initialise the SVE state directly in registers instead of accessing multiple copies of the register state in memory. Instead in that common case do the conversion in the registers and update the task metadata so that we can return to userspace without spilling the register state to memory unless there is some other reason to do so. Signed-off-by: Mark Brown <broonie@kernel.org> Link: https://lore.kernel.org/r/20210312190313.24598-1-broonie@kernel.org Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2021-03-12 19:03:13 +00:00
SYM_FUNC_START(sve_set_vq)
sve_load_vq x0, x1, x2
ret
SYM_FUNC_END(sve_set_vq)
arm64/sve: Implement a helper to load SVE registers from FPSIMD state In a follow-up patch, we may save the FPSIMD rather than the full SVE state when the state has to be zeroed on return to userspace (e.g during a syscall). Introduce an helper to load SVE vectors from FPSIMD state and zero the rest of SVE registers. Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-7-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:53 +01:00
/*
* Load SVE state from FPSIMD state.
*
* x0 = pointer to struct fpsimd_state
* x1 = VQ - 1
*
* Each SVE vector will be loaded with the first 128-bits taken from FPSIMD
* and the rest zeroed. All the other SVE registers will be zeroed.
*/
SYM_FUNC_START(sve_load_from_fpsimd_state)
arm64/sve: Use the sve_flush macros in sve_load_from_fpsimd_state() This makes the code a bit clearer and as a result we can also make the indentation more normal, there is no change to the generated code. Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20210512151131.27877-3-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-12 16:11:30 +01:00
sve_load_vq x1, x2, x3
fpsimd_restore x0, 8
sve_flush_p_ffr
ret
arm64/sve: Implement a helper to load SVE registers from FPSIMD state In a follow-up patch, we may save the FPSIMD rather than the full SVE state when the state has to be zeroed on return to userspace (e.g during a syscall). Introduce an helper to load SVE vectors from FPSIMD state and zero the rest of SVE registers. Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-7-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:53 +01:00
SYM_FUNC_END(sve_load_from_fpsimd_state)
arm64/sve: Skip flushing Z registers with 128 bit vectors When the SVE vector length is 128 bits then there are no bits in the Z registers which are not shared with the V registers so we can skip them when zeroing state not shared with FPSIMD, this results in a minor performance improvement. Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20210512151131.27877-4-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-12 16:11:31 +01:00
/*
* Zero all SVE registers but the first 128-bits of each vector
*
* VQ must already be configured by caller, any further updates of VQ
* will need to ensure that the register state remains valid.
*
* x0 = VQ - 1
*/
arm64/sve: Implement a helper to flush SVE registers Introduce a new helper that will zero all SVE registers but the first 128-bits of each vector. This will be used by subsequent patches to avoid costly store/maipulate/reload sequences in places like do_sve_acc(). Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-6-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:52 +01:00
SYM_FUNC_START(sve_flush_live)
arm64/sve: Skip flushing Z registers with 128 bit vectors When the SVE vector length is 128 bits then there are no bits in the Z registers which are not shared with the V registers so we can skip them when zeroing state not shared with FPSIMD, this results in a minor performance improvement. Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20210512151131.27877-4-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-12 16:11:31 +01:00
cbz x0, 1f // A VQ-1 of 0 is 128 bits so no extra Z state
arm64/sve: Split _sve_flush macro into separate Z and predicate flushes Trivial refactoring to support further work, no change to generated code. Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20210512151131.27877-2-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-12 16:11:29 +01:00
sve_flush_z
arm64/sve: Skip flushing Z registers with 128 bit vectors When the SVE vector length is 128 bits then there are no bits in the Z registers which are not shared with the V registers so we can skip them when zeroing state not shared with FPSIMD, this results in a minor performance improvement. Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Link: https://lore.kernel.org/r/20210512151131.27877-4-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-12 16:11:31 +01:00
1: sve_flush_p_ffr
arm64/sve: Implement a helper to flush SVE registers Introduce a new helper that will zero all SVE registers but the first 128-bits of each vector. This will be used by subsequent patches to avoid costly store/maipulate/reload sequences in places like do_sve_acc(). Signed-off-by: Julien Grall <julien.grall@arm.com> Signed-off-by: Mark Brown <broonie@kernel.org> Reviewed-by: Dave Martin <Dave.Martin@arm.com> Link: https://lore.kernel.org/r/20200828181155.17745-6-broonie@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2020-08-28 19:11:52 +01:00
ret
SYM_FUNC_END(sve_flush_live)
arm64/sve: Low-level SVE architectural state manipulation functions Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin <Dave.Martin@arm.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2017-10-31 15:51:01 +00:00
#endif /* CONFIG_ARM64_SVE */
Reference in New Issue Copy Permalink