2019-06-03 07:44:50 +02:00
/* SPDX-License-Identifier: GPL-2.0-only */
2014-04-28 13:11:33 +08:00
/ *
2022-03-01 10:14:33 +00:00
* Copyright ( c ) 2 0 1 2 - 2 0 2 2 , A r m L i m i t e d .
2014-04-28 13:11:33 +08:00
*
2021-05-27 16:34:42 +01:00
* Adapted f r o m t h e o r i g i n a l a t :
2022-03-01 10:14:33 +00:00
* https : / / github. c o m / A R M - s o f t w a r e / o p t i m i z e d - r o u t i n e s / b l o b / 1 8 9 d f e f e 3 7 d54 c5 b / s t r i n g / a a r c h64 / s t r c m p . S
2014-04-28 13:11:33 +08:00
* /
# include < l i n u x / l i n k a g e . h >
# include < a s m / a s s e m b l e r . h >
2021-05-27 16:34:42 +01:00
/ * Assumptions :
2014-04-28 13:11:33 +08:00
*
2022-03-01 10:14:33 +00:00
* ARMv8 - a , A A r c h64 .
* MTE c o m p a t i b l e .
2014-04-28 13:11:33 +08:00
* /
2021-05-27 16:34:42 +01:00
# define L ( l a b e l ) . L ## l a b e l
2014-04-28 13:11:33 +08:00
# define R E P 8 _ 0 1 0 x01 0 1 0 1 0 1 0 1 0 1 0 1 0 1
# define R E P 8 _ 7 f 0 x7 f7 f7 f7 f7 f7 f7 f7 f
2021-05-27 16:34:42 +01:00
# define s r c1 x0
# define s r c2 x1
# define r e s u l t x0
2014-04-28 13:11:33 +08:00
2021-05-27 16:34:42 +01:00
# define d a t a1 x2
# define d a t a1 w w2
# define d a t a2 x3
# define d a t a2 w w3
# define h a s _ n u l x4
# define d i f f x5
2022-03-01 10:14:33 +00:00
# define o f f1 x5
2021-05-27 16:34:42 +01:00
# define s y n d r o m e x6
2022-03-01 10:14:33 +00:00
# define t m p x6
# define d a t a3 x7
# define z e r o o n e s x8
# define s h i f t x9
# define o f f2 x10
/ * On b i g - e n d i a n e a r l y b y t e s a r e a t M S B a n d o n l i t t l e - e n d i a n L S B .
LS_ F W m e a n s s h i f t i n g t o w a r d s e a r l y b y t e s . * /
# ifdef _ _ A A R C H 6 4 E B _ _
# define L S _ F W l s l
# else
# define L S _ F W l s r
# endif
/ * NUL d e t e c t i o n w o r k s o n t h e p r i n c i p l e t h a t ( X - 1 ) & ( ~ X ) & 0 x80
( = > ( X - 1 ) & ~ ( X | 0 x7 f ) ) i s n o n - z e r o i f f a b y t e i s z e r o , a n d
can b e d o n e i n p a r a l l e l a c r o s s t h e e n t i r e w o r d .
Since c a r r y p r o p a g a t i o n m a k e s 0 x1 b y t e s b e f o r e a N U L b y t e a p p e a r
NUL t o o i n b i g - e n d i a n , b y t e - r e v e r s e t h e d a t a b e f o r e t h e N U L c h e c k . * /
arm64: clean up symbol aliasing
Now that we have SYM_FUNC_ALIAS() and SYM_FUNC_ALIAS_WEAK(), use those
to simplify and more consistently define function aliases across
arch/arm64.
Aliases are now defined in terms of a canonical function name. For
position-independent functions I've made the __pi_<func> name the
canonical name, and defined other alises in terms of this.
The SYM_FUNC_{START,END}_PI(func) macros obscure the __pi_<func> name,
and make this hard to seatch for. The SYM_FUNC_START_WEAK_PI() macro
also obscures the fact that the __pi_<func> fymbol is global and the
<func> symbol is weak. For clarity, I have removed these macros and used
SYM_FUNC_{START,END}() directly with the __pi_<func> name.
For example:
SYM_FUNC_START_WEAK_PI(func)
... asm insns ...
SYM_FUNC_END_PI(func)
EXPORT_SYMBOL(func)
... becomes:
SYM_FUNC_START(__pi_func)
... asm insns ...
SYM_FUNC_END(__pi_func)
SYM_FUNC_ALIAS_WEAK(func, __pi_func)
EXPORT_SYMBOL(func)
For clarity, where there are multiple annotations such as
EXPORT_SYMBOL(), I've tried to keep annotations grouped by symbol. For
example, where a function has a name and an alias which are both
exported, this is organised as:
SYM_FUNC_START(func)
... asm insns ...
SYM_FUNC_END(func)
EXPORT_SYMBOL(func)
SYM_FUNC_ALIAS(alias, func)
EXPORT_SYMBOL(alias)
For consistency with the other string functions, I've defined strrchr as
a position-independent function, as it can safely be used as such even
though we have no users today.
As we no longer use SYM_FUNC_{START,END}_ALIAS(), our local copies are
removed. The common versions will be removed by a subsequent patch.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Acked-by: Mark Brown <broonie@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: Will Deacon <will@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220216162229.1076788-3-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
2022-02-16 16:22:27 +00:00
SYM_ F U N C _ S T A R T ( _ _ p i _ s t r c m p )
2022-03-01 10:14:33 +00:00
sub o f f2 , s r c2 , s r c1
mov z e r o o n e s , R E P 8 _ 0 1
and t m p , s r c1 , 7
tst o f f2 , 7
2021-05-27 16:34:42 +01:00
b. n e L ( m i s a l i g n e d8 )
2022-03-01 10:14:33 +00:00
cbnz t m p , L ( m u t u a l _ a l i g n )
.p2align 4
2021-05-27 16:34:42 +01:00
L( l o o p _ a l i g n e d ) :
2022-03-01 10:14:33 +00:00
ldr d a t a2 , [ s r c1 , o f f2 ]
ldr d a t a1 , [ s r c1 ] , 8
2021-05-27 16:34:42 +01:00
L( s t a r t _ r e a l i g n e d ) :
2022-03-01 10:14:33 +00:00
# ifdef _ _ A A R C H 6 4 E B _ _
rev t m p , d a t a1
sub h a s _ n u l , t m p , z e r o o n e s
orr t m p , t m p , R E P 8 _ 7 f
# else
sub h a s _ n u l , d a t a1 , z e r o o n e s
orr t m p , d a t a1 , R E P 8 _ 7 f
# endif
bics h a s _ n u l , h a s _ n u l , t m p / * N o n - z e r o i f N U L t e r m i n a t o r . * /
ccmp d a t a1 , d a t a2 , 0 , e q
b. e q L ( l o o p _ a l i g n e d )
# ifdef _ _ A A R C H 6 4 E B _ _
rev h a s _ n u l , h a s _ n u l
# endif
eor d i f f , d a t a1 , d a t a2
2014-04-28 13:11:33 +08:00
orr s y n d r o m e , d i f f , h a s _ n u l
2021-05-27 16:34:42 +01:00
L( e n d ) :
2022-03-01 10:14:33 +00:00
# ifndef _ _ A A R C H 6 4 E B _ _
2021-05-27 16:34:42 +01:00
rev s y n d r o m e , s y n d r o m e
rev d a t a1 , d a t a1
rev d a t a2 , d a t a2
2022-03-01 10:14:33 +00:00
# endif
clz s h i f t , s y n d r o m e
/ * The m o s t - s i g n i f i c a n t - n o n - z e r o b i t o f t h e s y n d r o m e m a r k s e i t h e r t h e
first b i t t h a t i s d i f f e r e n t , o r t h e t o p b i t o f t h e f i r s t z e r o b y t e .
2021-05-27 16:34:42 +01:00
Shifting l e f t n o w w i l l b r i n g t h e c r i t i c a l i n f o r m a t i o n i n t o t h e
top b i t s . * /
2022-03-01 10:14:33 +00:00
lsl d a t a1 , d a t a1 , s h i f t
lsl d a t a2 , d a t a2 , s h i f t
2021-05-27 16:34:42 +01:00
/ * But w e n e e d t o z e r o - e x t e n d ( c h a r i s u n s i g n e d ) t h e v a l u e a n d t h e n
perform a s i g n e d 3 2 - b i t s u b t r a c t i o n . * /
2022-03-01 10:14:33 +00:00
lsr d a t a1 , d a t a1 , 5 6
sub r e s u l t , d a t a1 , d a t a2 , l s r 5 6
2021-05-27 16:34:42 +01:00
ret
2022-03-01 10:14:33 +00:00
.p2align 4
2014-04-28 13:11:33 +08:00
2021-05-27 16:34:42 +01:00
L( m u t u a l _ a l i g n ) :
/ * Sources a r e m u t u a l l y a l i g n e d , b u t a r e n o t c u r r e n t l y a t a n
alignment b o u n d a r y . R o u n d d o w n t h e a d d r e s s e s a n d t h e n m a s k o f f
2022-03-01 10:14:33 +00:00
the b y t e s t h a t p r e c e d e t h e s t a r t p o i n t . * /
bic s r c1 , s r c1 , 7
ldr d a t a2 , [ s r c1 , o f f2 ]
ldr d a t a1 , [ s r c1 ] , 8
neg s h i f t , s r c2 , l s l 3 / * B i t s t o a l i g n m e n t - 6 4 . * /
mov t m p , - 1
LS_ F W t m p , t m p , s h i f t
orr d a t a1 , d a t a1 , t m p
orr d a t a2 , d a t a2 , t m p
2021-05-27 16:34:42 +01:00
b L ( s t a r t _ r e a l i g n e d )
L( m i s a l i g n e d8 ) :
/ * Align S R C 1 t o 8 b y t e s a n d t h e n c o m p a r e 8 b y t e s a t a t i m e , a l w a y s
2022-03-01 10:14:33 +00:00
checking t o m a k e s u r e t h a t w e d o n ' t a c c e s s b e y o n d t h e e n d o f S R C 2 . * /
cbz t m p , L ( s r c1 _ a l i g n e d )
2021-05-27 16:34:42 +01:00
L( d o _ m i s a l i g n e d ) :
2022-03-01 10:14:33 +00:00
ldrb d a t a1 w , [ s r c1 ] , 1
ldrb d a t a2 w , [ s r c2 ] , 1
cmp d a t a1 w , 0
ccmp d a t a1 w , d a t a2 w , 0 , n e / * N Z C V = 0 b00 0 0 . * /
2021-05-27 16:34:42 +01:00
b. n e L ( d o n e )
2022-03-01 10:14:33 +00:00
tst s r c1 , 7
2021-05-27 16:34:42 +01:00
b. n e L ( d o _ m i s a l i g n e d )
2022-03-01 10:14:33 +00:00
L( s r c1 _ a l i g n e d ) :
neg s h i f t , s r c2 , l s l 3
bic s r c2 , s r c2 , 7
ldr d a t a3 , [ s r c2 ] , 8
# ifdef _ _ A A R C H 6 4 E B _ _
rev d a t a3 , d a t a3
# endif
lsr t m p , z e r o o n e s , s h i f t
orr d a t a3 , d a t a3 , t m p
sub h a s _ n u l , d a t a3 , z e r o o n e s
orr t m p , d a t a3 , R E P 8 _ 7 f
bics h a s _ n u l , h a s _ n u l , t m p
b. n e L ( t a i l )
sub o f f1 , s r c2 , s r c1
.p2align 4
L( l o o p _ u n a l i g n e d ) :
ldr d a t a3 , [ s r c1 , o f f1 ]
ldr d a t a2 , [ s r c1 , o f f2 ]
# ifdef _ _ A A R C H 6 4 E B _ _
rev d a t a3 , d a t a3
# endif
sub h a s _ n u l , d a t a3 , z e r o o n e s
orr t m p , d a t a3 , R E P 8 _ 7 f
ldr d a t a1 , [ s r c1 ] , 8
bics h a s _ n u l , h a s _ n u l , t m p
ccmp d a t a1 , d a t a2 , 0 , e q
b. e q L ( l o o p _ u n a l i g n e d )
lsl t m p , h a s _ n u l , s h i f t
# ifdef _ _ A A R C H 6 4 E B _ _
rev t m p , t m p
# endif
eor d i f f , d a t a1 , d a t a2
orr s y n d r o m e , d i f f , t m p
cbnz s y n d r o m e , L ( e n d )
L( t a i l ) :
ldr d a t a1 , [ s r c1 ]
neg s h i f t , s h i f t
lsr d a t a2 , d a t a3 , s h i f t
lsr h a s _ n u l , h a s _ n u l , s h i f t
# ifdef _ _ A A R C H 6 4 E B _ _
rev d a t a2 , d a t a2
rev h a s _ n u l , h a s _ n u l
# endif
eor d i f f , d a t a1 , d a t a2
2014-04-28 13:11:33 +08:00
orr s y n d r o m e , d i f f , h a s _ n u l
2021-05-27 16:34:42 +01:00
b L ( e n d )
2014-04-28 13:11:33 +08:00
2021-05-27 16:34:42 +01:00
L( d o n e ) :
sub r e s u l t , d a t a1 , d a t a2
2014-04-28 13:11:33 +08:00
ret
arm64: clean up symbol aliasing
Now that we have SYM_FUNC_ALIAS() and SYM_FUNC_ALIAS_WEAK(), use those
to simplify and more consistently define function aliases across
arch/arm64.
Aliases are now defined in terms of a canonical function name. For
position-independent functions I've made the __pi_<func> name the
canonical name, and defined other alises in terms of this.
The SYM_FUNC_{START,END}_PI(func) macros obscure the __pi_<func> name,
and make this hard to seatch for. The SYM_FUNC_START_WEAK_PI() macro
also obscures the fact that the __pi_<func> fymbol is global and the
<func> symbol is weak. For clarity, I have removed these macros and used
SYM_FUNC_{START,END}() directly with the __pi_<func> name.
For example:
SYM_FUNC_START_WEAK_PI(func)
... asm insns ...
SYM_FUNC_END_PI(func)
EXPORT_SYMBOL(func)
... becomes:
SYM_FUNC_START(__pi_func)
... asm insns ...
SYM_FUNC_END(__pi_func)
SYM_FUNC_ALIAS_WEAK(func, __pi_func)
EXPORT_SYMBOL(func)
For clarity, where there are multiple annotations such as
EXPORT_SYMBOL(), I've tried to keep annotations grouped by symbol. For
example, where a function has a name and an alias which are both
exported, this is organised as:
SYM_FUNC_START(func)
... asm insns ...
SYM_FUNC_END(func)
EXPORT_SYMBOL(func)
SYM_FUNC_ALIAS(alias, func)
EXPORT_SYMBOL(alias)
For consistency with the other string functions, I've defined strrchr as
a position-independent function, as it can safely be used as such even
though we have no users today.
As we no longer use SYM_FUNC_{START,END}_ALIAS(), our local copies are
removed. The common versions will be removed by a subsequent patch.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Josh Poimboeuf <jpoimboe@redhat.com>
Acked-by: Mark Brown <broonie@kernel.org>
Cc: Joey Gouly <joey.gouly@arm.com>
Cc: Will Deacon <will@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220216162229.1076788-3-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
2022-02-16 16:22:27 +00:00
SYM_ F U N C _ E N D ( _ _ p i _ s t r c m p )
SYM_ F U N C _ A L I A S _ W E A K ( s t r c m p , _ _ p i _ s t r c m p )
2022-03-01 10:14:35 +00:00
EXPORT_ S Y M B O L _ N O K A S A N ( s t r c m p )
