2019-05-27 09:55:01 +03:00
/* SPDX-License-Identifier: GPL-2.0-or-later */
2005-10-10 16:36:14 +04:00
/ *
* PowerPC v e r s i o n
* Copyright ( C ) 1 9 9 5 - 1 9 9 6 G a r y T h o m a s ( g d t @linuxppc.org)
* Rewritten b y C o r t D o u g a n ( c o r t @fsmlabs.com) for PReP
* Copyright ( C ) 1 9 9 6 C o r t D o u g a n < c o r t @fsmlabs.com>
* Adapted f o r P o w e r M a c i n t o s h b y P a u l M a c k e r r a s .
* Low- l e v e l e x c e p t i o n h a n d l e r s a n d M M U s u p p o r t
* rewritten b y P a u l M a c k e r r a s .
* Copyright ( C ) 1 9 9 6 P a u l M a c k e r r a s .
* MPC8 x x m o d i f i c a t i o n s C o p y r i g h t ( C ) 1 9 9 7 D a n M a l e k ( d m a l e k @jlc.net).
*
* This f i l e c o n t a i n s t h e s y s t e m c a l l e n t r y c o d e , c o n t e x t s w i t c h
* code, a n d e x c e p t i o n / i n t e r r u p t r e t u r n c o d e f o r P o w e r P C .
* /
# include < l i n u x / e r r n o . h >
powerpc/kernel: Switch to using MAX_ERRNO
Currently on powerpc we have our own #define for the highest (negative)
errno value, called _LAST_ERRNO. This is defined to be 516, for reasons
which are not clear.
The generic code, and x86, use MAX_ERRNO, which is defined to be 4095.
In particular seccomp uses MAX_ERRNO to restrict the value that a
seccomp filter can return.
Currently with the mismatch between _LAST_ERRNO and MAX_ERRNO, a seccomp
tracer wanting to return 600, expecting it to be seen as an error, would
instead find on powerpc that userspace sees a successful syscall with a
return value of 600.
To avoid this inconsistency, switch powerpc to use MAX_ERRNO.
We are somewhat confident that generic syscalls that can return a
non-error value above negative MAX_ERRNO have already been updated to
use force_successful_syscall_return().
I have also checked all the powerpc specific syscalls, and believe that
none of them expect to return a non-error value between -MAX_ERRNO and
-516. So this change should be safe ...
Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Reviewed-by: Kees Cook <keescook@chromium.org>
2015-07-23 13:21:01 +03:00
# include < l i n u x / e r r . h >
2005-10-10 16:36:14 +04:00
# include < l i n u x / s y s . h >
# include < l i n u x / t h r e a d s . h >
# include < a s m / r e g . h >
# include < a s m / p a g e . h >
# include < a s m / m m u . h >
# include < a s m / c p u t a b l e . h >
# include < a s m / t h r e a d _ i n f o . h >
# include < a s m / p p c _ a s m . h >
# include < a s m / a s m - o f f s e t s . h >
# include < a s m / u n i s t d . h >
2010-11-18 18:06:17 +03:00
# include < a s m / p t r a c e . h >
2016-01-14 07:33:46 +03:00
# include < a s m / e x p o r t . h >
2018-07-05 19:25:01 +03:00
# include < a s m / f e a t u r e - f i x u p s . h >
2018-07-28 02:06:38 +03:00
# include < a s m / b a r r i e r . h >
2019-03-11 11:30:31 +03:00
# include < a s m / k u p . h >
2019-04-30 15:39:01 +03:00
# include < a s m / b u g . h >
2021-06-04 17:54:14 +03:00
# include < a s m / i n t e r r u p t . h >
2005-10-10 16:36:14 +04:00
2019-04-30 15:38:51 +03:00
# include " h e a d _ 3 2 . h "
2005-10-10 16:36:14 +04:00
2020-07-16 04:35:22 +03:00
/ *
* powerpc r e l i e s o n r e t u r n f r o m i n t e r r u p t / s y s c a l l b e i n g c o n t e x t s y n c h r o n i s i n g
* ( which r f i i s ) t o s u p p o r t A R C H _ H A S _ M E M B A R R I E R _ S Y N C _ C O R E w i t h o u t a d d i t i o n a l
* synchronisation i n s t r u c t i o n s .
* /
2017-07-12 13:08:49 +03:00
/ *
* Align t o 4 k i n o r d e r t o e n s u r e t h a t a l l f u n c t i o n s m o d y f i n g s r r0 / s r r1
* fit i n t o o n e p a g e i n o r d e r t o n o t e n c o u n t e r a T L B m i s s b e t w e e n t h e
* modification o f s r r0 / s r r1 a n d t h e a s s o c i a t e d r f i .
* /
.align 12
2021-03-12 15:50:46 +03:00
# if d e f i n e d ( C O N F I G _ P P C _ B O O K 3 S _ 3 2 ) | | d e f i n e d ( C O N F I G _ E 5 0 0 )
2021-03-12 15:50:39 +03:00
.globl prepare_transfer_to_handler
prepare_transfer_to_handler :
2021-03-12 15:50:21 +03:00
/* if from kernel, check interrupted DOZE/NAP mode */
2019-01-31 13:09:04 +03:00
lwz r12 ,T I _ L O C A L _ F L A G S ( r2 )
2006-04-18 15:49:11 +04:00
mtcrf 0 x01 ,r12
bt- 3 1 - T L F _ N A P P I N G ,4 f
2008-05-14 08:30:48 +04:00
bt- 3 1 - T L F _ S L E E P I N G ,7 f
2021-03-12 15:50:32 +03:00
blr
2019-04-30 15:39:01 +03:00
2020-03-31 19:03:45 +03:00
4 : rlwinm r12 ,r12 ,0 ,~ _ T L F _ N A P P I N G
stw r12 ,T I _ L O C A L _ F L A G S ( r2 )
b p o w e r _ s a v e _ p p c32 _ r e s t o r e
7 : rlwinm r12 ,r12 ,0 ,~ _ T L F _ S L E E P I N G
stw r12 ,T I _ L O C A L _ F L A G S ( r2 )
lwz r9 ,_ M S R ( r11 ) / * i f s l e e p i n g , c l e a r M S R . E E * /
rlwinm r9 ,r9 ,0 ,~ M S R _ E E
lwz r12 ,_ L I N K ( r11 ) / * a n d r e t u r n t o a d d r e s s i n L R * /
lwz r2 , G P R 2 ( r11 )
b f a s t _ e x c e p t i o n _ r e t u r n
2021-03-12 15:50:39 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( p r e p a r e _ t r a n s f e r _ t o _ h a n d l e r )
2021-03-12 15:50:46 +03:00
# endif / * C O N F I G _ P P C _ B O O K 3 S _ 3 2 | | C O N F I G _ E 5 0 0 * /
2020-03-31 19:03:45 +03:00
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
# if d e f i n e d ( C O N F I G _ P P C _ K U E P ) & & d e f i n e d ( C O N F I G _ P P C _ B O O K 3 S _ 3 2 )
.globl __kuep_lock
__kuep_lock :
2021-10-19 10:29:18 +03:00
lwz r9 , T H R E A D + T H S R 0 ( r2 )
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
update_ u s e r _ s e g m e n t s _ b y _ 4 r9 , r10 , r11 , r12
blr
__kuep_unlock :
2021-10-19 10:29:18 +03:00
lwz r9 , T H R E A D + T H S R 0 ( r2 )
rlwinm r9 ,r9 ,0 ,~ S R _ N X
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
update_ u s e r _ s e g m e n t s _ b y _ 4 r9 , r10 , r11 , r12
blr
.macro kuep_lock
bl _ _ k u e p _ l o c k
.endm
.macro kuep_unlock
bl _ _ k u e p _ u n l o c k
.endm
# else
.macro kuep_lock
.endm
.macro kuep_unlock
.endm
# endif
2019-04-30 15:39:02 +03:00
.globl transfer_to_syscall
transfer_to_syscall :
2021-06-04 17:54:14 +03:00
stw r11 , G P R 1 ( r1 )
stw r11 , 0 ( r1 )
mflr r12
stw r12 , _ L I N K ( r1 )
2021-10-19 10:29:25 +03:00
# ifdef C O N F I G _ B O O K E _ O R _ 4 0 x
2021-06-04 17:54:14 +03:00
rlwinm r9 ,r9 ,0 ,1 4 ,1 2 / * c l e a r M S R _ W E ( n e c e s s a r y ? ) * /
# endif
lis r12 ,S T A C K _ F R A M E _ R E G S _ M A R K E R @ha /* exception frame marker */
SAVE_ G P R ( 2 , r1 )
addi r12 ,r12 ,S T A C K _ F R A M E _ R E G S _ M A R K E R @l
stw r9 ,_ M S R ( r1 )
li r2 , I N T E R R U P T _ S Y S C A L L
stw r12 ,8 ( r1 )
stw r2 ,_ T R A P ( r1 )
SAVE_ G P R ( 0 , r1 )
2021-10-22 09:13:22 +03:00
SAVE_ G P R S ( 3 , 8 , r1 )
2021-06-04 17:54:14 +03:00
addi r2 ,r10 ,- T H R E A D
2021-02-08 18:10:32 +03:00
SAVE_ N V G P R S ( r1 )
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
kuep_ l o c k
2019-04-30 15:39:02 +03:00
2021-02-08 18:10:33 +03:00
/* Calling convention has r9 = orig r0, r10 = regs */
addi r10 ,r1 ,S T A C K _ F R A M E _ O V E R H E A D
mr r9 ,r0
bl s y s t e m _ c a l l _ e x c e p t i o n
2018-07-28 02:06:38 +03:00
2005-10-10 16:36:14 +04:00
ret_from_syscall :
2021-02-08 18:10:33 +03:00
addi r4 ,r1 ,S T A C K _ F R A M E _ O V E R H E A D
li r5 ,0
bl s y s c a l l _ e x i t _ p r e p a r e
2020-10-18 20:25:18 +03:00
# ifdef C O N F I G _ P P C _ 4 7 x
2007-10-31 08:42:19 +03:00
lis r4 ,i c a c h e _ 4 4 x _ n e e d _ f l u s h @ha
lwz r5 ,i c a c h e _ 4 4 x _ n e e d _ f l u s h @l(r4)
cmplwi c r0 ,r5 ,0
bne- 2 f
2020-10-18 20:25:18 +03:00
# endif / * C O N F I G _ P P C _ 4 7 x * /
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
kuep_ u n l o c k
2005-10-10 16:36:14 +04:00
lwz r4 ,_ L I N K ( r1 )
lwz r5 ,_ C C R ( r1 )
mtlr r4
lwz r7 ,_ N I P ( r1 )
2021-02-08 18:10:33 +03:00
lwz r8 ,_ M S R ( r1 )
cmpwi r3 ,0
lwz r3 ,G P R 3 ( r1 )
2020-03-31 19:03:46 +03:00
syscall_exit_finish :
2005-10-10 16:36:14 +04:00
mtspr S P R N _ S R R 0 ,r7
mtspr S P R N _ S R R 1 ,r8
2021-02-08 18:10:33 +03:00
bne 3 f
mtcr r5
1 : lwz r2 ,G P R 2 ( r1 )
lwz r1 ,G P R 1 ( r1 )
2020-11-08 19:57:37 +03:00
rfi
# ifdef C O N F I G _ 4 0 x
b . / * P r e v e n t p r e f e t c h p a s t r f i * /
# endif
2021-02-08 18:10:33 +03:00
3 : mtcr r5
lwz r4 ,_ C T R ( r1 )
lwz r5 ,_ X E R ( r1 )
REST_ N V G P R S ( r1 )
mtctr r4
mtxer r5
lwz r0 ,G P R 0 ( r1 )
lwz r3 ,G P R 3 ( r1 )
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 4 , 1 1 , r1 )
2021-02-08 18:10:33 +03:00
lwz r12 ,G P R 1 2 ( r1 )
b 1 b
2007-10-31 08:42:19 +03:00
# ifdef C O N F I G _ 4 4 x
2 : li r7 ,0
iccci r0 ,r0
stw r7 ,i c a c h e _ 4 4 x _ n e e d _ f l u s h @l(r4)
b 1 b
# endif / * C O N F I G _ 4 4 x * /
2005-10-10 16:36:14 +04:00
.globl ret_from_fork
ret_from_fork :
REST_ N V G P R S ( r1 )
bl s c h e d u l e _ t a i l
li r3 ,0
b r e t _ f r o m _ s y s c a l l
2012-09-13 02:32:42 +04:00
.globl ret_from_kernel_thread
ret_from_kernel_thread :
REST_ N V G P R S ( r1 )
bl s c h e d u l e _ t a i l
2021-08-20 08:16:05 +03:00
mtctr r14
2012-09-13 02:32:42 +04:00
mr r3 ,r15
PPC4 4 0 E P _ E R R 4 2
2021-08-20 08:16:05 +03:00
bctrl
2012-09-13 02:32:42 +04:00
li r3 ,0
2012-08-31 23:48:05 +04:00
b r e t _ f r o m _ s y s c a l l
2005-10-10 16:36:14 +04:00
/ *
* This r o u t i n e s w i t c h e s b e t w e e n t w o d i f f e r e n t t a s k s . T h e p r o c e s s
* state o f o n e i s s a v e d o n i t s k e r n e l s t a c k . T h e n t h e s t a t e
* of t h e o t h e r i s r e s t o r e d f r o m i t s k e r n e l s t a c k . T h e m e m o r y
* management h a r d w a r e i s u p d a t e d t o t h e s e c o n d p r o c e s s ' s s t a t e .
* Finally, w e c a n r e t u r n t o t h e s e c o n d p r o c e s s .
* On e n t r y , r3 p o i n t s t o t h e T H R E A D f o r t h e c u r r e n t t a s k , r4
* points t o t h e T H R E A D f o r t h e n e w t a s k .
*
* This r o u t i n e i s a l w a y s c a l l e d w i t h i n t e r r u p t s d i s a b l e d .
*
* Note : there a r e t w o w a y s t o g e t t o t h e " g o i n g o u t " p o r t i o n
* of t h i s c o d e ; either by coming in via the entry (_switch)
* or v i a " f o r k " w h i c h m u s t s e t u p a n e n v i r o n m e n t e q u i v a l e n t
* to t h e " _ s w i t c h " p a t h . I f y o u c h a n g e t h i s , y o u ' l l h a v e t o
* change t h e f o r k c o d e a l s o .
*
* The c o d e w h i c h c r e a t e s t h e n e w t a s k c o n t e x t i s i n ' c o p y _ t h r e a d '
* in a r c h / p p c / k e r n e l / p r o c e s s . c
* /
_ GLOBAL( _ s w i t c h )
stwu r1 ,- I N T _ F R A M E _ S I Z E ( r1 )
mflr r0
stw r0 ,I N T _ F R A M E _ S I Z E + 4 ( r1 )
/* r3-r12 are caller saved -- Cort */
SAVE_ N V G P R S ( r1 )
stw r0 ,_ N I P ( r1 ) / * R e t u r n t o s w i t c h c a l l e r * /
mfcr r10
stw r10 ,_ C C R ( r1 )
stw r1 ,K S P ( r3 ) / * S e t o l d s t a c k p o i n t e r * /
# ifdef C O N F I G _ S M P
/ * We n e e d a s y n c s o m e w h e r e h e r e t o m a k e s u r e t h a t i f t h e
* previous t a s k g e t s r e s c h e d u l e d o n a n o t h e r C P U , i t s e e s a l l
* stores i t h a s p e r f o r m e d o n t h i s o n e .
* /
sync
# endif / * C O N F I G _ S M P * /
tophys( r0 ,r4 )
2009-07-15 00:52:54 +04:00
mtspr S P R N _ S P R G _ T H R E A D ,r0 / * U p d a t e c u r r e n t T H R E A D p h y s a d d r * /
2005-10-10 16:36:14 +04:00
lwz r1 ,K S P ( r4 ) / * L o a d n e w s t a c k p o i n t e r * /
/* save the old current 'last' for return value */
mr r3 ,r2
addi r2 ,r4 ,- T H R E A D / * U p d a t e c u r r e n t * /
lwz r0 ,_ C C R ( r1 )
mtcrf 0 x F F ,r0
/* r3-r12 are destroyed -- Cort */
REST_ N V G P R S ( r1 )
lwz r4 ,_ N I P ( r1 ) / * R e t u r n t o _ s w i t c h c a l l e r i n n e w t a s k * /
mtlr r4
addi r1 ,r1 ,I N T _ F R A M E _ S I Z E
blr
.globl fast_exception_return
fast_exception_return :
# if ! ( d e f i n e d ( C O N F I G _ 4 x x ) | | d e f i n e d ( C O N F I G _ B O O K E ) )
andi. r10 ,r9 ,M S R _ R I / * c h e c k f o r r e c o v e r a b l e i n t e r r u p t * /
2021-03-12 15:50:34 +03:00
beq 3 f / * i f n o t , w e ' v e g o t p r o b l e m s * /
2005-10-10 16:36:14 +04:00
# endif
2021-10-22 09:13:22 +03:00
2 : REST_ G P R S ( 3 , 6 , r11 )
2005-10-10 16:36:14 +04:00
lwz r10 ,_ C C R ( r11 )
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 1 , 2 , r11 )
2005-10-10 16:36:14 +04:00
mtcr r10
lwz r10 ,_ L I N K ( r11 )
mtlr r10
2019-02-27 14:45:30 +03:00
/* Clear the exception_marker on the stack to avoid confusing stacktrace */
li r10 , 0
stw r10 , 8 ( r11 )
2005-10-10 16:36:14 +04:00
REST_ G P R ( 1 0 , r11 )
2018-01-12 15:45:23 +03:00
# if d e f i n e d ( C O N F I G _ P P C _ 8 x x ) & & d e f i n e d ( C O N F I G _ P E R F _ E V E N T S )
powerpc/8xx: Perf events on PPC 8xx
This patch has been reworked since RFC version. In the RFC, this patch
was preceded by a patch clearing MSR RI for all PPC32 at all time at
exception prologs. Now MSR RI clearing is done only when this 8xx perf
events functionality is compiled in, it is therefore limited to 8xx
and merged inside this patch.
Other main changes have been to take into account detailed review from
Peter Zijlstra. The instructions counter has been reworked to behave
as a free running counter like the three other counters.
The 8xx has no PMU, however some events can be emulated by other means.
This patch implements the following events (as reported by 'perf list'):
cpu-cycles OR cycles [Hardware event]
instructions [Hardware event]
dTLB-load-misses [Hardware cache event]
iTLB-load-misses [Hardware cache event]
'cycles' event is implemented using the timebase clock. Timebase clock
corresponds to CPU clock divided by 16, so number of cycles is
approximatly 16 times the number of TB ticks
On the 8xx, TLB misses are handled by software. It is therefore
easy to count all TLB misses each time the TLB miss exception is
called.
'instructions' is calculated by using instruction watchpoint counter.
This patch sets counter A to count instructions at address greater
than 0, hence we count all instructions executed while MSR RI bit is
set. The counter is set to the maximum which is 0xffff. Every 65535
instructions, debug instruction breakpoint exception fires. The
exception handler increments a counter in memory which then
represent the upper part of the instruction counter. We therefore
end up with a 48 bits counter. In order to avoid unnecessary overhead
while no perf event is active, this counter is started when the first
event referring to this counter is added, and the counter is stopped
when the last event referring to it is deleted. In order to properly
support breakpoint exceptions, MSR RI bit has to be unset in exception
epilogs in order to avoid breakpoint exceptions during critical
sections during changes to SRR0 and SRR1 would be problematic.
All counters are handled as free running counters.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Scott Wood <oss@buserror.net>
2016-12-15 15:42:18 +03:00
mtspr S P R N _ N R I , r0
# endif
2005-10-10 16:36:14 +04:00
mtspr S P R N _ S R R 1 ,r9
mtspr S P R N _ S R R 0 ,r12
REST_ G P R ( 9 , r11 )
REST_ G P R ( 1 2 , r11 )
lwz r11 ,G P R 1 1 ( r11 )
2020-11-08 19:57:37 +03:00
rfi
# ifdef C O N F I G _ 4 0 x
b . / * P r e v e n t p r e f e t c h p a s t r f i * /
# endif
2020-03-31 19:03:47 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( f a s t _ e x c e p t i o n _ r e t u r n )
2005-10-10 16:36:14 +04:00
/* aargh, a nonrecoverable interrupt, panic */
/* aargh, we don't know which trap this is */
3 :
li r10 ,- 1
2005-10-28 16:45:25 +04:00
stw r10 ,_ T R A P ( r11 )
2021-03-12 15:50:41 +03:00
prepare_ t r a n s f e r _ t o _ h a n d l e r
2021-03-12 15:50:32 +03:00
bl u n r e c o v e r a b l e _ e x c e p t i o n
2021-03-12 15:50:34 +03:00
trap / * s h o u l d n o t g e t h e r e * /
2005-10-10 16:36:14 +04:00
2021-03-12 15:50:34 +03:00
.globl interrupt_return
interrupt_return :
lwz r4 ,_ M S R ( r1 )
addi r3 ,r1 ,S T A C K _ F R A M E _ O V E R H E A D
andi. r0 ,r4 ,M S R _ P R
beq . L k e r n e l _ i n t e r r u p t _ r e t u r n
bl i n t e r r u p t _ e x i t _ u s e r _ p r e p a r e
cmpwi r3 ,0
powerpc/32s: Do kuep_lock() and kuep_unlock() in assembly
When interrupt and syscall entries where converted to C, KUEP locking
and unlocking was also converted. It improved performance by unrolling
the loop, and allowed easily implementing boot time deactivation of
KUEP.
However, null_syscall selftest shows that KUEP is still heavy
(361 cycles with KUEP, 212 cycles without).
A way to improve more is to group 'mtsr's together, instead of
repeating 'addi' + 'mtsr' several times.
In order to do that, more registers need to be available. In C, GCC
will always be able to provide the requested number of registers, but
at the cost of saving some data on the stack, which is counter
performant here.
So let's do it in assembly, when we have full control of which
register can be used. It also has the advantage of locking earlier
and unlocking later and it helps GCC generating less tricky code.
The only drawback is to make boot time deactivation less straight
forward and require 'hand' instruction patching.
Group 'mtsr's by 4.
With this change, null_syscall selftest reports 336 cycles. Without
the change it was 361 cycles, that's a 7% reduction.
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Link: https://lore.kernel.org/r/115cb279e9b9948dfd93a065e047081c59e3a2a6.1634627931.git.christophe.leroy@csgroup.eu
2021-10-19 10:29:17 +03:00
kuep_ u n l o c k
2021-03-12 15:50:34 +03:00
bne- . L r e s t o r e _ n v g p r s
.Lfast_user_interrupt_return :
lwz r11 ,_ N I P ( r1 )
lwz r12 ,_ M S R ( r1 )
mtspr S P R N _ S R R 0 ,r11
mtspr S P R N _ S R R 1 ,r12
2005-10-10 16:36:14 +04:00
2021-03-12 15:50:34 +03:00
BEGIN_ F T R _ S E C T I O N
stwcx. r0 ,0 ,r1 / * t o c l e a r t h e r e s e r v a t i o n * /
FTR_ S E C T I O N _ E L S E
lwarx r0 ,0 ,r1
ALT_ F T R _ S E C T I O N _ E N D _ I F C L R ( C P U _ F T R _ S T C X _ C H E C K S _ A D D R E S S )
2012-09-17 03:54:30 +04:00
2021-03-12 15:50:34 +03:00
lwz r3 ,_ C C R ( r1 )
lwz r4 ,_ L I N K ( r1 )
lwz r5 ,_ C T R ( r1 )
lwz r6 ,_ X E R ( r1 )
li r0 ,0
2012-09-17 03:54:30 +04:00
2021-03-12 15:50:34 +03:00
/ *
* Leaving a s t a l e e x c e p t i o n _ m a r k e r o n t h e s t a c k c a n c o n f u s e
* the r e l i a b l e s t a c k u n w i n d e r l a t e r o n . C l e a r i t .
* /
stw r0 ,8 ( r1 )
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 7 , 1 2 , r1 )
2012-09-17 03:54:30 +04:00
2021-03-12 15:50:34 +03:00
mtcr r3
mtlr r4
2012-09-17 03:54:30 +04:00
mtctr r5
2021-03-12 15:50:34 +03:00
mtspr S P R N _ X E R ,r6
2005-10-10 16:36:14 +04:00
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 2 , 6 , r1 )
2021-03-12 15:50:34 +03:00
REST_ G P R ( 0 , r1 )
REST_ G P R ( 1 , r1 )
rfi
# ifdef C O N F I G _ 4 0 x
b . / * P r e v e n t p r e f e t c h p a s t r f i * /
# endif
2009-06-17 21:43:59 +04:00
2021-03-12 15:50:34 +03:00
.Lrestore_nvgprs :
REST_ N V G P R S ( r1 )
b . L f a s t _ u s e r _ i n t e r r u p t _ r e t u r n
2009-06-17 21:43:59 +04:00
2021-03-12 15:50:34 +03:00
.Lkernel_interrupt_return :
bl i n t e r r u p t _ e x i t _ k e r n e l _ p r e p a r e
2005-10-10 16:36:14 +04:00
2021-03-12 15:50:34 +03:00
.Lfast_kernel_interrupt_return :
cmpwi c r1 ,r3 ,0
lwz r11 ,_ N I P ( r1 )
lwz r12 ,_ M S R ( r1 )
mtspr S P R N _ S R R 0 ,r11
mtspr S P R N _ S R R 1 ,r12
2005-10-10 16:36:14 +04:00
2007-11-10 01:17:49 +03:00
BEGIN_ F T R _ S E C T I O N
2021-03-12 15:50:34 +03:00
stwcx. r0 ,0 ,r1 / * t o c l e a r t h e r e s e r v a t i o n * /
FTR_ S E C T I O N _ E L S E
lwarx r0 ,0 ,r1
ALT_ F T R _ S E C T I O N _ E N D _ I F C L R ( C P U _ F T R _ S T C X _ C H E C K S _ A D D R E S S )
2005-10-10 16:36:14 +04:00
2021-03-12 15:50:34 +03:00
lwz r3 ,_ L I N K ( r1 )
lwz r4 ,_ C T R ( r1 )
lwz r5 ,_ X E R ( r1 )
lwz r6 ,_ C C R ( r1 )
li r0 ,0
2005-10-10 16:36:14 +04:00
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 7 , 1 2 , r1 )
2021-03-12 15:50:34 +03:00
mtlr r3
mtctr r4
mtspr S P R N _ X E R ,r5
2005-10-10 16:36:14 +04:00
/ *
2021-03-12 15:50:34 +03:00
* Leaving a s t a l e e x c e p t i o n _ m a r k e r o n t h e s t a c k c a n c o n f u s e
* the r e l i a b l e s t a c k u n w i n d e r l a t e r o n . C l e a r i t .
2005-10-10 16:36:14 +04:00
* /
2021-03-12 15:50:34 +03:00
stw r0 ,8 ( r1 )
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 2 , 5 , r1 )
2021-03-12 15:50:34 +03:00
bne- c r1 ,1 f / * e m u l a t e s t a c k s t o r e * /
mtcr r6
REST_ G P R ( 6 , r1 )
REST_ G P R ( 0 , r1 )
REST_ G P R ( 1 , r1 )
2020-11-08 19:57:36 +03:00
rfi
2021-03-12 15:50:34 +03:00
# ifdef C O N F I G _ 4 0 x
b . / * P r e v e n t p r e f e t c h p a s t r f i * /
# endif
2005-10-10 16:36:14 +04:00
2021-03-12 15:50:34 +03:00
1 : / *
* Emulate s t a c k s t o r e w i t h u p d a t e . N e w r1 v a l u e w a s a l r e a d y c a l c u l a t e d
* and u p d a t e d i n o u r i n t e r r u p t r e g s b y e m u l a t e _ l o a d s t o r e , b u t w e c a n ' t
* store t h e p r e v i o u s v a l u e o f r1 t o t h e s t a c k b e f o r e r e - l o a d i n g o u r
* registers f r o m i t , o t h e r w i s e t h e y c o u l d b e c l o b b e r e d . U s e
* SPRG S c r a t c h0 a s t e m p o r a r y s t o r a g e t o h o l d t h e s t o r e
* data, a s i n t e r r u p t s a r e d i s a b l e d h e r e s o i t w o n ' t b e c l o b b e r e d .
2005-10-10 16:36:14 +04:00
* /
2021-03-12 15:50:34 +03:00
mtcr r6
# ifdef C O N F I G _ B O O K E
mtspr S P R N _ S P R G _ W S C R A T C H 0 , r9
# else
mtspr S P R N _ S P R G _ S C R A T C H 0 , r9
# endif
addi r9 ,r1 ,I N T _ F R A M E _ S I Z E / * g e t o r i g i n a l r1 * /
REST_ G P R ( 6 , r1 )
REST_ G P R ( 0 , r1 )
REST_ G P R ( 1 , r1 )
stw r9 ,0 ( r1 ) / * p e r f o r m s t o r e c o m p o n e n t o f s t w u * /
# ifdef C O N F I G _ B O O K E
mfspr r9 , S P R N _ S P R G _ R S C R A T C H 0
# else
mfspr r9 , S P R N _ S P R G _ S C R A T C H 0
# endif
2005-10-10 16:36:14 +04:00
rfi
2021-03-12 15:50:34 +03:00
# ifdef C O N F I G _ 4 0 x
b . / * P r e v e n t p r e f e t c h p a s t r f i * /
# endif
_ ASM_ N O K P R O B E _ S Y M B O L ( i n t e r r u p t _ r e t u r n )
# if d e f i n e d ( C O N F I G _ 4 x x ) | | d e f i n e d ( C O N F I G _ B O O K E )
2005-10-10 16:36:14 +04:00
/ *
* Returning f r o m a c r i t i c a l i n t e r r u p t i n u s e r m o d e d o e s n ' t n e e d
* to b e a n y d i f f e r e n t f r o m a n o r m a l e x c e p t i o n . F o r a c r i t i c a l
* interrupt i n t h e k e r n e l , w e j u s t r e t u r n ( w i t h o u t c h e c k i n g f o r
* preemption) s i n c e t h e i n t e r r u p t m a y h a v e h a p p e n e d a t s o m e c r u c i a l
* place ( e . g . i n s i d e t h e T L B m i s s h a n d l e r ) , a n d b e c a u s e w e w i l l b e
* running w i t h r1 p o i n t i n g i n t o c r i t i c a l _ s t a c k , n o t t h e c u r r e n t
* process' s k e r n e l s t a c k ( a n d t h e r e f o r e c u r r e n t _ t h r e a d _ i n f o ( ) w i l l
* give t h e w r o n g a n s w e r ) .
* We h a v e t o r e s t o r e v a r i o u s S P R s t h a t m a y h a v e b e e n i n u s e a t t h e
* time o f t h e c r i t i c a l i n t e r r u p t .
*
* /
# ifdef C O N F I G _ 4 0 x
# define P P C _ 4 0 x _ T U R N _ O F F _ M S R _ D R \
/ * avoid a n y p o s s i b l e T L B m i s s e s h e r e b y t u r n i n g o f f M S R . D R , w e \
* assume t h e i n s t r u c t i o n s h e r e a r e m a p p e d b y a p i n n e d T L B e n t r y * / \
li r10 ,M S R _ I R ; \
mtmsr r10 ; \
isync; \
tophys( r1 , r1 ) ;
# else
# define P P C _ 4 0 x _ T U R N _ O F F _ M S R _ D R
# endif
# define R E T _ F R O M _ E X C _ L E V E L ( e x c _ l v l _ s r r0 , e x c _ l v l _ s r r1 , e x c _ l v l _ r f i ) \
REST_ N V G P R S ( r1 ) ; \
lwz r3 ,_ M S R ( r1 ) ; \
andi. r3 ,r3 ,M S R _ P R ; \
2021-03-12 15:50:34 +03:00
bne i n t e r r u p t _ r e t u r n ; \
2005-10-10 16:36:14 +04:00
lwz r0 ,G P R 0 ( r1 ) ; \
lwz r2 ,G P R 2 ( r1 ) ; \
2021-10-22 09:13:22 +03:00
REST_ G P R S ( 3 , 8 , r1 ) ; \
2005-10-10 16:36:14 +04:00
lwz r10 ,_ X E R ( r1 ) ; \
lwz r11 ,_ C T R ( r1 ) ; \
mtspr S P R N _ X E R ,r10 ; \
mtctr r11 ; \
stwcx. r0 ,0 ,r1 ; /* to clear the reservation */ \
lwz r11 ,_ L I N K ( r1 ) ; \
mtlr r11 ; \
lwz r10 ,_ C C R ( r1 ) ; \
mtcrf 0 x f f ,r10 ; \
PPC_ 4 0 x _ T U R N _ O F F _ M S R _ D R ; \
lwz r9 ,_ D E A R ( r1 ) ; \
lwz r10 ,_ E S R ( r1 ) ; \
mtspr S P R N _ D E A R ,r9 ; \
mtspr S P R N _ E S R ,r10 ; \
lwz r11 ,_ N I P ( r1 ) ; \
lwz r12 ,_ M S R ( r1 ) ; \
mtspr e x c _ l v l _ s r r0 ,r11 ; \
mtspr e x c _ l v l _ s r r1 ,r12 ; \
lwz r9 ,G P R 9 ( r1 ) ; \
lwz r12 ,G P R 1 2 ( r1 ) ; \
lwz r10 ,G P R 1 0 ( r1 ) ; \
lwz r11 ,G P R 1 1 ( r1 ) ; \
lwz r1 ,G P R 1 ( r1 ) ; \
exc_ l v l _ r f i ; \
b . ; /* prevent prefetch past exc_lvl_rfi */
2008-04-30 14:23:21 +04:00
# define R E S T O R E _ x S R R ( e x c _ l v l _ s r r0 , e x c _ l v l _ s r r1 ) \
lwz r9 ,_ ## e x c _ l v l _ s r r 0 ( r1 ) ; \
lwz r10 ,_ ## e x c _ l v l _ s r r 1 ( r1 ) ; \
mtspr S P R N _ ## e x c _ l v l _ s r r 0 ,r9 ; \
mtspr S P R N _ ## e x c _ l v l _ s r r 1 ,r10 ;
2009-02-13 01:12:40 +03:00
# if d e f i n e d ( C O N F I G _ P P C _ B O O K 3 E _ M M U )
2008-04-30 14:23:21 +04:00
# ifdef C O N F I G _ P H Y S _ 6 4 B I T
# define R E S T O R E _ M A S 7 \
lwz r11 ,M A S 7 ( r1 ) ; \
mtspr S P R N _ M A S 7 ,r11 ;
# else
# define R E S T O R E _ M A S 7
# endif / * C O N F I G _ P H Y S _ 6 4 B I T * /
# define R E S T O R E _ M M U _ R E G S \
lwz r9 ,M A S 0 ( r1 ) ; \
lwz r10 ,M A S 1 ( r1 ) ; \
lwz r11 ,M A S 2 ( r1 ) ; \
mtspr S P R N _ M A S 0 ,r9 ; \
lwz r9 ,M A S 3 ( r1 ) ; \
mtspr S P R N _ M A S 1 ,r10 ; \
lwz r10 ,M A S 6 ( r1 ) ; \
mtspr S P R N _ M A S 2 ,r11 ; \
mtspr S P R N _ M A S 3 ,r9 ; \
mtspr S P R N _ M A S 6 ,r10 ; \
RESTORE_ M A S 7 ;
# elif d e f i n e d ( C O N F I G _ 4 4 x )
# define R E S T O R E _ M M U _ R E G S \
lwz r9 ,M M U C R ( r1 ) ; \
mtspr S P R N _ M M U C R ,r9 ;
# else
# define R E S T O R E _ M M U _ R E G S
# endif
# ifdef C O N F I G _ 4 0 x
2005-10-10 16:36:14 +04:00
.globl ret_from_crit_exc
ret_from_crit_exc :
2008-04-30 14:23:21 +04:00
lis r9 ,c r i t _ s r r0 @ha;
lwz r9 ,c r i t _ s r r0 @l(r9);
lis r10 ,c r i t _ s r r1 @ha;
lwz r10 ,c r i t _ s r r1 @l(r10);
mtspr S P R N _ S R R 0 ,r9 ;
mtspr S P R N _ S R R 1 ,r10 ;
2009-02-10 23:10:44 +03:00
RET_ F R O M _ E X C _ L E V E L ( S P R N _ C S R R 0 , S P R N _ C S R R 1 , P P C _ R F C I )
2020-03-31 19:03:47 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( r e t _ f r o m _ c r i t _ e x c )
2008-04-30 14:23:21 +04:00
# endif / * C O N F I G _ 4 0 x * /
2005-10-10 16:36:14 +04:00
# ifdef C O N F I G _ B O O K E
2008-04-30 14:23:21 +04:00
.globl ret_from_crit_exc
ret_from_crit_exc :
RESTORE_ x S R R ( S R R 0 ,S R R 1 ) ;
RESTORE_ M M U _ R E G S ;
2009-02-10 23:10:44 +03:00
RET_ F R O M _ E X C _ L E V E L ( S P R N _ C S R R 0 , S P R N _ C S R R 1 , P P C _ R F C I )
2020-03-31 19:03:47 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( r e t _ f r o m _ c r i t _ e x c )
2008-04-30 14:23:21 +04:00
2005-10-10 16:36:14 +04:00
.globl ret_from_debug_exc
ret_from_debug_exc :
2008-04-30 14:23:21 +04:00
RESTORE_ x S R R ( S R R 0 ,S R R 1 ) ;
RESTORE_ x S R R ( C S R R 0 ,C S R R 1 ) ;
RESTORE_ M M U _ R E G S ;
2009-02-10 23:10:44 +03:00
RET_ F R O M _ E X C _ L E V E L ( S P R N _ D S R R 0 , S P R N _ D S R R 1 , P P C _ R F D I )
2020-03-31 19:03:47 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( r e t _ f r o m _ d e b u g _ e x c )
2005-10-10 16:36:14 +04:00
.globl ret_from_mcheck_exc
ret_from_mcheck_exc :
2008-04-30 14:23:21 +04:00
RESTORE_ x S R R ( S R R 0 ,S R R 1 ) ;
RESTORE_ x S R R ( C S R R 0 ,C S R R 1 ) ;
RESTORE_ x S R R ( D S R R 0 ,D S R R 1 ) ;
RESTORE_ M M U _ R E G S ;
2009-02-10 23:10:44 +03:00
RET_ F R O M _ E X C _ L E V E L ( S P R N _ M C S R R 0 , S P R N _ M C S R R 1 , P P C _ R F M C I )
2020-03-31 19:03:47 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( r e t _ f r o m _ m c h e c k _ e x c )
2005-10-10 16:36:14 +04:00
# endif / * C O N F I G _ B O O K E * /
# endif / * ! ( C O N F I G _ 4 x x | | C O N F I G _ B O O K E ) * /
/ *
* PROM c o d e f o r s p e c i f i c m a c h i n e s f o l l o w s . P u t i t
* here s o i t ' s e a s y t o a d d a r c h - s p e c i f i c s e c t i o n s l a t e r .
* - - Cort
* /
2005-10-26 11:05:24 +04:00
# ifdef C O N F I G _ P P C _ R T A S
2005-10-10 16:36:14 +04:00
/ *
* On C H R P , t h e R u n - T i m e A b s t r a c t i o n S e r v i c e s ( R T A S ) h a v e t o b e
* called w i t h t h e M M U o f f .
* /
_ GLOBAL( e n t e r _ r t a s )
stwu r1 ,- I N T _ F R A M E _ S I Z E ( r1 )
mflr r0
stw r0 ,I N T _ F R A M E _ S I Z E + 4 ( r1 )
2006-01-13 06:56:25 +03:00
LOAD_ R E G _ A D D R ( r4 , r t a s )
2005-10-10 16:36:14 +04:00
lis r6 ,1 f @ha /* physical return address for rtas */
addi r6 ,r6 ,1 f @l
tophys( r6 ,r6 )
2005-10-26 11:05:24 +04:00
lwz r8 ,R T A S E N T R Y ( r4 )
lwz r4 ,R T A S B A S E ( r4 )
2005-10-10 16:36:14 +04:00
mfmsr r9
stw r9 ,8 ( r1 )
2019-08-20 17:34:13 +03:00
LOAD_ R E G _ I M M E D I A T E ( r0 ,M S R _ K E R N E L )
2020-09-29 09:48:34 +03:00
mtmsr r0 / * d i s a b l e i n t e r r u p t s s o S R R 0 / 1 d o n ' t g e t t r a s h e d * /
2005-10-10 16:36:14 +04:00
li r9 ,M S R _ K E R N E L & ~ ( M S R _ I R | M S R _ D R )
mtlr r6
2021-03-12 15:50:22 +03:00
stw r1 , T H R E A D + R T A S _ S P ( r2 )
2005-10-10 16:36:14 +04:00
mtspr S P R N _ S R R 0 ,r8
mtspr S P R N _ S R R 1 ,r9
2020-11-08 19:57:36 +03:00
rfi
2021-03-12 15:50:22 +03:00
1 :
2021-03-12 15:50:24 +03:00
lis r8 , 1 f @h
ori r8 , r8 , 1 f @l
LOAD_ R E G _ I M M E D I A T E ( r9 ,M S R _ K E R N E L )
mtspr S P R N _ S R R 0 ,r8
mtspr S P R N _ S R R 1 ,r9
rfi / * R e a c t i v a t e M M U t r a n s l a t i o n * /
1 :
2021-03-12 15:50:22 +03:00
lwz r8 ,I N T _ F R A M E _ S I Z E + 4 ( r1 ) / * g e t r e t u r n a d d r e s s * /
lwz r9 ,8 ( r1 ) / * o r i g i n a l m s r v a l u e * /
2005-10-10 16:36:14 +04:00
addi r1 ,r1 ,I N T _ F R A M E _ S I Z E
li r0 ,0
2021-03-12 15:50:22 +03:00
stw r0 , T H R E A D + R T A S _ S P ( r2 )
2021-03-12 15:50:24 +03:00
mtlr r8
mtmsr r9
blr / * r e t u r n t o c a l l e r * /
2020-03-31 19:03:44 +03:00
_ ASM_ N O K P R O B E _ S Y M B O L ( e n t e r _ r t a s )
2005-10-26 11:05:24 +04:00
# endif / * C O N F I G _ P P C _ R T A S * /
