x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()
In reaction to a proposal to introduce a memcpy_mcsafe_fast()
implementation Linus points out that memcpy_mcsafe() is poorly named
relative to communicating the scope of the interface. Specifically what
addresses are valid to pass as source, destination, and what faults /
exceptions are handled.
Of particular concern is that even though x86 might be able to handle
the semantics of copy_mc_to_user() with its common copy_user_generic()
implementation other archs likely need / want an explicit path for this
case:
On Fri, May 1, 2020 at 11:28 AM Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 30, 2020 at 6:21 PM Dan Williams <dan.j.williams@intel.com> wrote:
> >
> > However now I see that copy_user_generic() works for the wrong reason.
> > It works because the exception on the source address due to poison
> > looks no different than a write fault on the user address to the
> > caller, it's still just a short copy. So it makes copy_to_user() work
> > for the wrong reason relative to the name.
>
> Right.
>
> And it won't work that way on other architectures. On x86, we have a
> generic function that can take faults on either side, and we use it
> for both cases (and for the "in_user" case too), but that's an
> artifact of the architecture oddity.
>
> In fact, it's probably wrong even on x86 - because it can hide bugs -
> but writing those things is painful enough that everybody prefers
> having just one function.
Replace a single top-level memcpy_mcsafe() with either
copy_mc_to_user(), or copy_mc_to_kernel().
Introduce an x86 copy_mc_fragile() name as the rename for the
low-level x86 implementation formerly named memcpy_mcsafe(). It is used
as the slow / careful backend that is supplanted by a fast
copy_mc_generic() in a follow-on patch.
One side-effect of this reorganization is that separating copy_mc_64.S
to its own file means that perf no longer needs to track dependencies
for its memcpy_64.S benchmarks.
[ bp: Massage a bit. ]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <stable@vger.kernel.org>
Link: http://lore.kernel.org/r/CAHk-=wjSqtXAqfUJxFtWNwmguFASTgB0dz1dT3V-78Quiezqbg@mail.gmail.com
Link: https://lkml.kernel.org/r/160195561680.2163339.11574962055305783722.stgit@dwillia2-desk3.amr.corp.intel.com
2020-10-06 06:40:16 +03:00
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright(c) 2016-2020 Intel Corporation. All rights reserved. */
# include < l i n u x / l i n k a g e . h >
# include < a s m / a s m . h >
# ifndef C O N F I G _ U M L
# ifdef C O N F I G _ X 8 6 _ M C E
/ *
* copy_ m c _ f r a g i l e - c o p y m e m o r y w i t h i n d i c a t i o n i f a n e x c e p t i o n / f a u l t h a p p e n e d
*
* The ' f r a g i l e ' v e r s i o n i s o p t e d i n t o b y p l a t f o r m q u i r k s a n d t a k e s
* pains t o a v o i d u n r e c o v e r a b l e c o r n e r c a s e s l i k e ' f a s t - s t r i n g '
* instruction s e q u e n c e s , a n d c o n s u m i n g p o i s o n a c r o s s a c a c h e l i n e
* boundary. T h e n o n - f r a g i l e v e r s i o n i s e q u i v a l e n t t o m e m c p y ( )
* regardless o f C P U m a c h i n e - c h e c k - r e c o v e r y c a p a b i l i t y .
* /
SYM_ F U N C _ S T A R T ( c o p y _ m c _ f r a g i l e )
cmpl $ 8 , % e d x
/* Less than 8 bytes? Go to byte copy loop */
jb . L _ n o _ w h o l e _ w o r d s
/* Check for bad alignment of source */
testl $ 7 , % e s i
/* Already aligned */
jz . L _ 8 b y t e _ a l i g n e d
/* Copy one byte at a time until source is 8-byte aligned */
movl % e s i , % e c x
andl $ 7 , % e c x
subl $ 8 , % e c x
negl % e c x
subl % e c x , % e d x
.L_read_leading_bytes :
movb ( % r s i ) , % a l
.L_write_leading_bytes :
movb % a l , ( % r d i )
incq % r s i
incq % r d i
decl % e c x
jnz . L _ r e a d _ l e a d i n g _ b y t e s
.L_8byte_aligned :
movl % e d x , % e c x
andl $ 7 , % e d x
shrl $ 3 , % e c x
jz . L _ n o _ w h o l e _ w o r d s
.L_read_words :
movq ( % r s i ) , % r8
.L_write_words :
movq % r8 , ( % r d i )
addq $ 8 , % r s i
addq $ 8 , % r d i
decl % e c x
jnz . L _ r e a d _ w o r d s
/* Any trailing bytes? */
.L_no_whole_words :
andl % e d x , % e d x
jz . L _ d o n e _ m e m c p y _ t r a p
/* Copy trailing bytes */
movl % e d x , % e c x
.L_read_trailing_bytes :
movb ( % r s i ) , % a l
.L_write_trailing_bytes :
movb % a l , ( % r d i )
incq % r s i
incq % r d i
decl % e c x
jnz . L _ r e a d _ t r a i l i n g _ b y t e s
/* Copy successful. Return zero */
.L_done_memcpy_trap :
xorl % e a x , % e a x
.L_done :
2021-12-04 16:43:40 +03:00
RET
x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()
In reaction to a proposal to introduce a memcpy_mcsafe_fast()
implementation Linus points out that memcpy_mcsafe() is poorly named
relative to communicating the scope of the interface. Specifically what
addresses are valid to pass as source, destination, and what faults /
exceptions are handled.
Of particular concern is that even though x86 might be able to handle
the semantics of copy_mc_to_user() with its common copy_user_generic()
implementation other archs likely need / want an explicit path for this
case:
On Fri, May 1, 2020 at 11:28 AM Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 30, 2020 at 6:21 PM Dan Williams <dan.j.williams@intel.com> wrote:
> >
> > However now I see that copy_user_generic() works for the wrong reason.
> > It works because the exception on the source address due to poison
> > looks no different than a write fault on the user address to the
> > caller, it's still just a short copy. So it makes copy_to_user() work
> > for the wrong reason relative to the name.
>
> Right.
>
> And it won't work that way on other architectures. On x86, we have a
> generic function that can take faults on either side, and we use it
> for both cases (and for the "in_user" case too), but that's an
> artifact of the architecture oddity.
>
> In fact, it's probably wrong even on x86 - because it can hide bugs -
> but writing those things is painful enough that everybody prefers
> having just one function.
Replace a single top-level memcpy_mcsafe() with either
copy_mc_to_user(), or copy_mc_to_kernel().
Introduce an x86 copy_mc_fragile() name as the rename for the
low-level x86 implementation formerly named memcpy_mcsafe(). It is used
as the slow / careful backend that is supplanted by a fast
copy_mc_generic() in a follow-on patch.
One side-effect of this reorganization is that separating copy_mc_64.S
to its own file means that perf no longer needs to track dependencies
for its memcpy_64.S benchmarks.
[ bp: Massage a bit. ]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <stable@vger.kernel.org>
Link: http://lore.kernel.org/r/CAHk-=wjSqtXAqfUJxFtWNwmguFASTgB0dz1dT3V-78Quiezqbg@mail.gmail.com
Link: https://lkml.kernel.org/r/160195561680.2163339.11574962055305783722.stgit@dwillia2-desk3.amr.corp.intel.com
2020-10-06 06:40:16 +03:00
/ *
* Return n u m b e r o f b y t e s n o t c o p i e d f o r a n y f a i l u r e . N o t e t h a t
* there i s n o " t a i l " h a n d l i n g s i n c e t h e s o u r c e b u f f e r i s 8 - b y t e
* aligned a n d p o i s o n i s c a c h e l i n e a l i g n e d .
* /
.E_read_words :
shll $ 3 , % e c x
.E_leading_bytes :
addl % e d x , % e c x
.E_trailing_bytes :
mov % e c x , % e a x
jmp . L _ d o n e
/ *
* For w r i t e f a u l t h a n d l i n g , g i v e n t h e d e s t i n a t i o n i s u n a l i g n e d ,
* we h a n d l e f a u l t s o n m u l t i - b y t e w r i t e s w i t h a b y t e - b y - b y t e
* copy u p t o t h e w r i t e - p r o t e c t e d p a g e .
* /
.E_write_words :
shll $ 3 , % e c x
addl % e d x , % e c x
movl % e c x , % e d x
jmp c o p y _ m c _ f r a g i l e _ h a n d l e _ t a i l
2021-09-08 16:29:21 +03:00
_ ASM_ E X T A B L E _ T Y P E ( . L _ r e a d _ l e a d i n g _ b y t e s , . E _ l e a d i n g _ b y t e s , E X _ T Y P E _ D E F A U L T _ M C E _ S A F E )
_ ASM_ E X T A B L E _ T Y P E ( . L _ r e a d _ w o r d s , . E _ r e a d _ w o r d s , E X _ T Y P E _ D E F A U L T _ M C E _ S A F E )
_ ASM_ E X T A B L E _ T Y P E ( . L _ r e a d _ t r a i l i n g _ b y t e s , . E _ t r a i l i n g _ b y t e s , E X _ T Y P E _ D E F A U L T _ M C E _ S A F E )
x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()
In reaction to a proposal to introduce a memcpy_mcsafe_fast()
implementation Linus points out that memcpy_mcsafe() is poorly named
relative to communicating the scope of the interface. Specifically what
addresses are valid to pass as source, destination, and what faults /
exceptions are handled.
Of particular concern is that even though x86 might be able to handle
the semantics of copy_mc_to_user() with its common copy_user_generic()
implementation other archs likely need / want an explicit path for this
case:
On Fri, May 1, 2020 at 11:28 AM Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 30, 2020 at 6:21 PM Dan Williams <dan.j.williams@intel.com> wrote:
> >
> > However now I see that copy_user_generic() works for the wrong reason.
> > It works because the exception on the source address due to poison
> > looks no different than a write fault on the user address to the
> > caller, it's still just a short copy. So it makes copy_to_user() work
> > for the wrong reason relative to the name.
>
> Right.
>
> And it won't work that way on other architectures. On x86, we have a
> generic function that can take faults on either side, and we use it
> for both cases (and for the "in_user" case too), but that's an
> artifact of the architecture oddity.
>
> In fact, it's probably wrong even on x86 - because it can hide bugs -
> but writing those things is painful enough that everybody prefers
> having just one function.
Replace a single top-level memcpy_mcsafe() with either
copy_mc_to_user(), or copy_mc_to_kernel().
Introduce an x86 copy_mc_fragile() name as the rename for the
low-level x86 implementation formerly named memcpy_mcsafe(). It is used
as the slow / careful backend that is supplanted by a fast
copy_mc_generic() in a follow-on patch.
One side-effect of this reorganization is that separating copy_mc_64.S
to its own file means that perf no longer needs to track dependencies
for its memcpy_64.S benchmarks.
[ bp: Massage a bit. ]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <stable@vger.kernel.org>
Link: http://lore.kernel.org/r/CAHk-=wjSqtXAqfUJxFtWNwmguFASTgB0dz1dT3V-78Quiezqbg@mail.gmail.com
Link: https://lkml.kernel.org/r/160195561680.2163339.11574962055305783722.stgit@dwillia2-desk3.amr.corp.intel.com
2020-10-06 06:40:16 +03:00
_ ASM_ E X T A B L E ( . L _ w r i t e _ l e a d i n g _ b y t e s , . E _ l e a d i n g _ b y t e s )
_ ASM_ E X T A B L E ( . L _ w r i t e _ w o r d s , . E _ w r i t e _ w o r d s )
_ ASM_ E X T A B L E ( . L _ w r i t e _ t r a i l i n g _ b y t e s , . E _ t r a i l i n g _ b y t e s )
2021-11-10 13:01:06 +03:00
SYM_ F U N C _ E N D ( c o p y _ m c _ f r a g i l e )
x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()
In reaction to a proposal to introduce a memcpy_mcsafe_fast()
implementation Linus points out that memcpy_mcsafe() is poorly named
relative to communicating the scope of the interface. Specifically what
addresses are valid to pass as source, destination, and what faults /
exceptions are handled.
Of particular concern is that even though x86 might be able to handle
the semantics of copy_mc_to_user() with its common copy_user_generic()
implementation other archs likely need / want an explicit path for this
case:
On Fri, May 1, 2020 at 11:28 AM Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 30, 2020 at 6:21 PM Dan Williams <dan.j.williams@intel.com> wrote:
> >
> > However now I see that copy_user_generic() works for the wrong reason.
> > It works because the exception on the source address due to poison
> > looks no different than a write fault on the user address to the
> > caller, it's still just a short copy. So it makes copy_to_user() work
> > for the wrong reason relative to the name.
>
> Right.
>
> And it won't work that way on other architectures. On x86, we have a
> generic function that can take faults on either side, and we use it
> for both cases (and for the "in_user" case too), but that's an
> artifact of the architecture oddity.
>
> In fact, it's probably wrong even on x86 - because it can hide bugs -
> but writing those things is painful enough that everybody prefers
> having just one function.
Replace a single top-level memcpy_mcsafe() with either
copy_mc_to_user(), or copy_mc_to_kernel().
Introduce an x86 copy_mc_fragile() name as the rename for the
low-level x86 implementation formerly named memcpy_mcsafe(). It is used
as the slow / careful backend that is supplanted by a fast
copy_mc_generic() in a follow-on patch.
One side-effect of this reorganization is that separating copy_mc_64.S
to its own file means that perf no longer needs to track dependencies
for its memcpy_64.S benchmarks.
[ bp: Massage a bit. ]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <stable@vger.kernel.org>
Link: http://lore.kernel.org/r/CAHk-=wjSqtXAqfUJxFtWNwmguFASTgB0dz1dT3V-78Quiezqbg@mail.gmail.com
Link: https://lkml.kernel.org/r/160195561680.2163339.11574962055305783722.stgit@dwillia2-desk3.amr.corp.intel.com
2020-10-06 06:40:16 +03:00
# endif / * C O N F I G _ X 8 6 _ M C E * /
2020-10-06 06:40:25 +03:00
/ *
* copy_ m c _ e n h a n c e d _ f a s t _ s t r i n g - m e m o r y c o p y w i t h e x c e p t i o n h a n d l i n g
*
* Fast s t r i n g c o p y + f a u l t / e x c e p t i o n h a n d l i n g . I f t h e C P U d o e s
* support m a c h i n e c h e c k e x c e p t i o n r e c o v e r y , b u t d o e s n o t s u p p o r t
* recovering f r o m f a s t - s t r i n g e x c e p t i o n s t h e n t h i s C P U n e e d s t o b e
* added t o t h e c o p y _ m c _ f r a g i l e _ k e y s e t o f q u i r k s . O t h e r w i s e , a b s e n t a n y
* machine c h e c k r e c o v e r y s u p p o r t t h i s v e r s i o n s h o u l d b e n o s l o w e r t h a n
* standard m e m c p y .
* /
SYM_ F U N C _ S T A R T ( c o p y _ m c _ e n h a n c e d _ f a s t _ s t r i n g )
movq % r d i , % r a x
movq % r d x , % r c x
.L_copy :
rep m o v s b
/* Copy successful. Return zero */
xorl % e a x , % e a x
2021-12-04 16:43:40 +03:00
RET
2020-10-06 06:40:25 +03:00
.E_copy :
/ *
* On f a u l t % r c x i s u p d a t e d s u c h t h a t t h e c o p y i n s t r u c t i o n c o u l d
* optionally b e r e s t a r t e d a t t h e f a u l t p o s i t i o n , i . e . i t
* contains ' b y t e s r e m a i n i n g ' . A n o n - z e r o r e t u r n i n d i c a t e s e r r o r
* to c o p y _ m c _ g e n e r i c ( ) u s e r s , o r i n d i c a t e s h o r t t r a n s f e r s t o
* user- c o p y r o u t i n e s .
* /
movq % r c x , % r a x
2021-12-04 16:43:40 +03:00
RET
2020-10-06 06:40:25 +03:00
2021-09-08 16:29:21 +03:00
_ ASM_ E X T A B L E _ T Y P E ( . L _ c o p y , . E _ c o p y , E X _ T Y P E _ D E F A U L T _ M C E _ S A F E )
2021-11-10 13:01:06 +03:00
SYM_ F U N C _ E N D ( c o p y _ m c _ e n h a n c e d _ f a s t _ s t r i n g )
x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()
In reaction to a proposal to introduce a memcpy_mcsafe_fast()
implementation Linus points out that memcpy_mcsafe() is poorly named
relative to communicating the scope of the interface. Specifically what
addresses are valid to pass as source, destination, and what faults /
exceptions are handled.
Of particular concern is that even though x86 might be able to handle
the semantics of copy_mc_to_user() with its common copy_user_generic()
implementation other archs likely need / want an explicit path for this
case:
On Fri, May 1, 2020 at 11:28 AM Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Thu, Apr 30, 2020 at 6:21 PM Dan Williams <dan.j.williams@intel.com> wrote:
> >
> > However now I see that copy_user_generic() works for the wrong reason.
> > It works because the exception on the source address due to poison
> > looks no different than a write fault on the user address to the
> > caller, it's still just a short copy. So it makes copy_to_user() work
> > for the wrong reason relative to the name.
>
> Right.
>
> And it won't work that way on other architectures. On x86, we have a
> generic function that can take faults on either side, and we use it
> for both cases (and for the "in_user" case too), but that's an
> artifact of the architecture oddity.
>
> In fact, it's probably wrong even on x86 - because it can hide bugs -
> but writing those things is painful enough that everybody prefers
> having just one function.
Replace a single top-level memcpy_mcsafe() with either
copy_mc_to_user(), or copy_mc_to_kernel().
Introduce an x86 copy_mc_fragile() name as the rename for the
low-level x86 implementation formerly named memcpy_mcsafe(). It is used
as the slow / careful backend that is supplanted by a fast
copy_mc_generic() in a follow-on patch.
One side-effect of this reorganization is that separating copy_mc_64.S
to its own file means that perf no longer needs to track dependencies
for its memcpy_64.S benchmarks.
[ bp: Massage a bit. ]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <stable@vger.kernel.org>
Link: http://lore.kernel.org/r/CAHk-=wjSqtXAqfUJxFtWNwmguFASTgB0dz1dT3V-78Quiezqbg@mail.gmail.com
Link: https://lkml.kernel.org/r/160195561680.2163339.11574962055305783722.stgit@dwillia2-desk3.amr.corp.intel.com
2020-10-06 06:40:16 +03:00
# endif / * ! C O N F I G _ U M L * /
