2013-09-07 12:56:26 +10:00
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Implement f a s t C R C - T 1 0 D I F c o m p u t a t i o n w i t h S S E a n d P C L M U L Q D Q i n s t r u c t i o n s
#
# Copyright ( c ) 2 0 1 3 , I n t e l C o r p o r a t i o n
#
# Authors :
# Erdinc O z t u r k < e r d i n c . o z t u r k @intel.com>
# Vinodh G o p a l < v i n o d h . g o p a l @intel.com>
# James G u i l f o r d < j a m e s . g u i l f o r d @intel.com>
# Tim C h e n < t i m . c . c h e n @linux.intel.com>
#
# This s o f t w a r e i s a v a i l a b l e t o y o u u n d e r a c h o i c e o f o n e o f t w o
# licenses. Y o u m a y c h o o s e t o b e l i c e n s e d u n d e r t h e t e r m s o f t h e G N U
# General P u b l i c L i c e n s e ( G P L ) V e r s i o n 2 , a v a i l a b l e f r o m t h e f i l e
# COPYING i n t h e m a i n d i r e c t o r y o f t h i s s o u r c e t r e e , o r t h e
# OpenIB. o r g B S D l i c e n s e b e l o w :
#
# Redistribution a n d u s e i n s o u r c e a n d b i n a r y f o r m s , w i t h o r w i t h o u t
# modification, a r e p e r m i t t e d p r o v i d e d t h a t t h e f o l l o w i n g c o n d i t i o n s a r e
# met :
#
# * Redistributions o f s o u r c e c o d e m u s t r e t a i n t h e a b o v e c o p y r i g h t
# notice, t h i s l i s t o f c o n d i t i o n s a n d t h e f o l l o w i n g d i s c l a i m e r .
#
# * Redistributions i n b i n a r y f o r m m u s t r e p r o d u c e t h e a b o v e c o p y r i g h t
# notice, t h i s l i s t o f c o n d i t i o n s a n d t h e f o l l o w i n g d i s c l a i m e r i n t h e
# documentation a n d / o r o t h e r m a t e r i a l s p r o v i d e d w i t h t h e
# distribution.
#
# * Neither t h e n a m e o f t h e I n t e l C o r p o r a t i o n n o r t h e n a m e s o f i t s
# contributors m a y b e u s e d t o e n d o r s e o r p r o m o t e p r o d u c t s d e r i v e d f r o m
# this s o f t w a r e w i t h o u t s p e c i f i c p r i o r w r i t t e n p e r m i s s i o n .
#
#
# THIS S O F T W A R E I S P R O V I D E D B Y I N T E L C O R P O R A T I O N " " A S I S " " A N D A N Y
# EXPRESS O R I M P L I E D W A R R A N T I E S , I N C L U D I N G , B U T N O T L I M I T E D T O , T H E
# IMPLIED W A R R A N T I E S O F M E R C H A N T A B I L I T Y A N D F I T N E S S F O R A P A R T I C U L A R
# PURPOSE A R E D I S C L A I M E D . I N N O E V E N T S H A L L I N T E L C O R P O R A T I O N O R
# CONTRIBUTORS B E L I A B L E F O R A N Y D I R E C T , I N D I R E C T , I N C I D E N T A L , S P E C I A L ,
# EXEMPLARY, O R C O N S E Q U E N T I A L D A M A G E S ( I N C L U D I N G , B U T N O T L I M I T E D T O ,
# PROCUREMENT O F S U B S T I T U T E G O O D S O R S E R V I C E S ; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, W H E T H E R I N C O N T R A C T , S T R I C T L I A B I L I T Y , O R T O R T ( I N C L U D I N G
# NEGLIGENCE O R O T H E R W I S E ) A R I S I N G I N A N Y W A Y O U T O F T H E U S E O F T H I S
# SOFTWARE, E V E N I F A D V I S E D O F T H E P O S S I B I L I T Y O F S U C H D A M A G E .
#
# Reference p a p e r t i t l e d " F a s t C R C C o m p u t a t i o n f o r G e n e r i c
# Polynomials U s i n g P C L M U L Q D Q I n s t r u c t i o n "
# URL : http : / / www. i n t e l . c o m / c o n t e n t / d a m / w w w / p u b l i c / u s / e n / d o c u m e n t s
# / white- p a p e r s / f a s t - c r c - c o m p u t a t i o n - g e n e r i c - p o l y n o m i a l s - p c l m u l q d q - p a p e r . p d f
#
# include < l i n u x / l i n k a g e . h >
.text
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# define i n i t _ c r c % e d i
# define b u f % r s i
# define l e n % r d x
# define F O L D _ C O N S T S % x m m 1 0
# define B S W A P _ M A S K % x m m 1 1
# Fold r e g 1 , r e g 2 i n t o t h e n e x t 3 2 d a t a b y t e s , s t o r i n g t h e r e s u l t b a c k i n t o
# reg1 , r e g 2 .
.macro fold_32_bytes offset, r e g 1 , r e g 2
movdqu \ o f f s e t ( b u f ) , % x m m 9
movdqu \ o f f s e t + 1 6 ( b u f ) , % x m m 1 2
pshufb B S W A P _ M A S K , % x m m 9
pshufb B S W A P _ M A S K , % x m m 1 2
movdqa \ r e g 1 , % x m m 8
movdqa \ r e g 2 , % x m m 1 3
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , \ r e g 1
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 8
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , \ r e g 2
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 1 3
pxor % x m m 9 , \ r e g 1
xorps % x m m 8 , \ r e g 1
pxor % x m m 1 2 , \ r e g 2
xorps % x m m 1 3 , \ r e g 2
.endm
# Fold s r c _ r e g i n t o d s t _ r e g .
.macro fold_16_bytes src_ r e g , d s t _ r e g
movdqa \ s r c _ r e g , % x m m 8
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , \ s r c _ r e g
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , % x m m 8
pxor % x m m 8 , \ d s t _ r e g
xorps \ s r c _ r e g , \ d s t _ r e g
.endm
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
#
# u1 6 c r c _ t 1 0 d i f _ p c l ( u 1 6 i n i t _ c r c , c o n s t * u 8 b u f , s i z e _ t l e n ) ;
#
# Assumes l e n > = 1 6 .
#
2013-09-07 12:56:26 +10:00
.align 16
2019-10-11 13:51:04 +02:00
SYM_ F U N C _ S T A R T ( c r c _ t 1 0 d i f _ p c l )
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
movdqa . L b s w a p _ m a s k ( % r i p ) , B S W A P _ M A S K
# For s i z e s l e s s t h a n 2 5 6 b y t e s , w e c a n ' t f o l d 1 2 8 b y t e s a t a t i m e .
cmp $ 2 5 6 , l e n
jl . L l e s s _ t h a n _ 2 5 6 _ b y t e s
# Load t h e f i r s t 1 2 8 d a t a b y t e s . B y t e s w a p p i n g i s n e c e s s a r y t o m a k e t h e
# bit o r d e r m a t c h t h e p o l y n o m i a l c o e f f i c i e n t o r d e r .
movdqu 1 6 * 0 ( b u f ) , % x m m 0
movdqu 1 6 * 1 ( b u f ) , % x m m 1
movdqu 1 6 * 2 ( b u f ) , % x m m 2
movdqu 1 6 * 3 ( b u f ) , % x m m 3
movdqu 1 6 * 4 ( b u f ) , % x m m 4
movdqu 1 6 * 5 ( b u f ) , % x m m 5
movdqu 1 6 * 6 ( b u f ) , % x m m 6
movdqu 1 6 * 7 ( b u f ) , % x m m 7
add $ 1 2 8 , b u f
pshufb B S W A P _ M A S K , % x m m 0
pshufb B S W A P _ M A S K , % x m m 1
pshufb B S W A P _ M A S K , % x m m 2
pshufb B S W A P _ M A S K , % x m m 3
pshufb B S W A P _ M A S K , % x m m 4
pshufb B S W A P _ M A S K , % x m m 5
pshufb B S W A P _ M A S K , % x m m 6
pshufb B S W A P _ M A S K , % x m m 7
# XOR t h e f i r s t 1 6 d a t a * b i t s * w i t h t h e i n i t i a l C R C v a l u e .
pxor % x m m 8 , % x m m 8
pinsrw $ 7 , i n i t _ c r c , % x m m 8
pxor % x m m 8 , % x m m 0
movdqa . L f o l d _ a c r o s s _ 1 2 8 _ b y t e s _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
# Subtract 1 2 8 f o r t h e 1 2 8 d a t a b y t e s j u s t c o n s u m e d . S u b t r a c t a n o t h e r
# 1 2 8 to s i m p l i f y t h e t e r m i n a t i o n c o n d i t i o n o f t h e f o l l o w i n g l o o p .
sub $ 2 5 6 , l e n
# While > = 1 2 8 d a t a b y t e s r e m a i n ( n o t c o u n t i n g x m m 0 - 7 ) , f o l d t h e 1 2 8
# bytes x m m 0 - 7 i n t o t h e m , s t o r i n g t h e r e s u l t b a c k i n t o x m m 0 - 7 .
.Lfold_128_bytes_loop :
fold_ 3 2 _ b y t e s 0 , % x m m 0 , % x m m 1
fold_ 3 2 _ b y t e s 3 2 , % x m m 2 , % x m m 3
fold_ 3 2 _ b y t e s 6 4 , % x m m 4 , % x m m 5
fold_ 3 2 _ b y t e s 9 6 , % x m m 6 , % x m m 7
add $ 1 2 8 , b u f
sub $ 1 2 8 , l e n
jge . L f o l d _ 1 2 8 _ b y t e s _ l o o p
# Now f o l d t h e 1 1 2 b y t e s i n x m m 0 - x m m 6 i n t o t h e 1 6 b y t e s i n x m m 7 .
# Fold a c r o s s 6 4 b y t e s .
movdqa . L f o l d _ a c r o s s _ 6 4 _ b y t e s _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
fold_ 1 6 _ b y t e s % x m m 0 , % x m m 4
fold_ 1 6 _ b y t e s % x m m 1 , % x m m 5
fold_ 1 6 _ b y t e s % x m m 2 , % x m m 6
fold_ 1 6 _ b y t e s % x m m 3 , % x m m 7
# Fold a c r o s s 3 2 b y t e s .
movdqa . L f o l d _ a c r o s s _ 3 2 _ b y t e s _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
fold_ 1 6 _ b y t e s % x m m 4 , % x m m 6
fold_ 1 6 _ b y t e s % x m m 5 , % x m m 7
# Fold a c r o s s 1 6 b y t e s .
movdqa . L f o l d _ a c r o s s _ 1 6 _ b y t e s _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
fold_ 1 6 _ b y t e s % x m m 6 , % x m m 7
# Add 1 2 8 t o g e t t h e c o r r e c t n u m b e r o f d a t a b y t e s r e m a i n i n g i n 0 . . . 1 2 7
# ( not c o u n t i n g x m m 7 ) , f o l l o w i n g t h e p r e v i o u s e x t r a s u b t r a c t i o n b y 1 2 8 .
# Then s u b t r a c t 1 6 t o s i m p l i f y t h e t e r m i n a t i o n c o n d i t i o n o f t h e
# following l o o p .
add $ 1 2 8 - 1 6 , l e n
# While > = 1 6 d a t a b y t e s r e m a i n ( n o t c o u n t i n g x m m 7 ) , f o l d t h e 1 6 b y t e s
# xmm7 i n t o t h e m , s t o r i n g t h e r e s u l t b a c k i n t o x m m 7 .
jl . L f o l d _ 1 6 _ b y t e s _ l o o p _ d o n e
.Lfold_16_bytes_loop :
2013-09-07 12:56:26 +10:00
movdqa % x m m 7 , % x m m 8
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 7
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , % x m m 8
2013-09-07 12:56:26 +10:00
pxor % x m m 8 , % x m m 7
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
movdqu ( b u f ) , % x m m 0
pshufb B S W A P _ M A S K , % x m m 0
2013-09-07 12:56:26 +10:00
pxor % x m m 0 , % x m m 7
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
add $ 1 6 , b u f
sub $ 1 6 , l e n
jge . L f o l d _ 1 6 _ b y t e s _ l o o p
.Lfold_16_bytes_loop_done :
# Add 1 6 t o g e t t h e c o r r e c t n u m b e r o f d a t a b y t e s r e m a i n i n g i n 0 . . . 1 5
# ( not c o u n t i n g x m m 7 ) , f o l l o w i n g t h e p r e v i o u s e x t r a s u b t r a c t i o n b y 1 6 .
add $ 1 6 , l e n
je . L r e d u c e _ f i n a l _ 1 6 _ b y t e s
.Lhandle_partial_segment :
# Reduce t h e l a s t ' 1 6 + l e n ' b y t e s w h e r e 1 < = l e n < = 1 5 a n d t h e f i r s t 1 6
# bytes a r e i n x m m 7 a n d t h e r e s t a r e t h e r e m a i n i n g d a t a i n ' b u f ' . T o d o
# this w i t h o u t n e e d i n g a f o l d c o n s t a n t f o r e a c h p o s s i b l e ' l e n ' , r e d i v i d e
# the b y t e s i n t o a f i r s t c h u n k o f ' l e n ' b y t e s a n d a s e c o n d c h u n k o f 1 6
# bytes, t h e n f o l d t h e f i r s t c h u n k i n t o t h e s e c o n d .
2013-09-07 12:56:26 +10:00
movdqa % x m m 7 , % x m m 2
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# xmm1 = l a s t 1 6 o r i g i n a l d a t a b y t e s
movdqu - 1 6 ( b u f , l e n ) , % x m m 1
pshufb B S W A P _ M A S K , % x m m 1
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# xmm2 = h i g h o r d e r p a r t o f s e c o n d c h u n k : x m m 7 l e f t - s h i f t e d b y ' l e n ' b y t e s .
lea . L b y t e s h i f t _ t a b l e + 1 6 ( % r i p ) , % r a x
sub l e n , % r a x
2013-09-07 12:56:26 +10:00
movdqu ( % r a x ) , % x m m 0
pshufb % x m m 0 , % x m m 2
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# xmm7 = f i r s t c h u n k : x m m 7 r i g h t - s h i f t e d b y ' 1 6 - l e n ' b y t e s .
pxor . L m a s k 1 ( % r i p ) , % x m m 0
2013-09-07 12:56:26 +10:00
pshufb % x m m 0 , % x m m 7
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# xmm1 = s e c o n d c h u n k : ' l e n ' b y t e s f r o m x m m 1 ( l o w - o r d e r b y t e s ) ,
# then ' 1 6 - l e n ' b y t e s f r o m x m m 2 ( h i g h - o r d e r b y t e s ) .
2013-09-07 12:56:26 +10:00
pblendvb % x m m 2 , % x m m 1 #x m m 0 i s i m p l i c i t
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Fold t h e f i r s t c h u n k i n t o t h e s e c o n d c h u n k , s t o r i n g t h e r e s u l t i n x m m 7 .
2013-09-07 12:56:26 +10:00
movdqa % x m m 7 , % x m m 8
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 7
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , % x m m 8
2013-09-07 12:56:26 +10:00
pxor % x m m 8 , % x m m 7
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
pxor % x m m 1 , % x m m 7
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
.Lreduce_final_16_bytes :
# Reduce t h e 1 2 8 - b i t v a l u e M ( x ) , s t o r e d i n x m m 7 , t o t h e f i n a l 1 6 - b i t C R C
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Load ' x ^ 4 8 * ( x ^ 4 8 m o d G ( x ) ) ' a n d ' x ^ 4 8 * ( x ^ 8 0 m o d G ( x ) ) ' .
movdqa . L f i n a l _ f o l d _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Fold t h e h i g h 6 4 b i t s i n t o t h e l o w 6 4 b i t s , w h i l e a l s o m u l t i p l y i n g b y
# x^ 6 4 . T h i s p r o d u c e s a 1 2 8 - b i t v a l u e c o n g r u e n t t o x ^ 6 4 * M ( x ) a n d
# whose l o w 4 8 b i t s a r e 0 .
2013-09-07 12:56:26 +10:00
movdqa % x m m 7 , % x m m 0
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 7 # h i g h b i t s * x ^ 48 * ( x ^ 8 0 m o d G ( x ) )
pslldq $ 8 , % x m m 0
pxor % x m m 0 , % x m m 7 # + l o w b i t s * x ^ 64
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Fold t h e h i g h 3 2 b i t s i n t o t h e l o w 9 6 b i t s . T h i s p r o d u c e s a 9 6 - b i t
# value c o n g r u e n t t o x ^ 6 4 * M ( x ) a n d w h o s e l o w 4 8 b i t s a r e 0 .
2013-09-07 12:56:26 +10:00
movdqa % x m m 7 , % x m m 0
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
pand . L m a s k 2 ( % r i p ) , % x m m 0 # z e r o h i g h 32 b i t s
psrldq $ 1 2 , % x m m 7 # e x t r a c t h i g h 32 b i t s
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , % x m m 7 # h i g h 32 b i t s * x ^ 4 8 * ( x ^ 4 8 m o d G ( x ) )
pxor % x m m 0 , % x m m 7 # + l o w b i t s
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Load G ( x ) a n d f l o o r ( x ^ 4 8 / G ( x ) ) .
movdqa . L b a r r e t t _ r e d u c t i o n _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Use B a r r e t t r e d u c t i o n t o c o m p u t e t h e f i n a l C R C v a l u e .
movdqa % x m m 7 , % x m m 0
pclmulqdq $ 0 x11 , F O L D _ C O N S T S , % x m m 7 # h i g h 32 b i t s * f l o o r ( x ^ 4 8 / G ( x ) )
psrlq $ 3 2 , % x m m 7 # / = x ^ 32
pclmulqdq $ 0 x00 , F O L D _ C O N S T S , % x m m 7 # * = G ( x )
psrlq $ 4 8 , % x m m 0
pxor % x m m 7 , % x m m 0 # + l o w 16 n o n z e r o b i t s
# Final C R C v a l u e ( x ^ 1 6 * M ( x ) ) m o d G ( x ) i s i n l o w 1 6 b i t s o f x m m 0 .
pextrw $ 0 , % x m m 0 , % e a x
2013-09-07 12:56:26 +10:00
ret
.align 16
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
.Lless_than_256_bytes :
# Checksumming a b u f f e r o f l e n g t h 1 6 . . . 2 5 5 b y t e s
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Load t h e f i r s t 1 6 d a t a b y t e s .
movdqu ( b u f ) , % x m m 7
pshufb B S W A P _ M A S K , % x m m 7
add $ 1 6 , b u f
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# XOR t h e f i r s t 1 6 d a t a * b i t s * w i t h t h e i n i t i a l C R C v a l u e .
pxor % x m m 0 , % x m m 0
pinsrw $ 7 , i n i t _ c r c , % x m m 0
2013-09-07 12:56:26 +10:00
pxor % x m m 0 , % x m m 7
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
movdqa . L f o l d _ a c r o s s _ 1 6 _ b y t e s _ c o n s t s ( % r i p ) , F O L D _ C O N S T S
cmp $ 1 6 , l e n
je . L r e d u c e _ f i n a l _ 1 6 _ b y t e s # l e n = = 16
sub $ 3 2 , l e n
jge . L f o l d _ 1 6 _ b y t e s _ l o o p # 32 < = l e n < = 2 5 5
add $ 1 6 , l e n
jmp . L h a n d l e _ p a r t i a l _ s e g m e n t # 17 < = l e n < = 3 1
2019-10-11 13:51:04 +02:00
SYM_ F U N C _ E N D ( c r c _ t 1 0 d i f _ p c l )
2013-09-07 12:56:26 +10:00
crypto: x86 - make constants readonly, allow linker to merge them
A lot of asm-optimized routines in arch/x86/crypto/ keep its
constants in .data. This is wrong, they should be on .rodata.
Mnay of these constants are the same in different modules.
For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
exists in at least half a dozen places.
There is a way to let linker merge them and use just one copy.
The rules are as follows: mergeable objects of different sizes
should not share sections. You can't put them all in one .rodata
section, they will lose "mergeability".
GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
This patch does the same:
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
It is important that all data in such section consists of
16-byte elements, not larger ones, and there are no implicit
use of one element from another.
When this is not the case, use non-mergeable section:
.section .rodata[.VAR_NAME], "a", @progbits
This reduces .data by ~15 kbytes:
text data bss dec hex filename
11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
11112095 2690672 2630712 16433479 fac147 vmlinux.o
Merged objects are visible in System.map:
ffffffff81a28810 r POLY
ffffffff81a28810 r POLY
ffffffff81a28820 r TWOONE
ffffffff81a28820 r TWOONE
ffffffff81a28830 r PSHUFFLE_BYTE_FLIP_MASK <- merged regardless of
ffffffff81a28830 r SHUF_MASK <------------- the name difference
ffffffff81a28830 r SHUF_MASK
ffffffff81a28830 r SHUF_MASK
..
ffffffff81a28d00 r K512 <- merged three identical 640-byte tables
ffffffff81a28d00 r K512
ffffffff81a28d00 r K512
Use of object names in section name suffixes is not strictly necessary,
but might help if someday link stage will use garbage collection
to eliminate unused sections (ld --gc-sections).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Josh Poimboeuf <jpoimboe@redhat.com>
CC: Xiaodong Liu <xiaodong.liu@intel.com>
CC: Megha Dey <megha.dey@intel.com>
CC: linux-crypto@vger.kernel.org
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2017-01-19 22:33:04 +01:00
.section .rodata , " a" , @progbits
.align 16
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# Fold c o n s t a n t s p r e c o m p u t e d f r o m t h e p o l y n o m i a l 0 x18 b b7
# G( x ) = x ^ 1 6 + x ^ 1 5 + x ^ 1 1 + x ^ 9 + x ^ 8 + x ^ 7 + x ^ 5 + x ^ 4 + x ^ 2 + x ^ 1 + x ^ 0
.Lfold_across_128_bytes_consts :
.quad 0x0000000000006123 # x^ ( 8 * 1 2 8 ) m o d G ( x )
.quad 0x0000000000002295 # x^ ( 8 * 1 2 8 + 6 4 ) m o d G ( x )
.Lfold_across_64_bytes_consts :
.quad 0x0000000000001069 # x^ ( 4 * 1 2 8 ) m o d G ( x )
.quad 0x000000000000dd31 # x^ ( 4 * 1 2 8 + 6 4 ) m o d G ( x )
.Lfold_across_32_bytes_consts :
.quad 0x000000000000857d # x^ ( 2 * 1 2 8 ) m o d G ( x )
.quad 0x0000000000007acc # x^ ( 2 * 1 2 8 + 6 4 ) m o d G ( x )
.Lfold_across_16_bytes_consts :
.quad 0x000000000000a010 # x^ ( 1 * 1 2 8 ) m o d G ( x )
.quad 0x0000000000001faa # x^ ( 1 * 1 2 8 + 6 4 ) m o d G ( x )
.Lfinal_fold_consts :
.quad 0x1368000000000000 # x^ 4 8 * ( x ^ 4 8 m o d G ( x ) )
.quad 0x2d56000000000000 # x^ 4 8 * ( x ^ 8 0 m o d G ( x ) )
.Lbarrett_reduction_consts :
.quad 0x0000000000018bb7 # G( x )
.quad 0x00000001f65a57f8 # floor( x ^ 4 8 / G ( x ) )
2013-09-07 12:56:26 +10:00
crypto: x86 - make constants readonly, allow linker to merge them
A lot of asm-optimized routines in arch/x86/crypto/ keep its
constants in .data. This is wrong, they should be on .rodata.
Mnay of these constants are the same in different modules.
For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
exists in at least half a dozen places.
There is a way to let linker merge them and use just one copy.
The rules are as follows: mergeable objects of different sizes
should not share sections. You can't put them all in one .rodata
section, they will lose "mergeability".
GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
This patch does the same:
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
It is important that all data in such section consists of
16-byte elements, not larger ones, and there are no implicit
use of one element from another.
When this is not the case, use non-mergeable section:
.section .rodata[.VAR_NAME], "a", @progbits
This reduces .data by ~15 kbytes:
text data bss dec hex filename
11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
11112095 2690672 2630712 16433479 fac147 vmlinux.o
Merged objects are visible in System.map:
ffffffff81a28810 r POLY
ffffffff81a28810 r POLY
ffffffff81a28820 r TWOONE
ffffffff81a28820 r TWOONE
ffffffff81a28830 r PSHUFFLE_BYTE_FLIP_MASK <- merged regardless of
ffffffff81a28830 r SHUF_MASK <------------- the name difference
ffffffff81a28830 r SHUF_MASK
ffffffff81a28830 r SHUF_MASK
..
ffffffff81a28d00 r K512 <- merged three identical 640-byte tables
ffffffff81a28d00 r K512
ffffffff81a28d00 r K512
Use of object names in section name suffixes is not strictly necessary,
but might help if someday link stage will use garbage collection
to eliminate unused sections (ld --gc-sections).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Josh Poimboeuf <jpoimboe@redhat.com>
CC: Xiaodong Liu <xiaodong.liu@intel.com>
CC: Megha Dey <megha.dey@intel.com>
CC: linux-crypto@vger.kernel.org
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2017-01-19 22:33:04 +01:00
.section .rodata .cst16 .mask1 , " aM" , @progbits, 16
.align 16
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
.Lmask1 :
.octa 0x80808080808080808080808080808080
crypto: x86 - make constants readonly, allow linker to merge them
A lot of asm-optimized routines in arch/x86/crypto/ keep its
constants in .data. This is wrong, they should be on .rodata.
Mnay of these constants are the same in different modules.
For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
exists in at least half a dozen places.
There is a way to let linker merge them and use just one copy.
The rules are as follows: mergeable objects of different sizes
should not share sections. You can't put them all in one .rodata
section, they will lose "mergeability".
GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
This patch does the same:
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
It is important that all data in such section consists of
16-byte elements, not larger ones, and there are no implicit
use of one element from another.
When this is not the case, use non-mergeable section:
.section .rodata[.VAR_NAME], "a", @progbits
This reduces .data by ~15 kbytes:
text data bss dec hex filename
11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
11112095 2690672 2630712 16433479 fac147 vmlinux.o
Merged objects are visible in System.map:
ffffffff81a28810 r POLY
ffffffff81a28810 r POLY
ffffffff81a28820 r TWOONE
ffffffff81a28820 r TWOONE
ffffffff81a28830 r PSHUFFLE_BYTE_FLIP_MASK <- merged regardless of
ffffffff81a28830 r SHUF_MASK <------------- the name difference
ffffffff81a28830 r SHUF_MASK
ffffffff81a28830 r SHUF_MASK
..
ffffffff81a28d00 r K512 <- merged three identical 640-byte tables
ffffffff81a28d00 r K512
ffffffff81a28d00 r K512
Use of object names in section name suffixes is not strictly necessary,
but might help if someday link stage will use garbage collection
to eliminate unused sections (ld --gc-sections).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Josh Poimboeuf <jpoimboe@redhat.com>
CC: Xiaodong Liu <xiaodong.liu@intel.com>
CC: Megha Dey <megha.dey@intel.com>
CC: linux-crypto@vger.kernel.org
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2017-01-19 22:33:04 +01:00
.section .rodata .cst16 .mask2 , " aM" , @progbits, 16
.align 16
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
.Lmask2 :
.octa 0x00000000FFFFFFFFFFFFFFFFFFFFFFFF
.section .rodata .cst16 .bswap_mask , " aM" , @progbits, 16
.align 16
.Lbswap_mask :
.octa 0x000102030405060708090A0B0C0D0E0F
2013-09-07 12:56:26 +10:00
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
.section .rodata .cst32 .byteshift_table , " aM" , @progbits, 32
crypto: x86 - make constants readonly, allow linker to merge them
A lot of asm-optimized routines in arch/x86/crypto/ keep its
constants in .data. This is wrong, they should be on .rodata.
Mnay of these constants are the same in different modules.
For example, 128-bit shuffle mask 0x000102030405060708090A0B0C0D0E0F
exists in at least half a dozen places.
There is a way to let linker merge them and use just one copy.
The rules are as follows: mergeable objects of different sizes
should not share sections. You can't put them all in one .rodata
section, they will lose "mergeability".
GCC puts its mergeable constants in ".rodata.cstSIZE" sections,
or ".rodata.cstSIZE.<object_name>" if -fdata-sections is used.
This patch does the same:
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
It is important that all data in such section consists of
16-byte elements, not larger ones, and there are no implicit
use of one element from another.
When this is not the case, use non-mergeable section:
.section .rodata[.VAR_NAME], "a", @progbits
This reduces .data by ~15 kbytes:
text data bss dec hex filename
11097415 2705840 2630712 16433967 fac32f vmlinux-prev.o
11112095 2690672 2630712 16433479 fac147 vmlinux.o
Merged objects are visible in System.map:
ffffffff81a28810 r POLY
ffffffff81a28810 r POLY
ffffffff81a28820 r TWOONE
ffffffff81a28820 r TWOONE
ffffffff81a28830 r PSHUFFLE_BYTE_FLIP_MASK <- merged regardless of
ffffffff81a28830 r SHUF_MASK <------------- the name difference
ffffffff81a28830 r SHUF_MASK
ffffffff81a28830 r SHUF_MASK
..
ffffffff81a28d00 r K512 <- merged three identical 640-byte tables
ffffffff81a28d00 r K512
ffffffff81a28d00 r K512
Use of object names in section name suffixes is not strictly necessary,
but might help if someday link stage will use garbage collection
to eliminate unused sections (ld --gc-sections).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Josh Poimboeuf <jpoimboe@redhat.com>
CC: Xiaodong Liu <xiaodong.liu@intel.com>
CC: Megha Dey <megha.dey@intel.com>
CC: linux-crypto@vger.kernel.org
CC: x86@kernel.org
CC: linux-kernel@vger.kernel.org
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2017-01-19 22:33:04 +01:00
.align 16
crypto: x86/crct10dif-pcl - cleanup and optimizations
The x86, arm, and arm64 asm implementations of crct10dif are very
difficult to understand partly because many of the comments, labels, and
macros are named incorrectly: the lengths mentioned are usually off by a
factor of two from the actual code. Many other things are unnecessarily
convoluted as well, e.g. there are many more fold constants than
actually needed and some aren't fully reduced.
This series therefore cleans up all these implementations to be much
more maintainable. I also made some small optimizations where I saw
opportunities, resulting in slightly better performance.
This patch cleans up the x86 version.
As part of this, I removed support for len < 16 from the x86 assembly;
now the glue code falls back to the generic table-based implementation
in this case. Due to the overhead of kernel_fpu_begin(), this actually
significantly improves performance on these lengths. (And even if
kernel_fpu_begin() were free, the generic code is still faster for about
len < 11.) This removal also eliminates error-prone special cases and
makes the x86, arm32, and arm64 ports of the code match more closely.
Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2019-01-30 20:42:40 -08:00
# For 1 < = l e n < = 1 5 , t h e 1 6 - b y t e v e c t o r b e g i n n i n g a t & b y t e s h i f t _ t a b l e [ 1 6 - l e n ]
# is t h e i n d e x v e c t o r t o s h i f t l e f t b y ' l e n ' b y t e s , a n d i s a l s o { 0 x80 , . . . ,
# 0 x8 0 } X O R t h e i n d e x v e c t o r t o s h i f t r i g h t b y ' 1 6 - l e n ' b y t e s .
.Lbyteshift_table :
.byte 0 x0 , 0 x81 , 0 x82 , 0 x83 , 0 x84 , 0 x85 , 0 x86 , 0 x87
.byte 0 x8 8 , 0 x89 , 0 x8 a , 0 x8 b , 0 x8 c , 0 x8 d , 0 x8 e , 0 x8 f
.byte 0 x0 , 0 x1 , 0 x2 , 0 x3 , 0 x4 , 0 x5 , 0 x6 , 0 x7
.byte 0 x8 , 0 x9 , 0 x a , 0 x b , 0 x c , 0 x d , 0 x e , 0 x0
