iv/linux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

Browse Source 
Pull crypto update from Herbert Xu:
 "API:
   - Add helper for simple skcipher modes.
   - Add helper to register multiple templates.
   - Set CRYPTO_TFM_NEED_KEY when setkey fails.
   - Require neither or both of export/import in shash.
   - AEAD decryption test vectors are now generated from encryption
     ones.
   - New option CONFIG_CRYPTO_MANAGER_EXTRA_TESTS that includes random
     fuzzing.

  Algorithms:
   - Conversions to skcipher and helper for many templates.
   - Add more test vectors for nhpoly1305 and adiantum.

  Drivers:
   - Add crypto4xx prng support.
   - Add xcbc/cmac/ecb support in caam.
   - Add AES support for Exynos5433 in s5p.
   - Remove sha384/sha512 from artpec7 as hardware cannot do partial
     hash"

[ There is a merge of the Freescale SoC tree in order to pull in changes
  required by patches to the caam/qi2 driver. ]

* 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (174 commits)
  crypto: s5p - add AES support for Exynos5433
  dt-bindings: crypto: document Exynos5433 SlimSSS
  crypto: crypto4xx - add missing of_node_put after of_device_is_available
  crypto: cavium/zip - fix collision with generic cra_driver_name
  crypto: af_alg - use struct_size() in sock_kfree_s()
  crypto: caam - remove redundant likely/unlikely annotation
  crypto: s5p - update iv after AES-CBC op end
  crypto: x86/poly1305 - Clear key material from stack in SSE2 variant
  crypto: caam - generate hash keys in-place
  crypto: caam - fix DMA mapping xcbc key twice
  crypto: caam - fix hash context DMA unmap size
  hwrng: bcm2835 - fix probe as platform device
  crypto: s5p-sss - Use AES_BLOCK_SIZE define instead of number
  crypto: stm32 - drop pointless static qualifier in stm32_hash_remove()
  crypto: chelsio - Fixed Traffic Stall
  crypto: marvell - Remove set but not used variable 'ivsize'
  crypto: ccp - Update driver messages to remove some confusion
  crypto: adiantum - add 1536 and 4096-byte test vectors
  crypto: nhpoly1305 - add a test vector with len % 16 != 0
  crypto: arm/aes-ce - update IV after partial final CTR block
  ...
This commit is contained in:
Linus Torvalds 2019-03-05 09:09:55 -08:00
commit 63bdf4284c
170 changed files with 8942 additions and 11459 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
Documentation/devicetree/bindings/crypto/samsung-slimsss.txt Normal file
View File

@ -0,0 +1,19 @@
Samsung SoC SlimSSS (Slim Security SubSystem) module
The SlimSSS module in Exynos5433 SoC supports the following:
-- Feeder (FeedCtrl)
-- Advanced Encryption Standard (AES) with ECB,CBC,CTR,XTS and (CBC/XTS)/CTS
-- SHA-1/SHA-256 and (SHA-1/SHA-256)/HMAC
Required properties:
- compatible : Should contain entry for slimSSS version:
- "samsung,exynos5433-slim-sss" for Exynos5433 SoC.
- reg : Offset and length of the register set for the module
- interrupts : interrupt specifiers of SlimSSS module interrupts (one feed
control interrupt).
- clocks : list of clock phandle and specifier pairs for all clocks listed in
clock-names property.
- clock-names : list of device clock input names; should contain "pclk" and
"aclk" for slim-sss in Exynos5433.

1
MAINTAINERS
View File

@ -3529,7 +3529,6 @@ F: include/linux/spi/cc2520.h
F: Documentation/devicetree/bindings/net/ieee802154/cc2520.txt
CCREE ARM TRUSTZONE CRYPTOCELL REE DRIVER
M: Yael Chemla <yael.chemla@foss.arm.com>
M: Gilad Ben-Yossef <gilad@benyossef.com>
L: linux-crypto@vger.kernel.org
S: Supported

26
arch/arm/crypto/aes-ce-core.S
View File

@ -317,25 +317,27 @@ ENTRY(ce_aes_ctr_encrypt)
.Lctrloop:
vmov q0, q6
bl aes_encrypt
subs r4, r4, #1
bmi .Lctrtailblock @ blocks < 0 means tail block
vld1.8 {q3}, [r1]!
veor q3, q0, q3
vst1.8 {q3}, [r0]!
adds r6, r6, #1 @ increment BE ctr
rev ip, r6
vmov s27, ip
bcs .Lctrcarry
teq r4, #0
.Lctrcarrydone:
subs r4, r4, #1
bmi .Lctrtailblock @ blocks < 0 means tail block
vld1.8 {q3}, [r1]!
veor q3, q0, q3
vst1.8 {q3}, [r0]!
bne .Lctrloop
.Lctrout:
vst1.8 {q6}, [r5]
vst1.8 {q6}, [r5] @ return next CTR value
pop {r4-r6, pc}
.Lctrtailblock:
vst1.8 {q0}, [r0, :64] @ return just the key stream
pop {r4-r6, pc}
vst1.8 {q0}, [r0, :64] @ return the key stream
b .Lctrout
.Lctrcarry:
.irp sreg, s26, s25, s24
@ -344,11 +346,9 @@ ENTRY(ce_aes_ctr_encrypt)
adds ip, ip, #1
rev ip, ip
vmov \sreg, ip
bcc 0f
bcc .Lctrcarrydone
.endr
0: teq r4, #0
beq .Lctrout
b .Lctrloop
b .Lctrcarrydone
ENDPROC(ce_aes_ctr_encrypt)
/*

524
arch/arm/crypto/crct10dif-ce-core.S
View File

@ -2,12 +2,14 @@
// Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions
//
// Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
// Copyright (C) 2019 Google LLC <ebiggers@google.com>
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License version 2 as
// published by the Free Software Foundation.
//
// Derived from the x86 version:
//
// Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions
//
@ -54,19 +56,11 @@
// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
// Function API:
// UINT16 crc_t10dif_pcl(
// UINT16 init_crc, //initial CRC value, 16 bits
// const unsigned char *buf, //buffer pointer to calculate CRC on
// UINT64 len //buffer length in bytes (64-bit data)
// );
//
// Reference paper titled "Fast CRC Computation for Generic
// Polynomials Using PCLMULQDQ Instruction"
// URL: http://www.intel.com/content/dam/www/public/us/en/documents
// /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
//
//
#include <linux/linkage.h>
#include <asm/assembler.h>
@ -78,13 +72,14 @@
#endif
.text
.arch armv7-a
.fpu crypto-neon-fp-armv8
arg1_low32 .req r0
arg2 .req r1
arg3 .req r2
init_crc .req r0
buf .req r1
len .req r2
qzr .req q13
fold_consts_ptr .req ip
q0l .req d0
q0h .req d1
@ -102,82 +97,35 @@
q6h .req d13
q7l .req d14
q7h .req d15
q8l .req d16
q8h .req d17
q9l .req d18
q9h .req d19
q10l .req d20
q10h .req d21
q11l .req d22
q11h .req d23
q12l .req d24
q12h .req d25
ENTRY(crc_t10dif_pmull)
vmov.i8 qzr, #0 // init zero register
FOLD_CONSTS .req q10
FOLD_CONST_L .req q10l
FOLD_CONST_H .req q10h
// adjust the 16-bit initial_crc value, scale it to 32 bits
lsl arg1_low32, arg1_low32, #16
// Fold reg1, reg2 into the next 32 data bytes, storing the result back
// into reg1, reg2.
.macro fold_32_bytes, reg1, reg2
vld1.64 {q11-q12}, [buf]!
// check if smaller than 256
cmp arg3, #256
vmull.p64 q8, \reg1\()h, FOLD_CONST_H
vmull.p64 \reg1, \reg1\()l, FOLD_CONST_L
vmull.p64 q9, \reg2\()h, FOLD_CONST_H
vmull.p64 \reg2, \reg2\()l, FOLD_CONST_L
// for sizes less than 128, we can't fold 64B at a time...
blt _less_than_128
// load the initial crc value
// crc value does not need to be byte-reflected, but it needs
// to be moved to the high part of the register.
// because data will be byte-reflected and will align with
// initial crc at correct place.
vmov s0, arg1_low32 // initial crc
vext.8 q10, qzr, q0, #4
// receive the initial 64B data, xor the initial crc value
vld1.64 {q0-q1}, [arg2, :128]!
vld1.64 {q2-q3}, [arg2, :128]!
vld1.64 {q4-q5}, [arg2, :128]!
vld1.64 {q6-q7}, [arg2, :128]!
CPU_LE( vrev64.8 q0, q0 )
CPU_LE( vrev64.8 q1, q1 )
CPU_LE( vrev64.8 q2, q2 )
CPU_LE( vrev64.8 q3, q3 )
CPU_LE( vrev64.8 q4, q4 )
CPU_LE( vrev64.8 q5, q5 )
CPU_LE( vrev64.8 q6, q6 )
CPU_LE( vrev64.8 q7, q7 )
vswp d0, d1
vswp d2, d3
vswp d4, d5
vswp d6, d7
vswp d8, d9
vswp d10, d11
vswp d12, d13
vswp d14, d15
// XOR the initial_crc value
veor.8 q0, q0, q10
adr ip, rk3
vld1.64 {q10}, [ip, :128] // xmm10 has rk3 and rk4
//
// we subtract 256 instead of 128 to save one instruction from the loop
//
sub arg3, arg3, #256
// at this section of the code, there is 64*x+y (0<=y<64) bytes of
// buffer. The _fold_64_B_loop will fold 64B at a time
// until we have 64+y Bytes of buffer
// fold 64B at a time. This section of the code folds 4 vector
// registers in parallel
_fold_64_B_loop:
.macro fold64, reg1, reg2
vld1.64 {q11-q12}, [arg2, :128]!
vmull.p64 q8, \reg1\()h, d21
vmull.p64 \reg1, \reg1\()l, d20
vmull.p64 q9, \reg2\()h, d21
vmull.p64 \reg2, \reg2\()l, d20
CPU_LE( vrev64.8 q11, q11 )
CPU_LE( vrev64.8 q12, q12 )
vswp d22, d23
vswp d24, d25
CPU_LE( vrev64.8 q11, q11 )
CPU_LE( vrev64.8 q12, q12 )
vswp q11l, q11h
vswp q12l, q12h
veor.8 \reg1, \reg1, q8
veor.8 \reg2, \reg2, q9
@ -185,242 +133,248 @@ CPU_LE( vrev64.8 q12, q12 )
veor.8 \reg2, \reg2, q12
.endm
fold64 q0, q1
fold64 q2, q3
fold64 q4, q5
fold64 q6, q7
subs arg3, arg3, #128
// check if there is another 64B in the buffer to be able to fold
bge _fold_64_B_loop
// at this point, the buffer pointer is pointing at the last y Bytes
// of the buffer the 64B of folded data is in 4 of the vector
// registers: v0, v1, v2, v3
// fold the 8 vector registers to 1 vector register with different
// constants
adr ip, rk9
vld1.64 {q10}, [ip, :128]!
.macro fold16, reg, rk
vmull.p64 q8, \reg\()l, d20
vmull.p64 \reg, \reg\()h, d21
.ifnb \rk
vld1.64 {q10}, [ip, :128]!
// Fold src_reg into dst_reg, optionally loading the next fold constants
.macro fold_16_bytes, src_reg, dst_reg, load_next_consts
vmull.p64 q8, \src_reg\()l, FOLD_CONST_L
vmull.p64 \src_reg, \src_reg\()h, FOLD_CONST_H
.ifnb \load_next_consts
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
.endif
veor.8 q7, q7, q8
veor.8 q7, q7, \reg
veor.8 \dst_reg, \dst_reg, q8
veor.8 \dst_reg, \dst_reg, \src_reg
.endm
fold16 q0, rk11
fold16 q1, rk13
fold16 q2, rk15
fold16 q3, rk17
fold16 q4, rk19
fold16 q5, rk1
fold16 q6
.macro __adrl, out, sym
movw \out, #:lower16:\sym
movt \out, #:upper16:\sym
.endm
// instead of 64, we add 48 to the loop counter to save 1 instruction
// from the loop instead of a cmp instruction, we use the negative
// flag with the jl instruction
adds arg3, arg3, #(128-16)
blt _final_reduction_for_128
//
// u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
ENTRY(crc_t10dif_pmull)
// now we have 16+y bytes left to reduce. 16 Bytes is in register v7
// and the rest is in memory. We can fold 16 bytes at a time if y>=16
// continue folding 16B at a time
// For sizes less than 256 bytes, we can't fold 128 bytes at a time.
cmp len, #256
blt .Lless_than_256_bytes
_16B_reduction_loop:
vmull.p64 q8, d14, d20
vmull.p64 q7, d15, d21
__adrl fold_consts_ptr, .Lfold_across_128_bytes_consts
// Load the first 128 data bytes. Byte swapping is necessary to make
// the bit order match the polynomial coefficient order.
vld1.64 {q0-q1}, [buf]!
vld1.64 {q2-q3}, [buf]!
vld1.64 {q4-q5}, [buf]!
vld1.64 {q6-q7}, [buf]!
CPU_LE( vrev64.8 q0, q0 )
CPU_LE( vrev64.8 q1, q1 )
CPU_LE( vrev64.8 q2, q2 )
CPU_LE( vrev64.8 q3, q3 )
CPU_LE( vrev64.8 q4, q4 )
CPU_LE( vrev64.8 q5, q5 )
CPU_LE( vrev64.8 q6, q6 )
CPU_LE( vrev64.8 q7, q7 )
vswp q0l, q0h
vswp q1l, q1h
vswp q2l, q2h
vswp q3l, q3h
vswp q4l, q4h
vswp q5l, q5h
vswp q6l, q6h
vswp q7l, q7h
// XOR the first 16 data *bits* with the initial CRC value.
vmov.i8 q8h, #0
vmov.u16 q8h[3], init_crc
veor q0h, q0h, q8h
// Load the constants for folding across 128 bytes.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
// Subtract 128 for the 128 data bytes just consumed. Subtract another
// 128 to simplify the termination condition of the following loop.
sub len, len, #256
// While >= 128 data bytes remain (not counting q0-q7), fold the 128
// bytes q0-q7 into them, storing the result back into q0-q7.
.Lfold_128_bytes_loop:
fold_32_bytes q0, q1
fold_32_bytes q2, q3
fold_32_bytes q4, q5
fold_32_bytes q6, q7
subs len, len, #128
bge .Lfold_128_bytes_loop
// Now fold the 112 bytes in q0-q6 into the 16 bytes in q7.
// Fold across 64 bytes.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
fold_16_bytes q0, q4
fold_16_bytes q1, q5
fold_16_bytes q2, q6
fold_16_bytes q3, q7, 1
// Fold across 32 bytes.
fold_16_bytes q4, q6
fold_16_bytes q5, q7, 1
// Fold across 16 bytes.
fold_16_bytes q6, q7
// Add 128 to get the correct number of data bytes remaining in 0...127
// (not counting q7), following the previous extra subtraction by 128.
// Then subtract 16 to simplify the termination condition of the
// following loop.
adds len, len, #(128-16)
// While >= 16 data bytes remain (not counting q7), fold the 16 bytes q7
// into them, storing the result back into q7.
blt .Lfold_16_bytes_loop_done
.Lfold_16_bytes_loop:
vmull.p64 q8, q7l, FOLD_CONST_L
vmull.p64 q7, q7h, FOLD_CONST_H
veor.8 q7, q7, q8
vld1.64 {q0}, [arg2, :128]!
CPU_LE( vrev64.8 q0, q0 )
vswp d0, d1
vld1.64 {q0}, [buf]!
CPU_LE( vrev64.8 q0, q0 )
vswp q0l, q0h
veor.8 q7, q7, q0
subs arg3, arg3, #16
subs len, len, #16
bge .Lfold_16_bytes_loop
// instead of a cmp instruction, we utilize the flags with the
// jge instruction equivalent of: cmp arg3, 16-16
// check if there is any more 16B in the buffer to be able to fold
bge _16B_reduction_loop
.Lfold_16_bytes_loop_done:
// Add 16 to get the correct number of data bytes remaining in 0...15
// (not counting q7), following the previous extra subtraction by 16.
adds len, len, #16
beq .Lreduce_final_16_bytes
// now we have 16+z bytes left to reduce, where 0<= z < 16.
// first, we reduce the data in the xmm7 register
.Lhandle_partial_segment:
// Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first
// 16 bytes are in q7 and the rest are the remaining data in 'buf'. To
// do this without needing a fold constant for each possible 'len',
// redivide the bytes into a first chunk of 'len' bytes and a second
// chunk of 16 bytes, then fold the first chunk into the second.
_final_reduction_for_128:
// check if any more data to fold. If not, compute the CRC of
// the final 128 bits
adds arg3, arg3, #16
beq _128_done
// q0 = last 16 original data bytes
add buf, buf, len
sub buf, buf, #16
vld1.64 {q0}, [buf]
CPU_LE( vrev64.8 q0, q0 )
vswp q0l, q0h
// here we are getting data that is less than 16 bytes.
// since we know that there was data before the pointer, we can
// offset the input pointer before the actual point, to receive
// exactly 16 bytes. after that the registers need to be adjusted.
_get_last_two_regs:
add arg2, arg2, arg3
sub arg2, arg2, #16
vld1.64 {q1}, [arg2]
CPU_LE( vrev64.8 q1, q1 )
vswp d2, d3
// q1 = high order part of second chunk: q7 left-shifted by 'len' bytes.
__adrl r3, .Lbyteshift_table + 16
sub r3, r3, len
vld1.8 {q2}, [r3]
vtbl.8 q1l, {q7l-q7h}, q2l
vtbl.8 q1h, {q7l-q7h}, q2h
// get rid of the extra data that was loaded before
// load the shift constant
adr ip, tbl_shf_table + 16
sub ip, ip, arg3
vld1.8 {q0}, [ip]
// q3 = first chunk: q7 right-shifted by '16-len' bytes.
vmov.i8 q3, #0x80
veor.8 q2, q2, q3
vtbl.8 q3l, {q7l-q7h}, q2l
vtbl.8 q3h, {q7l-q7h}, q2h
// shift v2 to the left by arg3 bytes
vtbl.8 d4, {d14-d15}, d0
vtbl.8 d5, {d14-d15}, d1
// Convert to 8-bit masks: 'len' 0x00 bytes, then '16-len' 0xff bytes.
vshr.s8 q2, q2, #7
// shift v7 to the right by 16-arg3 bytes
vmov.i8 q9, #0x80
veor.8 q0, q0, q9
vtbl.8 d18, {d14-d15}, d0
vtbl.8 d19, {d14-d15}, d1
// q2 = second chunk: 'len' bytes from q0 (low-order bytes),
// then '16-len' bytes from q1 (high-order bytes).
vbsl.8 q2, q1, q0
// blend
vshr.s8 q0, q0, #7 // convert to 8-bit mask
vbsl.8 q0, q2, q1
// fold 16 Bytes
vmull.p64 q8, d18, d20
vmull.p64 q7, d19, d21
veor.8 q7, q7, q8
// Fold the first chunk into the second chunk, storing the result in q7.
vmull.p64 q0, q3l, FOLD_CONST_L
vmull.p64 q7, q3h, FOLD_CONST_H
veor.8 q7, q7, q0
veor.8 q7, q7, q2
_128_done:
// compute crc of a 128-bit value
vldr d20, rk5
vldr d21, rk6 // rk5 and rk6 in xmm10
.Lreduce_final_16_bytes:
// Reduce the 128-bit value M(x), stored in q7, to the final 16-bit CRC.
// 64b fold
vext.8 q0, qzr, q7, #8
vmull.p64 q7, d15, d20
veor.8 q7, q7, q0
// Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
// 32b fold
vext.8 q0, q7, qzr, #12
vmov s31, s3
vmull.p64 q0, d0, d21
veor.8 q7, q0, q7
// Fold the high 64 bits into the low 64 bits, while also multiplying by
// x^64. This produces a 128-bit value congruent to x^64 * M(x) and
// whose low 48 bits are 0.
vmull.p64 q0, q7h, FOLD_CONST_H // high bits * x^48 * (x^80 mod G(x))
veor.8 q0h, q0h, q7l // + low bits * x^64
// barrett reduction
_barrett:
vldr d20, rk7
vldr d21, rk8
// Fold the high 32 bits into the low 96 bits. This produces a 96-bit
// value congruent to x^64 * M(x) and whose low 48 bits are 0.
vmov.i8 q1, #0
vmov s4, s3 // extract high 32 bits
vmov s3, s5 // zero high 32 bits
vmull.p64 q1, q1l, FOLD_CONST_L // high 32 bits * x^48 * (x^48 mod G(x))
veor.8 q0, q0, q1 // + low bits
vmull.p64 q0, d15, d20
vext.8 q0, qzr, q0, #12
vmull.p64 q0, d1, d21
vext.8 q0, qzr, q0, #12
veor.8 q7, q7, q0
vmov r0, s29
// Load G(x) and floor(x^48 / G(x)).
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]
_cleanup:
// scale the result back to 16 bits
lsr r0, r0, #16
// Use Barrett reduction to compute the final CRC value.
vmull.p64 q1, q0h, FOLD_CONST_H // high 32 bits * floor(x^48 / G(x))
vshr.u64 q1l, q1l, #32 // /= x^32
vmull.p64 q1, q1l, FOLD_CONST_L // *= G(x)
vshr.u64 q0l, q0l, #48
veor.8 q0l, q0l, q1l // + low 16 nonzero bits
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of q0.
vmov.u16 r0, q0l[0]
bx lr
_less_than_128:
teq arg3, #0
beq _cleanup
.Lless_than_256_bytes:
// Checksumming a buffer of length 16...255 bytes
vmov.i8 q0, #0
vmov s3, arg1_low32 // get the initial crc value
__adrl fold_consts_ptr, .Lfold_across_16_bytes_consts
vld1.64 {q7}, [arg2, :128]!
CPU_LE( vrev64.8 q7, q7 )
vswp d14, d15
veor.8 q7, q7, q0
// Load the first 16 data bytes.
vld1.64 {q7}, [buf]!
CPU_LE( vrev64.8 q7, q7 )
vswp q7l, q7h
cmp arg3, #16
beq _128_done // exactly 16 left
blt _less_than_16_left
// XOR the first 16 data *bits* with the initial CRC value.
vmov.i8 q0h, #0
vmov.u16 q0h[3], init_crc
veor.8 q7h, q7h, q0h
// now if there is, load the constants
vldr d20, rk1
vldr d21, rk2 // rk1 and rk2 in xmm10
// Load the fold-across-16-bytes constants.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
// check if there is enough buffer to be able to fold 16B at a time
subs arg3, arg3, #32
addlt arg3, arg3, #16
blt _get_last_two_regs
b _16B_reduction_loop
_less_than_16_left:
// shl r9, 4
adr ip, tbl_shf_table + 16
sub ip, ip, arg3
vld1.8 {q0}, [ip]
vmov.i8 q9, #0x80
veor.8 q0, q0, q9
vtbl.8 d18, {d14-d15}, d0
vtbl.8 d15, {d14-d15}, d1
vmov d14, d18
b _128_done
cmp len, #16
beq .Lreduce_final_16_bytes // len == 16
subs len, len, #32
addlt len, len, #16
blt .Lhandle_partial_segment // 17 <= len <= 31
b .Lfold_16_bytes_loop // 32 <= len <= 255
ENDPROC(crc_t10dif_pmull)
// precomputed constants
// these constants are precomputed from the poly:
// 0x8bb70000 (0x8bb7 scaled to 32 bits)
.section ".rodata", "a"
.align 4
// Q = 0x18BB70000
// rk1 = 2^(32*3) mod Q << 32
// rk2 = 2^(32*5) mod Q << 32
// rk3 = 2^(32*15) mod Q << 32
// rk4 = 2^(32*17) mod Q << 32
// rk5 = 2^(32*3) mod Q << 32
// rk6 = 2^(32*2) mod Q << 32
// rk7 = floor(2^64/Q)
// rk8 = Q
rk3: .quad 0x9d9d000000000000
rk4: .quad 0x7cf5000000000000
rk5: .quad 0x2d56000000000000
rk6: .quad 0x1368000000000000
rk7: .quad 0x00000001f65a57f8
rk8: .quad 0x000000018bb70000
rk9: .quad 0xceae000000000000
rk10: .quad 0xbfd6000000000000
rk11: .quad 0x1e16000000000000
rk12: .quad 0x713c000000000000
rk13: .quad 0xf7f9000000000000
rk14: .quad 0x80a6000000000000
rk15: .quad 0x044c000000000000
rk16: .quad 0xe658000000000000
rk17: .quad 0xad18000000000000
rk18: .quad 0xa497000000000000
rk19: .quad 0x6ee3000000000000
rk20: .quad 0xe7b5000000000000
rk1: .quad 0x2d56000000000000
rk2: .quad 0x06df000000000000
tbl_shf_table:
// use these values for shift constants for the tbl/tbx instruction
// different alignments result in values as shown:
// DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1
// DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2
// DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3
// DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4
// DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5
// DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6
// DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
// DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
// DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
// DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
// DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
// DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
// DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
// DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
// DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
// Fold constants precomputed from the polynomial 0x18bb7
// G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
.Lfold_across_128_bytes_consts:
.quad 0x0000000000006123 // x^(8*128) mod G(x)
.quad 0x0000000000002295 // x^(8*128+64) mod G(x)
// .Lfold_across_64_bytes_consts:
.quad 0x0000000000001069 // x^(4*128) mod G(x)
.quad 0x000000000000dd31 // x^(4*128+64) mod G(x)
// .Lfold_across_32_bytes_consts:
.quad 0x000000000000857d // x^(2*128) mod G(x)
.quad 0x0000000000007acc // x^(2*128+64) mod G(x)
.Lfold_across_16_bytes_consts:
.quad 0x000000000000a010 // x^(1*128) mod G(x)
.quad 0x0000000000001faa // x^(1*128+64) mod G(x)
// .Lfinal_fold_consts:
.quad 0x1368000000000000 // x^48 * (x^48 mod G(x))
.quad 0x2d56000000000000 // x^48 * (x^80 mod G(x))
// .Lbarrett_reduction_consts:
.quad 0x0000000000018bb7 // G(x)
.quad 0x00000001f65a57f8 // floor(x^48 / G(x))
// For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 -
// len] is the index vector to shift left by 'len' bytes, and is also {0x80,
// ..., 0x80} XOR the index vector to shift right by '16 - len' bytes.
.Lbyteshift_table:
.byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7

25
arch/arm/crypto/crct10dif-ce-glue.c
View File

@ -21,7 +21,7 @@
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 buf[], u32 len);
asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
static int crct10dif_init(struct shash_desc *desc)
{
@ -35,26 +35,15 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data,
unsigned int length)
{
u16 *crc = shash_desc_ctx(desc);
unsigned int l;
if (!may_use_simd()) {
*crc = crc_t10dif_generic(*crc, data, length);
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
} else {
if (unlikely((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) {
l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE -
((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE));
*crc = crc_t10dif_generic(*crc, data, l);
length -= l;
data += l;
}
if (length > 0) {
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
}
*crc = crc_t10dif_generic(*crc, data, length);
}
return 0;
}

5
arch/arm64/crypto/aes-ce-ccm-core.S
View File

@ -74,12 +74,13 @@ ENTRY(ce_aes_ccm_auth_data)
beq 10f
ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */
b 7b
8: mov w7, w8
8: cbz w8, 91f
mov w7, w8
add w8, w8, #16
9: ext v1.16b, v1.16b, v1.16b, #1
adds w7, w7, #1
bne 9b
eor v0.16b, v0.16b, v1.16b
91: eor v0.16b, v0.16b, v1.16b
st1 {v0.16b}, [x0]
10: str w8, [x3]
ret

8
arch/arm64/crypto/aes-ce-ccm-glue.c
View File

@ -125,7 +125,7 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
abytes -= added;
}
while (abytes > AES_BLOCK_SIZE) {
while (abytes >= AES_BLOCK_SIZE) {
__aes_arm64_encrypt(key->key_enc, mac, mac,
num_rounds(key));
crypto_xor(mac, in, AES_BLOCK_SIZE);
@ -139,8 +139,6 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
num_rounds(key));
crypto_xor(mac, in, abytes);
*macp = abytes;
} else {
*macp = 0;
}
}
}
@ -255,7 +253,7 @@ static int ccm_encrypt(struct aead_request *req)
/* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_encrypt(&walk, req, true);
err = skcipher_walk_aead_encrypt(&walk, req, false);
if (may_use_simd()) {
while (walk.nbytes) {
@ -313,7 +311,7 @@ static int ccm_decrypt(struct aead_request *req)
/* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_decrypt(&walk, req, true);
err = skcipher_walk_aead_decrypt(&walk, req, false);
if (may_use_simd()) {
while (walk.nbytes) {

3
arch/arm64/crypto/aes-modes.S
View File

@ -320,8 +320,7 @@ AES_ENTRY(aes_ctr_encrypt)
.Lctrtailblock:
st1 {v0.16b}, [x0]
ldp x29, x30, [sp], #16
ret
b .Lctrout
.Lctrcarry:
umov x7, v4.d[0] /* load upper word of ctr */

8
arch/arm64/crypto/aes-neonbs-core.S
View File

@ -971,18 +971,22 @@ CPU_LE( rev x8, x8 )
8: next_ctr v0
st1 {v0.16b}, [x24]
cbz x23, 0f
cbz x23, .Lctr_done
cond_yield_neon 98b
b 99b
0: frame_pop
.Lctr_done:
frame_pop
ret
/*
* If we are handling the tail of the input (x6 != NULL), return the
* final keystream block back to the caller.
*/
0: cbz x25, 8b
st1 {v0.16b}, [x25]
b 8b
1: cbz x25, 8b
st1 {v1.16b}, [x25]
b 8b

477
arch/arm64/crypto/crct10dif-ce-core.S
View File

@ -2,12 +2,14 @@
// Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions
//
// Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
// Copyright (C) 2019 Google LLC <ebiggers@google.com>
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License version 2 as
// published by the Free Software Foundation.
//
// Derived from the x86 version:
//
// Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions
//
@ -54,19 +56,11 @@
// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
// Function API:
// UINT16 crc_t10dif_pcl(
// UINT16 init_crc, //initial CRC value, 16 bits
// const unsigned char *buf, //buffer pointer to calculate CRC on
// UINT64 len //buffer length in bytes (64-bit data)
// );
//
// Reference paper titled "Fast CRC Computation for Generic
// Polynomials Using PCLMULQDQ Instruction"
// URL: http://www.intel.com/content/dam/www/public/us/en/documents
// /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
//
//
#include <linux/linkage.h>
#include <asm/assembler.h>
@ -74,14 +68,14 @@
.text
.cpu generic+crypto
arg1_low32 .req w19
arg2 .req x20
arg3 .req x21
init_crc .req w19
buf .req x20
len .req x21
fold_consts_ptr .req x22
vzr .req v13
fold_consts .req v10
ad .req v14
bd .req v10
k00_16 .req v15
k32_48 .req v16
@ -143,11 +137,11 @@ __pmull_p8_core:
ext t5.8b, ad.8b, ad.8b, #2 // A2
ext t6.8b, ad.8b, ad.8b, #3 // A3
pmull t4.8h, t4.8b, bd.8b // F = A1*B
pmull t4.8h, t4.8b, fold_consts.8b // F = A1*B
pmull t8.8h, ad.8b, bd1.8b // E = A*B1
pmull t5.8h, t5.8b, bd.8b // H = A2*B
pmull t5.8h, t5.8b, fold_consts.8b // H = A2*B
pmull t7.8h, ad.8b, bd2.8b // G = A*B2
pmull t6.8h, t6.8b, bd.8b // J = A3*B
pmull t6.8h, t6.8b, fold_consts.8b // J = A3*B
pmull t9.8h, ad.8b, bd3.8b // I = A*B3
pmull t3.8h, ad.8b, bd4.8b // K = A*B4
b 0f
@ -157,11 +151,11 @@ __pmull_p8_core:
tbl t5.16b, {ad.16b}, perm2.16b // A2
tbl t6.16b, {ad.16b}, perm3.16b // A3
pmull2 t4.8h, t4.16b, bd.16b // F = A1*B
pmull2 t4.8h, t4.16b, fold_consts.16b // F = A1*B
pmull2 t8.8h, ad.16b, bd1.16b // E = A*B1
pmull2 t5.8h, t5.16b, bd.16b // H = A2*B
pmull2 t5.8h, t5.16b, fold_consts.16b // H = A2*B
pmull2 t7.8h, ad.16b, bd2.16b // G = A*B2
pmull2 t6.8h, t6.16b, bd.16b // J = A3*B
pmull2 t6.8h, t6.16b, fold_consts.16b // J = A3*B
pmull2 t9.8h, ad.16b, bd3.16b // I = A*B3
pmull2 t3.8h, ad.16b, bd4.16b // K = A*B4
@ -203,14 +197,14 @@ __pmull_p8_core:
ENDPROC(__pmull_p8_core)
.macro __pmull_p8, rq, ad, bd, i
.ifnc \bd, v10
.ifnc \bd, fold_consts
.err
.endif
mov ad.16b, \ad\().16b
.ifb \i
pmull \rq\().8h, \ad\().8b, bd.8b // D = A*B
pmull \rq\().8h, \ad\().8b, \bd\().8b // D = A*B
.else
pmull2 \rq\().8h, \ad\().16b, bd.16b // D = A*B
pmull2 \rq\().8h, \ad\().16b, \bd\().16b // D = A*B
.endif
bl .L__pmull_p8_core\i
@ -219,17 +213,19 @@ ENDPROC(__pmull_p8_core)
eor \rq\().16b, \rq\().16b, t6.16b
.endm
.macro fold64, p, reg1, reg2
ldp q11, q12, [arg2], #0x20
// Fold reg1, reg2 into the next 32 data bytes, storing the result back
// into reg1, reg2.
.macro fold_32_bytes, p, reg1, reg2
ldp q11, q12, [buf], #0x20
__pmull_\p v8, \reg1, v10, 2
__pmull_\p \reg1, \reg1, v10
__pmull_\p v8, \reg1, fold_consts, 2
__pmull_\p \reg1, \reg1, fold_consts
CPU_LE( rev64 v11.16b, v11.16b )
CPU_LE( rev64 v12.16b, v12.16b )
__pmull_\p v9, \reg2, v10, 2
__pmull_\p \reg2, \reg2, v10
__pmull_\p v9, \reg2, fold_consts, 2
__pmull_\p \reg2, \reg2, fold_consts
CPU_LE( ext v11.16b, v11.16b, v11.16b, #8 )
CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
@ -240,15 +236,16 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
eor \reg2\().16b, \reg2\().16b, v12.16b
.endm
.macro fold16, p, reg, rk
__pmull_\p v8, \reg, v10
__pmull_\p \reg, \reg, v10, 2
.ifnb \rk
ldr_l q10, \rk, x8
__pmull_pre_\p v10
// Fold src_reg into dst_reg, optionally loading the next fold constants
.macro fold_16_bytes, p, src_reg, dst_reg, load_next_consts
__pmull_\p v8, \src_reg, fold_consts
__pmull_\p \src_reg, \src_reg, fold_consts, 2
.ifnb \load_next_consts
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
.endif
eor v7.16b, v7.16b, v8.16b
eor v7.16b, v7.16b, \reg\().16b
eor \dst_reg\().16b, \dst_reg\().16b, v8.16b
eor \dst_reg\().16b, \dst_reg\().16b, \src_reg\().16b
.endm
.macro __pmull_p64, rd, rn, rm, n
@ -260,40 +257,27 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
.endm
.macro crc_t10dif_pmull, p
frame_push 3, 128
frame_push 4, 128
mov arg1_low32, w0
mov arg2, x1
mov arg3, x2
movi vzr.16b, #0 // init zero register
mov init_crc, w0
mov buf, x1
mov len, x2
__pmull_init_\p
// adjust the 16-bit initial_crc value, scale it to 32 bits
lsl arg1_low32, arg1_low32, #16
// For sizes less than 256 bytes, we can't fold 128 bytes at a time.
cmp len, #256
b.lt .Lless_than_256_bytes_\@
// check if smaller than 256
cmp arg3, #256
// for sizes less than 128, we can't fold 64B at a time...
b.lt .L_less_than_128_\@
// load the initial crc value
// crc value does not need to be byte-reflected, but it needs
// to be moved to the high part of the register.
// because data will be byte-reflected and will align with
// initial crc at correct place.
movi v10.16b, #0
mov v10.s[3], arg1_low32 // initial crc
// receive the initial 64B data, xor the initial crc value
ldp q0, q1, [arg2]
ldp q2, q3, [arg2, #0x20]
ldp q4, q5, [arg2, #0x40]
ldp q6, q7, [arg2, #0x60]
add arg2, arg2, #0x80
adr_l fold_consts_ptr, .Lfold_across_128_bytes_consts
// Load the first 128 data bytes. Byte swapping is necessary to make
// the bit order match the polynomial coefficient order.
ldp q0, q1, [buf]
ldp q2, q3, [buf, #0x20]
ldp q4, q5, [buf, #0x40]
ldp q6, q7, [buf, #0x60]
add buf, buf, #0x80
CPU_LE( rev64 v0.16b, v0.16b )
CPU_LE( rev64 v1.16b, v1.16b )
CPU_LE( rev64 v2.16b, v2.16b )
@ -302,7 +286,6 @@ CPU_LE( rev64 v4.16b, v4.16b )
CPU_LE( rev64 v5.16b, v5.16b )
CPU_LE( rev64 v6.16b, v6.16b )
CPU_LE( rev64 v7.16b, v7.16b )
CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
CPU_LE( ext v1.16b, v1.16b, v1.16b, #8 )
CPU_LE( ext v2.16b, v2.16b, v2.16b, #8 )
@ -312,36 +295,29 @@ CPU_LE( ext v5.16b, v5.16b, v5.16b, #8 )
CPU_LE( ext v6.16b, v6.16b, v6.16b, #8 )
CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
// XOR the initial_crc value
eor v0.16b, v0.16b, v10.16b
// XOR the first 16 data *bits* with the initial CRC value.
movi v8.16b, #0
mov v8.h[7], init_crc
eor v0.16b, v0.16b, v8.16b
ldr_l q10, rk3, x8 // xmm10 has rk3 and rk4
// type of pmull instruction
// will determine which constant to use
__pmull_pre_\p v10
// Load the constants for folding across 128 bytes.
ld1 {fold_consts.2d}, [fold_consts_ptr]
__pmull_pre_\p fold_consts
//
// we subtract 256 instead of 128 to save one instruction from the loop
//
sub arg3, arg3, #256
// Subtract 128 for the 128 data bytes just consumed. Subtract another
// 128 to simplify the termination condition of the following loop.
sub len, len, #256
// at this section of the code, there is 64*x+y (0<=y<64) bytes of
// buffer. The _fold_64_B_loop will fold 64B at a time
// until we have 64+y Bytes of buffer
// While >= 128 data bytes remain (not counting v0-v7), fold the 128
// bytes v0-v7 into them, storing the result back into v0-v7.
.Lfold_128_bytes_loop_\@:
fold_32_bytes \p, v0, v1
fold_32_bytes \p, v2, v3
fold_32_bytes \p, v4, v5
fold_32_bytes \p, v6, v7
// fold 64B at a time. This section of the code folds 4 vector
// registers in parallel
.L_fold_64_B_loop_\@:
fold64 \p, v0, v1
fold64 \p, v2, v3
fold64 \p, v4, v5
fold64 \p, v6, v7
subs arg3, arg3, #128
// check if there is another 64B in the buffer to be able to fold
b.lt .L_fold_64_B_end_\@
subs len, len, #128
b.lt .Lfold_128_bytes_loop_done_\@
if_will_cond_yield_neon
stp q0, q1, [sp, #.Lframe_local_offset]
@ -353,228 +329,207 @@ CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
ldp q2, q3, [sp, #.Lframe_local_offset + 32]
ldp q4, q5, [sp, #.Lframe_local_offset + 64]
ldp q6, q7, [sp, #.Lframe_local_offset + 96]
ldr_l q10, rk3, x8
movi vzr.16b, #0 // init zero register
ld1 {fold_consts.2d}, [fold_consts_ptr]
__pmull_init_\p
__pmull_pre_\p v10
__pmull_pre_\p fold_consts
endif_yield_neon
b .L_fold_64_B_loop_\@
b .Lfold_128_bytes_loop_\@
.L_fold_64_B_end_\@:
// at this point, the buffer pointer is pointing at the last y Bytes
// of the buffer the 64B of folded data is in 4 of the vector
// registers: v0, v1, v2, v3
.Lfold_128_bytes_loop_done_\@:
// fold the 8 vector registers to 1 vector register with different
// constants
// Now fold the 112 bytes in v0-v6 into the 16 bytes in v7.
ldr_l q10, rk9, x8
__pmull_pre_\p v10
// Fold across 64 bytes.
add fold_consts_ptr, fold_consts_ptr, #16
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
fold_16_bytes \p, v0, v4
fold_16_bytes \p, v1, v5
fold_16_bytes \p, v2, v6
fold_16_bytes \p, v3, v7, 1
// Fold across 32 bytes.
fold_16_bytes \p, v4, v6
fold_16_bytes \p, v5, v7, 1
// Fold across 16 bytes.
fold_16_bytes \p, v6, v7
fold16 \p, v0, rk11
fold16 \p, v1, rk13
fold16 \p, v2, rk15
fold16 \p, v3, rk17
fold16 \p, v4, rk19
fold16 \p, v5, rk1
fold16 \p, v6
// Add 128 to get the correct number of data bytes remaining in 0...127
// (not counting v7), following the previous extra subtraction by 128.
// Then subtract 16 to simplify the termination condition of the
// following loop.
adds len, len, #(128-16)
// instead of 64, we add 48 to the loop counter to save 1 instruction
// from the loop instead of a cmp instruction, we use the negative
// flag with the jl instruction
adds arg3, arg3, #(128-16)
b.lt .L_final_reduction_for_128_\@
// now we have 16+y bytes left to reduce. 16 Bytes is in register v7
// and the rest is in memory. We can fold 16 bytes at a time if y>=16
// continue folding 16B at a time
.L_16B_reduction_loop_\@:
__pmull_\p v8, v7, v10
__pmull_\p v7, v7, v10, 2
// While >= 16 data bytes remain (not counting v7), fold the 16 bytes v7
// into them, storing the result back into v7.
b.lt .Lfold_16_bytes_loop_done_\@
.Lfold_16_bytes_loop_\@:
__pmull_\p v8, v7, fold_consts
__pmull_\p v7, v7, fold_consts, 2
eor v7.16b, v7.16b, v8.16b
ldr q0, [arg2], #16
ldr q0, [buf], #16
CPU_LE( rev64 v0.16b, v0.16b )
CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
eor v7.16b, v7.16b, v0.16b
subs arg3, arg3, #16
subs len, len, #16
b.ge .Lfold_16_bytes_loop_\@
// instead of a cmp instruction, we utilize the flags with the
// jge instruction equivalent of: cmp arg3, 16-16
// check if there is any more 16B in the buffer to be able to fold
b.ge .L_16B_reduction_loop_\@
.Lfold_16_bytes_loop_done_\@:
// Add 16 to get the correct number of data bytes remaining in 0...15
// (not counting v7), following the previous extra subtraction by 16.
adds len, len, #16
b.eq .Lreduce_final_16_bytes_\@
// now we have 16+z bytes left to reduce, where 0<= z < 16.
// first, we reduce the data in the xmm7 register
.Lhandle_partial_segment_\@:
// Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first
// 16 bytes are in v7 and the rest are the remaining data in 'buf'. To
// do this without needing a fold constant for each possible 'len',
// redivide the bytes into a first chunk of 'len' bytes and a second
// chunk of 16 bytes, then fold the first chunk into the second.
.L_final_reduction_for_128_\@:
// check if any more data to fold. If not, compute the CRC of
// the final 128 bits
adds arg3, arg3, #16
b.eq .L_128_done_\@
// v0 = last 16 original data bytes
add buf, buf, len
ldr q0, [buf, #-16]
CPU_LE( rev64 v0.16b, v0.16b )
CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
// here we are getting data that is less than 16 bytes.
// since we know that there was data before the pointer, we can
// offset the input pointer before the actual point, to receive
// exactly 16 bytes. after that the registers need to be adjusted.
.L_get_last_two_regs_\@:
add arg2, arg2, arg3
ldr q1, [arg2, #-16]
CPU_LE( rev64 v1.16b, v1.16b )
CPU_LE( ext v1.16b, v1.16b, v1.16b, #8 )
// v1 = high order part of second chunk: v7 left-shifted by 'len' bytes.
adr_l x4, .Lbyteshift_table + 16
sub x4, x4, len
ld1 {v2.16b}, [x4]
tbl v1.16b, {v7.16b}, v2.16b
// get rid of the extra data that was loaded before
// load the shift constant
adr_l x4, tbl_shf_table + 16
sub x4, x4, arg3
ld1 {v0.16b}, [x4]
// v3 = first chunk: v7 right-shifted by '16-len' bytes.
movi v3.16b, #0x80
eor v2.16b, v2.16b, v3.16b
tbl v3.16b, {v7.16b}, v2.16b
// shift v2 to the left by arg3 bytes
tbl v2.16b, {v7.16b}, v0.16b
// Convert to 8-bit masks: 'len' 0x00 bytes, then '16-len' 0xff bytes.
sshr v2.16b, v2.16b, #7
// shift v7 to the right by 16-arg3 bytes
movi v9.16b, #0x80
eor v0.16b, v0.16b, v9.16b
tbl v7.16b, {v7.16b}, v0.16b
// v2 = second chunk: 'len' bytes from v0 (low-order bytes),
// then '16-len' bytes from v1 (high-order bytes).
bsl v2.16b, v1.16b, v0.16b
// blend
sshr v0.16b, v0.16b, #7 // convert to 8-bit mask
bsl v0.16b, v2.16b, v1.16b
// fold 16 Bytes
__pmull_\p v8, v7, v10
__pmull_\p v7, v7, v10, 2
eor v7.16b, v7.16b, v8.16b
// Fold the first chunk into the second chunk, storing the result in v7.
__pmull_\p v0, v3, fold_consts
__pmull_\p v7, v3, fold_consts, 2
eor v7.16b, v7.16b, v0.16b
eor v7.16b, v7.16b, v2.16b
.L_128_done_\@:
// compute crc of a 128-bit value
ldr_l q10, rk5, x8 // rk5 and rk6 in xmm10
__pmull_pre_\p v10
.Lreduce_final_16_bytes_\@:
// Reduce the 128-bit value M(x), stored in v7, to the final 16-bit CRC.
// 64b fold
ext v0.16b, vzr.16b, v7.16b, #8
mov v7.d[0], v7.d[1]
__pmull_\p v7, v7, v10
eor v7.16b, v7.16b, v0.16b
movi v2.16b, #0 // init zero register
// 32b fold
ext v0.16b, v7.16b, vzr.16b, #4
mov v7.s[3], vzr.s[0]
__pmull_\p v0, v0, v10, 2
eor v7.16b, v7.16b, v0.16b
// Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
// barrett reduction
ldr_l q10, rk7, x8
__pmull_pre_\p v10
mov v0.d[0], v7.d[1]
// Fold the high 64 bits into the low 64 bits, while also multiplying by
// x^64. This produces a 128-bit value congruent to x^64 * M(x) and
// whose low 48 bits are 0.
ext v0.16b, v2.16b, v7.16b, #8
__pmull_\p v7, v7, fold_consts, 2 // high bits * x^48 * (x^80 mod G(x))
eor v0.16b, v0.16b, v7.16b // + low bits * x^64
__pmull_\p v0, v0, v10
ext v0.16b, vzr.16b, v0.16b, #12
__pmull_\p v0, v0, v10, 2
ext v0.16b, vzr.16b, v0.16b, #12
eor v7.16b, v7.16b, v0.16b
mov w0, v7.s[1]
// Fold the high 32 bits into the low 96 bits. This produces a 96-bit
// value congruent to x^64 * M(x) and whose low 48 bits are 0.
ext v1.16b, v0.16b, v2.16b, #12 // extract high 32 bits
mov v0.s[3], v2.s[0] // zero high 32 bits
__pmull_\p v1, v1, fold_consts // high 32 bits * x^48 * (x^48 mod G(x))
eor v0.16b, v0.16b, v1.16b // + low bits
.L_cleanup_\@:
// scale the result back to 16 bits
lsr x0, x0, #16
// Load G(x) and floor(x^48 / G(x)).
ld1 {fold_consts.2d}, [fold_consts_ptr]
__pmull_pre_\p fold_consts
// Use Barrett reduction to compute the final CRC value.
__pmull_\p v1, v0, fold_consts, 2 // high 32 bits * floor(x^48 / G(x))
ushr v1.2d, v1.2d, #32 // /= x^32
__pmull_\p v1, v1, fold_consts // *= G(x)
ushr v0.2d, v0.2d, #48
eor v0.16b, v0.16b, v1.16b // + low 16 nonzero bits
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of v0.
umov w0, v0.h[0]
frame_pop
ret
.L_less_than_128_\@:
cbz arg3, .L_cleanup_\@
.Lless_than_256_bytes_\@:
// Checksumming a buffer of length 16...255 bytes
movi v0.16b, #0
mov v0.s[3], arg1_low32 // get the initial crc value
adr_l fold_consts_ptr, .Lfold_across_16_bytes_consts
ldr q7, [arg2], #0x10
// Load the first 16 data bytes.
ldr q7, [buf], #0x10
CPU_LE( rev64 v7.16b, v7.16b )
CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
eor v7.16b, v7.16b, v0.16b // xor the initial crc value
cmp arg3, #16
b.eq .L_128_done_\@ // exactly 16 left
b.lt .L_less_than_16_left_\@
// XOR the first 16 data *bits* with the initial CRC value.
movi v0.16b, #0
mov v0.h[7], init_crc
eor v7.16b, v7.16b, v0.16b
ldr_l q10, rk1, x8 // rk1 and rk2 in xmm10
__pmull_pre_\p v10
// Load the fold-across-16-bytes constants.
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
// update the counter. subtract 32 instead of 16 to save one
// instruction from the loop
subs arg3, arg3, #32
b.ge .L_16B_reduction_loop_\@
add arg3, arg3, #16
b .L_get_last_two_regs_\@
.L_less_than_16_left_\@:
// shl r9, 4
adr_l x0, tbl_shf_table + 16
sub x0, x0, arg3
ld1 {v0.16b}, [x0]
movi v9.16b, #0x80
eor v0.16b, v0.16b, v9.16b
tbl v7.16b, {v7.16b}, v0.16b
b .L_128_done_\@
cmp len, #16
b.eq .Lreduce_final_16_bytes_\@ // len == 16
subs len, len, #32
b.ge .Lfold_16_bytes_loop_\@ // 32 <= len <= 255
add len, len, #16
b .Lhandle_partial_segment_\@ // 17 <= len <= 31
.endm
//
// u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
ENTRY(crc_t10dif_pmull_p8)
crc_t10dif_pmull p8
ENDPROC(crc_t10dif_pmull_p8)
.align 5
//
// u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
ENTRY(crc_t10dif_pmull_p64)
crc_t10dif_pmull p64
ENDPROC(crc_t10dif_pmull_p64)
// precomputed constants
// these constants are precomputed from the poly:
// 0x8bb70000 (0x8bb7 scaled to 32 bits)
.section ".rodata", "a"
.align 4
// Q = 0x18BB70000
// rk1 = 2^(32*3) mod Q << 32
// rk2 = 2^(32*5) mod Q << 32
// rk3 = 2^(32*15) mod Q << 32
// rk4 = 2^(32*17) mod Q << 32
// rk5 = 2^(32*3) mod Q << 32
// rk6 = 2^(32*2) mod Q << 32
// rk7 = floor(2^64/Q)
// rk8 = Q
rk1: .octa 0x06df0000000000002d56000000000000
rk3: .octa 0x7cf50000000000009d9d000000000000
rk5: .octa 0x13680000000000002d56000000000000
rk7: .octa 0x000000018bb7000000000001f65a57f8
rk9: .octa 0xbfd6000000000000ceae000000000000
rk11: .octa 0x713c0000000000001e16000000000000
rk13: .octa 0x80a6000000000000f7f9000000000000
rk15: .octa 0xe658000000000000044c000000000000
rk17: .octa 0xa497000000000000ad18000000000000
rk19: .octa 0xe7b50000000000006ee3000000000000
tbl_shf_table:
// use these values for shift constants for the tbl/tbx instruction
// different alignments result in values as shown:
// DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1
// DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2
// DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3
// DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4
// DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5
// DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6
// DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
// DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
// DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
// DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
// DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
// DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
// DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
// DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
// DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
// Fold constants precomputed from the polynomial 0x18bb7
// G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
.Lfold_across_128_bytes_consts:
.quad 0x0000000000006123 // x^(8*128) mod G(x)
.quad 0x0000000000002295 // x^(8*128+64) mod G(x)
// .Lfold_across_64_bytes_consts:
.quad 0x0000000000001069 // x^(4*128) mod G(x)
.quad 0x000000000000dd31 // x^(4*128+64) mod G(x)
// .Lfold_across_32_bytes_consts:
.quad 0x000000000000857d // x^(2*128) mod G(x)
.quad 0x0000000000007acc // x^(2*128+64) mod G(x)
.Lfold_across_16_bytes_consts:
.quad 0x000000000000a010 // x^(1*128) mod G(x)
.quad 0x0000000000001faa // x^(1*128+64) mod G(x)
// .Lfinal_fold_consts:
.quad 0x1368000000000000 // x^48 * (x^48 mod G(x))
.quad 0x2d56000000000000 // x^48 * (x^80 mod G(x))
// .Lbarrett_reduction_consts:
.quad 0x0000000000018bb7 // G(x)
.quad 0x00000001f65a57f8 // floor(x^48 / G(x))
// For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 -
// len] is the index vector to shift left by 'len' bytes, and is also {0x80,
// ..., 0x80} XOR the index vector to shift right by '16 - len' bytes.
.Lbyteshift_table:
.byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7

77
arch/arm64/crypto/crct10dif-ce-glue.c
View File

@ -22,10 +22,8 @@
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 buf[], u64 len);
asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 buf[], u64 len);
static u16 (*crc_t10dif_pmull)(u16 init_crc, const u8 buf[], u64 len);
asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len);
asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
static int crct10dif_init(struct shash_desc *desc)
{
@ -35,30 +33,33 @@ static int crct10dif_init(struct shash_desc *desc)
return 0;
}
static int crct10dif_update(struct shash_desc *desc, const u8 *data,
static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data,
unsigned int length)
{
u16 *crc = shash_desc_ctx(desc);
unsigned int l;
if (unlikely((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) {
l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE -
((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE));
*crc = crc_t10dif_generic(*crc, data, l);
length -= l;
data += l;
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull_p8(*crc, data, length);
kernel_neon_end();
} else {
*crc = crc_t10dif_generic(*crc, data, length);
}
if (length > 0) {
if (may_use_simd()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
} else {
*crc = crc_t10dif_generic(*crc, data, length);
}
return 0;
}
static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data,
unsigned int length)
{
u16 *crc = shash_desc_ctx(desc);
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull_p64(*crc, data, length);
kernel_neon_end();
} else {
*crc = crc_t10dif_generic(*crc, data, length);
}
return 0;
@ -72,10 +73,22 @@ static int crct10dif_final(struct shash_desc *desc, u8 *out)
return 0;
}
static struct shash_alg crc_t10dif_alg = {
static struct shash_alg crc_t10dif_alg[] = {{
.digestsize = CRC_T10DIF_DIGEST_SIZE,
.init = crct10dif_init,
.update = crct10dif_update,
.update = crct10dif_update_pmull_p8,
.final = crct10dif_final,
.descsize = CRC_T10DIF_DIGEST_SIZE,
.base.cra_name = "crct10dif",
.base.cra_driver_name = "crct10dif-arm64-neon",
.base.cra_priority = 100,
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
}, {
.digestsize = CRC_T10DIF_DIGEST_SIZE,
.init = crct10dif_init,
.update = crct10dif_update_pmull_p64,
.final = crct10dif_final,
.descsize = CRC_T10DIF_DIGEST_SIZE,
@ -84,21 +97,25 @@ static struct shash_alg crc_t10dif_alg = {
.base.cra_priority = 200,
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
};
}};
static int __init crc_t10dif_mod_init(void)
{
if (elf_hwcap & HWCAP_PMULL)
crc_t10dif_pmull = crc_t10dif_pmull_p64;
return crypto_register_shashes(crc_t10dif_alg,
ARRAY_SIZE(crc_t10dif_alg));
else
crc_t10dif_pmull = crc_t10dif_pmull_p8;
return crypto_register_shash(&crc_t10dif_alg);
/* only register the first array element */
return crypto_register_shash(crc_t10dif_alg);
}
static void __exit crc_t10dif_mod_exit(void)
{
crypto_unregister_shash(&crc_t10dif_alg);
if (elf_hwcap & HWCAP_PMULL)
crypto_unregister_shashes(crc_t10dif_alg,
ARRAY_SIZE(crc_t10dif_alg));
else
crypto_unregister_shash(crc_t10dif_alg);
}
module_cpu_feature_match(ASIMD, crc_t10dif_mod_init);

118
arch/arm64/crypto/ghash-ce-glue.c
View File

@ -60,10 +60,6 @@ asmlinkage void pmull_ghash_update_p8(int blocks, u64 dg[], const char *src,
struct ghash_key const *k,
const char *head);
static void (*pmull_ghash_update)(int blocks, u64 dg[], const char *src,
struct ghash_key const *k,
const char *head);
asmlinkage void pmull_gcm_encrypt(int blocks, u64 dg[], u8 dst[],
const u8 src[], struct ghash_key const *k,
u8 ctr[], u32 const rk[], int rounds,
@ -87,11 +83,15 @@ static int ghash_init(struct shash_desc *desc)
}
static void ghash_do_update(int blocks, u64 dg[], const char *src,
struct ghash_key *key, const char *head)
struct ghash_key *key, const char *head,
void (*simd_update)(int blocks, u64 dg[],
const char *src,
struct ghash_key const *k,
const char *head))
{
if (likely(may_use_simd())) {
kernel_neon_begin();
pmull_ghash_update(blocks, dg, src, key, head);
simd_update(blocks, dg, src, key, head);
kernel_neon_end();
} else {
be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) };
@ -119,8 +119,12 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,
/* avoid hogging the CPU for too long */
#define MAX_BLOCKS (SZ_64K / GHASH_BLOCK_SIZE)
static int ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int len)
static int __ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int len,
void (*simd_update)(int blocks, u64 dg[],
const char *src,
struct ghash_key const *k,
const char *head))
{
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
@ -146,7 +150,8 @@ static int ghash_update(struct shash_desc *desc, const u8 *src,
int chunk = min(blocks, MAX_BLOCKS);
ghash_do_update(chunk, ctx->digest, src, key,
partial ? ctx->buf : NULL);
partial ? ctx->buf : NULL,
simd_update);
blocks -= chunk;
src += chunk * GHASH_BLOCK_SIZE;
@ -158,7 +163,19 @@ static int ghash_update(struct shash_desc *desc, const u8 *src,
return 0;
}
static int ghash_final(struct shash_desc *desc, u8 *dst)
static int ghash_update_p8(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
return __ghash_update(desc, src, len, pmull_ghash_update_p8);
}
static int ghash_update_p64(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
return __ghash_update(desc, src, len, pmull_ghash_update_p64);
}
static int ghash_final_p8(struct shash_desc *desc, u8 *dst)
{
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
@ -168,7 +185,28 @@ static int ghash_final(struct shash_desc *desc, u8 *dst)
memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial);
ghash_do_update(1, ctx->digest, ctx->buf, key, NULL);
ghash_do_update(1, ctx->digest, ctx->buf, key, NULL,
pmull_ghash_update_p8);
}
put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8);
*ctx = (struct ghash_desc_ctx){};
return 0;
}
static int ghash_final_p64(struct shash_desc *desc, u8 *dst)
{
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
if (partial) {
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial);
ghash_do_update(1, ctx->digest, ctx->buf, key, NULL,
pmull_ghash_update_p64);
}
put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8);
@ -224,7 +262,21 @@ static int ghash_setkey(struct crypto_shash *tfm,
return __ghash_setkey(key, inkey, keylen);
}
static struct shash_alg ghash_alg = {
static struct shash_alg ghash_alg[] = {{
.base.cra_name = "ghash",
.base.cra_driver_name = "ghash-neon",
.base.cra_priority = 100,
.base.cra_blocksize = GHASH_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct ghash_key),
.base.cra_module = THIS_MODULE,
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update_p8,
.final = ghash_final_p8,
.setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx),
}, {
.base.cra_name = "ghash",
.base.cra_driver_name = "ghash-ce",
.base.cra_priority = 200,
@ -234,11 +286,11 @@ static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.final = ghash_final,
.update = ghash_update_p64,
.final = ghash_final_p64,
.setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx),
};
}};
static int num_rounds(struct crypto_aes_ctx *ctx)
{
@ -301,7 +353,8 @@ static void gcm_update_mac(u64 dg[], const u8 *src, int count, u8 buf[],
int blocks = count / GHASH_BLOCK_SIZE;
ghash_do_update(blocks, dg, src, &ctx->ghash_key,
*buf_count ? buf : NULL);
*buf_count ? buf : NULL,
pmull_ghash_update_p64);
src += blocks * GHASH_BLOCK_SIZE;
count %= GHASH_BLOCK_SIZE;
@ -345,7 +398,8 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u64 dg[])
if (buf_count) {
memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count);
ghash_do_update(1, dg, buf, &ctx->ghash_key, NULL);
ghash_do_update(1, dg, buf, &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
}
}
@ -358,7 +412,8 @@ static void gcm_final(struct aead_request *req, struct gcm_aes_ctx *ctx,
lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(cryptlen * 8);
ghash_do_update(1, dg, (void *)&lengths, &ctx->ghash_key, NULL);
ghash_do_update(1, dg, (void *)&lengths, &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
put_unaligned_be64(dg[1], mac);
put_unaligned_be64(dg[0], mac + 8);
@ -434,7 +489,7 @@ static int gcm_encrypt(struct aead_request *req)
ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg,
walk.dst.virt.addr, &ctx->ghash_key,
NULL);
NULL, pmull_ghash_update_p64);
err = skcipher_walk_done(&walk,
walk.nbytes % (2 * AES_BLOCK_SIZE));
@ -469,7 +524,8 @@ static int gcm_encrypt(struct aead_request *req)
memcpy(buf, dst, nbytes);
memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head,
pmull_ghash_update_p64);
err = skcipher_walk_done(&walk, 0);
}
@ -558,7 +614,8 @@ static int gcm_decrypt(struct aead_request *req)
u8 *src = walk.src.virt.addr;
ghash_do_update(blocks, dg, walk.src.virt.addr,
&ctx->ghash_key, NULL);
&ctx->ghash_key, NULL,
pmull_ghash_update_p64);
do {
__aes_arm64_encrypt(ctx->aes_key.key_enc,
@ -602,7 +659,8 @@ static int gcm_decrypt(struct aead_request *req)
memcpy(buf, src, nbytes);
memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head,
pmull_ghash_update_p64);
crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, iv,
walk.nbytes);
@ -650,26 +708,30 @@ static int __init ghash_ce_mod_init(void)
return -ENODEV;
if (elf_hwcap & HWCAP_PMULL)
pmull_ghash_update = pmull_ghash_update_p64;
ret = crypto_register_shashes(ghash_alg,
ARRAY_SIZE(ghash_alg));
else
pmull_ghash_update = pmull_ghash_update_p8;
/* only register the first array element */
ret = crypto_register_shash(ghash_alg);
ret = crypto_register_shash(&ghash_alg);
if (ret)
return ret;
if (elf_hwcap & HWCAP_PMULL) {
ret = crypto_register_aead(&gcm_aes_alg);
if (ret)
crypto_unregister_shash(&ghash_alg);
crypto_unregister_shashes(ghash_alg,
ARRAY_SIZE(ghash_alg));
}
return ret;
}
static void __exit ghash_ce_mod_exit(void)
{
crypto_unregister_shash(&ghash_alg);
if (elf_hwcap & HWCAP_PMULL)
crypto_unregister_shashes(ghash_alg, ARRAY_SIZE(ghash_alg));
else
crypto_unregister_shash(ghash_alg);
crypto_unregister_aead(&gcm_aes_alg);
}

4
arch/s390/crypto/des_s390.c
View File

@ -38,7 +38,7 @@ static int des_setkey(struct crypto_tfm *tfm, const u8 *key,
/* check for weak keys */
if (!des_ekey(tmp, key) &&
(tfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
(tfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}
@ -228,7 +228,7 @@ static int des3_setkey(struct crypto_tfm *tfm, const u8 *key,
if (!(crypto_memneq(key, &key[DES_KEY_SIZE], DES_KEY_SIZE) &&
crypto_memneq(&key[DES_KEY_SIZE], &key[DES_KEY_SIZE * 2],
DES_KEY_SIZE)) &&
(tfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
(tfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}

4
arch/sparc/crypto/des_glue.c
View File

@ -53,7 +53,7 @@ static int des_set_key(struct crypto_tfm *tfm, const u8 *key,
* weak key detection code.
*/
ret = des_ekey(tmp, key);
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}
@ -209,7 +209,7 @@ static int des3_ede_set_key(struct crypto_tfm *tfm, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
(*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}

38
arch/x86/crypto/aegis128-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis128_aesni_process_ad(
}
static void crypto_aegis128_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req,
struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize, base;
while (walk->nbytes >= AEGIS128_BLOCK_SIZE) {
ops->crypt_blocks(state,
round_down(walk->nbytes, AEGIS128_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS128_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS128_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS128_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
walk->dst.virt.addr);
skcipher_walk_done(walk, 0);
}
}
@ -186,13 +175,16 @@ static void crypto_aegis128_aesni_crypt(struct aead_request *req,
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin();
crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv);
crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis128_aesni_process_crypt(&state, req, ops);
crypto_aegis128_aesni_process_crypt(&state, &walk, ops);
crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();

38
arch/x86/crypto/aegis128l-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis128l_aesni_process_ad(
}
static void crypto_aegis128l_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req,
struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize, base;
while (walk->nbytes >= AEGIS128L_BLOCK_SIZE) {
ops->crypt_blocks(state, round_down(walk->nbytes,
AEGIS128L_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS128L_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS128L_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS128L_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
walk->dst.virt.addr);
skcipher_walk_done(walk, 0);
}
}
@ -186,13 +175,16 @@ static void crypto_aegis128l_aesni_crypt(struct aead_request *req,
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis128l_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin();
crypto_aegis128l_aesni_init(&state, ctx->key.bytes, req->iv);
crypto_aegis128l_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis128l_aesni_process_crypt(&state, req, ops);
crypto_aegis128l_aesni_process_crypt(&state, &walk, ops);
crypto_aegis128l_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();

38
arch/x86/crypto/aegis256-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis256_aesni_process_ad(
}
static void crypto_aegis256_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req,
struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize, base;
while (walk->nbytes >= AEGIS256_BLOCK_SIZE) {
ops->crypt_blocks(state,
round_down(walk->nbytes, AEGIS256_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS256_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS256_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS256_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
walk->dst.virt.addr);
skcipher_walk_done(walk, 0);
}
}
@ -186,13 +175,16 @@ static void crypto_aegis256_aesni_crypt(struct aead_request *req,
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis256_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin();
crypto_aegis256_aesni_init(&state, ctx->key, req->iv);
crypto_aegis256_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis256_aesni_process_crypt(&state, req, ops);
crypto_aegis256_aesni_process_crypt(&state, &walk, ops);
crypto_aegis256_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();

47
arch/x86/crypto/aesni-intel_glue.c
View File

@ -175,26 +175,18 @@ asmlinkage void aesni_gcm_finalize(void *ctx,
struct gcm_context_data *gdata,
u8 *auth_tag, unsigned long auth_tag_len);
static struct aesni_gcm_tfm_s {
void (*init)(void *ctx,
struct gcm_context_data *gdata,
u8 *iv,
u8 *hash_subkey, const u8 *aad,
unsigned long aad_len);
void (*enc_update)(void *ctx,
struct gcm_context_data *gdata, u8 *out,
const u8 *in,
unsigned long plaintext_len);
void (*dec_update)(void *ctx,
struct gcm_context_data *gdata, u8 *out,
const u8 *in,
unsigned long ciphertext_len);
void (*finalize)(void *ctx,
struct gcm_context_data *gdata,
u8 *auth_tag, unsigned long auth_tag_len);
static const struct aesni_gcm_tfm_s {
void (*init)(void *ctx, struct gcm_context_data *gdata, u8 *iv,
u8 *hash_subkey, const u8 *aad, unsigned long aad_len);
void (*enc_update)(void *ctx, struct gcm_context_data *gdata, u8 *out,
const u8 *in, unsigned long plaintext_len);
void (*dec_update)(void *ctx, struct gcm_context_data *gdata, u8 *out,
const u8 *in, unsigned long ciphertext_len);
void (*finalize)(void *ctx, struct gcm_context_data *gdata,
u8 *auth_tag, unsigned long auth_tag_len);
} *aesni_gcm_tfm;
struct aesni_gcm_tfm_s aesni_gcm_tfm_sse = {
static const struct aesni_gcm_tfm_s aesni_gcm_tfm_sse = {
.init = &aesni_gcm_init,
.enc_update = &aesni_gcm_enc_update,
.dec_update = &aesni_gcm_dec_update,
@ -243,7 +235,7 @@ asmlinkage void aesni_gcm_dec_avx_gen2(void *ctx,
const u8 *aad, unsigned long aad_len,
u8 *auth_tag, unsigned long auth_tag_len);
struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen2 = {
static const struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen2 = {
.init = &aesni_gcm_init_avx_gen2,
.enc_update = &aesni_gcm_enc_update_avx_gen2,
.dec_update = &aesni_gcm_dec_update_avx_gen2,
@ -288,7 +280,7 @@ asmlinkage void aesni_gcm_dec_avx_gen4(void *ctx,
const u8 *aad, unsigned long aad_len,
u8 *auth_tag, unsigned long auth_tag_len);
struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen4 = {
static const struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen4 = {
.init = &aesni_gcm_init_avx_gen4,
.enc_update = &aesni_gcm_enc_update_avx_gen4,
.dec_update = &aesni_gcm_dec_update_avx_gen4,
@ -778,7 +770,7 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned long auth_tag_len = crypto_aead_authsize(tfm);
struct aesni_gcm_tfm_s *gcm_tfm = aesni_gcm_tfm;
const struct aesni_gcm_tfm_s *gcm_tfm = aesni_gcm_tfm;
struct gcm_context_data data AESNI_ALIGN_ATTR;
struct scatter_walk dst_sg_walk = {};
unsigned long left = req->cryptlen;
@ -821,11 +813,14 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
scatterwalk_map_and_copy(assoc, req->src, 0, assoclen, 0);
}
src_sg = scatterwalk_ffwd(src_start, req->src, req->assoclen);
scatterwalk_start(&src_sg_walk, src_sg);
if (req->src != req->dst) {
dst_sg = scatterwalk_ffwd(dst_start, req->dst, req->assoclen);
scatterwalk_start(&dst_sg_walk, dst_sg);
if (left) {
src_sg = scatterwalk_ffwd(src_start, req->src, req->assoclen);
scatterwalk_start(&src_sg_walk, src_sg);
if (req->src != req->dst) {
dst_sg = scatterwalk_ffwd(dst_start, req->dst,
req->assoclen);
scatterwalk_start(&dst_sg_walk, dst_sg);
}
}
kernel_fpu_begin();

752
arch/x86/crypto/crct10dif-pcl-asm_64.S
View File

@ -43,609 +43,291 @@
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
########################################################################
# Function API:
# UINT16 crc_t10dif_pcl(
# UINT16 init_crc, //initial CRC value, 16 bits
# const unsigned char *buf, //buffer pointer to calculate CRC on
# UINT64 len //buffer length in bytes (64-bit data)
# );
#
# Reference paper titled "Fast CRC Computation for Generic
# Polynomials Using PCLMULQDQ Instruction"
# URL: http://www.intel.com/content/dam/www/public/us/en/documents
# /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
#
#
#include <linux/linkage.h>
.text
#define arg1 %rdi
#define arg2 %rsi
#define arg3 %rdx
#define init_crc %edi
#define buf %rsi
#define len %rdx
#define arg1_low32 %edi
#define FOLD_CONSTS %xmm10
#define BSWAP_MASK %xmm11
ENTRY(crc_t10dif_pcl)
# Fold reg1, reg2 into the next 32 data bytes, storing the result back into
# reg1, reg2.
.macro fold_32_bytes offset, reg1, reg2
movdqu \offset(buf), %xmm9
movdqu \offset+16(buf), %xmm12
pshufb BSWAP_MASK, %xmm9
pshufb BSWAP_MASK, %xmm12
movdqa \reg1, %xmm8
movdqa \reg2, %xmm13
pclmulqdq $0x00, FOLD_CONSTS, \reg1
pclmulqdq $0x11, FOLD_CONSTS, %xmm8
pclmulqdq $0x00, FOLD_CONSTS, \reg2
pclmulqdq $0x11, FOLD_CONSTS, %xmm13
pxor %xmm9 , \reg1
xorps %xmm8 , \reg1
pxor %xmm12, \reg2
xorps %xmm13, \reg2
.endm
# Fold src_reg into dst_reg.
.macro fold_16_bytes src_reg, dst_reg
movdqa \src_reg, %xmm8
pclmulqdq $0x11, FOLD_CONSTS, \src_reg
pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, \dst_reg
xorps \src_reg, \dst_reg
.endm
#
# u16 crc_t10dif_pcl(u16 init_crc, const *u8 buf, size_t len);
#
# Assumes len >= 16.
#
.align 16
ENTRY(crc_t10dif_pcl)
# adjust the 16-bit initial_crc value, scale it to 32 bits
shl $16, arg1_low32
movdqa .Lbswap_mask(%rip), BSWAP_MASK
# Allocate Stack Space
mov %rsp, %rcx
sub $16*2, %rsp
# align stack to 16 byte boundary
and $~(0x10 - 1), %rsp
# For sizes less than 256 bytes, we can't fold 128 bytes at a time.
cmp $256, len
jl .Lless_than_256_bytes
# check if smaller than 256
cmp $256, arg3
# Load the first 128 data bytes. Byte swapping is necessary to make the
# bit order match the polynomial coefficient order.
movdqu 16*0(buf), %xmm0
movdqu 16*1(buf), %xmm1
movdqu 16*2(buf), %xmm2
movdqu 16*3(buf), %xmm3
movdqu 16*4(buf), %xmm4
movdqu 16*5(buf), %xmm5
movdqu 16*6(buf), %xmm6
movdqu 16*7(buf), %xmm7
add $128, buf
pshufb BSWAP_MASK, %xmm0
pshufb BSWAP_MASK, %xmm1
pshufb BSWAP_MASK, %xmm2
pshufb BSWAP_MASK, %xmm3
pshufb BSWAP_MASK, %xmm4
pshufb BSWAP_MASK, %xmm5
pshufb BSWAP_MASK, %xmm6
pshufb BSWAP_MASK, %xmm7
# for sizes less than 128, we can't fold 64B at a time...
jl _less_than_128
# XOR the first 16 data *bits* with the initial CRC value.
pxor %xmm8, %xmm8
pinsrw $7, init_crc, %xmm8
pxor %xmm8, %xmm0
movdqa .Lfold_across_128_bytes_consts(%rip), FOLD_CONSTS
# load the initial crc value
movd arg1_low32, %xmm10 # initial crc
# Subtract 128 for the 128 data bytes just consumed. Subtract another
# 128 to simplify the termination condition of the following loop.
sub $256, len
# crc value does not need to be byte-reflected, but it needs
# to be moved to the high part of the register.
# because data will be byte-reflected and will align with
# initial crc at correct place.
pslldq $12, %xmm10
# While >= 128 data bytes remain (not counting xmm0-7), fold the 128
# bytes xmm0-7 into them, storing the result back into xmm0-7.
.Lfold_128_bytes_loop:
fold_32_bytes 0, %xmm0, %xmm1
fold_32_bytes 32, %xmm2, %xmm3
fold_32_bytes 64, %xmm4, %xmm5
fold_32_bytes 96, %xmm6, %xmm7
add $128, buf
sub $128, len
jge .Lfold_128_bytes_loop
movdqa SHUF_MASK(%rip), %xmm11
# receive the initial 64B data, xor the initial crc value
movdqu 16*0(arg2), %xmm0
movdqu 16*1(arg2), %xmm1
movdqu 16*2(arg2), %xmm2
movdqu 16*3(arg2), %xmm3
movdqu 16*4(arg2), %xmm4
movdqu 16*5(arg2), %xmm5
movdqu 16*6(arg2), %xmm6
movdqu 16*7(arg2), %xmm7
# Now fold the 112 bytes in xmm0-xmm6 into the 16 bytes in xmm7.
pshufb %xmm11, %xmm0
# XOR the initial_crc value
pxor %xmm10, %xmm0
pshufb %xmm11, %xmm1
pshufb %xmm11, %xmm2
pshufb %xmm11, %xmm3
pshufb %xmm11, %xmm4
pshufb %xmm11, %xmm5
pshufb %xmm11, %xmm6
pshufb %xmm11, %xmm7
# Fold across 64 bytes.
movdqa .Lfold_across_64_bytes_consts(%rip), FOLD_CONSTS
fold_16_bytes %xmm0, %xmm4
fold_16_bytes %xmm1, %xmm5
fold_16_bytes %xmm2, %xmm6
fold_16_bytes %xmm3, %xmm7
# Fold across 32 bytes.
movdqa .Lfold_across_32_bytes_consts(%rip), FOLD_CONSTS
fold_16_bytes %xmm4, %xmm6
fold_16_bytes %xmm5, %xmm7
# Fold across 16 bytes.
movdqa .Lfold_across_16_bytes_consts(%rip), FOLD_CONSTS
fold_16_bytes %xmm6, %xmm7
movdqa rk3(%rip), %xmm10 #xmm10 has rk3 and rk4
#imm value of pclmulqdq instruction
#will determine which constant to use
# Add 128 to get the correct number of data bytes remaining in 0...127
# (not counting xmm7), following the previous extra subtraction by 128.
# Then subtract 16 to simplify the termination condition of the
# following loop.
add $128-16, len
#################################################################
# we subtract 256 instead of 128 to save one instruction from the loop
sub $256, arg3
# at this section of the code, there is 64*x+y (0<=y<64) bytes of
# buffer. The _fold_64_B_loop will fold 64B at a time
# until we have 64+y Bytes of buffer
# fold 64B at a time. This section of the code folds 4 xmm
# registers in parallel
_fold_64_B_loop:
# update the buffer pointer
add $128, arg2 # buf += 64#
movdqu 16*0(arg2), %xmm9
movdqu 16*1(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm0, %xmm8
movdqa %xmm1, %xmm13
pclmulqdq $0x0 , %xmm10, %xmm0
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0 , %xmm10, %xmm1
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm0
xorps %xmm8 , %xmm0
pxor %xmm12, %xmm1
xorps %xmm13, %xmm1
movdqu 16*2(arg2), %xmm9
movdqu 16*3(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm2, %xmm8
movdqa %xmm3, %xmm13
pclmulqdq $0x0, %xmm10, %xmm2
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0, %xmm10, %xmm3
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm2
xorps %xmm8 , %xmm2
pxor %xmm12, %xmm3
xorps %xmm13, %xmm3
movdqu 16*4(arg2), %xmm9
movdqu 16*5(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm4, %xmm8
movdqa %xmm5, %xmm13
pclmulqdq $0x0, %xmm10, %xmm4
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0, %xmm10, %xmm5
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm4
xorps %xmm8 , %xmm4
pxor %xmm12, %xmm5
xorps %xmm13, %xmm5
movdqu 16*6(arg2), %xmm9
movdqu 16*7(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm6 , %xmm8
movdqa %xmm7 , %xmm13
pclmulqdq $0x0 , %xmm10, %xmm6
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0 , %xmm10, %xmm7
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm6
xorps %xmm8 , %xmm6
pxor %xmm12, %xmm7
xorps %xmm13, %xmm7
sub $128, arg3
# check if there is another 64B in the buffer to be able to fold
jge _fold_64_B_loop
##################################################################
add $128, arg2
# at this point, the buffer pointer is pointing at the last y Bytes
# of the buffer the 64B of folded data is in 4 of the xmm
# registers: xmm0, xmm1, xmm2, xmm3
# fold the 8 xmm registers to 1 xmm register with different constants
movdqa rk9(%rip), %xmm10
movdqa %xmm0, %xmm8
pclmulqdq $0x11, %xmm10, %xmm0
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm0, %xmm7
movdqa rk11(%rip), %xmm10
movdqa %xmm1, %xmm8
pclmulqdq $0x11, %xmm10, %xmm1
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm1, %xmm7
movdqa rk13(%rip), %xmm10
movdqa %xmm2, %xmm8
pclmulqdq $0x11, %xmm10, %xmm2
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm2, %xmm7
movdqa rk15(%rip), %xmm10
movdqa %xmm3, %xmm8
pclmulqdq $0x11, %xmm10, %xmm3
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm3, %xmm7
movdqa rk17(%rip), %xmm10
movdqa %xmm4, %xmm8
pclmulqdq $0x11, %xmm10, %xmm4
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm4, %xmm7
movdqa rk19(%rip), %xmm10
movdqa %xmm5, %xmm8
pclmulqdq $0x11, %xmm10, %xmm5
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm5, %xmm7
movdqa rk1(%rip), %xmm10 #xmm10 has rk1 and rk2
#imm value of pclmulqdq instruction
#will determine which constant to use
movdqa %xmm6, %xmm8
pclmulqdq $0x11, %xmm10, %xmm6
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm6, %xmm7
# instead of 64, we add 48 to the loop counter to save 1 instruction
# from the loop instead of a cmp instruction, we use the negative
# flag with the jl instruction
add $128-16, arg3
jl _final_reduction_for_128
# now we have 16+y bytes left to reduce. 16 Bytes is in register xmm7
# and the rest is in memory. We can fold 16 bytes at a time if y>=16
# continue folding 16B at a time
_16B_reduction_loop:
# While >= 16 data bytes remain (not counting xmm7), fold the 16 bytes
# xmm7 into them, storing the result back into xmm7.
jl .Lfold_16_bytes_loop_done
.Lfold_16_bytes_loop:
movdqa %xmm7, %xmm8
pclmulqdq $0x11, %xmm10, %xmm7
pclmulqdq $0x0 , %xmm10, %xmm8
pclmulqdq $0x11, FOLD_CONSTS, %xmm7
pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, %xmm7
movdqu (arg2), %xmm0
pshufb %xmm11, %xmm0
movdqu (buf), %xmm0
pshufb BSWAP_MASK, %xmm0
pxor %xmm0 , %xmm7
add $16, arg2
sub $16, arg3
# instead of a cmp instruction, we utilize the flags with the
# jge instruction equivalent of: cmp arg3, 16-16
# check if there is any more 16B in the buffer to be able to fold
jge _16B_reduction_loop
add $16, buf
sub $16, len
jge .Lfold_16_bytes_loop
#now we have 16+z bytes left to reduce, where 0<= z < 16.
#first, we reduce the data in the xmm7 register
.Lfold_16_bytes_loop_done:
# Add 16 to get the correct number of data bytes remaining in 0...15
# (not counting xmm7), following the previous extra subtraction by 16.
add $16, len
je .Lreduce_final_16_bytes
.Lhandle_partial_segment:
# Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first 16
# bytes are in xmm7 and the rest are the remaining data in 'buf'. To do
# this without needing a fold constant for each possible 'len', redivide
# the bytes into a first chunk of 'len' bytes and a second chunk of 16
# bytes, then fold the first chunk into the second.
_final_reduction_for_128:
# check if any more data to fold. If not, compute the CRC of
# the final 128 bits
add $16, arg3
je _128_done
# here we are getting data that is less than 16 bytes.
# since we know that there was data before the pointer, we can
# offset the input pointer before the actual point, to receive
# exactly 16 bytes. after that the registers need to be adjusted.
_get_last_two_xmms:
movdqa %xmm7, %xmm2
movdqu -16(arg2, arg3), %xmm1
pshufb %xmm11, %xmm1
# xmm1 = last 16 original data bytes
movdqu -16(buf, len), %xmm1
pshufb BSWAP_MASK, %xmm1
# get rid of the extra data that was loaded before
# load the shift constant
lea pshufb_shf_table+16(%rip), %rax
sub arg3, %rax
# xmm2 = high order part of second chunk: xmm7 left-shifted by 'len' bytes.
lea .Lbyteshift_table+16(%rip), %rax
sub len, %rax
movdqu (%rax), %xmm0
# shift xmm2 to the left by arg3 bytes
pshufb %xmm0, %xmm2
# shift xmm7 to the right by 16-arg3 bytes
pxor mask1(%rip), %xmm0
# xmm7 = first chunk: xmm7 right-shifted by '16-len' bytes.
pxor .Lmask1(%rip), %xmm0
pshufb %xmm0, %xmm7
# xmm1 = second chunk: 'len' bytes from xmm1 (low-order bytes),
# then '16-len' bytes from xmm2 (high-order bytes).
pblendvb %xmm2, %xmm1 #xmm0 is implicit
# fold 16 Bytes
movdqa %xmm1, %xmm2
# Fold the first chunk into the second chunk, storing the result in xmm7.
movdqa %xmm7, %xmm8
pclmulqdq $0x11, %xmm10, %xmm7
pclmulqdq $0x0 , %xmm10, %xmm8
pclmulqdq $0x11, FOLD_CONSTS, %xmm7
pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, %xmm7
pxor %xmm2, %xmm7
pxor %xmm1, %xmm7
_128_done:
# compute crc of a 128-bit value
movdqa rk5(%rip), %xmm10 # rk5 and rk6 in xmm10
.Lreduce_final_16_bytes:
# Reduce the 128-bit value M(x), stored in xmm7, to the final 16-bit CRC
# Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
movdqa .Lfinal_fold_consts(%rip), FOLD_CONSTS
# Fold the high 64 bits into the low 64 bits, while also multiplying by
# x^64. This produces a 128-bit value congruent to x^64 * M(x) and
# whose low 48 bits are 0.
movdqa %xmm7, %xmm0
pclmulqdq $0x11, FOLD_CONSTS, %xmm7 # high bits * x^48 * (x^80 mod G(x))
pslldq $8, %xmm0
pxor %xmm0, %xmm7 # + low bits * x^64
#64b fold
pclmulqdq $0x1, %xmm10, %xmm7
pslldq $8 , %xmm0
pxor %xmm0, %xmm7
#32b fold
# Fold the high 32 bits into the low 96 bits. This produces a 96-bit
# value congruent to x^64 * M(x) and whose low 48 bits are 0.
movdqa %xmm7, %xmm0
pand .Lmask2(%rip), %xmm0 # zero high 32 bits
psrldq $12, %xmm7 # extract high 32 bits
pclmulqdq $0x00, FOLD_CONSTS, %xmm7 # high 32 bits * x^48 * (x^48 mod G(x))
pxor %xmm0, %xmm7 # + low bits
pand mask2(%rip), %xmm0
# Load G(x) and floor(x^48 / G(x)).
movdqa .Lbarrett_reduction_consts(%rip), FOLD_CONSTS
psrldq $12, %xmm7
pclmulqdq $0x10, %xmm10, %xmm7
pxor %xmm0, %xmm7
#barrett reduction
_barrett:
movdqa rk7(%rip), %xmm10 # rk7 and rk8 in xmm10
# Use Barrett reduction to compute the final CRC value.
movdqa %xmm7, %xmm0
pclmulqdq $0x01, %xmm10, %xmm7
pslldq $4, %xmm7
pclmulqdq $0x11, %xmm10, %xmm7
pclmulqdq $0x11, FOLD_CONSTS, %xmm7 # high 32 bits * floor(x^48 / G(x))
psrlq $32, %xmm7 # /= x^32
pclmulqdq $0x00, FOLD_CONSTS, %xmm7 # *= G(x)
psrlq $48, %xmm0
pxor %xmm7, %xmm0 # + low 16 nonzero bits
# Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of xmm0.
pslldq $4, %xmm7
pxor %xmm0, %xmm7
pextrd $1, %xmm7, %eax
_cleanup:
# scale the result back to 16 bits
shr $16, %eax
mov %rcx, %rsp
pextrw $0, %xmm0, %eax
ret
########################################################################
.align 16
_less_than_128:
.Lless_than_256_bytes:
# Checksumming a buffer of length 16...255 bytes
# check if there is enough buffer to be able to fold 16B at a time
cmp $32, arg3
jl _less_than_32
movdqa SHUF_MASK(%rip), %xmm11
# Load the first 16 data bytes.
movdqu (buf), %xmm7
pshufb BSWAP_MASK, %xmm7
add $16, buf
# now if there is, load the constants
movdqa rk1(%rip), %xmm10 # rk1 and rk2 in xmm10
movd arg1_low32, %xmm0 # get the initial crc value
pslldq $12, %xmm0 # align it to its correct place
movdqu (arg2), %xmm7 # load the plaintext
pshufb %xmm11, %xmm7 # byte-reflect the plaintext
# XOR the first 16 data *bits* with the initial CRC value.
pxor %xmm0, %xmm0
pinsrw $7, init_crc, %xmm0
pxor %xmm0, %xmm7
# update the buffer pointer
add $16, arg2
# update the counter. subtract 32 instead of 16 to save one
# instruction from the loop
sub $32, arg3
jmp _16B_reduction_loop
.align 16
_less_than_32:
# mov initial crc to the return value. this is necessary for
# zero-length buffers.
mov arg1_low32, %eax
test arg3, arg3
je _cleanup
movdqa SHUF_MASK(%rip), %xmm11
movd arg1_low32, %xmm0 # get the initial crc value
pslldq $12, %xmm0 # align it to its correct place
cmp $16, arg3
je _exact_16_left
jl _less_than_16_left
movdqu (arg2), %xmm7 # load the plaintext
pshufb %xmm11, %xmm7 # byte-reflect the plaintext
pxor %xmm0 , %xmm7 # xor the initial crc value
add $16, arg2
sub $16, arg3
movdqa rk1(%rip), %xmm10 # rk1 and rk2 in xmm10
jmp _get_last_two_xmms
.align 16
_less_than_16_left:
# use stack space to load data less than 16 bytes, zero-out
# the 16B in memory first.
pxor %xmm1, %xmm1
mov %rsp, %r11
movdqa %xmm1, (%r11)
cmp $4, arg3
jl _only_less_than_4
# backup the counter value
mov arg3, %r9
cmp $8, arg3
jl _less_than_8_left
# load 8 Bytes
mov (arg2), %rax
mov %rax, (%r11)
add $8, %r11
sub $8, arg3
add $8, arg2
_less_than_8_left:
cmp $4, arg3
jl _less_than_4_left
# load 4 Bytes
mov (arg2), %eax
mov %eax, (%r11)
add $4, %r11
sub $4, arg3
add $4, arg2
_less_than_4_left:
cmp $2, arg3
jl _less_than_2_left
# load 2 Bytes
mov (arg2), %ax
mov %ax, (%r11)
add $2, %r11
sub $2, arg3
add $2, arg2
_less_than_2_left:
cmp $1, arg3
jl _zero_left
# load 1 Byte
mov (arg2), %al
mov %al, (%r11)
_zero_left:
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
# shl r9, 4
lea pshufb_shf_table+16(%rip), %rax
sub %r9, %rax
movdqu (%rax), %xmm0
pxor mask1(%rip), %xmm0
pshufb %xmm0, %xmm7
jmp _128_done
.align 16
_exact_16_left:
movdqu (arg2), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
jmp _128_done
_only_less_than_4:
cmp $3, arg3
jl _only_less_than_3
# load 3 Bytes
mov (arg2), %al
mov %al, (%r11)
mov 1(arg2), %al
mov %al, 1(%r11)
mov 2(arg2), %al
mov %al, 2(%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $5, %xmm7
jmp _barrett
_only_less_than_3:
cmp $2, arg3
jl _only_less_than_2
# load 2 Bytes
mov (arg2), %al
mov %al, (%r11)
mov 1(arg2), %al
mov %al, 1(%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $6, %xmm7
jmp _barrett
_only_less_than_2:
# load 1 Byte
mov (arg2), %al
mov %al, (%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $7, %xmm7
jmp _barrett
movdqa .Lfold_across_16_bytes_consts(%rip), FOLD_CONSTS
cmp $16, len
je .Lreduce_final_16_bytes # len == 16
sub $32, len
jge .Lfold_16_bytes_loop # 32 <= len <= 255
add $16, len
jmp .Lhandle_partial_segment # 17 <= len <= 31
ENDPROC(crc_t10dif_pcl)
.section .rodata, "a", @progbits
.align 16
# precomputed constants
# these constants are precomputed from the poly:
# 0x8bb70000 (0x8bb7 scaled to 32 bits)
# Q = 0x18BB70000
# rk1 = 2^(32*3) mod Q << 32
# rk2 = 2^(32*5) mod Q << 32
# rk3 = 2^(32*15) mod Q << 32
# rk4 = 2^(32*17) mod Q << 32
# rk5 = 2^(32*3) mod Q << 32
# rk6 = 2^(32*2) mod Q << 32
# rk7 = floor(2^64/Q)
# rk8 = Q
rk1:
.quad 0x2d56000000000000
rk2:
.quad 0x06df000000000000
rk3:
.quad 0x9d9d000000000000
rk4:
.quad 0x7cf5000000000000
rk5:
.quad 0x2d56000000000000
rk6:
.quad 0x1368000000000000
rk7:
.quad 0x00000001f65a57f8
rk8:
.quad 0x000000018bb70000
rk9:
.quad 0xceae000000000000
rk10:
.quad 0xbfd6000000000000
rk11:
.quad 0x1e16000000000000
rk12:
.quad 0x713c000000000000
rk13:
.quad 0xf7f9000000000000
rk14:
.quad 0x80a6000000000000
rk15:
.quad 0x044c000000000000
rk16:
.quad 0xe658000000000000
rk17:
.quad 0xad18000000000000
rk18:
.quad 0xa497000000000000
rk19:
.quad 0x6ee3000000000000
rk20:
.quad 0xe7b5000000000000
# Fold constants precomputed from the polynomial 0x18bb7
# G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
.Lfold_across_128_bytes_consts:
.quad 0x0000000000006123 # x^(8*128) mod G(x)
.quad 0x0000000000002295 # x^(8*128+64) mod G(x)
.Lfold_across_64_bytes_consts:
.quad 0x0000000000001069 # x^(4*128) mod G(x)
.quad 0x000000000000dd31 # x^(4*128+64) mod G(x)
.Lfold_across_32_bytes_consts:
.quad 0x000000000000857d # x^(2*128) mod G(x)
.quad 0x0000000000007acc # x^(2*128+64) mod G(x)
.Lfold_across_16_bytes_consts:
.quad 0x000000000000a010 # x^(1*128) mod G(x)
.quad 0x0000000000001faa # x^(1*128+64) mod G(x)
.Lfinal_fold_consts:
.quad 0x1368000000000000 # x^48 * (x^48 mod G(x))
.quad 0x2d56000000000000 # x^48 * (x^80 mod G(x))
.Lbarrett_reduction_consts:
.quad 0x0000000000018bb7 # G(x)
.quad 0x00000001f65a57f8 # floor(x^48 / G(x))
.section .rodata.cst16.mask1, "aM", @progbits, 16
.align 16
mask1:
.octa 0x80808080808080808080808080808080
.Lmask1:
.octa 0x80808080808080808080808080808080
.section .rodata.cst16.mask2, "aM", @progbits, 16
.align 16
mask2:
.octa 0x00000000FFFFFFFFFFFFFFFFFFFFFFFF
.Lmask2:
.octa 0x00000000FFFFFFFFFFFFFFFFFFFFFFFF
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16
.section .rodata.cst16.bswap_mask, "aM", @progbits, 16
.align 16
SHUF_MASK:
.octa 0x000102030405060708090A0B0C0D0E0F
.Lbswap_mask:
.octa 0x000102030405060708090A0B0C0D0E0F
.section .rodata.cst32.pshufb_shf_table, "aM", @progbits, 32
.align 32
pshufb_shf_table:
# use these values for shift constants for the pshufb instruction
# different alignments result in values as shown:
# DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1
# DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2
# DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3
# DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4
# DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5
# DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6
# DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
# DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
# DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
# DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
# DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
# DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
# DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
# DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
# DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
.octa 0x8f8e8d8c8b8a89888786858483828100
.octa 0x000e0d0c0b0a09080706050403020100
.section .rodata.cst32.byteshift_table, "aM", @progbits, 32
.align 16
# For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 - len]
# is the index vector to shift left by 'len' bytes, and is also {0x80, ...,
# 0x80} XOR the index vector to shift right by '16 - len' bytes.
.Lbyteshift_table:
.byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7
.byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe , 0x0

12
arch/x86/crypto/crct10dif-pclmul_glue.c
View File

@ -33,18 +33,12 @@
#include <asm/cpufeatures.h>
#include <asm/cpu_device_id.h>
asmlinkage __u16 crc_t10dif_pcl(__u16 crc, const unsigned char *buf,
size_t len);
asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);
struct chksum_desc_ctx {
__u16 crc;
};
/*
* Steps through buffer one byte at at time, calculates reflected
* crc using table.
*/
static int chksum_init(struct shash_desc *desc)
{
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
@ -59,7 +53,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data,
{
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
if (irq_fpu_usable()) {
if (length >= 16 && irq_fpu_usable()) {
kernel_fpu_begin();
ctx->crc = crc_t10dif_pcl(ctx->crc, data, length);
kernel_fpu_end();
@ -79,7 +73,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)
static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len,
u8 *out)
{
if (irq_fpu_usable()) {
if (len >= 16 && irq_fpu_usable()) {
kernel_fpu_begin();
*(__u16 *)out = crc_t10dif_pcl(*crcp, data, len);
kernel_fpu_end();

40
arch/x86/crypto/morus1280_glue.c
View File

@ -85,31 +85,20 @@ static void crypto_morus1280_glue_process_ad(
static void crypto_morus1280_glue_process_crypt(struct morus1280_state *state,
struct morus1280_ops ops,
struct aead_request *req)
struct skcipher_walk *walk)
{
struct skcipher_walk walk;
u8 *cursor_src, *cursor_dst;
unsigned int chunksize, base;
while (walk->nbytes >= MORUS1280_BLOCK_SIZE) {
ops.crypt_blocks(state, walk->src.virt.addr,
walk->dst.virt.addr,
round_down(walk->nbytes,
MORUS1280_BLOCK_SIZE));
skcipher_walk_done(walk, walk->nbytes % MORUS1280_BLOCK_SIZE);
}
ops.skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
cursor_src = walk.src.virt.addr;
cursor_dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize);
base = chunksize & ~(MORUS1280_BLOCK_SIZE - 1);
cursor_src += base;
cursor_dst += base;
chunksize &= MORUS1280_BLOCK_SIZE - 1;
if (chunksize > 0)
ops.crypt_tail(state, cursor_src, cursor_dst,
chunksize);
skcipher_walk_done(&walk, 0);
if (walk->nbytes) {
ops.crypt_tail(state, walk->src.virt.addr, walk->dst.virt.addr,
walk->nbytes);
skcipher_walk_done(walk, 0);
}
}
@ -147,12 +136,15 @@ static void crypto_morus1280_glue_crypt(struct aead_request *req,
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct morus1280_ctx *ctx = crypto_aead_ctx(tfm);
struct morus1280_state state;
struct skcipher_walk walk;
ops.skcipher_walk_init(&walk, req, true);
kernel_fpu_begin();
ctx->ops->init(&state, &ctx->key, req->iv);
crypto_morus1280_glue_process_ad(&state, ctx->ops, req->src, req->assoclen);
crypto_morus1280_glue_process_crypt(&state, ops, req);
crypto_morus1280_glue_process_crypt(&state, ops, &walk);
ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();

39
arch/x86/crypto/morus640_glue.c
View File

@ -85,31 +85,19 @@ static void crypto_morus640_glue_process_ad(
static void crypto_morus640_glue_process_crypt(struct morus640_state *state,
struct morus640_ops ops,
struct aead_request *req)
struct skcipher_walk *walk)
{
struct skcipher_walk walk;
u8 *cursor_src, *cursor_dst;
unsigned int chunksize, base;
while (walk->nbytes >= MORUS640_BLOCK_SIZE) {
ops.crypt_blocks(state, walk->src.virt.addr,
walk->dst.virt.addr,
round_down(walk->nbytes, MORUS640_BLOCK_SIZE));
skcipher_walk_done(walk, walk->nbytes % MORUS640_BLOCK_SIZE);
}
ops.skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
cursor_src = walk.src.virt.addr;
cursor_dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize);
base = chunksize & ~(MORUS640_BLOCK_SIZE - 1);
cursor_src += base;
cursor_dst += base;
chunksize &= MORUS640_BLOCK_SIZE - 1;
if (chunksize > 0)
ops.crypt_tail(state, cursor_src, cursor_dst,
chunksize);
skcipher_walk_done(&walk, 0);
if (walk->nbytes) {
ops.crypt_tail(state, walk->src.virt.addr, walk->dst.virt.addr,
walk->nbytes);
skcipher_walk_done(walk, 0);
}
}
@ -143,12 +131,15 @@ static void crypto_morus640_glue_crypt(struct aead_request *req,
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct morus640_ctx *ctx = crypto_aead_ctx(tfm);
struct morus640_state state;
struct skcipher_walk walk;
ops.skcipher_walk_init(&walk, req, true);
kernel_fpu_begin();
ctx->ops->init(&state, &ctx->key, req->iv);
crypto_morus640_glue_process_ad(&state, ctx->ops, req->src, req->assoclen);
crypto_morus640_glue_process_crypt(&state, ops, req);
crypto_morus640_glue_process_crypt(&state, ops, &walk);
ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();

4
arch/x86/crypto/poly1305-sse2-x86_64.S
View File

@ -272,6 +272,10 @@ ENTRY(poly1305_block_sse2)
dec %rcx
jnz .Ldoblock
# Zeroing of key material
mov %rcx,0x00(%rsp)
mov %rcx,0x08(%rsp)
add $0x10,%rsp
pop %r12
pop %rbx

14
crypto/Kconfig
View File

@ -168,6 +168,16 @@ config CRYPTO_MANAGER_DISABLE_TESTS
Disable run-time self tests that normally take place at
algorithm registration.
config CRYPTO_MANAGER_EXTRA_TESTS
bool "Enable extra run-time crypto self tests"
depends on DEBUG_KERNEL && !CRYPTO_MANAGER_DISABLE_TESTS
help
Enable extra run-time self tests of registered crypto algorithms,
including randomized fuzz tests.
This is intended for developer use only, as these tests take much
longer to run than the normal self tests.
config CRYPTO_GF128MUL
tristate "GF(2^128) multiplication functions"
help
@ -642,7 +652,7 @@ config CRYPTO_CRC32_PCLMUL
From Intel Westmere and AMD Bulldozer processor with SSE4.2
and PCLMULQDQ supported, the processor will support
CRC32 PCLMULQDQ implementation using hardware accelerated PCLMULQDQ
instruction. This option will create 'crc32-plcmul' module,
instruction. This option will create 'crc32-pclmul' module,
which will enable any routine to use the CRC-32-IEEE 802.3 checksum
and gain better performance as compared with the table implementation.
@ -671,7 +681,7 @@ config CRYPTO_CRCT10DIF_PCLMUL
For x86_64 processors with SSE4.2 and PCLMULQDQ supported,
CRC T10 DIF PCLMULQDQ computation can be hardware
accelerated PCLMULQDQ instruction. This option will create
'crct10dif-plcmul' module, which is faster when computing the
'crct10dif-pclmul' module, which is faster when computing the
crct10dif checksum as compared with the generic table implementation.
config CRYPTO_CRCT10DIF_VPMSUM

4
crypto/aead.c
View File

@ -61,8 +61,10 @@ int crypto_aead_setkey(struct crypto_aead *tfm,
else
err = crypto_aead_alg(tfm)->setkey(tfm, key, keylen);
if (err)
if (unlikely(err)) {
crypto_aead_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
return err;
}
crypto_aead_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;

7
crypto/aegis.h
View File

@ -1,14 +1,9 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* AEGIS common definitions
*
* Copyright (c) 2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (c) 2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#ifndef _CRYPTO_AEGIS_H

20
crypto/aegis128.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The AEGIS-128 Authenticated-Encryption Algorithm
*
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#include <crypto/algapi.h>
@ -290,19 +286,19 @@ static void crypto_aegis128_process_crypt(struct aegis_state *state,
const struct aegis128_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
unsigned int nbytes = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize);
if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0);
ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
}

20
crypto/aegis128l.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The AEGIS-128L Authenticated-Encryption Algorithm
*
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#include <crypto/algapi.h>
@ -353,19 +349,19 @@ static void crypto_aegis128l_process_crypt(struct aegis_state *state,
const struct aegis128l_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
unsigned int nbytes = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize);
if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0);
ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
}

20
crypto/aegis256.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The AEGIS-256 Authenticated-Encryption Algorithm
*
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#include <crypto/algapi.h>
@ -303,19 +299,19 @@ static void crypto_aegis256_process_crypt(struct aegis_state *state,
const struct aegis256_ops *ops)
{
struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
unsigned int nbytes = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize);
if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0);
ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
}

36
crypto/af_alg.c
View File

@ -304,8 +304,6 @@ int af_alg_accept(struct sock *sk, struct socket *newsock, bool kern)
if (err)
goto unlock;
sk2->sk_family = PF_ALG;
if (nokey || !ask->refcnt++)
sock_hold(sk);
ask->nokey_refcnt += nokey;
@ -382,7 +380,6 @@ static int alg_create(struct net *net, struct socket *sock, int protocol,
sock->ops = &alg_proto_ops;
sock_init_data(sock, sk);
sk->sk_family = PF_ALG;
sk->sk_destruct = alg_sock_destruct;
return 0;
@ -427,12 +424,12 @@ int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len)
}
EXPORT_SYMBOL_GPL(af_alg_make_sg);
void af_alg_link_sg(struct af_alg_sgl *sgl_prev, struct af_alg_sgl *sgl_new)
static void af_alg_link_sg(struct af_alg_sgl *sgl_prev,
struct af_alg_sgl *sgl_new)
{
sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1);
sg_chain(sgl_prev->sg, sgl_prev->npages + 1, sgl_new->sg);
}
EXPORT_SYMBOL_GPL(af_alg_link_sg);
void af_alg_free_sg(struct af_alg_sgl *sgl)
{
@ -443,7 +440,7 @@ void af_alg_free_sg(struct af_alg_sgl *sgl)
}
EXPORT_SYMBOL_GPL(af_alg_free_sg);
int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con)
static int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con)
{
struct cmsghdr *cmsg;
@ -482,7 +479,6 @@ int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con)
return 0;
}
EXPORT_SYMBOL_GPL(af_alg_cmsg_send);
/**
* af_alg_alloc_tsgl - allocate the TX SGL
@ -490,7 +486,7 @@ EXPORT_SYMBOL_GPL(af_alg_cmsg_send);
* @sk socket of connection to user space
* @return: 0 upon success, < 0 upon error
*/
int af_alg_alloc_tsgl(struct sock *sk)
static int af_alg_alloc_tsgl(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
@ -519,7 +515,6 @@ int af_alg_alloc_tsgl(struct sock *sk)
return 0;
}
EXPORT_SYMBOL_GPL(af_alg_alloc_tsgl);
/**
* aead_count_tsgl - Count number of TX SG entries
@ -534,17 +529,17 @@ EXPORT_SYMBOL_GPL(af_alg_alloc_tsgl);
*/
unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset)
{
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
struct af_alg_tsgl *sgl, *tmp;
const struct alg_sock *ask = alg_sk(sk);
const struct af_alg_ctx *ctx = ask->private;
const struct af_alg_tsgl *sgl;
unsigned int i;
unsigned int sgl_count = 0;
if (!bytes)
return 0;
list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
struct scatterlist *sg = sgl->sg;
list_for_each_entry(sgl, &ctx->tsgl_list, list) {
const struct scatterlist *sg = sgl->sg;
for (i = 0; i < sgl->cur; i++) {
size_t bytes_count;
@ -642,8 +637,7 @@ void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
}
list_del(&sgl->list);
sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) *
(MAX_SGL_ENTS + 1));
sock_kfree_s(sk, sgl, struct_size(sgl, sg, MAX_SGL_ENTS + 1));
}
if (!ctx->used)
@ -656,7 +650,7 @@ EXPORT_SYMBOL_GPL(af_alg_pull_tsgl);
*
* @areq Request holding the TX and RX SGL
*/
void af_alg_free_areq_sgls(struct af_alg_async_req *areq)
static void af_alg_free_areq_sgls(struct af_alg_async_req *areq)
{
struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk);
@ -685,7 +679,6 @@ void af_alg_free_areq_sgls(struct af_alg_async_req *areq)
sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
}
}
EXPORT_SYMBOL_GPL(af_alg_free_areq_sgls);
/**
* af_alg_wait_for_wmem - wait for availability of writable memory
@ -694,7 +687,7 @@ EXPORT_SYMBOL_GPL(af_alg_free_areq_sgls);
* @flags If MSG_DONTWAIT is set, then only report if function would sleep
* @return 0 when writable memory is available, < 0 upon error
*/
int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags)
static int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags)
{
DEFINE_WAIT_FUNC(wait, woken_wake_function);
int err = -ERESTARTSYS;
@ -719,7 +712,6 @@ int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags)
return err;
}
EXPORT_SYMBOL_GPL(af_alg_wait_for_wmem);
/**
* af_alg_wmem_wakeup - wakeup caller when writable memory is available
@ -788,8 +780,7 @@ EXPORT_SYMBOL_GPL(af_alg_wait_for_data);
*
* @sk socket of connection to user space
*/
void af_alg_data_wakeup(struct sock *sk)
static void af_alg_data_wakeup(struct sock *sk)
{
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
@ -807,7 +798,6 @@ void af_alg_data_wakeup(struct sock *sk)
sk_wake_async(sk, SOCK_WAKE_SPACE, POLL_OUT);
rcu_read_unlock();
}
EXPORT_SYMBOL_GPL(af_alg_data_wakeup);
/**
* af_alg_sendmsg - implementation of sendmsg system call handler

42
crypto/ahash.c
View File

@ -86,17 +86,17 @@ static int hash_walk_new_entry(struct crypto_hash_walk *walk)
int crypto_hash_walk_done(struct crypto_hash_walk *walk, int err)
{
unsigned int alignmask = walk->alignmask;
unsigned int nbytes = walk->entrylen;
walk->data -= walk->offset;
if (nbytes && walk->offset & alignmask && !err) {
walk->offset = ALIGN(walk->offset, alignmask + 1);
nbytes = min(nbytes,
((unsigned int)(PAGE_SIZE)) - walk->offset);
walk->entrylen -= nbytes;
if (walk->entrylen && (walk->offset & alignmask) && !err) {
unsigned int nbytes;
walk->offset = ALIGN(walk->offset, alignmask + 1);
nbytes = min(walk->entrylen,
(unsigned int)(PAGE_SIZE - walk->offset));
if (nbytes) {
walk->entrylen -= nbytes;
walk->data += walk->offset;
return nbytes;
}
@ -116,7 +116,7 @@ int crypto_hash_walk_done(struct crypto_hash_walk *walk, int err)
if (err)
return err;
if (nbytes) {
if (walk->entrylen) {
walk->offset = 0;
walk->pg++;
return hash_walk_next(walk);
@ -190,6 +190,21 @@ static int ahash_setkey_unaligned(struct crypto_ahash *tfm, const u8 *key,
return ret;
}
static int ahash_nosetkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen)
{
return -ENOSYS;
}
static void ahash_set_needkey(struct crypto_ahash *tfm)
{
const struct hash_alg_common *alg = crypto_hash_alg_common(tfm);
if (tfm->setkey != ahash_nosetkey &&
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_ahash_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen)
{
@ -201,20 +216,16 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key,
else
err = tfm->setkey(tfm, key, keylen);
if (err)
if (unlikely(err)) {
ahash_set_needkey(tfm);
return err;
}
crypto_ahash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;
}
EXPORT_SYMBOL_GPL(crypto_ahash_setkey);
static int ahash_nosetkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen)
{
return -ENOSYS;
}
static inline unsigned int ahash_align_buffer_size(unsigned len,
unsigned long mask)
{
@ -489,8 +500,7 @@ static int crypto_ahash_init_tfm(struct crypto_tfm *tfm)
if (alg->setkey) {
hash->setkey = alg->setkey;
if (!(alg->halg.base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_ahash_set_flags(hash, CRYPTO_TFM_NEED_KEY);
ahash_set_needkey(hash);
}
return 0;

63
crypto/algapi.c
View File

@ -494,6 +494,24 @@ out:
}
EXPORT_SYMBOL_GPL(crypto_register_template);
int crypto_register_templates(struct crypto_template *tmpls, int count)
{
int i, err;
for (i = 0; i < count; i++) {
err = crypto_register_template(&tmpls[i]);
if (err)
goto out;
}
return 0;
out:
for (--i; i >= 0; --i)
crypto_unregister_template(&tmpls[i]);
return err;
}
EXPORT_SYMBOL_GPL(crypto_register_templates);
void crypto_unregister_template(struct crypto_template *tmpl)
{
struct crypto_instance *inst;
@ -523,6 +541,15 @@ void crypto_unregister_template(struct crypto_template *tmpl)
}
EXPORT_SYMBOL_GPL(crypto_unregister_template);
void crypto_unregister_templates(struct crypto_template *tmpls, int count)
{
int i;
for (i = count - 1; i >= 0; --i)
crypto_unregister_template(&tmpls[i]);
}
EXPORT_SYMBOL_GPL(crypto_unregister_templates);
static struct crypto_template *__crypto_lookup_template(const char *name)
{
struct crypto_template *q, *tmpl = NULL;
@ -608,6 +635,9 @@ int crypto_init_spawn(struct crypto_spawn *spawn, struct crypto_alg *alg,
{
int err = -EAGAIN;
if (WARN_ON_ONCE(inst == NULL))
return -EINVAL;
spawn->inst = inst;
spawn->mask = mask;
@ -845,8 +875,8 @@ int crypto_inst_setname(struct crypto_instance *inst, const char *name,
}
EXPORT_SYMBOL_GPL(crypto_inst_setname);
void *crypto_alloc_instance2(const char *name, struct crypto_alg *alg,
unsigned int head)
void *crypto_alloc_instance(const char *name, struct crypto_alg *alg,
unsigned int head)
{
struct crypto_instance *inst;
char *p;
@ -869,35 +899,6 @@ err_free_inst:
kfree(p);
return ERR_PTR(err);
}
EXPORT_SYMBOL_GPL(crypto_alloc_instance2);
struct crypto_instance *crypto_alloc_instance(const char *name,
struct crypto_alg *alg)
{
struct crypto_instance *inst;
struct crypto_spawn *spawn;
int err;
inst = crypto_alloc_instance2(name, alg, 0);
if (IS_ERR(inst))
goto out;
spawn = crypto_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, inst,
CRYPTO_ALG_TYPE_MASK | CRYPTO_ALG_ASYNC);
if (err)
goto err_free_inst;
return inst;
err_free_inst:
kfree(inst);
inst = ERR_PTR(err);
out:
return inst;
}
EXPORT_SYMBOL_GPL(crypto_alloc_instance);
void crypto_init_queue(struct crypto_queue *queue, unsigned int max_qlen)

87
crypto/arc4.c
View File

@ -12,14 +12,11 @@
*
*/
#include <linux/module.h>
#include <linux/init.h>
#include <linux/crypto.h>
#include <crypto/algapi.h>
#define ARC4_MIN_KEY_SIZE 1
#define ARC4_MAX_KEY_SIZE 256
#define ARC4_BLOCK_SIZE 1
#include <crypto/arc4.h>
#include <crypto/internal/skcipher.h>
#include <linux/init.h>
#include <linux/module.h>
struct arc4_ctx {
u32 S[256];
@ -50,6 +47,12 @@ static int arc4_set_key(struct crypto_tfm *tfm, const u8 *in_key,
return 0;
}
static int arc4_set_key_skcipher(struct crypto_skcipher *tfm, const u8 *in_key,
unsigned int key_len)
{
return arc4_set_key(&tfm->base, in_key, key_len);
}
static void arc4_crypt(struct arc4_ctx *ctx, u8 *out, const u8 *in,
unsigned int len)
{
@ -92,30 +95,25 @@ static void arc4_crypt_one(struct crypto_tfm *tfm, u8 *out, const u8 *in)
arc4_crypt(crypto_tfm_ctx(tfm), out, in, 1);
}
static int ecb_arc4_crypt(struct blkcipher_desc *desc, struct scatterlist *dst,
struct scatterlist *src, unsigned int nbytes)
static int ecb_arc4_crypt(struct skcipher_request *req)
{
struct arc4_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
struct blkcipher_walk walk;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct arc4_ctx *ctx = crypto_skcipher_ctx(tfm);
struct skcipher_walk walk;
int err;
blkcipher_walk_init(&walk, dst, src, nbytes);
err = blkcipher_walk_virt(desc, &walk);
err = skcipher_walk_virt(&walk, req, false);
while (walk.nbytes > 0) {
u8 *wsrc = walk.src.virt.addr;
u8 *wdst = walk.dst.virt.addr;
arc4_crypt(ctx, wdst, wsrc, walk.nbytes);
err = blkcipher_walk_done(desc, &walk, 0);
arc4_crypt(ctx, walk.dst.virt.addr, walk.src.virt.addr,
walk.nbytes);
err = skcipher_walk_done(&walk, 0);
}
return err;
}
static struct crypto_alg arc4_algs[2] = { {
static struct crypto_alg arc4_cipher = {
.cra_name = "arc4",
.cra_flags = CRYPTO_ALG_TYPE_CIPHER,
.cra_blocksize = ARC4_BLOCK_SIZE,
@ -130,34 +128,39 @@ static struct crypto_alg arc4_algs[2] = { {
.cia_decrypt = arc4_crypt_one,
},
},
}, {
.cra_name = "ecb(arc4)",
.cra_priority = 100,
.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
.cra_blocksize = ARC4_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct arc4_ctx),
.cra_alignmask = 0,
.cra_type = &crypto_blkcipher_type,
.cra_module = THIS_MODULE,
.cra_u = {
.blkcipher = {
.min_keysize = ARC4_MIN_KEY_SIZE,
.max_keysize = ARC4_MAX_KEY_SIZE,
.setkey = arc4_set_key,
.encrypt = ecb_arc4_crypt,
.decrypt = ecb_arc4_crypt,
},
},
} };
};
static struct skcipher_alg arc4_skcipher = {
.base.cra_name = "ecb(arc4)",
.base.cra_priority = 100,
.base.cra_blocksize = ARC4_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct arc4_ctx),
.base.cra_module = THIS_MODULE,
.min_keysize = ARC4_MIN_KEY_SIZE,
.max_keysize = ARC4_MAX_KEY_SIZE,
.setkey = arc4_set_key_skcipher,
.encrypt = ecb_arc4_crypt,
.decrypt = ecb_arc4_crypt,
};
static int __init arc4_init(void)
{
return crypto_register_algs(arc4_algs, ARRAY_SIZE(arc4_algs));
int err;
err = crypto_register_alg(&arc4_cipher);
if (err)
return err;
err = crypto_register_skcipher(&arc4_skcipher);
if (err)
crypto_unregister_alg(&arc4_cipher);
return err;
}
static void __exit arc4_exit(void)
{
crypto_unregister_algs(arc4_algs, ARRAY_SIZE(arc4_algs));
crypto_unregister_alg(&arc4_cipher);
crypto_unregister_skcipher(&arc4_skcipher);
}
module_init(arc4_init);

131
crypto/cbc.c
View File

@ -18,34 +18,11 @@
#include <linux/kernel.h>
#include <linux/log2.h>
#include <linux/module.h>
#include <linux/slab.h>
struct crypto_cbc_ctx {
struct crypto_cipher *child;
};
static int crypto_cbc_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static inline void crypto_cbc_encrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_cipher_encrypt_one(ctx->child, dst, src);
crypto_cipher_encrypt_one(skcipher_cipher_simple(tfm), dst, src);
}
static int crypto_cbc_encrypt(struct skcipher_request *req)
@ -56,9 +33,7 @@ static int crypto_cbc_encrypt(struct skcipher_request *req)
static inline void crypto_cbc_decrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_cipher_decrypt_one(ctx->child, dst, src);
crypto_cipher_decrypt_one(skcipher_cipher_simple(tfm), dst, src);
}
static int crypto_cbc_decrypt(struct skcipher_request *req)
@ -78,113 +53,33 @@ static int crypto_cbc_decrypt(struct skcipher_request *req)
return err;
}
static int crypto_cbc_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_cbc_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_cbc_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg;
u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER);
if (err)
return err;
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
algt = crypto_get_attr_type(tb);
err = PTR_ERR(algt);
if (IS_ERR(algt))
goto err_free_inst;
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "cbc", alg);
if (err)
goto err_drop_spawn;
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
err = -EINVAL;
if (!is_power_of_2(alg->cra_blocksize))
goto err_drop_spawn;
goto out_free_inst;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_cbc_ctx);
inst->alg.init = crypto_cbc_init_tfm;
inst->alg.exit = crypto_cbc_exit_tfm;
inst->alg.setkey = crypto_cbc_setkey;
inst->alg.encrypt = crypto_cbc_encrypt;
inst->alg.decrypt = crypto_cbc_decrypt;
inst->free = crypto_cbc_free;
err = skcipher_register_instance(tmpl, inst);
if (err)
goto err_drop_spawn;
crypto_mod_put(alg);
goto out_free_inst;
goto out_put_alg;
out:
out_free_inst:
inst->free(inst);
out_put_alg:
crypto_mod_put(alg);
return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
}
static struct crypto_template crypto_cbc_tmpl = {
@ -207,5 +102,5 @@ module_init(crypto_cbc_module_init);
module_exit(crypto_cbc_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CBC block cipher algorithm");
MODULE_DESCRIPTION("CBC block cipher mode of operation");
MODULE_ALIAS_CRYPTO("cbc");

78
crypto/ccm.c
View File

@ -589,12 +589,6 @@ static int crypto_ccm_create(struct crypto_template *tmpl, struct rtattr **tb)
mac_name);
}
static struct crypto_template crypto_ccm_tmpl = {
.name = "ccm",
.create = crypto_ccm_create,
.module = THIS_MODULE,
};
static int crypto_ccm_base_create(struct crypto_template *tmpl,
struct rtattr **tb)
{
@ -618,12 +612,6 @@ static int crypto_ccm_base_create(struct crypto_template *tmpl,
cipher_name);
}
static struct crypto_template crypto_ccm_base_tmpl = {
.name = "ccm_base",
.create = crypto_ccm_base_create,
.module = THIS_MODULE,
};
static int crypto_rfc4309_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen)
{
@ -854,12 +842,6 @@ out_free_inst:
goto out;
}
static struct crypto_template crypto_rfc4309_tmpl = {
.name = "rfc4309",
.create = crypto_rfc4309_create,
.module = THIS_MODULE,
};
static int crypto_cbcmac_digest_setkey(struct crypto_shash *parent,
const u8 *inkey, unsigned int keylen)
{
@ -999,51 +981,37 @@ out_put_alg:
return err;
}
static struct crypto_template crypto_cbcmac_tmpl = {
.name = "cbcmac",
.create = cbcmac_create,
.free = shash_free_instance,
.module = THIS_MODULE,
static struct crypto_template crypto_ccm_tmpls[] = {
{
.name = "cbcmac",
.create = cbcmac_create,
.free = shash_free_instance,
.module = THIS_MODULE,
}, {
.name = "ccm_base",
.create = crypto_ccm_base_create,
.module = THIS_MODULE,
}, {
.name = "ccm",
.create = crypto_ccm_create,
.module = THIS_MODULE,
}, {
.name = "rfc4309",
.create = crypto_rfc4309_create,
.module = THIS_MODULE,
},
};
static int __init crypto_ccm_module_init(void)
{
int err;
err = crypto_register_template(&crypto_cbcmac_tmpl);
if (err)
goto out;
err = crypto_register_template(&crypto_ccm_base_tmpl);
if (err)
goto out_undo_cbcmac;
err = crypto_register_template(&crypto_ccm_tmpl);
if (err)
goto out_undo_base;
err = crypto_register_template(&crypto_rfc4309_tmpl);
if (err)
goto out_undo_ccm;
out:
return err;
out_undo_ccm:
crypto_unregister_template(&crypto_ccm_tmpl);
out_undo_base:
crypto_unregister_template(&crypto_ccm_base_tmpl);
out_undo_cbcmac:
crypto_register_template(&crypto_cbcmac_tmpl);
goto out;
return crypto_register_templates(crypto_ccm_tmpls,
ARRAY_SIZE(crypto_ccm_tmpls));
}
static void __exit crypto_ccm_module_exit(void)
{
crypto_unregister_template(&crypto_rfc4309_tmpl);
crypto_unregister_template(&crypto_ccm_tmpl);
crypto_unregister_template(&crypto_ccm_base_tmpl);
crypto_unregister_template(&crypto_cbcmac_tmpl);
crypto_unregister_templates(crypto_ccm_tmpls,
ARRAY_SIZE(crypto_ccm_tmpls));
}
module_init(crypto_ccm_module_init);

139
crypto/cfb.c
View File

@ -25,28 +25,17 @@
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/types.h>
struct crypto_cfb_ctx {
struct crypto_cipher *child;
};
static unsigned int crypto_cfb_bsize(struct crypto_skcipher *tfm)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
return crypto_cipher_blocksize(child);
return crypto_cipher_blocksize(skcipher_cipher_simple(tfm));
}
static void crypto_cfb_encrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_cipher_encrypt_one(ctx->child, dst, src);
crypto_cipher_encrypt_one(skcipher_cipher_simple(tfm), dst, src);
}
/* final encrypt and decrypt is the same */
@ -77,12 +66,14 @@ static int crypto_cfb_encrypt_segment(struct skcipher_walk *walk,
do {
crypto_cfb_encrypt_one(tfm, iv, dst);
crypto_xor(dst, src, bsize);
memcpy(iv, dst, bsize);
iv = dst;
src += bsize;
dst += bsize;
} while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes;
}
@ -162,7 +153,7 @@ static int crypto_cfb_decrypt_inplace(struct skcipher_walk *walk,
const unsigned int bsize = crypto_cfb_bsize(tfm);
unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv;
u8 * const iv = walk->iv;
u8 tmp[MAX_CIPHER_BLOCKSIZE];
do {
@ -172,8 +163,6 @@ static int crypto_cfb_decrypt_inplace(struct skcipher_walk *walk,
src += bsize;
} while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes;
}
@ -186,22 +175,6 @@ static int crypto_cfb_decrypt_blocks(struct skcipher_walk *walk,
return crypto_cfb_decrypt_segment(walk, tfm);
}
static int crypto_cfb_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_cfb_decrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
@ -224,110 +197,34 @@ static int crypto_cfb_decrypt(struct skcipher_request *req)
return err;
}
static int crypto_cfb_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_cfb_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_cfb_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_cfb_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg;
u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER);
if (err)
return err;
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
algt = crypto_get_attr_type(tb);
err = PTR_ERR(algt);
if (IS_ERR(algt))
goto err_free_inst;
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "cfb", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_priority = alg->cra_priority;
/* we're a stream cipher independend of the crypto cra_blocksize */
/* CFB mode is a stream cipher. */
inst->alg.base.cra_blocksize = 1;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
/*
* To simplify the implementation, configure the skcipher walk to only
* give a partial block at the very end, never earlier.
*/
inst->alg.chunksize = alg->cra_blocksize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_cfb_ctx);
inst->alg.init = crypto_cfb_init_tfm;
inst->alg.exit = crypto_cfb_exit_tfm;
inst->alg.setkey = crypto_cfb_setkey;
inst->alg.encrypt = crypto_cfb_encrypt;
inst->alg.decrypt = crypto_cfb_decrypt;
inst->free = crypto_cfb_free;
err = skcipher_register_instance(tmpl, inst);
if (err)
goto err_drop_spawn;
crypto_mod_put(alg);
inst->free(inst);
out:
crypto_mod_put(alg);
return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
}
static struct crypto_template crypto_cfb_tmpl = {
@ -350,5 +247,5 @@ module_init(crypto_cfb_module_init);
module_exit(crypto_cfb_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CFB block cipher algorithm");
MODULE_DESCRIPTION("CFB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("cfb");

37
crypto/chacha20poly1305.c
View File

@ -701,37 +701,28 @@ static int rfc7539esp_create(struct crypto_template *tmpl, struct rtattr **tb)
return chachapoly_create(tmpl, tb, "rfc7539esp", 8);
}
static struct crypto_template rfc7539_tmpl = {
.name = "rfc7539",
.create = rfc7539_create,
.module = THIS_MODULE,
};
static struct crypto_template rfc7539esp_tmpl = {
.name = "rfc7539esp",
.create = rfc7539esp_create,
.module = THIS_MODULE,
static struct crypto_template rfc7539_tmpls[] = {
{
.name = "rfc7539",
.create = rfc7539_create,
.module = THIS_MODULE,
}, {
.name = "rfc7539esp",
.create = rfc7539esp_create,
.module = THIS_MODULE,
},
};
static int __init chacha20poly1305_module_init(void)
{
int err;
err = crypto_register_template(&rfc7539_tmpl);
if (err)
return err;
err = crypto_register_template(&rfc7539esp_tmpl);
if (err)
crypto_unregister_template(&rfc7539_tmpl);
return err;
return crypto_register_templates(rfc7539_tmpls,
ARRAY_SIZE(rfc7539_tmpls));
}
static void __exit chacha20poly1305_module_exit(void)
{
crypto_unregister_template(&rfc7539esp_tmpl);
crypto_unregister_template(&rfc7539_tmpl);
crypto_unregister_templates(rfc7539_tmpls,
ARRAY_SIZE(rfc7539_tmpls));
}
module_init(chacha20poly1305_module_init);

57
crypto/crypto_null.c
View File

@ -65,6 +65,10 @@ static int null_hash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen)
{ return 0; }
static int null_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int keylen)
{ return 0; }
static int null_setkey(struct crypto_tfm *tfm, const u8 *key,
unsigned int keylen)
{ return 0; }
@ -74,21 +78,18 @@ static void null_crypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
memcpy(dst, src, NULL_BLOCK_SIZE);
}
static int skcipher_null_crypt(struct blkcipher_desc *desc,
struct scatterlist *dst,
struct scatterlist *src, unsigned int nbytes)
static int null_skcipher_crypt(struct skcipher_request *req)
{
struct blkcipher_walk walk;
struct skcipher_walk walk;
int err;
blkcipher_walk_init(&walk, dst, src, nbytes);
err = blkcipher_walk_virt(desc, &walk);
err = skcipher_walk_virt(&walk, req, false);
while (walk.nbytes) {
if (walk.src.virt.addr != walk.dst.virt.addr)
memcpy(walk.dst.virt.addr, walk.src.virt.addr,
walk.nbytes);
err = blkcipher_walk_done(desc, &walk, 0);
err = skcipher_walk_done(&walk, 0);
}
return err;
@ -109,7 +110,22 @@ static struct shash_alg digest_null = {
}
};
static struct crypto_alg null_algs[3] = { {
static struct skcipher_alg skcipher_null = {
.base.cra_name = "ecb(cipher_null)",
.base.cra_driver_name = "ecb-cipher_null",
.base.cra_priority = 100,
.base.cra_blocksize = NULL_BLOCK_SIZE,
.base.cra_ctxsize = 0,
.base.cra_module = THIS_MODULE,
.min_keysize = NULL_KEY_SIZE,
.max_keysize = NULL_KEY_SIZE,
.ivsize = NULL_IV_SIZE,
.setkey = null_skcipher_setkey,
.encrypt = null_skcipher_crypt,
.decrypt = null_skcipher_crypt,
};
static struct crypto_alg null_algs[] = { {
.cra_name = "cipher_null",
.cra_flags = CRYPTO_ALG_TYPE_CIPHER,
.cra_blocksize = NULL_BLOCK_SIZE,
@ -121,22 +137,6 @@ static struct crypto_alg null_algs[3] = { {
.cia_setkey = null_setkey,
.cia_encrypt = null_crypt,
.cia_decrypt = null_crypt } }
}, {
.cra_name = "ecb(cipher_null)",
.cra_driver_name = "ecb-cipher_null",
.cra_priority = 100,
.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
.cra_blocksize = NULL_BLOCK_SIZE,
.cra_type = &crypto_blkcipher_type,
.cra_ctxsize = 0,
.cra_module = THIS_MODULE,
.cra_u = { .blkcipher = {
.min_keysize = NULL_KEY_SIZE,
.max_keysize = NULL_KEY_SIZE,
.ivsize = NULL_IV_SIZE,
.setkey = null_setkey,
.encrypt = skcipher_null_crypt,
.decrypt = skcipher_null_crypt } }
}, {
.cra_name = "compress_null",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
@ -199,8 +199,14 @@ static int __init crypto_null_mod_init(void)
if (ret < 0)
goto out_unregister_algs;
ret = crypto_register_skcipher(&skcipher_null);
if (ret < 0)
goto out_unregister_shash;
return 0;
out_unregister_shash:
crypto_unregister_shash(&digest_null);
out_unregister_algs:
crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs));
out:
@ -209,8 +215,9 @@ out:
static void __exit crypto_null_mod_fini(void)
{
crypto_unregister_shash(&digest_null);
crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs));
crypto_unregister_shash(&digest_null);
crypto_unregister_skcipher(&skcipher_null);
}
module_init(crypto_null_mod_init);

4
crypto/crypto_user_stat.c
View File

@ -20,10 +20,6 @@
#define null_terminated(x) (strnlen(x, sizeof(x)) < sizeof(x))
static DEFINE_MUTEX(crypto_cfg_mutex);
extern struct sock *crypto_nlsk;
struct crypto_dump_info {
struct sk_buff *in_skb;
struct sk_buff *out_skb;

200
crypto/ctr.c
View File

@ -17,14 +17,8 @@
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/random.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
struct crypto_ctr_ctx {
struct crypto_cipher *child;
};
struct crypto_rfc3686_ctx {
struct crypto_skcipher *child;
u8 nonce[CTR_RFC3686_NONCE_SIZE];
@ -35,24 +29,7 @@ struct crypto_rfc3686_req_ctx {
struct skcipher_request subreq CRYPTO_MINALIGN_ATTR;
};
static int crypto_ctr_setkey(struct crypto_tfm *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static void crypto_ctr_crypt_final(struct blkcipher_walk *walk,
static void crypto_ctr_crypt_final(struct skcipher_walk *walk,
struct crypto_cipher *tfm)
{
unsigned int bsize = crypto_cipher_blocksize(tfm);
@ -70,7 +47,7 @@ static void crypto_ctr_crypt_final(struct blkcipher_walk *walk,
crypto_inc(ctrblk, bsize);
}
static int crypto_ctr_crypt_segment(struct blkcipher_walk *walk,
static int crypto_ctr_crypt_segment(struct skcipher_walk *walk,
struct crypto_cipher *tfm)
{
void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
@ -96,7 +73,7 @@ static int crypto_ctr_crypt_segment(struct blkcipher_walk *walk,
return nbytes;
}
static int crypto_ctr_crypt_inplace(struct blkcipher_walk *walk,
static int crypto_ctr_crypt_inplace(struct skcipher_walk *walk,
struct crypto_cipher *tfm)
{
void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
@ -123,138 +100,77 @@ static int crypto_ctr_crypt_inplace(struct blkcipher_walk *walk,
return nbytes;
}
static int crypto_ctr_crypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
static int crypto_ctr_crypt(struct skcipher_request *req)
{
struct blkcipher_walk walk;
struct crypto_blkcipher *tfm = desc->tfm;
struct crypto_ctr_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
unsigned int bsize = crypto_cipher_blocksize(child);
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct skcipher_walk walk;
unsigned int nbytes;
int err;
blkcipher_walk_init(&walk, dst, src, nbytes);
err = blkcipher_walk_virt_block(desc, &walk, bsize);
err = skcipher_walk_virt(&walk, req, false);
while (walk.nbytes >= bsize) {
if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_ctr_crypt_inplace(&walk, child);
nbytes = crypto_ctr_crypt_inplace(&walk, cipher);
else
nbytes = crypto_ctr_crypt_segment(&walk, child);
nbytes = crypto_ctr_crypt_segment(&walk, cipher);
err = blkcipher_walk_done(desc, &walk, nbytes);
err = skcipher_walk_done(&walk, nbytes);
}
if (walk.nbytes) {
crypto_ctr_crypt_final(&walk, child);
err = blkcipher_walk_done(desc, &walk, 0);
crypto_ctr_crypt_final(&walk, cipher);
err = skcipher_walk_done(&walk, 0);
}
return err;
}
static int crypto_ctr_init_tfm(struct crypto_tfm *tfm)
static int crypto_ctr_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct crypto_instance *inst = (void *)tfm->__crt_alg;
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ctr_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_ctr_alloc(struct rtattr **tb)
{
struct crypto_instance *inst;
struct crypto_attr_type *algt;
struct skcipher_instance *inst;
struct crypto_alg *alg;
u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER);
if (err)
return ERR_PTR(err);
algt = crypto_get_attr_type(tb);
if (IS_ERR(algt))
return ERR_CAST(algt);
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, mask);
if (IS_ERR(alg))
return ERR_CAST(alg);
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
/* Block size must be >= 4 bytes. */
err = -EINVAL;
if (alg->cra_blocksize < 4)
goto out_put_alg;
goto out_free_inst;
/* If this is false we'd fail the alignment of crypto_inc. */
if (alg->cra_blocksize % 4)
goto out_put_alg;
goto out_free_inst;
inst = crypto_alloc_instance("ctr", alg);
if (IS_ERR(inst))
goto out;
/* CTR mode is a stream cipher. */
inst->alg.base.cra_blocksize = 1;
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER;
inst->alg.cra_priority = alg->cra_priority;
inst->alg.cra_blocksize = 1;
inst->alg.cra_alignmask = alg->cra_alignmask;
inst->alg.cra_type = &crypto_blkcipher_type;
/*
* To simplify the implementation, configure the skcipher walk to only
* give a partial block at the very end, never earlier.
*/
inst->alg.chunksize = alg->cra_blocksize;
inst->alg.cra_blkcipher.ivsize = alg->cra_blocksize;
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.encrypt = crypto_ctr_crypt;
inst->alg.decrypt = crypto_ctr_crypt;
inst->alg.cra_ctxsize = sizeof(struct crypto_ctr_ctx);
inst->alg.cra_init = crypto_ctr_init_tfm;
inst->alg.cra_exit = crypto_ctr_exit_tfm;
inst->alg.cra_blkcipher.setkey = crypto_ctr_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_ctr_crypt;
inst->alg.cra_blkcipher.decrypt = crypto_ctr_crypt;
out:
crypto_mod_put(alg);
return inst;
err = skcipher_register_instance(tmpl, inst);
if (err)
goto out_free_inst;
goto out_put_alg;
out_free_inst:
inst->free(inst);
out_put_alg:
inst = ERR_PTR(err);
goto out;
crypto_mod_put(alg);
return err;
}
static void crypto_ctr_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
}
static struct crypto_template crypto_ctr_tmpl = {
.name = "ctr",
.alloc = crypto_ctr_alloc,
.free = crypto_ctr_free,
.module = THIS_MODULE,
};
static int crypto_rfc3686_setkey(struct crypto_skcipher *parent,
const u8 *key, unsigned int keylen)
{
@ -444,42 +360,34 @@ err_free_inst:
goto out;
}
static struct crypto_template crypto_rfc3686_tmpl = {
.name = "rfc3686",
.create = crypto_rfc3686_create,
.module = THIS_MODULE,
static struct crypto_template crypto_ctr_tmpls[] = {
{
.name = "ctr",
.create = crypto_ctr_create,
.module = THIS_MODULE,
}, {
.name = "rfc3686",
.create = crypto_rfc3686_create,
.module = THIS_MODULE,
},
};
static int __init crypto_ctr_module_init(void)
{
int err;
err = crypto_register_template(&crypto_ctr_tmpl);
if (err)
goto out;
err = crypto_register_template(&crypto_rfc3686_tmpl);
if (err)
goto out_drop_ctr;
out:
return err;
out_drop_ctr:
crypto_unregister_template(&crypto_ctr_tmpl);
goto out;
return crypto_register_templates(crypto_ctr_tmpls,
ARRAY_SIZE(crypto_ctr_tmpls));
}
static void __exit crypto_ctr_module_exit(void)
{
crypto_unregister_template(&crypto_rfc3686_tmpl);
crypto_unregister_template(&crypto_ctr_tmpl);
crypto_unregister_templates(crypto_ctr_tmpls,
ARRAY_SIZE(crypto_ctr_tmpls));
}
module_init(crypto_ctr_module_init);
module_exit(crypto_ctr_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CTR Counter block mode");
MODULE_DESCRIPTION("CTR block cipher mode of operation");
MODULE_ALIAS_CRYPTO("rfc3686");
MODULE_ALIAS_CRYPTO("ctr");

4
crypto/des_generic.c
View File

@ -789,7 +789,7 @@ static int des_setkey(struct crypto_tfm *tfm, const u8 *key,
/* Expand to tmp */
ret = des_ekey(tmp, key);
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}
@ -866,7 +866,7 @@ int __des3_ede_setkey(u32 *expkey, u32 *flags, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
(*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}

151
crypto/ecb.c
View File

@ -11,162 +11,83 @@
*/
#include <crypto/algapi.h>
#include <crypto/internal/skcipher.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
struct crypto_ecb_ctx {
struct crypto_cipher *child;
};
static int crypto_ecb_setkey(struct crypto_tfm *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_ecb_crypt(struct blkcipher_desc *desc,
struct blkcipher_walk *walk,
struct crypto_cipher *tfm,
static int crypto_ecb_crypt(struct skcipher_request *req,
struct crypto_cipher *cipher,
void (*fn)(struct crypto_tfm *, u8 *, const u8 *))
{
int bsize = crypto_cipher_blocksize(tfm);
const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct skcipher_walk walk;
unsigned int nbytes;
int err;
err = blkcipher_walk_virt(desc, walk);
err = skcipher_walk_virt(&walk, req, false);
while ((nbytes = walk->nbytes)) {
u8 *wsrc = walk->src.virt.addr;
u8 *wdst = walk->dst.virt.addr;
while ((nbytes = walk.nbytes) != 0) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
do {
fn(crypto_cipher_tfm(tfm), wdst, wsrc);
fn(crypto_cipher_tfm(cipher), dst, src);
wsrc += bsize;
wdst += bsize;
src += bsize;
dst += bsize;
} while ((nbytes -= bsize) >= bsize);
err = blkcipher_walk_done(desc, walk, nbytes);
err = skcipher_walk_done(&walk, nbytes);
}
return err;
}
static int crypto_ecb_encrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
static int crypto_ecb_encrypt(struct skcipher_request *req)
{
struct blkcipher_walk walk;
struct crypto_blkcipher *tfm = desc->tfm;
struct crypto_ecb_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
blkcipher_walk_init(&walk, dst, src, nbytes);
return crypto_ecb_crypt(desc, &walk, child,
crypto_cipher_alg(child)->cia_encrypt);
return crypto_ecb_crypt(req, cipher,
crypto_cipher_alg(cipher)->cia_encrypt);
}
static int crypto_ecb_decrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
static int crypto_ecb_decrypt(struct skcipher_request *req)
{
struct blkcipher_walk walk;
struct crypto_blkcipher *tfm = desc->tfm;
struct crypto_ecb_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
blkcipher_walk_init(&walk, dst, src, nbytes);
return crypto_ecb_crypt(desc, &walk, child,
crypto_cipher_alg(child)->cia_decrypt);
return crypto_ecb_crypt(req, cipher,
crypto_cipher_alg(cipher)->cia_decrypt);
}
static int crypto_ecb_init_tfm(struct crypto_tfm *tfm)
static int crypto_ecb_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct crypto_instance *inst = (void *)tfm->__crt_alg;
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ecb_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_ecb_alloc(struct rtattr **tb)
{
struct crypto_instance *inst;
struct skcipher_instance *inst;
struct crypto_alg *alg;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER);
if (err)
return ERR_PTR(err);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER,
CRYPTO_ALG_TYPE_MASK);
if (IS_ERR(alg))
return ERR_CAST(alg);
inst = crypto_alloc_instance("ecb", alg);
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
goto out_put_alg;
return PTR_ERR(inst);
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER;
inst->alg.cra_priority = alg->cra_priority;
inst->alg.cra_blocksize = alg->cra_blocksize;
inst->alg.cra_alignmask = alg->cra_alignmask;
inst->alg.cra_type = &crypto_blkcipher_type;
inst->alg.ivsize = 0; /* ECB mode doesn't take an IV */
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.encrypt = crypto_ecb_encrypt;
inst->alg.decrypt = crypto_ecb_decrypt;
inst->alg.cra_ctxsize = sizeof(struct crypto_ecb_ctx);
inst->alg.cra_init = crypto_ecb_init_tfm;
inst->alg.cra_exit = crypto_ecb_exit_tfm;
inst->alg.cra_blkcipher.setkey = crypto_ecb_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_ecb_encrypt;
inst->alg.cra_blkcipher.decrypt = crypto_ecb_decrypt;
out_put_alg:
err = skcipher_register_instance(tmpl, inst);
if (err)
inst->free(inst);
crypto_mod_put(alg);
return inst;
}
static void crypto_ecb_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
return err;
}
static struct crypto_template crypto_ecb_tmpl = {
.name = "ecb",
.alloc = crypto_ecb_alloc,
.free = crypto_ecb_free,
.create = crypto_ecb_create,
.module = THIS_MODULE,
};
@ -184,5 +105,5 @@ module_init(crypto_ecb_module_init);
module_exit(crypto_ecb_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("ECB block cipher algorithm");
MODULE_DESCRIPTION("ECB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("ecb");

75
crypto/gcm.c
View File

@ -247,7 +247,7 @@ static int gcm_hash_len(struct aead_request *req, u32 flags)
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct ahash_request *ahreq = &pctx->u.ahreq;
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
u128 lengths;
be128 lengths;
lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(gctx->cryptlen * 8);
@ -727,12 +727,6 @@ static int crypto_gcm_create(struct crypto_template *tmpl, struct rtattr **tb)
ctr_name, "ghash");
}
static struct crypto_template crypto_gcm_tmpl = {
.name = "gcm",
.create = crypto_gcm_create,
.module = THIS_MODULE,
};
static int crypto_gcm_base_create(struct crypto_template *tmpl,
struct rtattr **tb)
{
@ -756,12 +750,6 @@ static int crypto_gcm_base_create(struct crypto_template *tmpl,
ctr_name, ghash_name);
}
static struct crypto_template crypto_gcm_base_tmpl = {
.name = "gcm_base",
.create = crypto_gcm_base_create,
.module = THIS_MODULE,
};
static int crypto_rfc4106_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen)
{
@ -989,12 +977,6 @@ out_free_inst:
goto out;
}
static struct crypto_template crypto_rfc4106_tmpl = {
.name = "rfc4106",
.create = crypto_rfc4106_create,
.module = THIS_MODULE,
};
static int crypto_rfc4543_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen)
{
@ -1231,10 +1213,24 @@ out_free_inst:
goto out;
}
static struct crypto_template crypto_rfc4543_tmpl = {
.name = "rfc4543",
.create = crypto_rfc4543_create,
.module = THIS_MODULE,
static struct crypto_template crypto_gcm_tmpls[] = {
{
.name = "gcm_base",
.create = crypto_gcm_base_create,
.module = THIS_MODULE,
}, {
.name = "gcm",
.create = crypto_gcm_create,
.module = THIS_MODULE,
}, {
.name = "rfc4106",
.create = crypto_rfc4106_create,
.module = THIS_MODULE,
}, {
.name = "rfc4543",
.create = crypto_rfc4543_create,
.module = THIS_MODULE,
},
};
static int __init crypto_gcm_module_init(void)
@ -1247,42 +1243,19 @@ static int __init crypto_gcm_module_init(void)
sg_init_one(&gcm_zeroes->sg, gcm_zeroes->buf, sizeof(gcm_zeroes->buf));
err = crypto_register_template(&crypto_gcm_base_tmpl);
err = crypto_register_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
if (err)
goto out;
kfree(gcm_zeroes);
err = crypto_register_template(&crypto_gcm_tmpl);
if (err)
goto out_undo_base;
err = crypto_register_template(&crypto_rfc4106_tmpl);
if (err)
goto out_undo_gcm;
err = crypto_register_template(&crypto_rfc4543_tmpl);
if (err)
goto out_undo_rfc4106;
return 0;
out_undo_rfc4106:
crypto_unregister_template(&crypto_rfc4106_tmpl);
out_undo_gcm:
crypto_unregister_template(&crypto_gcm_tmpl);
out_undo_base:
crypto_unregister_template(&crypto_gcm_base_tmpl);
out:
kfree(gcm_zeroes);
return err;
}
static void __exit crypto_gcm_module_exit(void)
{
kfree(gcm_zeroes);
crypto_unregister_template(&crypto_rfc4543_tmpl);
crypto_unregister_template(&crypto_rfc4106_tmpl);
crypto_unregister_template(&crypto_gcm_tmpl);
crypto_unregister_template(&crypto_gcm_base_tmpl);
crypto_unregister_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
}
module_init(crypto_gcm_module_init);

198
crypto/keywrap.c
View File

@ -56,7 +56,7 @@
* u8 *iv = data;
* u8 *pt = data + crypto_skcipher_ivsize(tfm);
* <ensure that pt contains the plaintext of size ptlen>
* sg_init_one(&sg, ptdata, ptlen);
* sg_init_one(&sg, pt, ptlen);
* skcipher_request_set_crypt(req, &sg, &sg, ptlen, iv);
*
* ==> After encryption, data now contains full KW result as per SP800-38F.
@ -70,8 +70,8 @@
* u8 *iv = data;
* u8 *ct = data + crypto_skcipher_ivsize(tfm);
* unsigned int ctlen = datalen - crypto_skcipher_ivsize(tfm);
* sg_init_one(&sg, ctdata, ctlen);
* skcipher_request_set_crypt(req, &sg, &sg, ptlen, iv);
* sg_init_one(&sg, ct, ctlen);
* skcipher_request_set_crypt(req, &sg, &sg, ctlen, iv);
*
* ==> After decryption (which hopefully does not return EBADMSG), the ct
* pointer now points to the plaintext of size ctlen.
@ -87,10 +87,6 @@
#include <crypto/scatterwalk.h>
#include <crypto/internal/skcipher.h>
struct crypto_kw_ctx {
struct crypto_cipher *child;
};
struct crypto_kw_block {
#define SEMIBSIZE 8
__be64 A;
@ -124,16 +120,13 @@ static void crypto_kw_scatterlist_ff(struct scatter_walk *walk,
}
}
static int crypto_kw_decrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
static int crypto_kw_decrypt(struct skcipher_request *req)
{
struct crypto_blkcipher *tfm = desc->tfm;
struct crypto_kw_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_kw_block block;
struct scatterlist *lsrc, *ldst;
u64 t = 6 * ((nbytes) >> 3);
struct scatterlist *src, *dst;
u64 t = 6 * ((req->cryptlen) >> 3);
unsigned int i;
int ret = 0;
@ -141,27 +134,27 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
* Require at least 2 semiblocks (note, the 3rd semiblock that is
* required by SP800-38F is the IV.
*/
if (nbytes < (2 * SEMIBSIZE) || nbytes % SEMIBSIZE)
if (req->cryptlen < (2 * SEMIBSIZE) || req->cryptlen % SEMIBSIZE)
return -EINVAL;
/* Place the IV into block A */
memcpy(&block.A, desc->info, SEMIBSIZE);
memcpy(&block.A, req->iv, SEMIBSIZE);
/*
* src scatterlist is read-only. dst scatterlist is r/w. During the
* first loop, lsrc points to src and ldst to dst. For any
* subsequent round, the code operates on dst only.
* first loop, src points to req->src and dst to req->dst. For any
* subsequent round, the code operates on req->dst only.
*/
lsrc = src;
ldst = dst;
src = req->src;
dst = req->dst;
for (i = 0; i < 6; i++) {
struct scatter_walk src_walk, dst_walk;
unsigned int tmp_nbytes = nbytes;
unsigned int nbytes = req->cryptlen;
while (tmp_nbytes) {
/* move pointer by tmp_nbytes in the SGL */
crypto_kw_scatterlist_ff(&src_walk, lsrc, tmp_nbytes);
while (nbytes) {
/* move pointer by nbytes in the SGL */
crypto_kw_scatterlist_ff(&src_walk, src, nbytes);
/* get the source block */
scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE,
false);
@ -170,21 +163,21 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
block.A ^= cpu_to_be64(t);
t--;
/* perform KW operation: decrypt block */
crypto_cipher_decrypt_one(child, (u8*)&block,
(u8*)&block);
crypto_cipher_decrypt_one(cipher, (u8 *)&block,
(u8 *)&block);
/* move pointer by tmp_nbytes in the SGL */
crypto_kw_scatterlist_ff(&dst_walk, ldst, tmp_nbytes);
/* move pointer by nbytes in the SGL */
crypto_kw_scatterlist_ff(&dst_walk, dst, nbytes);
/* Copy block->R into place */
scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE,
true);
tmp_nbytes -= SEMIBSIZE;
nbytes -= SEMIBSIZE;
}
/* we now start to operate on the dst SGL only */
lsrc = dst;
ldst = dst;
src = req->dst;
dst = req->dst;
}
/* Perform authentication check */
@ -196,15 +189,12 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
return ret;
}
static int crypto_kw_encrypt(struct blkcipher_desc *desc,
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
static int crypto_kw_encrypt(struct skcipher_request *req)
{
struct crypto_blkcipher *tfm = desc->tfm;
struct crypto_kw_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_kw_block block;
struct scatterlist *lsrc, *ldst;
struct scatterlist *src, *dst;
u64 t = 1;
unsigned int i;
@ -214,7 +204,7 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
* This means that the dst memory must be one semiblock larger than src.
* Also ensure that the given data is aligned to semiblock.
*/
if (nbytes < (2 * SEMIBSIZE) || nbytes % SEMIBSIZE)
if (req->cryptlen < (2 * SEMIBSIZE) || req->cryptlen % SEMIBSIZE)
return -EINVAL;
/*
@ -225,26 +215,26 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
/*
* src scatterlist is read-only. dst scatterlist is r/w. During the
* first loop, lsrc points to src and ldst to dst. For any
* subsequent round, the code operates on dst only.
* first loop, src points to req->src and dst to req->dst. For any
* subsequent round, the code operates on req->dst only.
*/
lsrc = src;
ldst = dst;
src = req->src;
dst = req->dst;
for (i = 0; i < 6; i++) {
struct scatter_walk src_walk, dst_walk;
unsigned int tmp_nbytes = nbytes;
unsigned int nbytes = req->cryptlen;
scatterwalk_start(&src_walk, lsrc);
scatterwalk_start(&dst_walk, ldst);
scatterwalk_start(&src_walk, src);
scatterwalk_start(&dst_walk, dst);
while (tmp_nbytes) {
while (nbytes) {
/* get the source block */
scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE,
false);
/* perform KW operation: encrypt block */
crypto_cipher_encrypt_one(child, (u8 *)&block,
crypto_cipher_encrypt_one(cipher, (u8 *)&block,
(u8 *)&block);
/* perform KW operation: modify IV with counter */
block.A ^= cpu_to_be64(t);
@ -254,117 +244,59 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE,
true);
tmp_nbytes -= SEMIBSIZE;
nbytes -= SEMIBSIZE;
}
/* we now start to operate on the dst SGL only */
lsrc = dst;
ldst = dst;
src = req->dst;
dst = req->dst;
}
/* establish the IV for the caller to pick up */
memcpy(desc->info, &block.A, SEMIBSIZE);
memcpy(req->iv, &block.A, SEMIBSIZE);
memzero_explicit(&block, sizeof(struct crypto_kw_block));
return 0;
}
static int crypto_kw_setkey(struct crypto_tfm *parent, const u8 *key,
unsigned int keylen)
static int crypto_kw_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(parent);
struct crypto_cipher *child = ctx->child;
struct skcipher_instance *inst;
struct crypto_alg *alg;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
static int crypto_kw_init_tfm(struct crypto_tfm *tfm)
{
struct crypto_instance *inst = crypto_tfm_alg_instance(tfm);
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_kw_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_kw_alloc(struct rtattr **tb)
{
struct crypto_instance *inst = NULL;
struct crypto_alg *alg = NULL;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER);
if (err)
return ERR_PTR(err);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER,
CRYPTO_ALG_TYPE_MASK);
if (IS_ERR(alg))
return ERR_CAST(alg);
inst = ERR_PTR(-EINVAL);
err = -EINVAL;
/* Section 5.1 requirement for KW */
if (alg->cra_blocksize != sizeof(struct crypto_kw_block))
goto err;
goto out_free_inst;
inst = crypto_alloc_instance("kw", alg);
if (IS_ERR(inst))
goto err;
inst->alg.base.cra_blocksize = SEMIBSIZE;
inst->alg.base.cra_alignmask = 0;
inst->alg.ivsize = SEMIBSIZE;
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER;
inst->alg.cra_priority = alg->cra_priority;
inst->alg.cra_blocksize = SEMIBSIZE;
inst->alg.cra_alignmask = 0;
inst->alg.cra_type = &crypto_blkcipher_type;
inst->alg.cra_blkcipher.ivsize = SEMIBSIZE;
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.encrypt = crypto_kw_encrypt;
inst->alg.decrypt = crypto_kw_decrypt;
inst->alg.cra_ctxsize = sizeof(struct crypto_kw_ctx);
err = skcipher_register_instance(tmpl, inst);
if (err)
goto out_free_inst;
goto out_put_alg;
inst->alg.cra_init = crypto_kw_init_tfm;
inst->alg.cra_exit = crypto_kw_exit_tfm;
inst->alg.cra_blkcipher.setkey = crypto_kw_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_kw_encrypt;
inst->alg.cra_blkcipher.decrypt = crypto_kw_decrypt;
err:
out_free_inst:
inst->free(inst);
out_put_alg:
crypto_mod_put(alg);
return inst;
}
static void crypto_kw_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
return err;
}
static struct crypto_template crypto_kw_tmpl = {
.name = "kw",
.alloc = crypto_kw_alloc,
.free = crypto_kw_free,
.create = crypto_kw_create,
.module = THIS_MODULE,
};

19
crypto/morus1280.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The MORUS-1280 Authenticated-Encryption Algorithm
*
* Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#include <asm/unaligned.h>
@ -366,18 +362,19 @@ static void crypto_morus1280_process_crypt(struct morus1280_state *state,
const struct morus1280_ops *ops)
{
struct skcipher_walk walk;
u8 *dst;
const u8 *src;
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
unsigned int nbytes = walk.nbytes;
ops->crypt_chunk(state, dst, src, walk.nbytes);
if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0);
ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
}

19
crypto/morus640.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The MORUS-640 Authenticated-Encryption Algorithm
*
* Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/
#include <asm/unaligned.h>
@ -365,18 +361,19 @@ static void crypto_morus640_process_crypt(struct morus640_state *state,
const struct morus640_ops *ops)
{
struct skcipher_walk walk;
u8 *dst;
const u8 *src;
ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) {
src = walk.src.virt.addr;
dst = walk.dst.virt.addr;
unsigned int nbytes = walk.nbytes;
ops->crypt_chunk(state, dst, src, walk.nbytes);
if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0);
ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
}

202
crypto/ofb.c
View File

@ -5,9 +5,6 @@
*
* Copyright (C) 2018 ARM Limited or its affiliates.
* All rights reserved.
*
* Based loosely on public domain code gleaned from libtomcrypt
* (https://github.com/libtom/libtomcrypt).
*/
#include <crypto/algapi.h>
@ -16,189 +13,70 @@
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
struct crypto_ofb_ctx {
struct crypto_cipher *child;
int cnt;
};
static int crypto_ofb_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
static int crypto_ofb_crypt(struct skcipher_request *req)
{
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct skcipher_walk walk;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
err = skcipher_walk_virt(&walk, req, false);
static int crypto_ofb_encrypt_segment(struct crypto_ofb_ctx *ctx,
struct skcipher_walk *walk,
struct crypto_cipher *tfm)
{
int bsize = crypto_cipher_blocksize(tfm);
int nbytes = walk->nbytes;
while (walk.nbytes >= bsize) {
const u8 *src = walk.src.virt.addr;
u8 *dst = walk.dst.virt.addr;
u8 * const iv = walk.iv;
unsigned int nbytes = walk.nbytes;
u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr;
u8 *iv = walk->iv;
do {
crypto_cipher_encrypt_one(cipher, iv, iv);
crypto_xor_cpy(dst, src, iv, bsize);
dst += bsize;
src += bsize;
} while ((nbytes -= bsize) >= bsize);
do {
if (ctx->cnt == bsize) {
if (nbytes < bsize)
break;
crypto_cipher_encrypt_one(tfm, iv, iv);
ctx->cnt = 0;
}
*dst = *src ^ iv[ctx->cnt];
src++;
dst++;
ctx->cnt++;
} while (--nbytes);
return nbytes;
}
static int crypto_ofb_encrypt(struct skcipher_request *req)
{
struct skcipher_walk walk;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
unsigned int bsize;
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
int ret = 0;
bsize = crypto_cipher_blocksize(child);
ctx->cnt = bsize;
ret = skcipher_walk_virt(&walk, req, false);
while (walk.nbytes) {
ret = crypto_ofb_encrypt_segment(ctx, &walk, child);
ret = skcipher_walk_done(&walk, ret);
err = skcipher_walk_done(&walk, nbytes);
}
return ret;
}
/* OFB encrypt and decrypt are identical */
static int crypto_ofb_decrypt(struct skcipher_request *req)
{
return crypto_ofb_encrypt(req);
}
static int crypto_ofb_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ofb_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_ofb_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
if (walk.nbytes) {
crypto_cipher_encrypt_one(cipher, walk.iv, walk.iv);
crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, walk.iv,
walk.nbytes);
err = skcipher_walk_done(&walk, 0);
}
return err;
}
static int crypto_ofb_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg;
u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER);
if (err)
return err;
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
/* OFB mode is a stream cipher. */
inst->alg.base.cra_blocksize = 1;
algt = crypto_get_attr_type(tb);
err = PTR_ERR(algt);
if (IS_ERR(algt))
goto err_free_inst;
/*
* To simplify the implementation, configure the skcipher walk to only
* give a partial block at the very end, never earlier.
*/
inst->alg.chunksize = alg->cra_blocksize;
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
crypto_mod_put(alg);
if (err)
goto err_free_inst;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "ofb", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
/* We access the data as u32s when xoring. */
inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_ofb_ctx);
inst->alg.init = crypto_ofb_init_tfm;
inst->alg.exit = crypto_ofb_exit_tfm;
inst->alg.setkey = crypto_ofb_setkey;
inst->alg.encrypt = crypto_ofb_encrypt;
inst->alg.decrypt = crypto_ofb_decrypt;
inst->free = crypto_ofb_free;
inst->alg.encrypt = crypto_ofb_crypt;
inst->alg.decrypt = crypto_ofb_crypt;
err = skcipher_register_instance(tmpl, inst);
if (err)
goto err_drop_spawn;
inst->free(inst);
out:
crypto_mod_put(alg);
return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_free_inst:
kfree(inst);
goto out;
}
static struct crypto_template crypto_ofb_tmpl = {
@ -221,5 +99,5 @@ module_init(crypto_ofb_module_init);
module_exit(crypto_ofb_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("OFB block cipher algorithm");
MODULE_DESCRIPTION("OFB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("ofb");

143
crypto/pcbc.c
View File

@ -20,28 +20,6 @@
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/compiler.h>
struct crypto_pcbc_ctx {
struct crypto_cipher *child;
};
static int crypto_pcbc_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_pcbc_encrypt_segment(struct skcipher_request *req,
struct skcipher_walk *walk,
@ -51,7 +29,7 @@ static int crypto_pcbc_encrypt_segment(struct skcipher_request *req,
unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr;
u8 *iv = walk->iv;
u8 * const iv = walk->iv;
do {
crypto_xor(iv, src, bsize);
@ -72,7 +50,7 @@ static int crypto_pcbc_encrypt_inplace(struct skcipher_request *req,
int bsize = crypto_cipher_blocksize(tfm);
unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv;
u8 * const iv = walk->iv;
u8 tmpbuf[MAX_CIPHER_BLOCKSIZE];
do {
@ -84,16 +62,13 @@ static int crypto_pcbc_encrypt_inplace(struct skcipher_request *req,
src += bsize;
} while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes;
}
static int crypto_pcbc_encrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct skcipher_walk walk;
unsigned int nbytes;
int err;
@ -103,10 +78,10 @@ static int crypto_pcbc_encrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes)) {
if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_encrypt_inplace(req, &walk,
child);
cipher);
else
nbytes = crypto_pcbc_encrypt_segment(req, &walk,
child);
cipher);
err = skcipher_walk_done(&walk, nbytes);
}
@ -121,7 +96,7 @@ static int crypto_pcbc_decrypt_segment(struct skcipher_request *req,
unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr;
u8 *iv = walk->iv;
u8 * const iv = walk->iv;
do {
crypto_cipher_decrypt_one(tfm, dst, src);
@ -132,8 +107,6 @@ static int crypto_pcbc_decrypt_segment(struct skcipher_request *req,
dst += bsize;
} while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes;
}
@ -144,7 +117,7 @@ static int crypto_pcbc_decrypt_inplace(struct skcipher_request *req,
int bsize = crypto_cipher_blocksize(tfm);
unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv;
u8 * const iv = walk->iv;
u8 tmpbuf[MAX_CIPHER_BLOCKSIZE] __aligned(__alignof__(u32));
do {
@ -156,16 +129,13 @@ static int crypto_pcbc_decrypt_inplace(struct skcipher_request *req,
src += bsize;
} while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes;
}
static int crypto_pcbc_decrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct skcipher_walk walk;
unsigned int nbytes;
int err;
@ -175,117 +145,34 @@ static int crypto_pcbc_decrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes)) {
if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_decrypt_inplace(req, &walk,
child);
cipher);
else
nbytes = crypto_pcbc_decrypt_segment(req, &walk,
child);
cipher);
err = skcipher_walk_done(&walk, nbytes);
}
return err;
}
static int crypto_pcbc_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_pcbc_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_pcbc_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_pcbc_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg;
int err;
algt = crypto_get_attr_type(tb);
if (IS_ERR(algt))
return PTR_ERR(algt);
inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(inst))
return PTR_ERR(inst);
if (((algt->type ^ CRYPTO_ALG_TYPE_SKCIPHER) & algt->mask) &
~CRYPTO_ALG_INTERNAL)
return -EINVAL;
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER |
(algt->type & CRYPTO_ALG_INTERNAL),
CRYPTO_ALG_TYPE_MASK |
(algt->mask & CRYPTO_ALG_INTERNAL));
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "pcbc", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_flags = alg->cra_flags & CRYPTO_ALG_INTERNAL;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_pcbc_ctx);
inst->alg.init = crypto_pcbc_init_tfm;
inst->alg.exit = crypto_pcbc_exit_tfm;
inst->alg.setkey = crypto_pcbc_setkey;
inst->alg.encrypt = crypto_pcbc_encrypt;
inst->alg.decrypt = crypto_pcbc_decrypt;
inst->free = crypto_pcbc_free;
err = skcipher_register_instance(tmpl, inst);
if (err)
goto err_drop_spawn;
inst->free(inst);
crypto_mod_put(alg);
out:
return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
}
static struct crypto_template crypto_pcbc_tmpl = {
@ -308,5 +195,5 @@ module_init(crypto_pcbc_module_init);
module_exit(crypto_pcbc_module_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("PCBC block cipher algorithm");
MODULE_DESCRIPTION("PCBC block cipher mode of operation");
MODULE_ALIAS_CRYPTO("pcbc");

1
crypto/rsa-pkcs1pad.c
View File

@ -12,6 +12,7 @@
#include <crypto/algapi.h>
#include <crypto/akcipher.h>
#include <crypto/internal/akcipher.h>
#include <crypto/internal/rsa.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/kernel.h>

7
crypto/seqiv.c
View File

@ -89,13 +89,12 @@ static int seqiv_aead_encrypt(struct aead_request *req)
if (unlikely(!IS_ALIGNED((unsigned long)info,
crypto_aead_alignmask(geniv) + 1))) {
info = kmalloc(ivsize, req->base.flags &
CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL:
GFP_ATOMIC);
info = kmemdup(req->iv, ivsize, req->base.flags &
CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC);
if (!info)
return -ENOMEM;
memcpy(info, req->iv, ivsize);
compl = seqiv_aead_encrypt_complete;
data = req;
}

27
crypto/shash.c
View File

@ -53,6 +53,13 @@ static int shash_setkey_unaligned(struct crypto_shash *tfm, const u8 *key,
return err;
}
static void shash_set_needkey(struct crypto_shash *tfm, struct shash_alg *alg)
{
if (crypto_shash_alg_has_setkey(alg) &&
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_shash_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
int crypto_shash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen)
{
@ -65,8 +72,10 @@ int crypto_shash_setkey(struct crypto_shash *tfm, const u8 *key,
else
err = shash->setkey(tfm, key, keylen);
if (err)
if (unlikely(err)) {
shash_set_needkey(tfm, shash);
return err;
}
crypto_shash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;
@ -373,15 +382,14 @@ int crypto_init_shash_ops_async(struct crypto_tfm *tfm)
crt->final = shash_async_final;
crt->finup = shash_async_finup;
crt->digest = shash_async_digest;
crt->setkey = shash_async_setkey;
if (crypto_shash_alg_has_setkey(alg))
crt->setkey = shash_async_setkey;
crypto_ahash_set_flags(crt, crypto_shash_get_flags(shash) &
CRYPTO_TFM_NEED_KEY);
if (alg->export)
crt->export = shash_async_export;
if (alg->import)
crt->import = shash_async_import;
crt->export = shash_async_export;
crt->import = shash_async_import;
crt->reqsize = sizeof(struct shash_desc) + crypto_shash_descsize(shash);
@ -395,9 +403,7 @@ static int crypto_shash_init_tfm(struct crypto_tfm *tfm)
hash->descsize = alg->descsize;
if (crypto_shash_alg_has_setkey(alg) &&
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_shash_set_flags(hash, CRYPTO_TFM_NEED_KEY);
shash_set_needkey(hash, alg);
return 0;
}
@ -464,6 +470,9 @@ static int shash_prepare_alg(struct shash_alg *alg)
alg->statesize > HASH_MAX_STATESIZE)
return -EINVAL;
if ((alg->export && !alg->import) || (alg->import && !alg->export))
return -EINVAL;
base->cra_type = &crypto_shash_type;
base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK;
base->cra_flags |= CRYPTO_ALG_TYPE_SHASH;

158
crypto/skcipher.c
View File

@ -585,6 +585,12 @@ static unsigned int crypto_skcipher_extsize(struct crypto_alg *alg)
return crypto_alg_extsize(alg);
}
static void skcipher_set_needkey(struct crypto_skcipher *tfm)
{
if (tfm->keysize)
crypto_skcipher_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
static int skcipher_setkey_blkcipher(struct crypto_skcipher *tfm,
const u8 *key, unsigned int keylen)
{
@ -598,8 +604,10 @@ static int skcipher_setkey_blkcipher(struct crypto_skcipher *tfm,
err = crypto_blkcipher_setkey(blkcipher, key, keylen);
crypto_skcipher_set_flags(tfm, crypto_blkcipher_get_flags(blkcipher) &
CRYPTO_TFM_RES_MASK);
if (err)
if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;
@ -677,8 +685,7 @@ static int crypto_init_skcipher_ops_blkcipher(struct crypto_tfm *tfm)
skcipher->ivsize = crypto_blkcipher_ivsize(blkcipher);
skcipher->keysize = calg->cra_blkcipher.max_keysize;
if (skcipher->keysize)
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
skcipher_set_needkey(skcipher);
return 0;
}
@ -698,8 +705,10 @@ static int skcipher_setkey_ablkcipher(struct crypto_skcipher *tfm,
crypto_skcipher_set_flags(tfm,
crypto_ablkcipher_get_flags(ablkcipher) &
CRYPTO_TFM_RES_MASK);
if (err)
if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;
@ -776,8 +785,7 @@ static int crypto_init_skcipher_ops_ablkcipher(struct crypto_tfm *tfm)
sizeof(struct ablkcipher_request);
skcipher->keysize = calg->cra_ablkcipher.max_keysize;
if (skcipher->keysize)
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
skcipher_set_needkey(skcipher);
return 0;
}
@ -820,8 +828,10 @@ static int skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
else
err = cipher->setkey(tfm, key, keylen);
if (err)
if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0;
@ -852,8 +862,7 @@ static int crypto_skcipher_init_tfm(struct crypto_tfm *tfm)
skcipher->ivsize = alg->ivsize;
skcipher->keysize = alg->max_keysize;
if (skcipher->keysize)
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
skcipher_set_needkey(skcipher);
if (alg->exit)
skcipher->base.exit = crypto_skcipher_exit_tfm;
@ -1058,5 +1067,136 @@ int skcipher_register_instance(struct crypto_template *tmpl,
}
EXPORT_SYMBOL_GPL(skcipher_register_instance);
static int skcipher_setkey_simple(struct crypto_skcipher *tfm, const u8 *key,
unsigned int keylen)
{
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
int err;
crypto_cipher_clear_flags(cipher, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(cipher, crypto_skcipher_get_flags(tfm) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(cipher, key, keylen);
crypto_skcipher_set_flags(tfm, crypto_cipher_get_flags(cipher) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int skcipher_init_tfm_simple(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct skcipher_ctx_simple *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->cipher = cipher;
return 0;
}
static void skcipher_exit_tfm_simple(struct crypto_skcipher *tfm)
{
struct skcipher_ctx_simple *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->cipher);
}
static void skcipher_free_instance_simple(struct skcipher_instance *inst)
{
crypto_drop_spawn(skcipher_instance_ctx(inst));
kfree(inst);
}
/**
* skcipher_alloc_instance_simple - allocate instance of simple block cipher mode
*
* Allocate an skcipher_instance for a simple block cipher mode of operation,
* e.g. cbc or ecb. The instance context will have just a single crypto_spawn,
* that for the underlying cipher. The {min,max}_keysize, ivsize, blocksize,
* alignmask, and priority are set from the underlying cipher but can be
* overridden if needed. The tfm context defaults to skcipher_ctx_simple, and
* default ->setkey(), ->init(), and ->exit() methods are installed.
*
* @tmpl: the template being instantiated
* @tb: the template parameters
* @cipher_alg_ret: on success, a pointer to the underlying cipher algorithm is
* returned here. It must be dropped with crypto_mod_put().
*
* Return: a pointer to the new instance, or an ERR_PTR(). The caller still
* needs to register the instance.
*/
struct skcipher_instance *
skcipher_alloc_instance_simple(struct crypto_template *tmpl, struct rtattr **tb,
struct crypto_alg **cipher_alg_ret)
{
struct crypto_attr_type *algt;
struct crypto_alg *cipher_alg;
struct skcipher_instance *inst;
struct crypto_spawn *spawn;
u32 mask;
int err;
algt = crypto_get_attr_type(tb);
if (IS_ERR(algt))
return ERR_CAST(algt);
if ((algt->type ^ CRYPTO_ALG_TYPE_SKCIPHER) & algt->mask)
return ERR_PTR(-EINVAL);
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
cipher_alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
if (IS_ERR(cipher_alg))
return ERR_CAST(cipher_alg);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst) {
err = -ENOMEM;
goto err_put_cipher_alg;
}
spawn = skcipher_instance_ctx(inst);
err = crypto_inst_setname(skcipher_crypto_instance(inst), tmpl->name,
cipher_alg);
if (err)
goto err_free_inst;
err = crypto_init_spawn(spawn, cipher_alg,
skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_free_inst;
inst->free = skcipher_free_instance_simple;
/* Default algorithm properties, can be overridden */
inst->alg.base.cra_blocksize = cipher_alg->cra_blocksize;
inst->alg.base.cra_alignmask = cipher_alg->cra_alignmask;
inst->alg.base.cra_priority = cipher_alg->cra_priority;
inst->alg.min_keysize = cipher_alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = cipher_alg->cra_cipher.cia_max_keysize;
inst->alg.ivsize = cipher_alg->cra_blocksize;
/* Use skcipher_ctx_simple by default, can be overridden */
inst->alg.base.cra_ctxsize = sizeof(struct skcipher_ctx_simple);
inst->alg.setkey = skcipher_setkey_simple;
inst->alg.init = skcipher_init_tfm_simple;
inst->alg.exit = skcipher_exit_tfm_simple;
*cipher_alg_ret = cipher_alg;
return inst;
err_free_inst:
kfree(inst);
err_put_cipher_alg:
crypto_mod_put(cipher_alg);
return ERR_PTR(err);
}
EXPORT_SYMBOL_GPL(skcipher_alloc_instance_simple);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Symmetric key cipher type");

2
crypto/streebog_generic.c
View File

@ -960,7 +960,7 @@ static int streebog_init(struct shash_desc *desc)
memset(ctx, 0, sizeof(struct streebog_state));
for (i = 0; i < 8; i++) {
if (digest_size == STREEBOG256_DIGEST_SIZE)
ctx->h.qword[i] = 0x0101010101010101ULL;
ctx->h.qword[i] = cpu_to_le64(0x0101010101010101ULL);
}
return 0;
}

2932
crypto/testmgr.c
View File

File diff suppressed because it is too large Load Diff

10841
crypto/testmgr.h
View File

File diff suppressed because it is too large Load Diff

6
crypto/tgr192.c
View File

@ -25,8 +25,9 @@
#include <linux/init.h>
#include <linux/module.h>
#include <linux/mm.h>
#include <asm/byteorder.h>
#include <linux/types.h>
#include <asm/byteorder.h>
#include <asm/unaligned.h>
#define TGR192_DIGEST_SIZE 24
#define TGR160_DIGEST_SIZE 20
@ -468,10 +469,9 @@ static void tgr192_transform(struct tgr192_ctx *tctx, const u8 * data)
u64 a, b, c, aa, bb, cc;
u64 x[8];
int i;
const __le64 *ptr = (const __le64 *)data;
for (i = 0; i < 8; i++)
x[i] = le64_to_cpu(ptr[i]);
x[i] = get_unaligned_le64(data + i * sizeof(__le64));
/* save */
a = aa = tctx->a;

11
drivers/bus/fsl-mc/fsl-mc-allocator.c
View File

@ -295,6 +295,14 @@ int __must_check fsl_mc_object_allocate(struct fsl_mc_device *mc_dev,
if (!mc_adev)
goto error;
mc_adev->consumer_link = device_link_add(&mc_dev->dev,
&mc_adev->dev,
DL_FLAG_AUTOREMOVE_CONSUMER);
if (!mc_adev->consumer_link) {
error = -EINVAL;
goto error;
}
*new_mc_adev = mc_adev;
return 0;
error:
@ -321,6 +329,9 @@ void fsl_mc_object_free(struct fsl_mc_device *mc_adev)
return;
fsl_mc_resource_free(resource);
device_link_del(mc_adev->consumer_link);
mc_adev->consumer_link = NULL;
}
EXPORT_SYMBOL_GPL(fsl_mc_object_free);

13
drivers/bus/fsl-mc/mc-io.c
View File

@ -209,9 +209,19 @@ int __must_check fsl_mc_portal_allocate(struct fsl_mc_device *mc_dev,
if (error < 0)
goto error_cleanup_resource;
dpmcp_dev->consumer_link = device_link_add(&mc_dev->dev,
&dpmcp_dev->dev,
DL_FLAG_AUTOREMOVE_CONSUMER);
if (!dpmcp_dev->consumer_link) {
error = -EINVAL;
goto error_cleanup_mc_io;
}
*new_mc_io = mc_io;
return 0;
error_cleanup_mc_io:
fsl_destroy_mc_io(mc_io);
error_cleanup_resource:
fsl_mc_resource_free(resource);
return error;
@ -244,6 +254,9 @@ void fsl_mc_portal_free(struct fsl_mc_io *mc_io)
fsl_destroy_mc_io(mc_io);
fsl_mc_resource_free(resource);
device_link_del(dpmcp_dev->consumer_link);
dpmcp_dev->consumer_link = NULL;
}
EXPORT_SYMBOL_GPL(fsl_mc_portal_free);

16
drivers/char/hw_random/bcm2835-rng.c
View File

@ -168,14 +168,16 @@ static int bcm2835_rng_probe(struct platform_device *pdev)
priv->rng.read = bcm2835_rng_read;
priv->rng.cleanup = bcm2835_rng_cleanup;
rng_id = of_match_node(bcm2835_rng_of_match, np);
if (!rng_id)
return -EINVAL;
if (dev_of_node(dev)) {
rng_id = of_match_node(bcm2835_rng_of_match, np);
if (!rng_id)
return -EINVAL;
/* Check for rng init function, execute it */
of_data = rng_id->data;
if (of_data)
priv->mask_interrupts = of_data->mask_interrupts;
/* Check for rng init function, execute it */
of_data = rng_id->data;
if (of_data)
priv->mask_interrupts = of_data->mask_interrupts;
}
/* register driver */
err = devm_hwrng_register(dev, &priv->rng);

2
drivers/char/hw_random/virtio-rng.c
View File

@ -73,7 +73,7 @@ static int virtio_read(struct hwrng *rng, void *buf, size_t size, bool wait)
if (!vi->busy) {
vi->busy = true;
init_completion(&vi->have_data);
reinit_completion(&vi->have_data);
register_buffer(vi, buf, size);
}

87
drivers/crypto/amcc/crypto4xx_core.c
View File

@ -40,9 +40,11 @@
#include <crypto/ctr.h>
#include <crypto/gcm.h>
#include <crypto/sha.h>
#include <crypto/rng.h>
#include <crypto/scatterwalk.h>
#include <crypto/skcipher.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h>
#include "crypto4xx_reg_def.h"
#include "crypto4xx_core.h"
@ -1035,6 +1037,10 @@ static int crypto4xx_register_alg(struct crypto4xx_device *sec_dev,
rc = crypto_register_ahash(&alg->alg.u.hash);
break;
case CRYPTO_ALG_TYPE_RNG:
rc = crypto_register_rng(&alg->alg.u.rng);
break;
default:
rc = crypto_register_skcipher(&alg->alg.u.cipher);
break;
@ -1064,6 +1070,10 @@ static void crypto4xx_unregister_alg(struct crypto4xx_device *sec_dev)
crypto_unregister_aead(&alg->alg.u.aead);
break;
case CRYPTO_ALG_TYPE_RNG:
crypto_unregister_rng(&alg->alg.u.rng);
break;
default:
crypto_unregister_skcipher(&alg->alg.u.cipher);
}
@ -1122,6 +1132,69 @@ static irqreturn_t crypto4xx_ce_interrupt_handler_revb(int irq, void *data)
PPC4XX_TMO_ERR_INT);
}
static int ppc4xx_prng_data_read(struct crypto4xx_device *dev,
u8 *data, unsigned int max)
{
unsigned int i, curr = 0;
u32 val[2];
do {
/* trigger PRN generation */
writel(PPC4XX_PRNG_CTRL_AUTO_EN,
dev->ce_base + CRYPTO4XX_PRNG_CTRL);
for (i = 0; i < 1024; i++) {
/* usually 19 iterations are enough */
if ((readl(dev->ce_base + CRYPTO4XX_PRNG_STAT) &
CRYPTO4XX_PRNG_STAT_BUSY))
continue;
val[0] = readl_be(dev->ce_base + CRYPTO4XX_PRNG_RES_0);
val[1] = readl_be(dev->ce_base + CRYPTO4XX_PRNG_RES_1);
break;
}
if (i == 1024)
return -ETIMEDOUT;
if ((max - curr) >= 8) {
memcpy(data, &val, 8);
data += 8;
curr += 8;
} else {
/* copy only remaining bytes */
memcpy(data, &val, max - curr);
break;
}
} while (curr < max);
return curr;
}
static int crypto4xx_prng_generate(struct crypto_rng *tfm,
const u8 *src, unsigned int slen,
u8 *dstn, unsigned int dlen)
{
struct rng_alg *alg = crypto_rng_alg(tfm);
struct crypto4xx_alg *amcc_alg;
struct crypto4xx_device *dev;
int ret;
amcc_alg = container_of(alg, struct crypto4xx_alg, alg.u.rng);
dev = amcc_alg->dev;
mutex_lock(&dev->core_dev->rng_lock);
ret = ppc4xx_prng_data_read(dev, dstn, dlen);
mutex_unlock(&dev->core_dev->rng_lock);
return ret;
}
static int crypto4xx_prng_seed(struct crypto_rng *tfm, const u8 *seed,
unsigned int slen)
{
return 0;
}
/**
* Supported Crypto Algorithms
*/
@ -1291,6 +1364,18 @@ static struct crypto4xx_alg_common crypto4xx_alg[] = {
.cra_module = THIS_MODULE,
},
} },
{ .type = CRYPTO_ALG_TYPE_RNG, .u.rng = {
.base = {
.cra_name = "stdrng",
.cra_driver_name = "crypto4xx_rng",
.cra_priority = 300,
.cra_ctxsize = 0,
.cra_module = THIS_MODULE,
},
.generate = crypto4xx_prng_generate,
.seed = crypto4xx_prng_seed,
.seedsize = 0,
} },
};
/**
@ -1360,6 +1445,7 @@ static int crypto4xx_probe(struct platform_device *ofdev)
core_dev->dev->core_dev = core_dev;
core_dev->dev->is_revb = is_revb;
core_dev->device = dev;
mutex_init(&core_dev->rng_lock);
spin_lock_init(&core_dev->lock);
INIT_LIST_HEAD(&core_dev->dev->alg_list);
ratelimit_default_init(&core_dev->dev->aead_ratelimit);
@ -1439,6 +1525,7 @@ static int crypto4xx_remove(struct platform_device *ofdev)
tasklet_kill(&core_dev->tasklet);
/* Un-register with Linux CryptoAPI */
crypto4xx_unregister_alg(core_dev->dev);
mutex_destroy(&core_dev->rng_lock);
/* Free all allocated memory */
crypto4xx_stop_all(core_dev);

4
drivers/crypto/amcc/crypto4xx_core.h
View File

@ -23,8 +23,10 @@
#define __CRYPTO4XX_CORE_H__
#include <linux/ratelimit.h>
#include <linux/mutex.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h>
#include "crypto4xx_reg_def.h"
#include "crypto4xx_sa.h"
@ -119,6 +121,7 @@ struct crypto4xx_core_device {
u32 irq;
struct tasklet_struct tasklet;
spinlock_t lock;
struct mutex rng_lock;
};
struct crypto4xx_ctx {
@ -143,6 +146,7 @@ struct crypto4xx_alg_common {
struct skcipher_alg cipher;
struct ahash_alg hash;
struct aead_alg aead;
struct rng_alg rng;
} u;
};

1
drivers/crypto/amcc/crypto4xx_reg_def.h
View File

@ -100,6 +100,7 @@
#define CRYPTO4XX_ENDIAN_CFG 0x000600d8
#define CRYPTO4XX_PRNG_STAT 0x00070000
#define CRYPTO4XX_PRNG_STAT_BUSY 0x1
#define CRYPTO4XX_PRNG_CTRL 0x00070004
#define CRYPTO4XX_PRNG_SEED_L 0x00070008
#define CRYPTO4XX_PRNG_SEED_H 0x0007000c

4
drivers/crypto/amcc/crypto4xx_trng.c
View File

@ -80,8 +80,10 @@ void ppc4xx_trng_probe(struct crypto4xx_core_device *core_dev)
/* Find the TRNG device node and map it */
trng = of_find_matching_node(NULL, ppc4xx_trng_match);
if (!trng || !of_device_is_available(trng))
if (!trng || !of_device_is_available(trng)) {
of_node_put(trng);
return;
}
dev->trng_base = of_iomap(trng, 0);
of_node_put(trng);

4
drivers/crypto/amcc/crypto4xx_trng.h
View File

@ -26,9 +26,9 @@ void ppc4xx_trng_probe(struct crypto4xx_core_device *core_dev);
void ppc4xx_trng_remove(struct crypto4xx_core_device *core_dev);
#else
static inline void ppc4xx_trng_probe(
struct crypto4xx_device *dev __maybe_unused) { }
struct crypto4xx_core_device *dev __maybe_unused) { }
static inline void ppc4xx_trng_remove(
struct crypto4xx_device *dev __maybe_unused) { }
struct crypto4xx_core_device *dev __maybe_unused) { }
#endif
#endif

2
drivers/crypto/atmel-tdes.c
View File

@ -785,7 +785,7 @@ static int atmel_des_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
}
err = des_ekey(tmp, key);
if (err == 0 && (ctfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
if (err == 0 && (ctfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
ctfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}

328
drivers/crypto/axis/artpec6_crypto.c
View File

@ -135,8 +135,6 @@
#define regk_crypto_ext 0x00000001
#define regk_crypto_hmac_sha1 0x00000007
#define regk_crypto_hmac_sha256 0x00000009
#define regk_crypto_hmac_sha384 0x0000000b
#define regk_crypto_hmac_sha512 0x0000000d
#define regk_crypto_init 0x00000000
#define regk_crypto_key_128 0x00000000
#define regk_crypto_key_192 0x00000001
@ -144,8 +142,6 @@
#define regk_crypto_null 0x00000000
#define regk_crypto_sha1 0x00000006
#define regk_crypto_sha256 0x00000008
#define regk_crypto_sha384 0x0000000a
#define regk_crypto_sha512 0x0000000c
/* DMA descriptor structures */
struct pdma_descr_ctrl {
@ -190,8 +186,6 @@ struct pdma_stat_descr {
/* Hash modes (including HMAC variants) */
#define ARTPEC6_CRYPTO_HASH_SHA1 1
#define ARTPEC6_CRYPTO_HASH_SHA256 2
#define ARTPEC6_CRYPTO_HASH_SHA384 3
#define ARTPEC6_CRYPTO_HASH_SHA512 4
/* Crypto modes */
#define ARTPEC6_CRYPTO_CIPHER_AES_ECB 1
@ -284,6 +278,7 @@ enum artpec6_crypto_hash_flags {
struct artpec6_crypto_req_common {
struct list_head list;
struct list_head complete_in_progress;
struct artpec6_crypto_dma_descriptors *dma;
struct crypto_async_request *req;
void (*complete)(struct crypto_async_request *req);
@ -291,11 +286,11 @@ struct artpec6_crypto_req_common {
};
struct artpec6_hash_request_context {
char partial_buffer[SHA512_BLOCK_SIZE];
char partial_buffer_out[SHA512_BLOCK_SIZE];
char key_buffer[SHA512_BLOCK_SIZE];
char pad_buffer[SHA512_BLOCK_SIZE + 32];
unsigned char digeststate[SHA512_DIGEST_SIZE];
char partial_buffer[SHA256_BLOCK_SIZE];
char partial_buffer_out[SHA256_BLOCK_SIZE];
char key_buffer[SHA256_BLOCK_SIZE];
char pad_buffer[SHA256_BLOCK_SIZE + 32];
unsigned char digeststate[SHA256_DIGEST_SIZE];
size_t partial_bytes;
u64 digcnt;
u32 key_md;
@ -305,8 +300,8 @@ struct artpec6_hash_request_context {
};
struct artpec6_hash_export_state {
char partial_buffer[SHA512_BLOCK_SIZE];
unsigned char digeststate[SHA512_DIGEST_SIZE];
char partial_buffer[SHA256_BLOCK_SIZE];
unsigned char digeststate[SHA256_DIGEST_SIZE];
size_t partial_bytes;
u64 digcnt;
int oper;
@ -314,7 +309,7 @@ struct artpec6_hash_export_state {
};
struct artpec6_hashalg_context {
char hmac_key[SHA512_BLOCK_SIZE];
char hmac_key[SHA256_BLOCK_SIZE];
size_t hmac_key_length;
struct crypto_shash *child_hash;
};
@ -670,8 +665,8 @@ artpec6_crypto_dma_map_descs(struct artpec6_crypto_req_common *common)
* to be written.
*/
return artpec6_crypto_dma_map_single(common,
dma->stat + dma->in_cnt - 1,
sizeof(dma->stat[0]),
dma->stat,
sizeof(dma->stat[0]) * dma->in_cnt,
DMA_BIDIRECTIONAL,
&dma->stat_dma_addr);
}
@ -1315,8 +1310,7 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
struct artpec6_hashalg_context *ctx = crypto_tfm_ctx(areq->base.tfm);
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(areq);
size_t digestsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(areq));
size_t contextsize = digestsize == SHA384_DIGEST_SIZE ?
SHA512_DIGEST_SIZE : digestsize;
size_t contextsize = digestsize;
size_t blocksize = crypto_tfm_alg_blocksize(
crypto_ahash_tfm(crypto_ahash_reqtfm(areq)));
struct artpec6_crypto_req_common *common = &req_ctx->common;
@ -1456,7 +1450,6 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
/* Finalize */
if (req_ctx->hash_flags & HASH_FLAG_FINALIZE) {
bool needtrim = contextsize != digestsize;
size_t hash_pad_len;
u64 digest_bits;
u32 oper;
@ -1502,19 +1495,10 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
/* Descriptor for the final result */
error = artpec6_crypto_setup_in_descr(common, areq->result,
digestsize,
!needtrim);
true);
if (error)
return error;
if (needtrim) {
/* Discard the extra context bytes for SHA-384 */
error = artpec6_crypto_setup_in_descr(common,
req_ctx->partial_buffer,
digestsize - contextsize, true);
if (error)
return error;
}
} else { /* This is not the final operation for this request */
if (!run_hw)
return ARTPEC6_CRYPTO_PREPARE_HASH_NO_START;
@ -1923,7 +1907,7 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
/* For the decryption, cryptlen includes the tag. */
input_length = areq->cryptlen;
if (req_ctx->decrypt)
input_length -= AES_BLOCK_SIZE;
input_length -= crypto_aead_authsize(cipher);
/* Prepare the context buffer */
req_ctx->hw_ctx.aad_length_bits =
@ -1988,7 +1972,7 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
size_t output_len = areq->cryptlen;
if (req_ctx->decrypt)
output_len -= AES_BLOCK_SIZE;
output_len -= crypto_aead_authsize(cipher);
artpec6_crypto_walk_init(&walk, areq->dst);
@ -2017,19 +2001,32 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
* the output ciphertext. For decryption it is put in a context
* buffer for later compare against the input tag.
*/
count = AES_BLOCK_SIZE;
if (req_ctx->decrypt) {
ret = artpec6_crypto_setup_in_descr(common,
req_ctx->decryption_tag, count, false);
req_ctx->decryption_tag, AES_BLOCK_SIZE, false);
if (ret)
return ret;
} else {
/* For encryption the requested tag size may be smaller
* than the hardware's generated tag.
*/
size_t authsize = crypto_aead_authsize(cipher);
ret = artpec6_crypto_setup_sg_descrs_in(common, &walk,
count);
authsize);
if (ret)
return ret;
if (authsize < AES_BLOCK_SIZE) {
count = AES_BLOCK_SIZE - authsize;
ret = artpec6_crypto_setup_in_descr(common,
ac->pad_buffer,
count, false);
if (ret)
return ret;
}
}
}
@ -2045,7 +2042,8 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
return artpec6_crypto_dma_map_descs(common);
}
static void artpec6_crypto_process_queue(struct artpec6_crypto *ac)
static void artpec6_crypto_process_queue(struct artpec6_crypto *ac,
struct list_head *completions)
{
struct artpec6_crypto_req_common *req;
@ -2056,7 +2054,7 @@ static void artpec6_crypto_process_queue(struct artpec6_crypto *ac)
list_move_tail(&req->list, &ac->pending);
artpec6_crypto_start_dma(req);
req->req->complete(req->req, -EINPROGRESS);
list_add_tail(&req->complete_in_progress, completions);
}
/*
@ -2086,6 +2084,11 @@ static void artpec6_crypto_task(unsigned long data)
struct artpec6_crypto *ac = (struct artpec6_crypto *)data;
struct artpec6_crypto_req_common *req;
struct artpec6_crypto_req_common *n;
struct list_head complete_done;
struct list_head complete_in_progress;
INIT_LIST_HEAD(&complete_done);
INIT_LIST_HEAD(&complete_in_progress);
if (list_empty(&ac->pending)) {
pr_debug("Spurious IRQ\n");
@ -2097,9 +2100,12 @@ static void artpec6_crypto_task(unsigned long data)
list_for_each_entry_safe(req, n, &ac->pending, list) {
struct artpec6_crypto_dma_descriptors *dma = req->dma;
u32 stat;
dma_addr_t stataddr;
dma_sync_single_for_cpu(artpec6_crypto_dev, dma->stat_dma_addr,
sizeof(dma->stat[0]),
stataddr = dma->stat_dma_addr + 4 * (req->dma->in_cnt - 1);
dma_sync_single_for_cpu(artpec6_crypto_dev,
stataddr,
4,
DMA_BIDIRECTIONAL);
stat = req->dma->stat[req->dma->in_cnt-1];
@ -2119,19 +2125,30 @@ static void artpec6_crypto_task(unsigned long data)
pr_debug("Completing request %p\n", req);
list_del(&req->list);
artpec6_crypto_dma_unmap_all(req);
artpec6_crypto_copy_bounce_buffers(req);
list_move_tail(&req->list, &complete_done);
ac->pending_count--;
}
artpec6_crypto_process_queue(ac, &complete_in_progress);
spin_unlock_bh(&ac->queue_lock);
/* Perform the completion callbacks without holding the queue lock
* to allow new request submissions from the callbacks.
*/
list_for_each_entry_safe(req, n, &complete_done, list) {
artpec6_crypto_dma_unmap_all(req);
artpec6_crypto_copy_bounce_buffers(req);
artpec6_crypto_common_destroy(req);
req->complete(req->req);
}
artpec6_crypto_process_queue(ac);
spin_unlock_bh(&ac->queue_lock);
list_for_each_entry_safe(req, n, &complete_in_progress,
complete_in_progress) {
req->req->complete(req->req, -EINPROGRESS);
}
}
static void artpec6_crypto_complete_crypto(struct crypto_async_request *req)
@ -2170,27 +2187,29 @@ static void artpec6_crypto_complete_aead(struct crypto_async_request *req)
/* Verify GCM hashtag. */
struct aead_request *areq = container_of(req,
struct aead_request, base);
struct crypto_aead *aead = crypto_aead_reqtfm(areq);
struct artpec6_crypto_aead_req_ctx *req_ctx = aead_request_ctx(areq);
if (req_ctx->decrypt) {
u8 input_tag[AES_BLOCK_SIZE];
unsigned int authsize = crypto_aead_authsize(aead);
sg_pcopy_to_buffer(areq->src,
sg_nents(areq->src),
input_tag,
AES_BLOCK_SIZE,
authsize,
areq->assoclen + areq->cryptlen -
AES_BLOCK_SIZE);
authsize);
if (memcmp(req_ctx->decryption_tag,
input_tag,
AES_BLOCK_SIZE)) {
if (crypto_memneq(req_ctx->decryption_tag,
input_tag,
authsize)) {
pr_debug("***EBADMSG:\n");
print_hex_dump_debug("ref:", DUMP_PREFIX_ADDRESS, 32, 1,
input_tag, AES_BLOCK_SIZE, true);
input_tag, authsize, true);
print_hex_dump_debug("out:", DUMP_PREFIX_ADDRESS, 32, 1,
req_ctx->decryption_tag,
AES_BLOCK_SIZE, true);
authsize, true);
result = -EBADMSG;
}
@ -2266,13 +2285,6 @@ artpec6_crypto_init_hash(struct ahash_request *req, u8 type, int hmac)
case ARTPEC6_CRYPTO_HASH_SHA256:
oper = hmac ? regk_crypto_hmac_sha256 : regk_crypto_sha256;
break;
case ARTPEC6_CRYPTO_HASH_SHA384:
oper = hmac ? regk_crypto_hmac_sha384 : regk_crypto_sha384;
break;
case ARTPEC6_CRYPTO_HASH_SHA512:
oper = hmac ? regk_crypto_hmac_sha512 : regk_crypto_sha512;
break;
default:
pr_err("%s: Unsupported hash type 0x%x\n", MODULE_NAME, type);
return -EINVAL;
@ -2368,53 +2380,11 @@ static int artpec6_crypto_sha256_digest(struct ahash_request *req)
return artpec6_crypto_prepare_submit_hash(req);
}
static int __maybe_unused artpec6_crypto_sha384_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 0);
}
static int __maybe_unused
artpec6_crypto_sha384_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 0);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_sha512_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 0);
}
static int artpec6_crypto_sha512_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 0);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_hmac_sha256_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA256, 1);
}
static int __maybe_unused
artpec6_crypto_hmac_sha384_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 1);
}
static int artpec6_crypto_hmac_sha512_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 1);
}
static int artpec6_crypto_hmac_sha256_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
@ -2425,27 +2395,6 @@ static int artpec6_crypto_hmac_sha256_digest(struct ahash_request *req)
return artpec6_crypto_prepare_submit_hash(req);
}
static int __maybe_unused
artpec6_crypto_hmac_sha384_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 1);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_hmac_sha512_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 1);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_ahash_init_common(struct crypto_tfm *tfm,
const char *base_hash_name)
{
@ -2480,17 +2429,6 @@ static int artpec6_crypto_ahash_init_hmac_sha256(struct crypto_tfm *tfm)
return artpec6_crypto_ahash_init_common(tfm, "sha256");
}
static int __maybe_unused
artpec6_crypto_ahash_init_hmac_sha384(struct crypto_tfm *tfm)
{
return artpec6_crypto_ahash_init_common(tfm, "sha384");
}
static int artpec6_crypto_ahash_init_hmac_sha512(struct crypto_tfm *tfm)
{
return artpec6_crypto_ahash_init_common(tfm, "sha512");
}
static void artpec6_crypto_ahash_exit(struct crypto_tfm *tfm)
{
struct artpec6_hashalg_context *tfm_ctx = crypto_tfm_ctx(tfm);
@ -2761,103 +2699,6 @@ static struct ahash_alg hash_algos[] = {
},
};
static struct ahash_alg artpec7_hash_algos[] = {
/* SHA-384 */
{
.init = artpec6_crypto_sha384_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_sha384_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.halg.digestsize = SHA384_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "sha384",
.cra_driver_name = "artpec-sha384",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA384_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* HMAC SHA-384 */
{
.init = artpec6_crypto_hmac_sha384_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_hmac_sha384_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.setkey = artpec6_crypto_hash_set_key,
.halg.digestsize = SHA384_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "hmac(sha384)",
.cra_driver_name = "artpec-hmac-sha384",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA384_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init_hmac_sha384,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* SHA-512 */
{
.init = artpec6_crypto_sha512_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_sha512_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.halg.digestsize = SHA512_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "sha512",
.cra_driver_name = "artpec-sha512",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA512_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* HMAC SHA-512 */
{
.init = artpec6_crypto_hmac_sha512_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_hmac_sha512_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.setkey = artpec6_crypto_hash_set_key,
.halg.digestsize = SHA512_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "hmac(sha512)",
.cra_driver_name = "artpec-hmac-sha512",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA512_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init_hmac_sha512,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
};
/* Crypto */
static struct skcipher_alg crypto_algos[] = {
/* AES - ECB */
@ -2984,12 +2825,6 @@ static void artpec6_crypto_init_debugfs(void)
{
dbgfs_root = debugfs_create_dir("artpec6_crypto", NULL);
if (!dbgfs_root || IS_ERR(dbgfs_root)) {
dbgfs_root = NULL;
pr_err("%s: Could not initialise debugfs!\n", MODULE_NAME);
return;
}
#ifdef CONFIG_FAULT_INJECTION
fault_create_debugfs_attr("fail_status_read", dbgfs_root,
&artpec6_crypto_fail_status_read);
@ -3001,9 +2836,6 @@ static void artpec6_crypto_init_debugfs(void)
static void artpec6_crypto_free_debugfs(void)
{
if (!dbgfs_root)
return;
debugfs_remove_recursive(dbgfs_root);
dbgfs_root = NULL;
}
@ -3104,19 +2936,10 @@ static int artpec6_crypto_probe(struct platform_device *pdev)
goto disable_hw;
}
if (variant != ARTPEC6_CRYPTO) {
err = crypto_register_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
if (err) {
dev_err(dev, "Failed to register ahashes\n");
goto unregister_ahashes;
}
}
err = crypto_register_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
if (err) {
dev_err(dev, "Failed to register ciphers\n");
goto unregister_a7_ahashes;
goto unregister_ahashes;
}
err = crypto_register_aeads(aead_algos, ARRAY_SIZE(aead_algos));
@ -3129,10 +2952,6 @@ static int artpec6_crypto_probe(struct platform_device *pdev)
unregister_algs:
crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
unregister_a7_ahashes:
if (variant != ARTPEC6_CRYPTO)
crypto_unregister_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
unregister_ahashes:
crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos));
disable_hw:
@ -3148,9 +2967,6 @@ static int artpec6_crypto_remove(struct platform_device *pdev)
int irq = platform_get_irq(pdev, 0);
crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos));
if (ac->variant != ARTPEC6_CRYPTO)
crypto_unregister_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
crypto_unregister_aeads(aead_algos, ARRAY_SIZE(aead_algos));

2
drivers/crypto/bcm/Makefile
View File

@ -11,5 +11,3 @@
obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) := bcm_crypto_spu.o
bcm_crypto_spu-objs := util.o spu.o spu2.o cipher.o
ccflags-y += -I. -DBCMDRIVER

10
drivers/crypto/bcm/cipher.c
View File

@ -717,7 +717,7 @@ static int handle_ahash_req(struct iproc_reqctx_s *rctx)
*/
unsigned int new_data_len;
unsigned int chunk_start = 0;
unsigned int __maybe_unused chunk_start = 0;
u32 db_size; /* Length of data field, incl gcm and hash padding */
int pad_len = 0; /* total pad len, including gcm, hash, stat padding */
u32 data_pad_len = 0; /* length of GCM/CCM padding */
@ -1675,8 +1675,6 @@ static void spu_rx_callback(struct mbox_client *cl, void *msg)
struct spu_hw *spu = &iproc_priv.spu;
struct brcm_message *mssg = msg;
struct iproc_reqctx_s *rctx;
struct iproc_ctx_s *ctx;
struct crypto_async_request *areq;
int err = 0;
rctx = mssg->ctx;
@ -1686,8 +1684,6 @@ static void spu_rx_callback(struct mbox_client *cl, void *msg)
err = -EFAULT;
goto cb_finish;
}
areq = rctx->parent;
ctx = rctx->ctx;
/* process the SPU status */
err = spu->spu_status_process(rctx->msg_buf.rx_stat);
@ -1822,7 +1818,7 @@ static int des_setkey(struct crypto_ablkcipher *cipher, const u8 *key,
if (keylen == DES_KEY_SIZE) {
if (des_ekey(tmp, key) == 0) {
if (crypto_ablkcipher_get_flags(cipher) &
CRYPTO_TFM_REQ_WEAK_KEY) {
CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) {
u32 flags = CRYPTO_TFM_RES_WEAK_KEY;
crypto_ablkcipher_set_flags(cipher, flags);
@ -2876,7 +2872,7 @@ static int aead_authenc_setkey(struct crypto_aead *cipher,
if (des_ekey(tmp, keys.enckey) == 0) {
if (crypto_aead_get_flags(cipher) &
CRYPTO_TFM_REQ_WEAK_KEY) {
CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) {
crypto_aead_set_flags(cipher, flags);
return -EINVAL;
}

4
drivers/crypto/bcm/cipher.h
View File

@ -23,6 +23,7 @@
#include <crypto/aes.h>
#include <crypto/internal/hash.h>
#include <crypto/aead.h>
#include <crypto/arc4.h>
#include <crypto/gcm.h>
#include <crypto/sha.h>
#include <crypto/sha3.h>
@ -34,9 +35,6 @@
/* Driver supports up to MAX_SPUS SPU blocks */
#define MAX_SPUS 16
#define ARC4_MIN_KEY_SIZE 1
#define ARC4_MAX_KEY_SIZE 256
#define ARC4_BLOCK_SIZE 1
#define ARC4_STATE_SIZE 4
#define CCM_AES_IV_SIZE 16

40
drivers/crypto/bcm/util.c
View File

@ -201,46 +201,6 @@ struct sdesc {
char ctx[];
};
/* do a synchronous decrypt operation */
int do_decrypt(char *alg_name,
void *key_ptr, unsigned int key_len,
void *iv_ptr, void *src_ptr, void *dst_ptr,
unsigned int block_len)
{
struct scatterlist sg_in[1], sg_out[1];
struct crypto_blkcipher *tfm =
crypto_alloc_blkcipher(alg_name, 0, CRYPTO_ALG_ASYNC);
struct blkcipher_desc desc = {.tfm = tfm, .flags = 0 };
int ret = 0;
void *iv;
int ivsize;
flow_log("%s() name:%s block_len:%u\n", __func__, alg_name, block_len);
if (IS_ERR(tfm))
return PTR_ERR(tfm);
crypto_blkcipher_setkey((void *)tfm, key_ptr, key_len);
sg_init_table(sg_in, 1);
sg_set_buf(sg_in, src_ptr, block_len);
sg_init_table(sg_out, 1);
sg_set_buf(sg_out, dst_ptr, block_len);
iv = crypto_blkcipher_crt(tfm)->iv;
ivsize = crypto_blkcipher_ivsize(tfm);
memcpy(iv, iv_ptr, ivsize);
ret = crypto_blkcipher_decrypt(&desc, sg_out, sg_in, block_len);
crypto_free_blkcipher(tfm);
if (ret < 0)
pr_err("aes_decrypt failed %d\n", ret);
return ret;
}
/**
* do_shash() - Do a synchronous hash operation in software
* @name: The name of the hash algorithm

6
drivers/crypto/bcm/util.h
View File

@ -95,12 +95,6 @@ u32 spu_msg_sg_add(struct scatterlist **to_sg,
void add_to_ctr(u8 *ctr_pos, unsigned int increment);
/* do a synchronous decrypt operation */
int do_decrypt(char *alg_name,
void *key_ptr, unsigned int key_len,
void *iv_ptr, void *src_ptr, void *dst_ptr,
unsigned int block_len);
/* produce a message digest from data of length n bytes */
int do_shash(unsigned char *name, unsigned char *result,
const u8 *data1, unsigned int data1_len,

1
drivers/crypto/caam/Kconfig
View File

@ -91,6 +91,7 @@ config CRYPTO_DEV_FSL_CAAM_CRYPTO_API
select CRYPTO_AEAD
select CRYPTO_AUTHENC
select CRYPTO_BLKCIPHER
select CRYPTO_DES
help
Selecting this will offload crypto for users of the
scatterlist crypto API (such as the linux native IPSec

238
drivers/crypto/caam/caamalg.c
View File

@ -3,7 +3,7 @@
* caam - Freescale FSL CAAM support for crypto API
*
* Copyright 2008-2011 Freescale Semiconductor, Inc.
* Copyright 2016-2018 NXP
* Copyright 2016-2019 NXP
*
* Based on talitos crypto API driver.
*
@ -766,6 +766,27 @@ static int skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
return 0;
}
static int des_skcipher_setkey(struct crypto_skcipher *skcipher,
const u8 *key, unsigned int keylen)
{
u32 tmp[DES3_EDE_EXPKEY_WORDS];
struct crypto_tfm *tfm = crypto_skcipher_tfm(skcipher);
if (keylen == DES3_EDE_KEY_SIZE &&
__des3_ede_setkey(tmp, &tfm->crt_flags, key, DES3_EDE_KEY_SIZE)) {
return -EINVAL;
}
if (!des_ekey(tmp, key) && (crypto_skcipher_get_flags(skcipher) &
CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
crypto_skcipher_set_flags(skcipher,
CRYPTO_TFM_RES_WEAK_KEY);
return -EINVAL;
}
return skcipher_setkey(skcipher, key, keylen);
}
static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
unsigned int keylen)
{
@ -802,6 +823,8 @@ static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
* aead_edesc - s/w-extended aead descriptor
* @src_nents: number of segments in input s/w scatterlist
* @dst_nents: number of segments in output s/w scatterlist
* @mapped_src_nents: number of segments in input h/w link table
* @mapped_dst_nents: number of segments in output h/w link table
* @sec4_sg_bytes: length of dma mapped sec4_sg space
* @sec4_sg_dma: bus physical mapped address of h/w link table
* @sec4_sg: pointer to h/w link table
@ -810,6 +833,8 @@ static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
struct aead_edesc {
int src_nents;
int dst_nents;
int mapped_src_nents;
int mapped_dst_nents;
int sec4_sg_bytes;
dma_addr_t sec4_sg_dma;
struct sec4_sg_entry *sec4_sg;
@ -820,6 +845,8 @@ struct aead_edesc {
* skcipher_edesc - s/w-extended skcipher descriptor
* @src_nents: number of segments in input s/w scatterlist
* @dst_nents: number of segments in output s/w scatterlist
* @mapped_src_nents: number of segments in input h/w link table
* @mapped_dst_nents: number of segments in output h/w link table
* @iv_dma: dma address of iv for checking continuity and link table
* @sec4_sg_bytes: length of dma mapped sec4_sg space
* @sec4_sg_dma: bus physical mapped address of h/w link table
@ -830,6 +857,8 @@ struct aead_edesc {
struct skcipher_edesc {
int src_nents;
int dst_nents;
int mapped_src_nents;
int mapped_dst_nents;
dma_addr_t iv_dma;
int sec4_sg_bytes;
dma_addr_t sec4_sg_dma;
@ -846,7 +875,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) {
if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
}
@ -961,8 +991,9 @@ static void skcipher_encrypt_done(struct device *jrdev, u32 *desc, u32 err,
* The crypto API expects us to set the IV (req->iv) to the last
* ciphertext block. This is used e.g. by the CTS mode.
*/
scatterwalk_map_and_copy(req->iv, req->dst, req->cryptlen - ivsize,
ivsize, 0);
if (ivsize)
scatterwalk_map_and_copy(req->iv, req->dst, req->cryptlen -
ivsize, ivsize, 0);
kfree(edesc);
@ -1023,11 +1054,12 @@ static void init_aead_job(struct aead_request *req,
init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE);
if (all_contig) {
src_dma = edesc->src_nents ? sg_dma_address(req->src) : 0;
src_dma = edesc->mapped_src_nents ? sg_dma_address(req->src) :
0;
in_options = 0;
} else {
src_dma = edesc->sec4_sg_dma;
sec4_sg_index += edesc->src_nents;
sec4_sg_index += edesc->mapped_src_nents;
in_options = LDST_SGF;
}
@ -1038,8 +1070,11 @@ static void init_aead_job(struct aead_request *req,
out_options = in_options;
if (unlikely(req->src != req->dst)) {
if (edesc->dst_nents == 1) {
if (!edesc->mapped_dst_nents) {
dst_dma = 0;
} else if (edesc->mapped_dst_nents == 1) {
dst_dma = sg_dma_address(req->dst);
out_options = 0;
} else {
dst_dma = edesc->sec4_sg_dma +
sec4_sg_index *
@ -1183,9 +1218,9 @@ static void init_skcipher_job(struct skcipher_request *req,
int ivsize = crypto_skcipher_ivsize(skcipher);
u32 *desc = edesc->hw_desc;
u32 *sh_desc;
u32 out_options = 0;
dma_addr_t dst_dma, ptr;
int len;
u32 in_options = 0, out_options = 0;
dma_addr_t src_dma, dst_dma, ptr;
int len, sec4_sg_index = 0;
#ifdef DEBUG
print_hex_dump(KERN_ERR, "presciv@"__stringify(__LINE__)": ",
@ -1203,21 +1238,27 @@ static void init_skcipher_job(struct skcipher_request *req,
len = desc_len(sh_desc);
init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE);
append_seq_in_ptr(desc, edesc->sec4_sg_dma, req->cryptlen + ivsize,
LDST_SGF);
if (ivsize || edesc->mapped_src_nents > 1) {
src_dma = edesc->sec4_sg_dma;
sec4_sg_index = edesc->mapped_src_nents + !!ivsize;
in_options = LDST_SGF;
} else {
src_dma = sg_dma_address(req->src);
}
append_seq_in_ptr(desc, src_dma, req->cryptlen + ivsize, in_options);
if (likely(req->src == req->dst)) {
dst_dma = edesc->sec4_sg_dma + sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
dst_dma = src_dma + !!ivsize * sizeof(struct sec4_sg_entry);
out_options = in_options;
} else if (edesc->mapped_dst_nents == 1) {
dst_dma = sg_dma_address(req->dst);
} else {
if (edesc->dst_nents == 1) {
dst_dma = sg_dma_address(req->dst);
} else {
dst_dma = edesc->sec4_sg_dma + (edesc->src_nents + 1) *
sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
dst_dma = edesc->sec4_sg_dma + sec4_sg_index *
sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
append_seq_out_ptr(desc, dst_dma, req->cryptlen, out_options);
}
@ -1289,12 +1330,19 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0;
}
mapped_dst_nents = dma_map_sg(jrdev, req->dst, dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(jrdev, "unable to map destination\n");
dma_unmap_sg(jrdev, req->src, src_nents, DMA_TO_DEVICE);
return ERR_PTR(-ENOMEM);
/* Cover also the case of null (zero length) output data */
if (dst_nents) {
mapped_dst_nents = dma_map_sg(jrdev, req->dst,
dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(jrdev, "unable to map destination\n");
dma_unmap_sg(jrdev, req->src, src_nents,
DMA_TO_DEVICE);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
}
}
@ -1313,6 +1361,8 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents;
edesc->mapped_src_nents = mapped_src_nents;
edesc->mapped_dst_nents = mapped_dst_nents;
edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
desc_bytes;
*all_contig_ptr = !(mapped_src_nents > 1);
@ -1586,7 +1636,7 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
GFP_KERNEL : GFP_ATOMIC;
int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
struct skcipher_edesc *edesc;
dma_addr_t iv_dma;
dma_addr_t iv_dma = 0;
u8 *iv;
int ivsize = crypto_skcipher_ivsize(skcipher);
int dst_sg_idx, sec4_sg_ents, sec4_sg_bytes;
@ -1621,7 +1671,6 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
dev_err(jrdev, "unable to map source\n");
return ERR_PTR(-ENOMEM);
}
mapped_dst_nents = dma_map_sg(jrdev, req->dst, dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
@ -1631,7 +1680,10 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
}
}
sec4_sg_ents = 1 + mapped_src_nents;
if (!ivsize && mapped_src_nents == 1)
sec4_sg_ents = 0; // no need for an input hw s/g table
else
sec4_sg_ents = mapped_src_nents + !!ivsize;
dst_sg_idx = sec4_sg_ents;
sec4_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0;
sec4_sg_bytes = sec4_sg_ents * sizeof(struct sec4_sg_entry);
@ -1650,39 +1702,48 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents;
edesc->mapped_src_nents = mapped_src_nents;
edesc->mapped_dst_nents = mapped_dst_nents;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (struct sec4_sg_entry *)((u8 *)edesc->hw_desc +
desc_bytes);
/* Make sure IV is located in a DMAable area */
iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes;
memcpy(iv, req->iv, ivsize);
if (ivsize) {
iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes;
memcpy(iv, req->iv, ivsize);
iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, iv_dma)) {
dev_err(jrdev, "unable to map IV\n");
caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0,
0, 0, 0);
kfree(edesc);
return ERR_PTR(-ENOMEM);
iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, iv_dma)) {
dev_err(jrdev, "unable to map IV\n");
caam_unmap(jrdev, req->src, req->dst, src_nents,
dst_nents, 0, 0, 0, 0);
kfree(edesc);
return ERR_PTR(-ENOMEM);
}
dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0);
}
dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0);
sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg + 1, 0);
if (dst_sg_idx)
sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg +
!!ivsize, 0);
if (mapped_dst_nents > 1) {
sg_to_sec4_sg_last(req->dst, mapped_dst_nents,
edesc->sec4_sg + dst_sg_idx, 0);
}
edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
sec4_sg_bytes, DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) {
dev_err(jrdev, "unable to map S/G table\n");
caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents,
iv_dma, ivsize, 0, 0);
kfree(edesc);
return ERR_PTR(-ENOMEM);
if (sec4_sg_bytes) {
edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
sec4_sg_bytes,
DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) {
dev_err(jrdev, "unable to map S/G table\n");
caam_unmap(jrdev, req->src, req->dst, src_nents,
dst_nents, iv_dma, ivsize, 0, 0);
kfree(edesc);
return ERR_PTR(-ENOMEM);
}
}
edesc->iv_dma = iv_dma;
@ -1749,8 +1810,9 @@ static int skcipher_decrypt(struct skcipher_request *req)
* The crypto API expects us to set the IV (req->iv) to the last
* ciphertext block.
*/
scatterwalk_map_and_copy(req->iv, req->src, req->cryptlen - ivsize,
ivsize, 0);
if (ivsize)
scatterwalk_map_and_copy(req->iv, req->src, req->cryptlen -
ivsize, ivsize, 0);
/* Create and submit job descriptor*/
init_skcipher_job(req, edesc, false);
@ -1796,7 +1858,7 @@ static struct caam_skcipher_alg driver_algs[] = {
.cra_driver_name = "cbc-3des-caam",
.cra_blocksize = DES3_EDE_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES3_EDE_KEY_SIZE,
@ -1812,7 +1874,7 @@ static struct caam_skcipher_alg driver_algs[] = {
.cra_driver_name = "cbc-des-caam",
.cra_blocksize = DES_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES_KEY_SIZE,
@ -1878,6 +1940,66 @@ static struct caam_skcipher_alg driver_algs[] = {
},
.caam.class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XTS,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(des)",
.cra_driver_name = "ecb-des-caam",
.cra_blocksize = DES_BLOCK_SIZE,
},
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES_KEY_SIZE,
.max_keysize = DES_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(aes)",
.cra_driver_name = "ecb-aes-caam",
.cra_blocksize = AES_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(des3_ede)",
.cra_driver_name = "ecb-des3-caam",
.cra_blocksize = DES3_EDE_BLOCK_SIZE,
},
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES3_EDE_KEY_SIZE,
.max_keysize = DES3_EDE_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(arc4)",
.cra_driver_name = "ecb-arc4-caam",
.cra_blocksize = ARC4_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = ARC4_MIN_KEY_SIZE,
.max_keysize = ARC4_MAX_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_ARC4 | OP_ALG_AAI_ECB,
},
};
static struct caam_aead_alg driver_aeads[] = {
@ -3337,6 +3459,7 @@ static int __init caam_algapi_init(void)
struct caam_drv_private *priv;
int i = 0, err = 0;
u32 aes_vid, aes_inst, des_inst, md_vid, md_inst, ccha_inst, ptha_inst;
u32 arc4_inst;
unsigned int md_limit = SHA512_DIGEST_SIZE;
bool registered = false;
@ -3381,6 +3504,8 @@ static int __init caam_algapi_init(void)
CHA_ID_LS_DES_SHIFT;
aes_inst = cha_inst & CHA_ID_LS_AES_MASK;
md_inst = (cha_inst & CHA_ID_LS_MD_MASK) >> CHA_ID_LS_MD_SHIFT;
arc4_inst = (cha_inst & CHA_ID_LS_ARC4_MASK) >>
CHA_ID_LS_ARC4_SHIFT;
ccha_inst = 0;
ptha_inst = 0;
} else {
@ -3397,6 +3522,7 @@ static int __init caam_algapi_init(void)
md_inst = mdha & CHA_VER_NUM_MASK;
ccha_inst = rd_reg32(&priv->ctrl->vreg.ccha) & CHA_VER_NUM_MASK;
ptha_inst = rd_reg32(&priv->ctrl->vreg.ptha) & CHA_VER_NUM_MASK;
arc4_inst = rd_reg32(&priv->ctrl->vreg.afha) & CHA_VER_NUM_MASK;
}
/* If MD is present, limit digest size based on LP256 */
@ -3417,6 +3543,10 @@ static int __init caam_algapi_init(void)
if (!aes_inst && (alg_sel == OP_ALG_ALGSEL_AES))
continue;
/* Skip ARC4 algorithms if not supported by device */
if (!arc4_inst && alg_sel == OP_ALG_ALGSEL_ARC4)
continue;
/*
* Check support for AES modes not available
* on LP devices.

18
drivers/crypto/caam/caamalg_desc.c
View File

@ -2,7 +2,7 @@
/*
* Shared descriptors for aead, skcipher algorithms
*
* Copyright 2016-2018 NXP
* Copyright 2016-2019 NXP
*/
#include "compat.h"
@ -1396,9 +1396,11 @@ void cnstr_shdsc_skcipher_encap(u32 * const desc, struct alginfo *cdata,
set_jump_tgt_here(desc, key_jump_cmd);
/* Load iv */
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off << LDST_OFFSET_SHIFT));
/* Load IV, if there is one */
if (ivsize)
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off <<
LDST_OFFSET_SHIFT));
/* Load counter into CONTEXT1 reg */
if (is_rfc3686)
@ -1462,9 +1464,11 @@ void cnstr_shdsc_skcipher_decap(u32 * const desc, struct alginfo *cdata,
set_jump_tgt_here(desc, key_jump_cmd);
/* load IV */
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off << LDST_OFFSET_SHIFT));
/* Load IV, if there is one */
if (ivsize)
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off <<
LDST_OFFSET_SHIFT));
/* Load counter into CONTEXT1 reg */
if (is_rfc3686)

29
drivers/crypto/caam/caamalg_qi.c
View File

@ -782,7 +782,7 @@ static struct caam_drv_ctx *get_drv_ctx(struct caam_ctx *ctx,
cpu = smp_processor_id();
drv_ctx = caam_drv_ctx_init(ctx->qidev, &cpu, desc);
if (likely(!IS_ERR_OR_NULL(drv_ctx)))
if (!IS_ERR_OR_NULL(drv_ctx))
drv_ctx->op_type = type;
ctx->drv_ctx[type] = drv_ctx;
@ -802,7 +802,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) {
if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
}
@ -892,7 +893,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
struct caam_drv_ctx *drv_ctx;
drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT);
if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
if (IS_ERR_OR_NULL(drv_ctx))
return (struct aead_edesc *)drv_ctx;
/* allocate space for base edesc and hw desc commands, link tables */
@ -955,13 +956,19 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0;
}
mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(qidev, "unable to map destination\n");
dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
if (dst_nents) {
mapped_dst_nents = dma_map_sg(qidev, req->dst,
dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(qidev, "unable to map destination\n");
dma_unmap_sg(qidev, req->src, src_nents,
DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
}
}
@ -1184,7 +1191,7 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
struct caam_drv_ctx *drv_ctx;
drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT);
if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
if (IS_ERR_OR_NULL(drv_ctx))
return (struct skcipher_edesc *)drv_ctx;
src_nents = sg_nents_for_len(req->src, req->cryptlen);

85
drivers/crypto/caam/caamalg_qi2.c
View File

@ -25,13 +25,6 @@
#define CAAM_MAX_KEY_SIZE (AES_MAX_KEY_SIZE + CTR_RFC3686_NONCE_SIZE + \
SHA512_DIGEST_SIZE * 2)
#if !IS_ENABLED(CONFIG_CRYPTO_DEV_FSL_CAAM)
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
#endif
/*
* This is a a cache of buffers, from which the users of CAAM QI driver
* can allocate short buffers. It's speedier than doing kmalloc on the hotpath.
@ -151,7 +144,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) {
if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
}
@ -392,13 +386,18 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0;
}
mapped_dst_nents = dma_map_sg(dev, req->dst, dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(dev, "unable to map destination\n");
dma_unmap_sg(dev, req->src, src_nents, DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
if (dst_nents) {
mapped_dst_nents = dma_map_sg(dev, req->dst, dst_nents,
DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) {
dev_err(dev, "unable to map destination\n");
dma_unmap_sg(dev, req->src, src_nents,
DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
}
} else {
src_nents = sg_nents_for_len(req->src, req->assoclen +
@ -4503,7 +4502,8 @@ static int __cold dpaa2_dpseci_dpio_setup(struct dpaa2_caam_priv *priv)
nctx->cb = dpaa2_caam_fqdan_cb;
/* Register notification callbacks */
err = dpaa2_io_service_register(NULL, nctx);
ppriv->dpio = dpaa2_io_service_select(cpu);
err = dpaa2_io_service_register(ppriv->dpio, nctx, dev);
if (unlikely(err)) {
dev_dbg(dev, "No affine DPIO for cpu %d\n", cpu);
nctx->cb = NULL;
@ -4536,7 +4536,7 @@ err:
ppriv = per_cpu_ptr(priv->ppriv, cpu);
if (!ppriv->nctx.cb)
break;
dpaa2_io_service_deregister(NULL, &ppriv->nctx);
dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx, dev);
}
for_each_online_cpu(cpu) {
@ -4556,7 +4556,8 @@ static void __cold dpaa2_dpseci_dpio_free(struct dpaa2_caam_priv *priv)
for_each_online_cpu(cpu) {
ppriv = per_cpu_ptr(priv->ppriv, cpu);
dpaa2_io_service_deregister(NULL, &ppriv->nctx);
dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx,
priv->dev);
dpaa2_io_store_destroy(ppriv->store);
if (++i == priv->num_pairs)
@ -4654,7 +4655,7 @@ static int dpaa2_caam_pull_fq(struct dpaa2_caam_priv_per_cpu *ppriv)
/* Retry while portal is busy */
do {
err = dpaa2_io_service_pull_fq(NULL, ppriv->rsp_fqid,
err = dpaa2_io_service_pull_fq(ppriv->dpio, ppriv->rsp_fqid,
ppriv->store);
} while (err == -EBUSY);
@ -4722,7 +4723,7 @@ static int dpaa2_dpseci_poll(struct napi_struct *napi, int budget)
if (cleaned < budget) {
napi_complete_done(napi, cleaned);
err = dpaa2_io_service_rearm(NULL, &ppriv->nctx);
err = dpaa2_io_service_rearm(ppriv->dpio, &ppriv->nctx);
if (unlikely(err))
dev_err(priv->dev, "Notification rearm failed: %d\n",
err);
@ -4863,21 +4864,31 @@ static int __cold dpaa2_dpseci_setup(struct fsl_mc_device *ls_dev)
i = 0;
for_each_online_cpu(cpu) {
dev_dbg(dev, "pair %d: rx queue %d, tx queue %d\n", i,
priv->rx_queue_attr[i].fqid,
priv->tx_queue_attr[i].fqid);
u8 j;
j = i % priv->num_pairs;
ppriv = per_cpu_ptr(priv->ppriv, cpu);
ppriv->req_fqid = priv->tx_queue_attr[i].fqid;
ppriv->rsp_fqid = priv->rx_queue_attr[i].fqid;
ppriv->prio = i;
ppriv->req_fqid = priv->tx_queue_attr[j].fqid;
/*
* Allow all cores to enqueue, while only some of them
* will take part in dequeuing.
*/
if (++i > priv->num_pairs)
continue;
ppriv->rsp_fqid = priv->rx_queue_attr[j].fqid;
ppriv->prio = j;
dev_dbg(dev, "pair %d: rx queue %d, tx queue %d\n", j,
priv->rx_queue_attr[j].fqid,
priv->tx_queue_attr[j].fqid);
ppriv->net_dev.dev = *dev;
INIT_LIST_HEAD(&ppriv->net_dev.napi_list);
netif_napi_add(&ppriv->net_dev, &ppriv->napi, dpaa2_dpseci_poll,
DPAA2_CAAM_NAPI_WEIGHT);
if (++i == priv->num_pairs)
break;
}
return 0;
@ -5229,7 +5240,8 @@ int dpaa2_caam_enqueue(struct device *dev, struct caam_request *req)
{
struct dpaa2_fd fd;
struct dpaa2_caam_priv *priv = dev_get_drvdata(dev);
int err = 0, i, id;
struct dpaa2_caam_priv_per_cpu *ppriv;
int err = 0, i;
if (IS_ERR(req))
return PTR_ERR(req);
@ -5259,23 +5271,18 @@ int dpaa2_caam_enqueue(struct device *dev, struct caam_request *req)
dpaa2_fd_set_len(&fd, dpaa2_fl_get_len(&req->fd_flt[1]));
dpaa2_fd_set_flc(&fd, req->flc_dma);
/*
* There is no guarantee that preemption is disabled here,
* thus take action.
*/
preempt_disable();
id = smp_processor_id() % priv->dpseci_attr.num_tx_queues;
ppriv = this_cpu_ptr(priv->ppriv);
for (i = 0; i < (priv->dpseci_attr.num_tx_queues << 1); i++) {
err = dpaa2_io_service_enqueue_fq(NULL,
priv->tx_queue_attr[id].fqid,
err = dpaa2_io_service_enqueue_fq(ppriv->dpio, ppriv->req_fqid,
&fd);
if (err != -EBUSY)
break;
cpu_relax();
}
preempt_enable();
if (unlikely(err)) {
dev_err(dev, "Error enqueuing frame: %d\n", err);
dev_err_ratelimited(dev, "Error enqueuing frame: %d\n", err);
goto err_out;
}

2
drivers/crypto/caam/caamalg_qi2.h
View File

@ -76,6 +76,7 @@ struct dpaa2_caam_priv {
* @nctx: notification context of response FQ
* @store: where dequeued frames are stored
* @priv: backpointer to dpaa2_caam_priv
* @dpio: portal used for data path operations
*/
struct dpaa2_caam_priv_per_cpu {
struct napi_struct napi;
@ -86,6 +87,7 @@ struct dpaa2_caam_priv_per_cpu {
struct dpaa2_io_notification_ctx nctx;
struct dpaa2_io_store *store;
struct dpaa2_caam_priv *priv;
struct dpaa2_io *dpio;
};
/*

429
drivers/crypto/caam/caamhash.c
View File

@ -3,7 +3,7 @@
* caam - Freescale FSL CAAM support for ahash functions of crypto API
*
* Copyright 2011 Freescale Semiconductor, Inc.
* Copyright 2018 NXP
* Copyright 2018-2019 NXP
*
* Based on caamalg.c crypto API driver.
*
@ -98,13 +98,14 @@ struct caam_hash_ctx {
u32 sh_desc_update_first[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u32 sh_desc_fin[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u32 sh_desc_digest[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u8 key[CAAM_MAX_HASH_KEY_SIZE] ____cacheline_aligned;
dma_addr_t sh_desc_update_dma ____cacheline_aligned;
dma_addr_t sh_desc_update_first_dma;
dma_addr_t sh_desc_fin_dma;
dma_addr_t sh_desc_digest_dma;
dma_addr_t key_dma;
enum dma_data_direction dir;
struct device *jrdev;
u8 key[CAAM_MAX_HASH_KEY_SIZE];
int ctx_len;
struct alginfo adata;
};
@ -113,6 +114,7 @@ struct caam_hash_ctx {
struct caam_hash_state {
dma_addr_t buf_dma;
dma_addr_t ctx_dma;
int ctx_dma_len;
u8 buf_0[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
int buflen_0;
u8 buf_1[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
@ -158,6 +160,11 @@ static inline int *alt_buflen(struct caam_hash_state *state)
return state->current_buf ? &state->buflen_0 : &state->buflen_1;
}
static inline bool is_cmac_aes(u32 algtype)
{
return (algtype & (OP_ALG_ALGSEL_MASK | OP_ALG_AAI_MASK)) ==
(OP_ALG_ALGSEL_AES | OP_ALG_AAI_CMAC);
}
/* Common job descriptor seq in/out ptr routines */
/* Map state->caam_ctx, and append seq_out_ptr command that points to it */
@ -165,6 +172,7 @@ static inline int map_seq_out_ptr_ctx(u32 *desc, struct device *jrdev,
struct caam_hash_state *state,
int ctx_len)
{
state->ctx_dma_len = ctx_len;
state->ctx_dma = dma_map_single(jrdev, state->caam_ctx,
ctx_len, DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, state->ctx_dma)) {
@ -178,18 +186,6 @@ static inline int map_seq_out_ptr_ctx(u32 *desc, struct device *jrdev,
return 0;
}
/* Map req->result, and append seq_out_ptr command that points to it */
static inline dma_addr_t map_seq_out_ptr_result(u32 *desc, struct device *jrdev,
u8 *result, int digestsize)
{
dma_addr_t dst_dma;
dst_dma = dma_map_single(jrdev, result, digestsize, DMA_FROM_DEVICE);
append_seq_out_ptr(desc, dst_dma, digestsize, 0);
return dst_dma;
}
/* Map current buffer in state (if length > 0) and put it in link table */
static inline int buf_map_to_sec4_sg(struct device *jrdev,
struct sec4_sg_entry *sec4_sg,
@ -218,6 +214,7 @@ static inline int ctx_map_to_sec4_sg(struct device *jrdev,
struct caam_hash_state *state, int ctx_len,
struct sec4_sg_entry *sec4_sg, u32 flag)
{
state->ctx_dma_len = ctx_len;
state->ctx_dma = dma_map_single(jrdev, state->caam_ctx, ctx_len, flag);
if (dma_mapping_error(jrdev, state->ctx_dma)) {
dev_err(jrdev, "unable to map ctx\n");
@ -292,14 +289,119 @@ static int ahash_set_sh_desc(struct crypto_ahash *ahash)
return 0;
}
static int axcbc_set_sh_desc(struct crypto_ahash *ahash)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
int digestsize = crypto_ahash_digestsize(ahash);
struct device *jrdev = ctx->jrdev;
u32 *desc;
/* key is loaded from memory for UPDATE and FINALIZE states */
ctx->adata.key_dma = ctx->key_dma;
/* shared descriptor for ahash_update */
desc = ctx->sh_desc_update;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_UPDATE,
ctx->ctx_len, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc update shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* shared descriptor for ahash_{final,finup} */
desc = ctx->sh_desc_fin;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_FINALIZE,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_fin_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc finup shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* key is immediate data for INIT and INITFINAL states */
ctx->adata.key_virt = ctx->key;
/* shared descriptor for first invocation of ahash_update */
desc = ctx->sh_desc_update_first;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INIT, ctx->ctx_len,
ctx->ctx_len, ctx->key_dma);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_first_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc update first shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* shared descriptor for ahash_digest */
desc = ctx->sh_desc_digest;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INITFINAL,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_digest_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc digest shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
return 0;
}
static int acmac_set_sh_desc(struct crypto_ahash *ahash)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
int digestsize = crypto_ahash_digestsize(ahash);
struct device *jrdev = ctx->jrdev;
u32 *desc;
/* shared descriptor for ahash_update */
desc = ctx->sh_desc_update;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_UPDATE,
ctx->ctx_len, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac update shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for ahash_{final,finup} */
desc = ctx->sh_desc_fin;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_FINALIZE,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_fin_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac finup shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for first invocation of ahash_update */
desc = ctx->sh_desc_update_first;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INIT, ctx->ctx_len,
ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_first_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac update first shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for ahash_digest */
desc = ctx->sh_desc_digest;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INITFINAL,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_digest_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac digest shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
return 0;
}
/* Digest hash size if it is too large */
static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
u32 *keylen, u8 *key_out, u32 digestsize)
static int hash_digest_key(struct caam_hash_ctx *ctx, u32 *keylen, u8 *key,
u32 digestsize)
{
struct device *jrdev = ctx->jrdev;
u32 *desc;
struct split_key_result result;
dma_addr_t src_dma, dst_dma;
dma_addr_t key_dma;
int ret;
desc = kmalloc(CAAM_CMD_SZ * 8 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA);
@ -310,18 +412,9 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
init_job_desc(desc, 0);
src_dma = dma_map_single(jrdev, (void *)key_in, *keylen,
DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, src_dma)) {
dev_err(jrdev, "unable to map key input memory\n");
kfree(desc);
return -ENOMEM;
}
dst_dma = dma_map_single(jrdev, (void *)key_out, digestsize,
DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, dst_dma)) {
dev_err(jrdev, "unable to map key output memory\n");
dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE);
key_dma = dma_map_single(jrdev, key, *keylen, DMA_BIDIRECTIONAL);
if (dma_mapping_error(jrdev, key_dma)) {
dev_err(jrdev, "unable to map key memory\n");
kfree(desc);
return -ENOMEM;
}
@ -329,16 +422,16 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
/* Job descriptor to perform unkeyed hash on key_in */
append_operation(desc, ctx->adata.algtype | OP_ALG_ENCRYPT |
OP_ALG_AS_INITFINAL);
append_seq_in_ptr(desc, src_dma, *keylen, 0);
append_seq_in_ptr(desc, key_dma, *keylen, 0);
append_seq_fifo_load(desc, *keylen, FIFOLD_CLASS_CLASS2 |
FIFOLD_TYPE_LAST2 | FIFOLD_TYPE_MSG);
append_seq_out_ptr(desc, dst_dma, digestsize, 0);
append_seq_out_ptr(desc, key_dma, digestsize, 0);
append_seq_store(desc, digestsize, LDST_CLASS_2_CCB |
LDST_SRCDST_BYTE_CONTEXT);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "key_in@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in, *keylen, 1);
DUMP_PREFIX_ADDRESS, 16, 4, key, *keylen, 1);
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
#endif
@ -354,12 +447,10 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
#ifdef DEBUG
print_hex_dump(KERN_ERR,
"digested key@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in,
digestsize, 1);
DUMP_PREFIX_ADDRESS, 16, 4, key, digestsize, 1);
#endif
}
dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE);
dma_unmap_single(jrdev, dst_dma, digestsize, DMA_FROM_DEVICE);
dma_unmap_single(jrdev, key_dma, *keylen, DMA_BIDIRECTIONAL);
*keylen = digestsize;
@ -383,13 +474,10 @@ static int ahash_setkey(struct crypto_ahash *ahash,
#endif
if (keylen > blocksize) {
hashed_key = kmalloc_array(digestsize,
sizeof(*hashed_key),
GFP_KERNEL | GFP_DMA);
hashed_key = kmemdup(key, keylen, GFP_KERNEL | GFP_DMA);
if (!hashed_key)
return -ENOMEM;
ret = hash_digest_key(ctx, key, &keylen, hashed_key,
digestsize);
ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize);
if (ret)
goto bad_free_key;
key = hashed_key;
@ -424,9 +512,39 @@ static int ahash_setkey(struct crypto_ahash *ahash,
return -EINVAL;
}
static int axcbc_setkey(struct crypto_ahash *ahash, const u8 *key,
unsigned int keylen)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct device *jrdev = ctx->jrdev;
memcpy(ctx->key, key, keylen);
dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
ctx->adata.keylen = keylen;
print_hex_dump_debug("axcbc ctx.key@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, ctx->key, keylen, 1);
return axcbc_set_sh_desc(ahash);
}
static int acmac_setkey(struct crypto_ahash *ahash, const u8 *key,
unsigned int keylen)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
/* key is immediate data for all cmac shared descriptors */
ctx->adata.key_virt = key;
ctx->adata.keylen = keylen;
print_hex_dump_debug("acmac ctx.key@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
return acmac_set_sh_desc(ahash);
}
/*
* ahash_edesc - s/w-extended ahash descriptor
* @dst_dma: physical mapped address of req->result
* @sec4_sg_dma: physical mapped address of h/w link table
* @src_nents: number of segments in input scatterlist
* @sec4_sg_bytes: length of dma mapped sec4_sg space
@ -434,7 +552,6 @@ static int ahash_setkey(struct crypto_ahash *ahash,
* @sec4_sg: h/w link table
*/
struct ahash_edesc {
dma_addr_t dst_dma;
dma_addr_t sec4_sg_dma;
int src_nents;
int sec4_sg_bytes;
@ -450,8 +567,6 @@ static inline void ahash_unmap(struct device *dev,
if (edesc->src_nents)
dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE);
if (edesc->dst_dma)
dma_unmap_single(dev, edesc->dst_dma, dst_len, DMA_FROM_DEVICE);
if (edesc->sec4_sg_bytes)
dma_unmap_single(dev, edesc->sec4_sg_dma,
@ -468,12 +583,10 @@ static inline void ahash_unmap_ctx(struct device *dev,
struct ahash_edesc *edesc,
struct ahash_request *req, int dst_len, u32 flag)
{
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
if (state->ctx_dma) {
dma_unmap_single(dev, state->ctx_dma, ctx->ctx_len, flag);
dma_unmap_single(dev, state->ctx_dma, state->ctx_dma_len, flag);
state->ctx_dma = 0;
}
ahash_unmap(dev, edesc, req, dst_len);
@ -486,9 +599,9 @@ static void ahash_done(struct device *jrdev, u32 *desc, u32 err,
struct ahash_edesc *edesc;
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
int digestsize = crypto_ahash_digestsize(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
#ifdef DEBUG
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
#endif
@ -497,17 +610,14 @@ static void ahash_done(struct device *jrdev, u32 *desc, u32 err,
if (err)
caam_jr_strstatus(jrdev, err);
ahash_unmap(jrdev, edesc, req, digestsize);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
memcpy(req->result, state->caam_ctx, digestsize);
kfree(edesc);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
ctx->ctx_len, 1);
if (req->result)
print_hex_dump(KERN_ERR, "result@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, req->result,
digestsize, 1);
#endif
req->base.complete(&req->base, err);
@ -555,9 +665,9 @@ static void ahash_done_ctx_src(struct device *jrdev, u32 *desc, u32 err,
struct ahash_edesc *edesc;
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
int digestsize = crypto_ahash_digestsize(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
#ifdef DEBUG
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
#endif
@ -566,17 +676,14 @@ static void ahash_done_ctx_src(struct device *jrdev, u32 *desc, u32 err,
if (err)
caam_jr_strstatus(jrdev, err);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_TO_DEVICE);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
memcpy(req->result, state->caam_ctx, digestsize);
kfree(edesc);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
ctx->ctx_len, 1);
if (req->result)
print_hex_dump(KERN_ERR, "result@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, req->result,
digestsize, 1);
#endif
req->base.complete(&req->base, err);
@ -688,6 +795,7 @@ static int ahash_update_ctx(struct ahash_request *req)
u8 *buf = current_buf(state);
int *buflen = current_buflen(state);
u8 *next_buf = alt_buf(state);
int blocksize = crypto_ahash_blocksize(ahash);
int *next_buflen = alt_buflen(state), last_buflen;
int in_len = *buflen + req->nbytes, to_hash;
u32 *desc;
@ -696,9 +804,20 @@ static int ahash_update_ctx(struct ahash_request *req)
int ret = 0;
last_buflen = *next_buflen;
*next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1);
*next_buflen = in_len & (blocksize - 1);
to_hash = in_len - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) {
src_nents = sg_nents_for_len(req->src,
req->nbytes - (*next_buflen));
@ -801,7 +920,7 @@ static int ahash_update_ctx(struct ahash_request *req)
#endif
return ret;
unmap_ctx:
unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len, DMA_BIDIRECTIONAL);
kfree(edesc);
return ret;
@ -837,7 +956,7 @@ static int ahash_final_ctx(struct ahash_request *req)
edesc->sec4_sg_bytes = sec4_sg_bytes;
ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len,
edesc->sec4_sg, DMA_TO_DEVICE);
edesc->sec4_sg, DMA_BIDIRECTIONAL);
if (ret)
goto unmap_ctx;
@ -857,14 +976,7 @@ static int ahash_final_ctx(struct ahash_request *req)
append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len + buflen,
LDST_SGF);
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = -ENOMEM;
goto unmap_ctx;
}
append_seq_out_ptr(desc, state->ctx_dma, digestsize, 0);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -877,7 +989,7 @@ static int ahash_final_ctx(struct ahash_request *req)
return -EINPROGRESS;
unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
kfree(edesc);
return ret;
}
@ -931,7 +1043,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
edesc->src_nents = src_nents;
ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len,
edesc->sec4_sg, DMA_TO_DEVICE);
edesc->sec4_sg, DMA_BIDIRECTIONAL);
if (ret)
goto unmap_ctx;
@ -945,13 +1057,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
if (ret)
goto unmap_ctx;
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = -ENOMEM;
goto unmap_ctx;
}
append_seq_out_ptr(desc, state->ctx_dma, digestsize, 0);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -964,7 +1070,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
return -EINPROGRESS;
unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
kfree(edesc);
return ret;
}
@ -1023,10 +1129,8 @@ static int ahash_digest(struct ahash_request *req)
desc = edesc->hw_desc;
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
if (ret) {
ahash_unmap(jrdev, edesc, req, digestsize);
kfree(edesc);
return -ENOMEM;
@ -1041,7 +1145,7 @@ static int ahash_digest(struct ahash_request *req)
if (!ret) {
ret = -EINPROGRESS;
} else {
ahash_unmap(jrdev, edesc, req, digestsize);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc);
}
@ -1083,12 +1187,9 @@ static int ahash_final_no_ctx(struct ahash_request *req)
append_seq_in_ptr(desc, state->buf_dma, buflen, 0);
}
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
if (ret)
goto unmap;
}
#ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -1099,7 +1200,7 @@ static int ahash_final_no_ctx(struct ahash_request *req)
if (!ret) {
ret = -EINPROGRESS;
} else {
ahash_unmap(jrdev, edesc, req, digestsize);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc);
}
@ -1122,6 +1223,7 @@ static int ahash_update_no_ctx(struct ahash_request *req)
GFP_KERNEL : GFP_ATOMIC;
u8 *buf = current_buf(state);
int *buflen = current_buflen(state);
int blocksize = crypto_ahash_blocksize(ahash);
u8 *next_buf = alt_buf(state);
int *next_buflen = alt_buflen(state);
int in_len = *buflen + req->nbytes, to_hash;
@ -1130,9 +1232,20 @@ static int ahash_update_no_ctx(struct ahash_request *req)
u32 *desc;
int ret = 0;
*next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1);
*next_buflen = in_len & (blocksize - 1);
to_hash = in_len - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) {
src_nents = sg_nents_for_len(req->src,
req->nbytes - *next_buflen);
@ -1298,12 +1411,9 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
goto unmap;
}
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
if (ret)
goto unmap;
}
#ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -1314,7 +1424,7 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
if (!ret) {
ret = -EINPROGRESS;
} else {
ahash_unmap(jrdev, edesc, req, digestsize);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc);
}
@ -1338,15 +1448,26 @@ static int ahash_update_first(struct ahash_request *req)
u8 *next_buf = alt_buf(state);
int *next_buflen = alt_buflen(state);
int to_hash;
int blocksize = crypto_ahash_blocksize(ahash);
u32 *desc;
int src_nents, mapped_nents;
struct ahash_edesc *edesc;
int ret = 0;
*next_buflen = req->nbytes & (crypto_tfm_alg_blocksize(&ahash->base) -
1);
*next_buflen = req->nbytes & (blocksize - 1);
to_hash = req->nbytes - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) {
src_nents = sg_nents_for_len(req->src,
req->nbytes - *next_buflen);
@ -1446,6 +1567,7 @@ static int ahash_init(struct ahash_request *req)
state->final = ahash_final_no_ctx;
state->ctx_dma = 0;
state->ctx_dma_len = 0;
state->current_buf = 0;
state->buf_dma = 0;
state->buflen_0 = 0;
@ -1654,6 +1776,44 @@ static struct caam_hash_template driver_hash[] = {
},
},
.alg_type = OP_ALG_ALGSEL_MD5,
}, {
.hmac_name = "xcbc(aes)",
.hmac_driver_name = "xcbc-aes-caam",
.blocksize = AES_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
.update = ahash_update,
.final = ahash_final,
.finup = ahash_finup,
.digest = ahash_digest,
.export = ahash_export,
.import = ahash_import,
.setkey = axcbc_setkey,
.halg = {
.digestsize = AES_BLOCK_SIZE,
.statesize = sizeof(struct caam_export_state),
},
},
.alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XCBC_MAC,
}, {
.hmac_name = "cmac(aes)",
.hmac_driver_name = "cmac-aes-caam",
.blocksize = AES_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
.update = ahash_update,
.final = ahash_final,
.finup = ahash_finup,
.digest = ahash_digest,
.export = ahash_export,
.import = ahash_import,
.setkey = acmac_setkey,
.halg = {
.digestsize = AES_BLOCK_SIZE,
.statesize = sizeof(struct caam_export_state),
},
},
.alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CMAC,
},
};
@ -1695,14 +1855,45 @@ static int caam_hash_cra_init(struct crypto_tfm *tfm)
}
priv = dev_get_drvdata(ctx->jrdev->parent);
ctx->dir = priv->era >= 6 ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE;
if (is_xcbc_aes(caam_hash->alg_type)) {
ctx->dir = DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS1_ALG | caam_hash->alg_type;
ctx->ctx_len = 48;
ctx->key_dma = dma_map_single_attrs(ctx->jrdev, ctx->key,
ARRAY_SIZE(ctx->key),
DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
if (dma_mapping_error(ctx->jrdev, ctx->key_dma)) {
dev_err(ctx->jrdev, "unable to map key\n");
caam_jr_free(ctx->jrdev);
return -ENOMEM;
}
} else if (is_cmac_aes(caam_hash->alg_type)) {
ctx->dir = DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS1_ALG | caam_hash->alg_type;
ctx->ctx_len = 32;
} else {
ctx->dir = priv->era >= 6 ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam_hash->alg_type;
ctx->ctx_len = runninglen[(ctx->adata.algtype &
OP_ALG_ALGSEL_SUBMASK) >>
OP_ALG_ALGSEL_SHIFT];
}
dma_addr = dma_map_single_attrs(ctx->jrdev, ctx->sh_desc_update,
offsetof(struct caam_hash_ctx,
sh_desc_update_dma),
offsetof(struct caam_hash_ctx, key),
ctx->dir, DMA_ATTR_SKIP_CPU_SYNC);
if (dma_mapping_error(ctx->jrdev, dma_addr)) {
dev_err(ctx->jrdev, "unable to map shared descriptors\n");
if (is_xcbc_aes(caam_hash->alg_type))
dma_unmap_single_attrs(ctx->jrdev, ctx->key_dma,
ARRAY_SIZE(ctx->key),
DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
caam_jr_free(ctx->jrdev);
return -ENOMEM;
}
@ -1716,16 +1907,14 @@ static int caam_hash_cra_init(struct crypto_tfm *tfm)
ctx->sh_desc_digest_dma = dma_addr + offsetof(struct caam_hash_ctx,
sh_desc_digest);
/* copy descriptor header template value */
ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam_hash->alg_type;
ctx->ctx_len = runninglen[(ctx->adata.algtype &
OP_ALG_ALGSEL_SUBMASK) >>
OP_ALG_ALGSEL_SHIFT];
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct caam_hash_state));
return ahash_set_sh_desc(ahash);
/*
* For keyed hash algorithms shared descriptors
* will be created later in setkey() callback
*/
return alg->setkey ? 0 : ahash_set_sh_desc(ahash);
}
static void caam_hash_cra_exit(struct crypto_tfm *tfm)
@ -1733,9 +1922,12 @@ static void caam_hash_cra_exit(struct crypto_tfm *tfm)
struct caam_hash_ctx *ctx = crypto_tfm_ctx(tfm);
dma_unmap_single_attrs(ctx->jrdev, ctx->sh_desc_update_dma,
offsetof(struct caam_hash_ctx,
sh_desc_update_dma),
offsetof(struct caam_hash_ctx, key),
ctx->dir, DMA_ATTR_SKIP_CPU_SYNC);
if (is_xcbc_aes(ctx->adata.algtype))
dma_unmap_single_attrs(ctx->jrdev, ctx->key_dma,
ARRAY_SIZE(ctx->key), DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
caam_jr_free(ctx->jrdev);
}
@ -1866,14 +2058,16 @@ static int __init caam_algapi_hash_init(void)
struct caam_hash_template *alg = driver_hash + i;
/* If MD size is not supported by device, skip registration */
if (alg->template_ahash.halg.digestsize > md_limit)
if (is_mdha(alg->alg_type) &&
alg->template_ahash.halg.digestsize > md_limit)
continue;
/* register hmac version */
t_alg = caam_hash_alloc(alg, true);
if (IS_ERR(t_alg)) {
err = PTR_ERR(t_alg);
pr_warn("%s alg allocation failed\n", alg->driver_name);
pr_warn("%s alg allocation failed\n",
alg->hmac_driver_name);
continue;
}
@ -1886,6 +2080,9 @@ static int __init caam_algapi_hash_init(void)
} else
list_add_tail(&t_alg->entry, &hash_list);
if ((alg->alg_type & OP_ALG_ALGSEL_MASK) == OP_ALG_ALGSEL_AES)
continue;
/* register unkeyed version */
t_alg = caam_hash_alloc(alg, false);
if (IS_ERR(t_alg)) {

68
drivers/crypto/caam/caamhash_desc.c
View File

@ -2,7 +2,7 @@
/*
* Shared descriptors for ahash algorithms
*
* Copyright 2017 NXP
* Copyright 2017-2019 NXP
*/
#include "compat.h"
@ -75,6 +75,72 @@ void cnstr_shdsc_ahash(u32 * const desc, struct alginfo *adata, u32 state,
}
EXPORT_SYMBOL(cnstr_shdsc_ahash);
/**
* cnstr_shdsc_sk_hash - shared descriptor for symmetric key cipher-based
* hash algorithms
* @desc: pointer to buffer used for descriptor construction
* @adata: pointer to authentication transform definitions.
* @state: algorithm state OP_ALG_AS_{INIT, FINALIZE, INITFINALIZE, UPDATE}
* @digestsize: algorithm's digest size
* @ctx_len: size of Context Register
* @key_dma: I/O Virtual Address of the key
*/
void cnstr_shdsc_sk_hash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, dma_addr_t key_dma)
{
u32 *skip_key_load;
init_sh_desc(desc, HDR_SHARE_SERIAL | HDR_SAVECTX);
/* Skip loading of key, context if already shared */
skip_key_load = append_jump(desc, JUMP_TEST_ALL | JUMP_COND_SHRD);
if (state == OP_ALG_AS_INIT || state == OP_ALG_AS_INITFINAL) {
append_key_as_imm(desc, adata->key_virt, adata->keylen,
adata->keylen, CLASS_1 | KEY_DEST_CLASS_REG);
} else { /* UPDATE, FINALIZE */
if (is_xcbc_aes(adata->algtype))
/* Load K1 */
append_key(desc, adata->key_dma, adata->keylen,
CLASS_1 | KEY_DEST_CLASS_REG | KEY_ENC);
else /* CMAC */
append_key_as_imm(desc, adata->key_virt, adata->keylen,
adata->keylen, CLASS_1 |
KEY_DEST_CLASS_REG);
/* Restore context */
append_seq_load(desc, ctx_len, LDST_CLASS_1_CCB |
LDST_SRCDST_BYTE_CONTEXT);
}
set_jump_tgt_here(desc, skip_key_load);
/* Class 1 operation */
append_operation(desc, adata->algtype | state | OP_ALG_ENCRYPT);
/*
* Load from buf and/or src and write to req->result or state->context
* Calculate remaining bytes to read
*/
append_math_add(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ);
/* Read remaining bytes */
append_seq_fifo_load(desc, 0, FIFOLD_CLASS_CLASS1 | FIFOLD_TYPE_LAST1 |
FIFOLD_TYPE_MSG | FIFOLDST_VLF);
/*
* Save context:
* - xcbc: partial hash, keys K2 and K3
* - cmac: partial hash, constant L = E(K,0)
*/
append_seq_store(desc, digestsize, LDST_CLASS_1_CCB |
LDST_SRCDST_BYTE_CONTEXT);
if (is_xcbc_aes(adata->algtype) && state == OP_ALG_AS_INIT)
/* Save K1 */
append_fifo_store(desc, key_dma, adata->keylen,
LDST_CLASS_1_CCB | FIFOST_TYPE_KEY_KEK);
}
EXPORT_SYMBOL(cnstr_shdsc_sk_hash);
MODULE_LICENSE("Dual BSD/GPL");
MODULE_DESCRIPTION("FSL CAAM ahash descriptors support");
MODULE_AUTHOR("NXP Semiconductors");

8
drivers/crypto/caam/caamhash_desc.h
View File

@ -15,7 +15,15 @@
#define DESC_AHASH_FINAL_LEN (DESC_AHASH_BASE + 5 * CAAM_CMD_SZ)
#define DESC_AHASH_DIGEST_LEN (DESC_AHASH_BASE + 4 * CAAM_CMD_SZ)
static inline bool is_xcbc_aes(u32 algtype)
{
return (algtype & (OP_ALG_ALGSEL_MASK | OP_ALG_AAI_MASK)) ==
(OP_ALG_ALGSEL_AES | OP_ALG_AAI_XCBC_MAC);
}
void cnstr_shdsc_ahash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, bool import_ctx, int era);
void cnstr_shdsc_sk_hash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, dma_addr_t key_dma);
#endif /* _CAAMHASH_DESC_H_ */

1
drivers/crypto/caam/compat.h
View File

@ -43,6 +43,7 @@
#include <crypto/akcipher.h>
#include <crypto/scatterwalk.h>
#include <crypto/skcipher.h>
#include <crypto/arc4.h>
#include <crypto/internal/skcipher.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/rsa.h>

25
drivers/crypto/caam/ctrl.c
View File

@ -18,12 +18,8 @@
#include "desc_constr.h"
#include "ctrl.h"
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_dpaa2;
EXPORT_SYMBOL(caam_dpaa2);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
#ifdef CONFIG_CAAM_QI
#include "qi.h"
@ -863,27 +859,18 @@ static int caam_probe(struct platform_device *pdev)
/* Internal covering keys (useful in non-secure mode only) */
ctrlpriv->ctl_kek_wrap.data = (__force void *)&ctrlpriv->ctrl->kek[0];
ctrlpriv->ctl_kek_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_kek = debugfs_create_blob("kek",
S_IRUSR |
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_kek_wrap);
debugfs_create_blob("kek", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
&ctrlpriv->ctl_kek_wrap);
ctrlpriv->ctl_tkek_wrap.data = (__force void *)&ctrlpriv->ctrl->tkek[0];
ctrlpriv->ctl_tkek_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_tkek = debugfs_create_blob("tkek",
S_IRUSR |
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_tkek_wrap);
debugfs_create_blob("tkek", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
&ctrlpriv->ctl_tkek_wrap);
ctrlpriv->ctl_tdsk_wrap.data = (__force void *)&ctrlpriv->ctrl->tdsk[0];
ctrlpriv->ctl_tdsk_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_tdsk = debugfs_create_blob("tdsk",
S_IRUSR |
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_tdsk_wrap);
debugfs_create_blob("tdsk", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
&ctrlpriv->ctl_tdsk_wrap);
#endif
return 0;

6
drivers/crypto/caam/error.c
View File

@ -50,6 +50,12 @@ void caam_dump_sg(const char *level, const char *prefix_str, int prefix_type,
#endif /* DEBUG */
EXPORT_SYMBOL(caam_dump_sg);
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
static const struct {
u8 value;
const char *error_text;

1
drivers/crypto/caam/intern.h
View File

@ -106,7 +106,6 @@ struct caam_drv_private {
struct dentry *dfs_root;
struct dentry *ctl; /* controller dir */
struct debugfs_blob_wrapper ctl_kek_wrap, ctl_tkek_wrap, ctl_tdsk_wrap;
struct dentry *ctl_kek, *ctl_tkek, *ctl_tdsk;
#endif
};

30
drivers/crypto/caam/key_gen.c
View File

@ -48,7 +48,7 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
{
u32 *desc;
struct split_key_result result;
dma_addr_t dma_addr_in, dma_addr_out;
dma_addr_t dma_addr;
int ret = -ENOMEM;
adata->keylen = split_key_len(adata->algtype & OP_ALG_ALGSEL_MASK);
@ -71,22 +71,17 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
return ret;
}
dma_addr_in = dma_map_single(jrdev, (void *)key_in, keylen,
DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, dma_addr_in)) {
dev_err(jrdev, "unable to map key input memory\n");
memcpy(key_out, key_in, keylen);
dma_addr = dma_map_single(jrdev, key_out, adata->keylen_pad,
DMA_BIDIRECTIONAL);
if (dma_mapping_error(jrdev, dma_addr)) {
dev_err(jrdev, "unable to map key memory\n");
goto out_free;
}
dma_addr_out = dma_map_single(jrdev, key_out, adata->keylen_pad,
DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, dma_addr_out)) {
dev_err(jrdev, "unable to map key output memory\n");
goto out_unmap_in;
}
init_job_desc(desc, 0);
append_key(desc, dma_addr_in, keylen, CLASS_2 | KEY_DEST_CLASS_REG);
append_key(desc, dma_addr, keylen, CLASS_2 | KEY_DEST_CLASS_REG);
/* Sets MDHA up into an HMAC-INIT */
append_operation(desc, (adata->algtype & OP_ALG_ALGSEL_MASK) |
@ -104,12 +99,10 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
* FIFO_STORE with the explicit split-key content store
* (0x26 output type)
*/
append_fifo_store(desc, dma_addr_out, adata->keylen,
append_fifo_store(desc, dma_addr, adata->keylen,
LDST_CLASS_2_CCB | FIFOST_TYPE_SPLIT_KEK);
#ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx.key@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in, keylen, 1);
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
#endif
@ -129,10 +122,7 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
#endif
}
dma_unmap_single(jrdev, dma_addr_out, adata->keylen_pad,
DMA_FROM_DEVICE);
out_unmap_in:
dma_unmap_single(jrdev, dma_addr_in, keylen, DMA_TO_DEVICE);
dma_unmap_single(jrdev, dma_addr, adata->keylen_pad, DMA_BIDIRECTIONAL);
out_free:
kfree(desc);
return ret;

4
drivers/crypto/caam/qi.c
View File

@ -318,7 +318,7 @@ int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc)
/* Create a new req FQ in parked state */
new_fq = create_caam_req_fq(drv_ctx->qidev, drv_ctx->rsp_fq,
drv_ctx->context_a, 0);
if (unlikely(IS_ERR_OR_NULL(new_fq))) {
if (IS_ERR_OR_NULL(new_fq)) {
dev_err(qidev, "FQ allocation for shdesc update failed\n");
return PTR_ERR(new_fq);
}
@ -431,7 +431,7 @@ struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev,
/* Attach request FQ */
drv_ctx->req_fq = create_caam_req_fq(qidev, drv_ctx->rsp_fq, hwdesc,
QMAN_INITFQ_FLAG_SCHED);
if (unlikely(IS_ERR_OR_NULL(drv_ctx->req_fq))) {
if (IS_ERR_OR_NULL(drv_ctx->req_fq)) {
dev_err(qidev, "create_caam_req_fq failed\n");
dma_unmap_single(qidev, hwdesc, size, DMA_BIDIRECTIONAL);
kfree(drv_ctx);

27
drivers/crypto/cavium/nitrox/nitrox_debugfs.c
View File

@ -55,31 +55,14 @@ void nitrox_debugfs_exit(struct nitrox_device *ndev)
ndev->debugfs_dir = NULL;
}
int nitrox_debugfs_init(struct nitrox_device *ndev)
void nitrox_debugfs_init(struct nitrox_device *ndev)
{
struct dentry *dir, *f;
struct dentry *dir;
dir = debugfs_create_dir(KBUILD_MODNAME, NULL);
if (!dir)
return -ENOMEM;
ndev->debugfs_dir = dir;
f = debugfs_create_file("firmware", 0400, dir, ndev,
&firmware_fops);
if (!f)
goto err;
f = debugfs_create_file("device", 0400, dir, ndev,
&device_fops);
if (!f)
goto err;
f = debugfs_create_file("stats", 0400, dir, ndev,
&stats_fops);
if (!f)
goto err;
return 0;
err:
nitrox_debugfs_exit(ndev);
return -ENODEV;
debugfs_create_file("firmware", 0400, dir, ndev, &firmware_fops);
debugfs_create_file("device", 0400, dir, ndev, &device_fops);
debugfs_create_file("stats", 0400, dir, ndev, &stats_fops);
}

5
drivers/crypto/cavium/nitrox/nitrox_debugfs.h
View File

@ -5,12 +5,11 @@
#include "nitrox_dev.h"
#ifdef CONFIG_DEBUG_FS
int nitrox_debugfs_init(struct nitrox_device *ndev);
void nitrox_debugfs_init(struct nitrox_device *ndev);
void nitrox_debugfs_exit(struct nitrox_device *ndev);
#else
static inline int nitrox_debugfs_init(struct nitrox_device *ndev)
static inline void nitrox_debugfs_init(struct nitrox_device *ndev)
{
return 0;
}
static inline void nitrox_debugfs_exit(struct nitrox_device *ndev)

4
drivers/crypto/cavium/nitrox/nitrox_main.c
View File

@ -404,9 +404,7 @@ static int nitrox_probe(struct pci_dev *pdev,
if (err)
goto pf_hw_fail;
err = nitrox_debugfs_init(ndev);
if (err)
goto pf_hw_fail;
nitrox_debugfs_init(ndev);
/* clear the statistics */
atomic64_set(&ndev->stats.posted, 0);

56
drivers/crypto/cavium/zip/zip_main.c
View File

@ -351,6 +351,7 @@ static struct pci_driver zip_driver = {
static struct crypto_alg zip_comp_deflate = {
.cra_name = "deflate",
.cra_driver_name = "deflate-cavium",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
.cra_ctxsize = sizeof(struct zip_kernel_ctx),
.cra_priority = 300,
@ -365,6 +366,7 @@ static struct crypto_alg zip_comp_deflate = {
static struct crypto_alg zip_comp_lzs = {
.cra_name = "lzs",
.cra_driver_name = "lzs-cavium",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
.cra_ctxsize = sizeof(struct zip_kernel_ctx),
.cra_priority = 300,
@ -384,7 +386,7 @@ static struct scomp_alg zip_scomp_deflate = {
.decompress = zip_scomp_decompress,
.base = {
.cra_name = "deflate",
.cra_driver_name = "deflate-scomp",
.cra_driver_name = "deflate-scomp-cavium",
.cra_module = THIS_MODULE,
.cra_priority = 300,
}
@ -397,7 +399,7 @@ static struct scomp_alg zip_scomp_lzs = {
.decompress = zip_scomp_decompress,
.base = {
.cra_name = "lzs",
.cra_driver_name = "lzs-scomp",
.cra_driver_name = "lzs-scomp-cavium",
.cra_module = THIS_MODULE,
.cra_priority = 300,
}
@ -618,41 +620,23 @@ static const struct file_operations zip_regs_fops = {
/* Root directory for thunderx_zip debugfs entry */
static struct dentry *zip_debugfs_root;
static int __init zip_debugfs_init(void)
static void __init zip_debugfs_init(void)
{
struct dentry *zip_stats, *zip_clear, *zip_regs;
if (!debugfs_initialized())
return -ENODEV;
return;
zip_debugfs_root = debugfs_create_dir("thunderx_zip", NULL);
if (!zip_debugfs_root)
return -ENOMEM;
/* Creating files for entries inside thunderx_zip directory */
zip_stats = debugfs_create_file("zip_stats", 0444,
zip_debugfs_root,
NULL, &zip_stats_fops);
if (!zip_stats)
goto failed_to_create;
debugfs_create_file("zip_stats", 0444, zip_debugfs_root, NULL,
&zip_stats_fops);
zip_clear = debugfs_create_file("zip_clear", 0444,
zip_debugfs_root,
NULL, &zip_clear_fops);
if (!zip_clear)
goto failed_to_create;
debugfs_create_file("zip_clear", 0444, zip_debugfs_root, NULL,
&zip_clear_fops);
zip_regs = debugfs_create_file("zip_regs", 0444,
zip_debugfs_root,
NULL, &zip_regs_fops);
if (!zip_regs)
goto failed_to_create;
debugfs_create_file("zip_regs", 0444, zip_debugfs_root, NULL,
&zip_regs_fops);
return 0;
failed_to_create:
debugfs_remove_recursive(zip_debugfs_root);
return -ENOENT;
}
static void __exit zip_debugfs_exit(void)
@ -661,13 +645,8 @@ static void __exit zip_debugfs_exit(void)
}
#else
static int __init zip_debugfs_init(void)
{
return 0;
}
static void __init zip_debugfs_init(void) { }
static void __exit zip_debugfs_exit(void) { }
#endif
/* debugfs - end */
@ -691,17 +670,10 @@ static int __init zip_init_module(void)
}
/* comp-decomp statistics are handled with debugfs interface */
ret = zip_debugfs_init();
if (ret < 0) {
zip_err("ZIP: debugfs initialization failed\n");
goto err_crypto_unregister;
}
zip_debugfs_init();
return ret;
err_crypto_unregister:
zip_unregister_compression_device();
err_pci_unregister:
pci_unregister_driver(&zip_driver);
return ret;

2
drivers/crypto/ccp/ccp-crypto-aes-cmac.c
View File

@ -1,7 +1,7 @@
/*
* AMD Cryptographic Coprocessor (CCP) AES CMAC crypto API support
*
* Copyright (C) 2013 Advanced Micro Devices, Inc.
* Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
*
* Author: Tom Lendacky <thomas.lendacky@amd.com>
*

2
drivers/crypto/ccp/ccp-crypto-des3.c
View File

@ -57,7 +57,7 @@ static int ccp_des3_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
(*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL;
}

2
drivers/crypto/ccp/ccp-crypto-sha.c
View File

@ -1,7 +1,7 @@
/*
* AMD Cryptographic Coprocessor (CCP) SHA crypto API support
*
* Copyright (C) 2013,2017 Advanced Micro Devices, Inc.
* Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
*
* Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com>

34
drivers/crypto/ccp/ccp-debugfs.c
View File

@ -286,10 +286,7 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
{
struct ccp_cmd_queue *cmd_q;
char name[MAX_NAME_LEN + 1];
struct dentry *debugfs_info;
struct dentry *debugfs_stats;
struct dentry *debugfs_q_instance;
struct dentry *debugfs_q_stats;
int i;
if (!debugfs_initialized())
@ -299,24 +296,14 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
if (!ccp_debugfs_dir)
ccp_debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL);
mutex_unlock(&ccp_debugfs_lock);
if (!ccp_debugfs_dir)
return;
ccp->debugfs_instance = debugfs_create_dir(ccp->name, ccp_debugfs_dir);
if (!ccp->debugfs_instance)
goto err;
debugfs_info = debugfs_create_file("info", 0400,
ccp->debugfs_instance, ccp,
&ccp_debugfs_info_ops);
if (!debugfs_info)
goto err;
debugfs_create_file("info", 0400, ccp->debugfs_instance, ccp,
&ccp_debugfs_info_ops);
debugfs_stats = debugfs_create_file("stats", 0600,
ccp->debugfs_instance, ccp,
&ccp_debugfs_stats_ops);
if (!debugfs_stats)
goto err;
debugfs_create_file("stats", 0600, ccp->debugfs_instance, ccp,
&ccp_debugfs_stats_ops);
for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i];
@ -325,21 +312,12 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
debugfs_q_instance =
debugfs_create_dir(name, ccp->debugfs_instance);
if (!debugfs_q_instance)
goto err;
debugfs_q_stats =
debugfs_create_file("stats", 0600,
debugfs_q_instance, cmd_q,
&ccp_debugfs_queue_ops);
if (!debugfs_q_stats)
goto err;
debugfs_create_file("stats", 0600, debugfs_q_instance, cmd_q,
&ccp_debugfs_queue_ops);
}
return;
err:
debugfs_remove_recursive(ccp->debugfs_instance);
}
void ccp5_debugfs_destroy(void)

2
drivers/crypto/ccp/ccp-ops.c
View File

@ -1,7 +1,7 @@
/*
* AMD Cryptographic Coprocessor (CCP) driver
*
* Copyright (C) 2013,2017 Advanced Micro Devices, Inc.
* Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
*
* Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com>

37
drivers/crypto/ccp/psp-dev.c
View File

@ -1,7 +1,7 @@
/*
* AMD Platform Security Processor (PSP) interface
*
* Copyright (C) 2016-2017 Advanced Micro Devices, Inc.
* Copyright (C) 2016,2018 Advanced Micro Devices, Inc.
*
* Author: Brijesh Singh <brijesh.singh@amd.com>
*
@ -437,6 +437,7 @@ static int sev_get_api_version(void)
psp_master->api_major = status->api_major;
psp_master->api_minor = status->api_minor;
psp_master->build = status->build;
psp_master->sev_state = status->state;
return 0;
}
@ -857,15 +858,15 @@ static int sev_misc_init(struct psp_device *psp)
return 0;
}
static int sev_init(struct psp_device *psp)
static int psp_check_sev_support(struct psp_device *psp)
{
/* Check if device supports SEV feature */
if (!(ioread32(psp->io_regs + psp->vdata->feature_reg) & 1)) {
dev_dbg(psp->dev, "device does not support SEV\n");
return 1;
dev_dbg(psp->dev, "psp does not support SEV\n");
return -ENODEV;
}
return sev_misc_init(psp);
return 0;
}
int psp_dev_init(struct sp_device *sp)
@ -890,6 +891,10 @@ int psp_dev_init(struct sp_device *sp)
psp->io_regs = sp->io_map;
ret = psp_check_sev_support(psp);
if (ret)
goto e_disable;
/* Disable and clear interrupts until ready */
iowrite32(0, psp->io_regs + psp->vdata->inten_reg);
iowrite32(-1, psp->io_regs + psp->vdata->intsts_reg);
@ -901,7 +906,7 @@ int psp_dev_init(struct sp_device *sp)
goto e_err;
}
ret = sev_init(psp);
ret = sev_misc_init(psp);
if (ret)
goto e_irq;
@ -922,6 +927,11 @@ e_err:
dev_notice(dev, "psp initialization failed\n");
return ret;
e_disable:
sp->psp_data = NULL;
return ret;
}
@ -964,6 +974,21 @@ void psp_pci_init(void)
if (sev_get_api_version())
goto err;
/*
* If platform is not in UNINIT state then firmware upgrade and/or
* platform INIT command will fail. These command require UNINIT state.
*
* In a normal boot we should never run into case where the firmware
* is not in UNINIT state on boot. But in case of kexec boot, a reboot
* may not go through a typical shutdown sequence and may leave the
* firmware in INIT or WORKING state.
*/
if (psp_master->sev_state != SEV_STATE_UNINIT) {
sev_platform_shutdown(NULL);
psp_master->sev_state = SEV_STATE_UNINIT;
}
if (SEV_VERSION_GREATER_OR_EQUAL(0, 15) &&
sev_update_firmware(psp_master->dev) == 0)
sev_get_api_version();

2
drivers/crypto/ccp/psp-dev.h
View File

@ -1,7 +1,7 @@
/*
* AMD Platform Security Processor (PSP) interface driver
*
* Copyright (C) 2017 Advanced Micro Devices, Inc.
* Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
*
* Author: Brijesh Singh <brijesh.singh@amd.com>
*

2
drivers/crypto/ccp/sp-dev.c
View File

@ -1,7 +1,7 @@
/*
* AMD Secure Processor driver
*
* Copyright (C) 2017 Advanced Micro Devices, Inc.
* Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
*
* Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com>

2
drivers/crypto/ccp/sp-dev.h
View File

@ -1,7 +1,7 @@
/*
* AMD Secure Processor driver
*
* Copyright (C) 2017 Advanced Micro Devices, Inc.
* Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
*
* Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com>

Some files were not shown because too many files have changed in this diff Show More