linux/arch/arm/crypto, branch master

lib/crypto: arm/ghash: Migrate optimized code into library

2026-03-23T22:24:54Z

Remove the "ghash-neon" crypto_shash algorithm. Move the corresponding assembly code into lib/crypto/, and wire it up to the GHASH library. This makes the GHASH library be optimized on arm (though only with NEON, not PMULL; for now the goal is just parity with crypto_shash). It greatly reduces the amount of arm-specific glue code that is needed, and it fixes the issue where this optimization was disabled by default. To integrate the assembly code correctly with the library, make the following tweaks: - Change the type of 'blocks' from int to size_t. - Change the types of 'dg' and 'h' to polyval_elem. Note that this simply reflects the format that the code was already using, at least on little endian CPUs. For big endian CPUs, add byte-swaps. - Remove the 'head' argument, which is no longer needed. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260319061723.1140720-8-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: arm/ghash - Move NEON GHASH assembly into its own file

2026-03-23T21:56:32Z

arch/arm/crypto/ghash-ce-core.S implements pmull_ghash_update_p8(), which is used only by a crypto_shash implementation of GHASH. It also implements other functions, including pmull_ghash_update_p64() and others, which are used only by a crypto_aead implementation of AES-GCM. While some code is shared between pmull_ghash_update_p8() and pmull_ghash_update_p64(), it's not very much. Since pmull_ghash_update_p8() will also need to be migrated into lib/crypto/ to achieve parity in the standalone GHASH support, let's move it into a separate file ghash-neon-core.S. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260319061723.1140720-7-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: arm/ghash - Make the "ghash" crypto_shash NEON-only

2026-03-23T21:56:32Z

arch/arm/crypto/ghash-ce-glue.c originally provided only a "ghash" crypto_shash algorithm using PMULL if available, else NEON. Significantly later, it was updated to also provide a full AES-GCM implementation using PMULL. This made the PMULL support in the "ghash" crypto_shash largely obsolete. Indeed, the arm64 equivalent of this file unconditionally uses only ASIMD in its "ghash" crypto_shash. Given that inconsistency and the fact that the NEON-only code is more easily separable into the GHASH library than the PMULL based code is, let's align with arm64 and just support NEON-only for the pure GHASH. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260319061723.1140720-6-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: arm/ghash - Use new AES library API

2026-01-15T22:09:08Z

Switch from the old AES library functions (which use struct crypto_aes_ctx) to the new ones (which use struct aes_enckey). This eliminates the unnecessary computation and caching of the decryption round keys. The new AES en/decryption functions are also much faster and use AES instructions when supported by the CPU. Note that in addition to the change in the key preparation function and the key struct type itself, the change in the type of the key struct results in aes_encrypt() (which is temporarily a type-generic macro) calling the new encryption function rather than the old one. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-24-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crypto: arm/aes: Migrate optimized code into library

2026-01-12T19:39:58Z

Move the ARM optimized single-block AES en/decryption code into lib/crypto/, wire it up to the AES library API, and remove the superseded "aes-arm" crypto_cipher algorithm. The result is that both the AES library and crypto_cipher APIs are now optimized for ARM, whereas previously only crypto_cipher was (and the optimizations weren't enabled by default, which this fixes as well). Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-11-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: aes - Replace aes-generic with wrapper around lib

2026-01-12T19:39:58Z

Now that the AES library's performance has been improved, replace aes_generic.c with a new file aes.c which wraps the AES library. In preparation for making the AES library actually utilize the kernel's existing architecture-optimized AES code including AES instructions, set the driver name to "aes-lib" instead of "aes-generic". This mirrors what's been done for the hash algorithms. Update testmgr.c accordingly. Since this removes the crypto_aes_set_key() helper function, add temporary replacements for it to arch/arm/crypto/aes-cipher-glue.c and arch/arm64/crypto/aes-cipher-glue.c. This is temporary, as that code will be migrated into lib/crypto/ in later commits. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-10-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: arm/aes - Switch to aes_enc_tab[] and aes_dec_tab[]

2026-01-12T19:39:58Z

Instead of crypto_ft_tab and crypto_it_tab from aes_generic.c, use aes_enc_tab and aes_dec_tab from lib/crypto/aes.c. These contain the same data in the first 1024 bytes (which is the part that this code uses), so the result is the same. This will allow aes_generic.c to eventually be removed. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-5-ebiggers@kernel.org Signed-off-by: Eric Biggers

crypto: arm/aes-neonbs - Use AES library for single blocks

2026-01-12T19:39:58Z

aes-neonbs-glue.c calls __aes_arm_encrypt() and __aes_arm_decrypt() to en/decrypt single blocks for CBC encryption, XTS tweak encryption, and XTS ciphertext stealing. In preparation for making the AES library use this same ARM-optimized single-block AES en/decryption code and making it an internal implementation detail of the AES library, replace the calls to these functions with calls to the AES library. Note that this reduces the size of the aesbs_cbc_ctx and aesbs_xts_ctx structs, since unnecessary decryption round keys are no longer included. Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20260112192035.10427-4-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crypto: arm/nh: Migrate optimized code into library

2026-01-12T19:07:49Z

Migrate the arm32 NEON implementation of NH into lib/crypto/. This makes the nh() function be optimized on arm32 kernels. Note: this temporarily makes the adiantum template not utilize the arm32 optimized NH code. This is resolved in a later commit that converts the adiantum template to use nh() instead of "nhpoly1305". Link: https://lore.kernel.org/r/20251211011846.8179-4-ebiggers@kernel.org Signed-off-by: Eric Biggers

lib/crypto: arm/blake2b: Migrate optimized code into library

2025-10-30T05:04:24Z

Migrate the arm-optimized BLAKE2b code from arch/arm/crypto/ to lib/crypto/arm/. This makes the BLAKE2b library able to use it, and it also simplifies the code because it's easier to integrate with the library than crypto_shash. This temporarily makes the arm-optimized BLAKE2b code unavailable via crypto_shash. A later commit reimplements the blake2b-* crypto_shash algorithms on top of the BLAKE2b library API, making it available again. Note that as per the lib/crypto/ convention, the optimized code is now enabled by default. So, this also fixes the longstanding issue where the optimized BLAKE2b code was not enabled by default. To see the diff from arch/arm/crypto/blake2b-neon-glue.c to lib/crypto/arm/blake2b.h, view this commit with 'git show -M10'. Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20251018043106.375964-8-ebiggers@kernel.org Signed-off-by: Eric Biggers