git/block-sha1, branch jch

sha1: do not redefine `platform_SHA_CTX` and friends

2024-09-27T18:27:47Z

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related macros to point at the opaque "context" type, init, update, and similar functions for each specific implementation. In hash.h, we use these platform_ variables to set up the function pointers for, e.g., the_hash_algo->init_fn(), etc. But while these header files have a header-specific macro that prevents them declaring their structs / functions multiple times, they unconditionally define the platform variables, making it impossible to load multiple SHA-1 implementations at once. As a prerequisite for loading a separate SHA-1 implementation for non-cryptographic uses, only define the platform_ variables if they have not already been defined. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

Makefile + hash.h: remove PPC_SHA1 implementation

2022-08-31T21:37:31Z

Remove the PPC_SHA1 implementation added in a6ef3518f9a ([PATCH] PPC assembly implementation of SHA1, 2005-04-22). When this was added Apple consumer hardware used the PPC architecture, and the implementation was intended to improve SHA-1 speed there. Since it was added we've moved to using sha1collisiondetection by default, and anyone wanting hard-rolled non-DC SHA-1 implementation can use OpenSSL's via the OPENSSL_SHA1 knob. The PPC_SHA1 originally originally targeted 32 bit PPC, and later the 64 bit PPC 970 (a.k.a. Apple PowerPC G5). See 926172c5e48 (block-sha1: improve code on large-register-set machines, 2009-08-10) for a reference about the performance on G5 (a comment in block-sha1/sha1.c being removed here). I can't get it to do anything but segfault on both the BE and LE POWER machines in the GCC compile farm[1]. Anyone who's concerned about performance on PPC these days is likely to be using the IBM POWER processors. There have been proposals to entirely remove non-sha1collisiondetection implementations from the tree[2]. I think per [3] that would be a bit overzealous. I.e. there are various set-ups git's speed is going to be more important than the relatively implausible SHA-1 collision attack, or where such attacks are entirely mitigated by other means (e.g. by incoming objects being checked with DC_SHA1). But that really doesn't apply to PPC_SHA1 in particular, which seems to have outlived its usefulness. As this gets rid of the only in-tree *.S assembly file we can remove the small bits of logic from the Makefile needed to build objects from *.S (as opposed to *.c) The code being removed here was also throwing warnings with the "-pedantic" flag, it could have been fixed as 544d93bc3b4 (block-sha1: remove use of obsolete x86 assembly, 2022-03-10) did for block-sha1/*, but as noted above let's remove it instead. 1. https://cfarm.tetaneutral.net/machines/list/ Tested on gcc{110,112,135,203}, a mixture of POWER [789] ppc64 and ppc64le. All segfault in anything needing object hashing (e.g. t/t1007-hash-object.sh) when compiled with PPC_SHA1=Y. 2. https://lore.kernel.org/git/20200223223758.120941-1-mh@glandium.org/ 3. https://lore.kernel.org/git/20200224044732.GK1018190@coredump.intra.peff.net/ Acked-by: brian m. carlson" Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

block-sha1: remove use of obsolete x86 assembly

2022-03-10T19:18:05Z

In the block SHA-1 code, we have special assembly code for i386 and amd64 to perform rotations with assembly. This is supposed to help pick the correct rotation operation depending on which rotation is smaller, which can help some systems perform slightly better, since any circular rotation can be specified as either a rotate left or a rotate right. However, this isn't needed, so we should remove it. First, SHA-1, like SHA-2, uses fixed constant rotates. Thus, all rotation amounts are known at compile time and are in fact baked into the code. Fortunately, peephole optimizers recognize rotations specified in the normal way and automatically emit the correct code, including a preference for choosing a rotate left versus a rotate right. This has been the case for well over a decade, and is a standard example of the utility of a peephole optimizer. Moreover, all modern CPUs, with the exception of extremely limited embedded CPUs such as some Cortex-M processors, provide a barrel shifter, which lets the CPU perform rotates of any bit amount in constant time. This is valuable for many cryptographic algorithms to improve performance, and is required to prevent timing attacks in algorithms which use data-dependent rotations (which don't include the hash algorithms we use). As a result, even though the compiler does the correct optimization, it isn't even needed here and either a left or a right rotate is equally acceptable. In fact, the SHA-256 code already takes this into account and just writes the simple code using an inline function to let the compiler optimize it for us. The downside of using this code, however, is that it uses a GCC extension, which makes the compiler complain when using -pedantic unless it's prefixed with __extension__. We could fix that, but since it's not needed, let's just remove it. We haven't noticed this because almost everyone uses the SHA1DC code instead, but it still shows up for some people. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano

block-sha1: drop trailing semicolon from macro definition

2021-03-17T17:20:01Z

23119ffb4e (block-sha1: put expanded macro parameters in parentheses, 2012-07-22) added a trailing semicolon to the definition of SHA_MIX without explanation. It doesn't matter with the current code, but make sure to avoid potential surprises by removing it again. This allows the macro to be used almost like a function: Users can combine it with operators of their choice, but still must not pass an expression with side-effects as a parameter, as it would be evaluated multiple times. Signed-off-by: René Scharfe Reviewed-by: Jonathan Nieder Signed-off-by: Junio C Hamano

block-sha1: take a size_t length parameter

2020-11-16T21:41:35Z

The block-sha1 implementation takes an "unsigned long" for the length of a buffer to hash, but our hash algorithm wrappers take a size_t, as do other implementations we support like openssl or sha1dc. On many systems, including Linux, these two are equivalent, but they are not on Windows (where only a "long long" is 64 bits). As a result, passing large chunks to a single the_hash_algo->update_fn() would produce wrong answers there. Note that we don't need to update any other sizes outside of the function interface. We store the cumulative size in a "long long" (which we must do since we hash things bigger than 4GB, like packfiles, even on 32-bit platforms). And internally, we break that size_t len down into 64-byte blocks to feed into the guts of the algorithm. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

sha1: provide another level of indirection for the SHA-1 functions

2015-11-05T18:35:11Z

The git source uses git_SHA1_Update() and friends to call into the code that computes the hashes. Traditionally, we used to map these directly to underlying implementation of the SHA-1 hash (e.g. SHA1_Update() from OpenSSL or blk_SHA1_Update() from block-sha1/). This arrangement however makes it hard to tweak behaviour of the underlying implementation without fully replacing. If we want to introduce a tweaked_SHA1_Update() wrapper to implement the "Update" in a slightly different way, for example, the implementation of the wrapper still would want to call into the underlying implementation, but tweaked_SHA1_Update() cannot call git_SHA1_Update() to get to the underlying implementation (often but not always SHA1_Update()). Add another level of indirection that maps platform_SHA1_Update() and friends to their underlying implementations, and by default make git_SHA1_Update() and friends map to platform_SHA1_* functions. Doing it this way will later allow us to map git_SHA1_Update() to tweaked_SHA1_Update(), and the latter can use platform_SHA1_Update() in its implementation. Signed-off-by: Atousa Pahlevan Duprat Signed-off-by: Junio C Hamano

Merge branch 'jk/pack-bitmap'

2014-02-27T22:01:48Z

Borrow the bitmap index into packfiles from JGit to speed up enumeration of objects involved in a commit range without having to fully traverse the history. * jk/pack-bitmap: (26 commits) ewah: unconditionally ntohll ewah data ewah: support platforms that require aligned reads read-cache: use get_be32 instead of hand-rolled ntoh_l block-sha1: factor out get_be and put_be wrappers do not discard revindex when re-preparing packfiles pack-bitmap: implement optional name_hash cache t/perf: add tests for pack bitmaps t: add basic bitmap functionality tests count-objects: recognize .bitmap in garbage-checking repack: consider bitmaps when performing repacks repack: handle optional files created by pack-objects repack: turn exts array into array-of-struct repack: stop using magic number for ARRAY_SIZE(exts) pack-objects: implement bitmap writing rev-list: add bitmap mode to speed up object lists pack-objects: use bitmaps when packing objects pack-objects: split add_object_entry pack-bitmap: add support for bitmap indexes documentation: add documentation for the bitmap format ewah: compressed bitmap implementation ...

block-sha1: factor out get_be and put_be wrappers

2014-01-23T22:03:21Z

The BLK_SHA1 code has optimized wrappers for doing endian conversions on memory that may not be aligned. Let's pull them out so that we can use them elsewhere, especially the time-tested list of platforms that prefer each strategy. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

block-sha1/sha1.c: have SP around arithmetic operators

2013-10-16T17:27:27Z

Signed-off-by: Junio C Hamano

Merge branch 'jn/block-sha1'

2012-07-24T03:56:47Z

The code to load a word one-byte-at-a-time was optimized into a word-wide load instruction even when the pointer was not aligned, which caused issues on architectures that do not like unaligned access. * jn/block-sha1: Makefile: BLK_SHA1 does not require fast htonl() and unaligned loads block-sha1: put expanded macro parameters in parentheses block-sha1: avoid pointer conversion that violates alignment constraints