git/object-file.c, branch v2.45.2

Merge branch 'eb/hash-transition'

2024-03-28T21:13:50Z

Work to support a repository that work with both SHA-1 and SHA-256 hash algorithms has started. * eb/hash-transition: (30 commits) t1016-compatObjectFormat: add tests to verify the conversion between objects t1006: test oid compatibility with cat-file t1006: rename sha1 to oid test-lib: compute the compatibility hash so tests may use it builtin/ls-tree: let the oid determine the output algorithm object-file: handle compat objects in check_object_signature tree-walk: init_tree_desc take an oid to get the hash algorithm builtin/cat-file: let the oid determine the output algorithm rev-parse: add an --output-object-format parameter repository: implement extensions.compatObjectFormat object-file: update object_info_extended to reencode objects object-file-convert: convert commits that embed signed tags object-file-convert: convert commit objects when writing object-file-convert: don't leak when converting tag objects object-file-convert: convert tag objects when writing object-file-convert: add a function to convert trees between algorithms object: factor out parse_mode out of fast-import and tree-walk into in object.h cache: add a function to read an OID of a specific algorithm tag: sign both hashes commit: export add_header_signature to support handling signatures on tags ...

treewide: remove unnecessary includes in source files

2023-12-26T20:04:31Z

Each of these were checked with gcc -E -I. ${SOURCE_FILE} | grep ${HEADER_FILE} to ensure that removing the direct inclusion of the header actually resulted in that header no longer being included at all (i.e. that no other header pulled it in transitively). ...except for a few cases where we verified that although the header was brought in transitively, nothing from it was directly used in that source file. These cases were: * builtin/credential-cache.c * builtin/pull.c * builtin/send-pack.c Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

object-file: handle compat objects in check_object_signature

2023-10-02T21:57:40Z

Update check_object_signature to find the hash algorithm the exising signature uses, and to use the same hash algorithm when recomputing it to check the signature is valid. This will be useful when teaching git ls-tree to display objects encoded with the compat hash algorithm. Signed-off-by: "Eric W. Biederman" Signed-off-by: Junio C Hamano

object-file: update object_info_extended to reencode objects

2023-10-02T21:57:40Z

oid_object_info_extended is updated to detect an oid encoding that does not match the current repository, use repo_oid_to_algop to find the correspoding oid in the current repository and to return the data for the oid. Signed-off-by: "Eric W. Biederman" Signed-off-by: Junio C Hamano

object-file: add a compat_oid_in parameter to write_object_file_flags

2023-10-02T21:57:39Z

To create the proper signatures for commit objects both versions of the commit object need to be generated and signed. After that it is a waste to throw away the work of generating the compatibility hash so update write_object_file_flags to take a compatibility hash input parameter that it can use to skip the work of generating the compatability hash. Update the places that don't generate the compatability hash to pass NULL so it is easy to tell write_object_file_flags should not attempt to use their compatability hash. Signed-off-by: "Eric W. Biederman" Signed-off-by: Junio C Hamano

object-file: update the loose object map when writing loose objects

2023-10-02T21:57:39Z

To implement SHA1 compatibility on SHA256 repositories the loose object map needs to be updated whenver a loose object is written. Updating the loose object map this way allows git to support the old hash algorithm in constant time. The functions write_loose_object, and stream_loose_object are the only two functions that write to the loose object store. Update stream_loose_object to compute the compatibiilty hash, update the loose object, and then call repo_add_loose_object_map to update the loose object map. Update write_object_file_flags to convert the object into it's compatibility encoding, hash the compatibility encoding, write the object, and then update the loose object map. Update force_object_loose to lookup the hash of the compatibility encoding, write the loose object, and then update the loose object map. Update write_object_file_literally to convert the object into it's compatibility hash encoding, hash the compatibility enconding, write the object, and then update the loose object map, when the type string is a known type. For objects with an unknown type this results in a partially broken repository, as the objects are not mapped. The point of write_object_file_literally is to generate a partially broken repository for testing. For testing skipping writing the loose object map is much more useful than refusing to write the broken object at all. Except that the loose objects are updated before the loose object map I have not done any analysis to see how robust this scheme is in the event of failure. Signed-off-by: "Eric W. Biederman" Signed-off-by: Junio C Hamano

bulk-checkin: only support blobs in index_bulk_checkin

2023-09-26T17:17:56Z

As the code is written today index_bulk_checkin only accepts blobs. Remove the enum object_type parameter and rename index_bulk_checkin to index_blob_bulk_checkin, index_stream to index_blob_stream, deflate_to_pack to deflate_blob_to_pack, stream_to_pack to stream_blob_to_pack, to make this explicit. Not supporting commits, tags, or trees has no downside as it is not currently supported now, and commits, tags, and trees being smaller by design do not have the problem that the problem that index_bulk_checkin was built to solve. Before we start adding code to support the hash function transition supporting additional objects types in index_bulk_checkin has no real additional cost, just an extra function parameter to know what the object type is. Once we begin the hash function transition this is not the case. The hash function transition document specifies that a repository with compatObjectFormat enabled will compute and store both the SHA-1 and SHA-256 hash of every object in the repository. What makes this a challenge is that it is not just an additional hash over the same object. Instead the hash function transition document specifies that the compatibility hash (specified with compatObjectFormat) be computed over the equivalent object that another git repository whose storage hash (specified with objectFormat) would store. When comparing equivalent repositories built with different storage hash functions, the oids embedded in objects used to refer to other objects differ and the location of signatures within objects differ. As blob objects have neither oids referring to other objects nor stored signatures their storage hash and their compatibility hash are computed over the same object. The other kinds of objects: trees, commits, and tags, all store oids referring to other objects. Signatures are stored in commit and tag objects. As oids and the tags to store signatures are not the same size in repositories built with different storage hashes the size of the equivalent objects are also different. A version of index_bulk_checkin that supports more than just blobs when computing both the SHA-1 and the SHA-256 of every object added would need a different, and more expensive structure. The structure is more expensive because it would be required to temporarily buffering the equivalent object the compatibility hash needs to be computed over. A temporary object is needed, because before a hash over an object can computed it's object header needs to be computed. One of the members of the object header is the entire size of the object. To know the size of an equivalent object an entire pass over the original object needs to be made, as trees, commits, and tags are composed of a variable number of variable sized pieces. Unfortunately there is no formula to compute the size of an equivalent object from just the size of the original object. Avoid all of those future complications by limiting index_bulk_checkin to only work on blobs. Inspired-by: brian m. carlson Signed-off-by: "Eric W. Biederman" Signed-off-by: Junio C Hamano

Merge branch 'jk/unused-parameter'

2023-07-25T19:05:24Z

Mark-up unused parameters in the code so that we can eventually enable -Wunused-parameter by default. * jk/unused-parameter: t/helper: mark unused callback void data parameters tag: mark unused parameters in each_tag_name_fn callbacks rev-parse: mark unused parameter in for_each_abbrev callback replace: mark unused parameter in each_mergetag_fn callback replace: mark unused parameter in ref callback merge-tree: mark unused parameter in traverse callback fsck: mark unused parameters in various fsck callbacks revisions: drop unused "opt" parameter in "tweak" callbacks count-objects: mark unused parameter in alternates callback am: mark unused keep_cr parameters http-push: mark unused parameter in xml callback http: mark unused parameters in curl callbacks do_for_each_ref_helper(): mark unused repository parameter test-ref-store: drop unimplemented reflog-expire command

fsck: mark unused parameters in various fsck callbacks

2023-07-14T00:24:00Z

There are a few callback functions which are used with the fsck code, but it's natural that not all callbacks need all parameters. For reporting, even something as obvious as "the oid of the object which had a problem" is not always used, as some callers are only checking a single object in the first place. And for both reporting and walking, things like void data pointers and the fsck_options aren't always necessary. But since each such parameter is used by _some_ callback, we have to keep them in the interface. Mark the unused ones in specific callbacks to avoid triggering -Wunused-parameter. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

git-compat-util: move alloc macros to git-compat-util.h

2023-07-05T18:42:31Z

alloc_nr, ALLOC_GROW, and ALLOC_GROW_BY are commonly used macros for dynamic array allocation. Moving these macros to git-compat-util.h with the other alloc macros focuses alloc.[ch] to allocation for Git objects and additionally allows us to remove inclusions to alloc.h from files that solely used the above macros. Signed-off-by: Calvin Wan Signed-off-by: Junio C Hamano