git/bundle.c, branch v2.36.2

bundle: output hash information in 'verify'

2022-03-23T20:14:09Z

The previous change moved the 'filter' capability to the end of the 'git bundle verify' output. Now, add the 'object-format' capability to the output, when it exists. This change makes 'git bundle verify' output the hash used in all cases, even if the capability is not in the bundle. This means that v2 bundles will always output that they use "sha1". This might look noisy to some users, but it does simplify the implementation and the test strategy for this feature. Since 'verify' ends early when a prerequisite commit is missing, we need to insert this hash message carefully into our expected test output throughout t6020. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

bundle: move capabilities to end of 'verify'

2022-03-23T20:13:59Z

The 'filter' capability was added in 105c6f14a (bundle: parse filter capability, 2022-03-09), but was added in a strange place in the 'git bundle verify' output. The tests for this show output like the following: The bundle contains these 2 refs: The bundle uses this filter: blob:none The bundle records a complete history. This looks very odd if we have a thin bundle that contains boundary commits instead of a complete history: The bundle contains these 2 refs: The bundle uses this filter: blob:none The bundle requires these 2 refs: This separation between tip refs and boundary refs is unfortunate. Move the filter capability output to the end of the output. Update the documentation to match. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

Merge branch 'ds/partial-bundles'

2022-03-21T22:14:24Z

Bundle file format gets extended to allow a partial bundle, filtered by similar criteria you would give when making a partial/lazy clone. * ds/partial-bundles: clone: fail gracefully when cloning filtered bundle bundle: unbundle promisor packs bundle: create filtered bundles rev-list: move --filter parsing into revision.c bundle: parse filter capability list-objects: handle NULL function pointers MyFirstObjectWalk: update recommended usage list-objects: consolidate traverse_commit_list[_filtered] pack-bitmap: drop filter in prepare_bitmap_walk() pack-objects: use rev.filter when possible revision: put object filter into struct rev_info list-objects-filter-options: create copy helper index-pack: document and test the --promisor option

bundle: unbundle promisor packs

2022-03-09T18:25:28Z

In order to have a valid pack-file after unbundling a bundle that has the 'filter' capability, we need to generate a .promisor file. The bundle does not promise _where_ the objects can be found, but we can expect that these bundles will be unbundled in repositories with appropriate promisor remotes that can find those missing objects. Use the 'git index-pack --promisor=' option to create this .promisor file. Add "from-bundle" as the message to help anyone diagnose issues with these promisor packs. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

bundle: create filtered bundles

2022-03-09T18:25:28Z

A previous change allowed Git to parse bundles with the 'filter' capability. Now, teach Git to create bundles with this option. Some rearranging of code is required to get the option parsing in the correct spot. There are now two reasons why we might need capabilities (a new hash algorithm or an object filter) so that is pulled out into a place where we can check both at the same time. The --filter option is parsed as part of setup_revisions(), but it expected the --objects flag, too. That flag is somewhat implied by 'git bundle' because it creates a pack-file walking objects, but there is also a walk that walks the revision range expecting only commits. Make this parsing work by setting 'revs.tree_objects' and 'revs.blob_objects' before the call to setup_revisions(). Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

bundle: parse filter capability

2022-03-09T18:25:27Z

The v3 bundle format has capabilities, allowing newer versions of Git to create bundles with newer features. Older versions that do not understand these new capabilities will fail with a helpful warning. Create a new capability allowing Git to understand that the contained pack-file is filtered according to some object filter. Typically, this filter will be "blob:none" for a blobless partial clone. This change teaches Git to parse this capability, place its value in the bundle header, and demonstrate this understanding by adding a message to 'git bundle verify'. Since we will use gently_parse_list_objects_filter() outside of list-objects-filter-options.c, make it an external method and move its API documentation to before its declaration. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

list-objects: handle NULL function pointers

2022-03-09T18:25:27Z

If a caller to traverse_commit_list() specifies the options for the --objects flag but does not specify a show_object function pointer, the result is a segfault. This is currently visible by running 'git bundle create --objects HEAD'. We could fix this problem by supplying a no-op callback in builtin/bundle.c, but that only solves the problem for one builtin, leaving this segfault open for other callers. Replace all callers of the show_commit and show_object function pointers in list-objects.c to call helper functions show_commit() and show_object() which check that the given context has non-NULL functions before passing the necessary data. One extra benefit is that it reduces duplication due to passing ctx->show_data to every caller. Test that this segfault no longer occurs for 'git bundle'. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

string-list API: change "nr" and "alloc" to "size_t"

2022-03-07T20:02:04Z

Change the "nr" and "alloc" members of "struct string_list" to use "size_t" instead of "nr". On some platforms the size of an "unsigned int" will be smaller than a "size_t", e.g. a 32 bit unsigned v.s. 64 bit unsigned. As "struct string_list" is a generic API we use in a lot of places this might cause overflows. As one example: code in "refs.c" keeps track of the number of refs with a "size_t", and auxiliary code in builtin/remote.c in get_ref_states() appends those to a "struct string_list". While we're at it split the "nr" and "alloc" in string-list.h across two lines, which is the case for most such struct member declarations (e.g. in "strbuf.h" and "strvec.h"). Changing e.g. "int i" to "size_t i" in run_and_feed_hook() isn't strictly necessary, and there are a lot more cases where we'll use a local "int", "unsigned int" etc. variable derived from the "nr" in the "struct string_list". But in that case as well as add_wrapped_shortlog_msg() in builtin/shortlog.c we need to adjust the printf format referring to "nr" anyway, so let's also change the other variables referring to it. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

bundle API: change "flags" to be "extra_index_pack_args"

2021-09-07T17:59:23Z

Since the "flags" parameter was added in be042aff24c (Teach progress eye-candy to fetch_refs_from_bundle(), 2011-09-18) there's never been more than the one flag: BUNDLE_VERBOSE. Let's have the only caller who cares about that pass "-v" itself instead through new "extra_index_pack_args" parameter. The flexibility of being able to pass arbitrary arguments to "unbundle" will be used in a subsequent commit. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

bundle: remove "ref_list" in favor of string-list.c API

2021-07-06T19:10:17Z

Move away from the "struct ref_list" in bundle.c in favor of the almost identical string-list.c API. That API fits this use-case perfectly, but did not exist in its current form when this code was added in 2e0afafebd (Add git-bundle: move objects and references by archive, 2007-02-22), with hindsight we could have used the path-list API, which later got renamed to string-list. See 8fd2cb4069 (Extract helper bits from c-merge-recursive work, 2006-07-25) We need to change "name" to "string" and "oid" to "util" to make this conversion, but other than that the APIs are pretty much identical for what bundle.c made use of. Let's also replace the memset(..,0,...) pattern with a more idiomatic "INIT" macro, and finally add a *_release() function so to free the allocated memory. Before this the add_to_ref_list() would leak memory, now e.g. "bundle list-heads" reports no memory leaks at all under valgrind. In the bundle_header_init() function we're using a clever trick to memcpy() what we'd get from the corresponding BUNDLE_HEADER_INIT. There is a concurrent series to make use of that pattern more generally, see [1]. 1. https://lore.kernel.org/git/cover-0.5-00000000000-20210701T104855Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Acked-by: Jeff King Signed-off-by: Junio C Hamano