git/midx-write.c, branch v2.47.2

Merge branch 'ps/leakfixes-part-5'

2024-09-03T16:15:00Z

Even more leak fixes. * ps/leakfixes-part-5: transport: fix leaking negotiation tips transport: fix leaking arguments when fetching from bundle builtin/fetch: fix leaking transaction with `--atomic` remote: fix leaking peer ref when expanding refmap remote: fix leaks when matching refspecs remote: fix leaking config strings builtin/fetch-pack: fix leaking refs sideband: fix leaks when configuring sideband colors builtin/send-pack: fix leaking refspecs transport: fix leaking OID arrays in git:// transport data t/helper: fix leaking multi-pack-indices in "read-midx" builtin/repack: fix leaks when computing packs to repack midx-write: fix leaking hashfile on error cases builtin/archive: fix leaking `OPT_FILENAME()` value builtin/upload-archive: fix leaking args passed to `write_archive()` builtin/merge-tree: fix leaking `-X` strategy options pretty: fix leaking key/value separator buffer pretty: fix memory leaks when parsing pretty formats convert: fix leaks when resetting attributes mailinfo: fix leaking header data

Merge branch 'tb/pseudo-merge-bitmap-fixes'

2024-08-26T18:32:21Z

We created a useless pseudo-merge reachability bitmap that is about 0 commits, and attempted to include commits that are not in packs, which made no sense. These bugs have been corrected. * tb/pseudo-merge-bitmap-fixes: pseudo-merge.c: ensure pseudo-merge groups are closed pseudo-merge.c: do not generate empty pseudo-merge commits t/t5333-pseudo-merge-bitmaps.sh: demonstrate empty pseudo-merge groups pack-bitmap-write.c: select pseudo-merges even for small bitmaps pack-bitmap: drop redundant args from `bitmap_writer_finish()` pack-bitmap: drop redundant args from `bitmap_writer_build()` pack-bitmap: drop redundant args from `bitmap_writer_build_type_index()` pack-bitmap: initialize `bitmap_writer_init()` with packing_data

midx-write: fix leaking hashfile on error cases

2024-08-22T16:18:04Z

When writing the MIDX file we first create the `struct hashfile` used to write the trailer hash, and then afterwards we verify whether we can actually write the MIDX in the first place. When we decide that we can't, this leads to a memory leak because we never free the hash file contents. We could fix this by freeing the hashfile on the exit path. There is a better option though: we can simply move the checks for the error condition earlier. As there is no early exit between creating the hashfile and finalizing it anymore this is sufficient to fix the memory leak. While at it, also move around the block checking for `ctx.entries_nr`. This change is not required to fix the memory leak, but it feels natural to move together all massaging of parameters before we go with them and execute the actual logic. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

Merge branch 'tb/incremental-midx-part-1'

2024-08-19T18:07:37Z

Incremental updates of multi-pack index files. * tb/incremental-midx-part-1: midx: implement support for writing incremental MIDX chains t/t5313-pack-bounds-checks.sh: prepare for sub-directories t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' midx: implement verification support for incremental MIDXs midx: support reading incremental MIDX chains midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs midx: teach `midx_preferred_pack()` about incremental MIDXs midx: teach `midx_contains_pack()` about incremental MIDXs midx: remove unused `midx_locate_pack()` midx: teach `fill_midx_entry()` about incremental MIDXs midx: teach `nth_midxed_offset()` about incremental MIDXs midx: teach `bsearch_midx()` about incremental MIDXs midx: introduce `bsearch_one_midx()` midx: teach `nth_bitmapped_pack()` about incremental MIDXs midx: teach `nth_midxed_object_oid()` about incremental MIDXs midx: teach `prepare_midx_pack()` about incremental MIDXs midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs midx: add new fields for incremental MIDX chains Documentation: describe incremental MIDX format

pack-bitmap: drop redundant args from `bitmap_writer_finish()`

2024-08-15T18:23:15Z

In a similar fashion as the previous commit, drop a redundant argument from the `bitmap_writer_finish()` function. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-bitmap: drop redundant args from `bitmap_writer_build()`

2024-08-15T18:22:27Z

In a similar fashion as the previous commit, drop a redundant argument from the `bitmap_writer_build()` function. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-bitmap: drop redundant args from `bitmap_writer_build_type_index()`

2024-08-15T18:20:24Z

The previous commit ensures that the bitmap_writer's "to_pack" field is initialized early on, so the "to_pack" and "index_nr" arguments to `bitmap_writer_build_type_index()` are redundant. Drop them and adjust the callers accordingly. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-bitmap: initialize `bitmap_writer_init()` with packing_data

2024-08-15T18:18:04Z

In order to determine its object order, the pack-bitmap machinery keeps a 'struct packing_data' corresponding to the pack or pseudo-pack (when writing a MIDX bitmap) being written. The to_pack field is provided to the bitmap machinery by callers of bitmap_writer_build() and assigned to the bitmap_writer struct at that point. But a subsequent commit will want to have access to that data earlier on during commit selection. Prepare for that by adding a 'to_pack' argument to 'bitmap_writer_init()', and initializing the field during that function. Subsequent commits will clean up other functions which take now-redundant arguments (like nr_objects, which is equivalent to pdata->objects_nr, or pdata itself). Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

refs: add referent to each_ref_fn

2024-08-09T15:47:34Z

Add a parameter to each_ref_fn so that callers to the ref APIs that use this function as a callback can have acess to the unresolved value of a symbolic ref. Signed-off-by: John Cai Signed-off-by: Junio C Hamano

midx: implement support for writing incremental MIDX chains

2024-08-06T19:01:39Z

Now that the rest of the MIDX subsystem and relevant callers have been updated to learn about how to read and process incremental MIDX chains, let's finally update the implementation in `write_midx_internal()` to be able to write incremental MIDX chains. This new feature is available behind the `--incremental` option for the `multi-pack-index` builtin, like so: $ git multi-pack-index write --incremental The implementation for doing so is relatively straightforward, and boils down to a handful of different kinds of changes implemented in this patch: - The `compute_sorted_entries()` function is taught to reject objects which appear in any existing MIDX layer. - Functions like `write_midx_revindex()` are adjusted to write pack_order values which are offset by the number of objects in the base MIDX layer. - The end of `write_midx_internal()` is adjusted to move non-incremental MIDX files when necessary (i.e. when creating an incremental chain with an existing non-incremental MIDX in the repository). There are a handful of other changes that are introduced, like new functions to clear incremental MIDX files that are unrelated to the current chain (using the same "keep_hash" mechanism as in the non-incremental case). The tests explicitly exercising the new incremental MIDX feature are relatively limited for two reasons: 1. Most of the "interesting" behavior is already thoroughly covered in t5319-multi-pack-index.sh, which handles the core logic of reading objects through a MIDX. The new tests in t5334-incremental-multi-pack-index.sh are mostly focused on creating and destroying incremental MIDXs, as well as stitching their results together across layers. 2. A new GIT_TEST environment variable is added called "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL", which modifies the entire test suite to write incremental MIDXs after repacking when combined with the "GIT_TEST_MULTI_PACK_INDEX" variable. This exercises the long tail of other interesting behavior that is defined implicitly throughout the rest of the CI suite. It is likewise added to the linux-TEST-vars job. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano