git/pack-bitmap-write.c, branch v2.36.2

core.fsync: introduce granular fsync control infrastructure

2022-03-10T23:10:22Z

This commit introduces the infrastructure for the core.fsync configuration knob. The repository components we want to sync are identified by flags so that we can turn on or off syncing for specific components. If core.fsyncObjectFiles is set and the core.fsync configuration also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any loose objects. This picks the strictest data integrity behavior if core.fsync and core.fsyncObjectFiles are set to conflicting values. This change introduces the currently unused fsync_component helper, which will be used by a later patch that adds fsyncing to the refs backend. Actual configuration and documentation of the fsync components list are in other patches in the series to separate review of the underlying mechanism from the policy of how it's configured. Helped-by: Patrick Steinhardt Signed-off-by: Neeraj Singh Signed-off-by: Junio C Hamano

Merge branch 'ab/only-single-progress-at-once'

2022-02-25T23:47:35Z

Further tweaks on progress API. * ab/only-single-progress-at-once: pack-bitmap-write.c: don't return without stop_progress() progress API: unify stop_progress{,_msg}(), fix trace2 bug progress.c: refactor stop_progress{,_msg}() to use helpers progress.c: use dereferenced "progress" variable, not "(*p_progress)" progress.h: format and be consistent with progress.c naming progress.c tests: test some invalid usage progress.c tests: make start/stop commands on stdin progress.c test helper: add missing braces leak tests: fix a memory leak in "test-progress" helper

pack-bitmap-write.c: don't return without stop_progress()

2022-02-03T23:40:00Z

Fix a bug that's been here since 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), we did not call stop_progress() if we reached the early exit in this function. We could call stop_progress() before we return, but better yet is to defer calling start_progress() until we need it. For now this only matters in practice because we'd previously omit the "region_leave" for the progress trace2 event. Suggested-by: SZEDER Gábor Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

pack-bitmap: read multi-pack bitmaps

2021-09-01T20:56:43Z

This prepares the code in pack-bitmap to interpret the new multi-pack bitmaps described in Documentation/technical/bitmap-format.txt, which mostly involves converting bit positions to accommodate looking them up in a MIDX. Note that there are currently no writers who write multi-pack bitmaps, and that this will be implemented in the subsequent commit. Note also that get_midx_checksum() and get_midx_filename() are made non-static so they can be called from pack-bitmap.c. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-bitmap-write.c: free existing bitmaps

2021-08-24T20:21:13Z

When writing a new bitmap, the bitmap writer code attempts to read the existing bitmap (if one is present). This is done in order to quickly permute the bits of any bitmaps for commits which appear in the existing bitmap, and were also selected for the new bitmap. But since this code was added in 341fa34887 (pack-bitmap-write: use existing bitmaps, 2020-12-08), the resources associated with opening an existing bitmap were never released. It's fine to ignore this, but it's bad hygiene. It will also cause a problem for the multi-pack-index builtin, which will be responsible not only for writing bitmaps, but also for expiring any old multi-pack bitmaps. If an existing bitmap was reused here, it will also be expired. That will cause a problem on platforms which require file resources to be closed before unlinking them, like Windows. Avoid this by ensuring we close reused bitmaps with free_bitmap_index() before removing them. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-bitmap-write.c: gracefully fail to write non-closed bitmaps

2021-08-24T20:21:13Z

The set of objects covered by a bitmap must be closed under reachability, since it must be the case that there is a valid bit position assigned for every possible reachable object (otherwise the bitmaps would be incomplete). Pack bitmaps are never written from 'git repack' unless repacking all-into-one, and so we never write non-closed bitmaps (except in the case of partial clones where we aren't guaranteed to have all objects). But multi-pack bitmaps change this, since it isn't known whether the set of objects in the MIDX is closed under reachability until walking them. Plumb through a bit that is set when a reachable object isn't found. As soon as a reachable object isn't found in the set of objects to include in the bitmap, bitmap_writer_build() knows that the set is not closed, and so it now fails gracefully. A test is added in t0410 to trigger a bitmap write without full reachability closure by removing local copies of some reachable objects from a promisor remote. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

oid_pos(): access table through const pointers

2021-01-28T20:03:26Z

When we are looking up an oid in an array, we obviously don't need to write to the array. Let's mark it as const in the function interfaces, as well as in the local variables we use to derference the void pointer (note a few cases use pointers-to-pointers, so we mark everything const). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

hash_pos(): convert to oid_pos()

2021-01-28T20:02:39Z

All of our callers are actually looking up an object_id, not a bare hash. Likewise, the arrays they are looking in are actual arrays of object_id (not just raw bytes of hashes, as we might find in a pack .idx; those are handled by bsearch_hash()). Using an object_id gives us more type safety, and makes the callers slightly shorter. It also gets rid of the word "sha1" from several access functions, though we could obviously also rename those with s/sha1/hash/. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'ma/sha1-is-a-hash'

2021-01-15T23:20:29Z

Retire more names with "sha1" in it. * ma/sha1-is-a-hash: hash-lookup: rename from sha1-lookup sha1-lookup: rename `sha1_pos()` as `hash_pos()` object-file.c: rename from sha1-file.c object-name.c: rename from sha1-name.c

hash-lookup: rename from sha1-lookup

2021-01-04T21:01:55Z

Change all remnants of "sha1" in hash-lookup.c and .h and rename them to reflect that we're not just able to handle SHA-1 these days. Signed-off-by: Martin Ågren Reviewed-by: Derrick Stolee Signed-off-by: Junio C Hamano