git/pack-bitmap.c, branch v2.40.2

pack-bitmap.c: trace bitmap ignore logs when midx-bitmap is found

2022-11-29T00:54:56Z

When we find a midx bitmap, we do not bother checking for pack bitmaps, since we can use only one. But since we will warn of unused bitmaps via trace2, let's continue looking for pack bitmaps when tracing is enabled. Signed-off-by: Jeff King Signed-off-by: Teng Long Signed-off-by: Junio C Hamano

pack-bitmap.c: break out of the bitmap loop early if not tracing

2022-11-29T00:54:56Z

After opening a bitmap successfully, we try opening others only because we want to report that other bitmap files are ignored in the trace2 log. When trace2 is not enabled, we do not have to do any of that. Signed-off-by: Jeff King Signed-off-by: Teng Long Signed-off-by: Junio C Hamano

pack-bitmap.c: avoid exposing absolute paths

2022-11-14T22:21:16Z

In "open_midx_bitmap_1()" and "open_pack_bitmap_1()", when we find that there are multiple bitmaps, we will only open the first one and then leave warnings about the remaining pack information, the information will contain the absolute path of the repository, for example in a alternates usage scenario. So let's hide this kind of potentially sensitive information in this commit. Found-by: XingXin Signed-off-by: Teng Long Signed-off-by: Taylor Blau

pack-bitmap.c: remove unnecessary "open_pack_index()" calls

2022-11-14T22:21:16Z

When trying to open a pack bitmap, we call open_pack_bitmap_1() in a loop, during which it tries to open up the pack index corresponding with each available pack. It's likely that we'll end up relying on objects in that pack later in the process (in which case we're doing the work of opening the pack index optimistically), but not guaranteed. For instance, consider a repository with a large number of small packs, and one large pack with a bitmap. If we see that bitmap pack last in our loop which calls open_pack_bitmap_1(), the current code will have opened *all* pack index files in the repository. If the request can be served out of the bitmapped pack alone, then the time spent opening these idx files was wasted.S Since open_pack_bitmap_1() calls is_pack_valid() later on (which in turns calls open_pack_index() itself), we can just drop the earlier call altogether. Signed-off-by: Teng Long Signed-off-by: Taylor Blau

Merge branch 'ds/bitmap-lookup-remove-tracing'

2022-09-27T04:46:51Z

Perf-fix. * ds/bitmap-lookup-remove-tracing: pack-bitmap: remove trace2 region from hot path

pack-bitmap: remove trace2 region from hot path

2022-09-26T19:09:18Z

The trace2 region around the call to lazy_bitmap_for_commit() in bitmap_for_commit() was added in 28cd730680d (pack-bitmap: prepare to read lookup table extension, 2022-08-14). While adding trace2 regions is typically helpful for tracking performance, this method is called possibly thousands of times as a commit walk explores commit history looking for a matching bitmap. When trace2 output is enabled, this region is emitted many times and performance is throttled by that output. For now, remove these regions entirely. This is a critical path, and it would be valuable to measure that the time spent in bitmap_for_commit() does not increase when using the commit lookup table. The best way to do that would be to use a mechanism that sums the time spent in a region and reports a single value at the end of the process. This technique was introduced but not merged by [1] so maybe this example presents some justification to revisit that approach. [1] https://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/ To help with the 'git blame' output in this region, add a comment that warns against adding a trace2 region. Delete a test from t5310 that used that trace output to check that this lookup optimization was activated. To create this kind of test again in the future, the stopwatch traces mentioned earlier could be used as a signal that we activated this code path. Helpedy-by: Junio C Hamano Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

pack-bitmap: improve grammar of "xor chain" error message

2022-09-23T15:54:05Z

Signed-off-by: Alex Henrie Signed-off-by: Junio C Hamano

pack-bitmap: prepare to read lookup table extension

2022-08-26T17:13:58Z

Earlier change teaches Git to write bitmap lookup table. But Git does not know how to parse them. Teach Git to parse the existing bitmap lookup table. The older versions of Git are not affected by it. Those versions ignore the lookup table. Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty Reviewed-by: Taylor Blau Signed-off-by: Junio C Hamano

Merge branch 'tl/pack-bitmap-error-messages'

2022-07-27T16:16:52Z

Tweak various messages that come from the pack-bitmap codepaths. * tl/pack-bitmap-error-messages: pack-bitmap.c: continue looping when first MIDX bitmap is found pack-bitmap.c: using error() instead of silently returning -1 pack-bitmap.c: do not ignore error when opening a bitmap file pack-bitmap.c: rename "idx_name" to "bitmap_name" pack-bitmap.c: mark more strings for translations pack-bitmap.c: fix formatting of error messages

pack-bitmap.c: continue looping when first MIDX bitmap is found

2022-07-18T18:20:52Z

In "open_midx_bitmap()", we do a loop with the MIDX(es) in repo, when the first one has been found, then will break out by a "return" directly. But actually, it's better to continue the loop until we have visited both the MIDX in our repository, as well as any alternates (along with _their_ alternates, recursively). The reason for this is, there may exist more than one MIDX file in a repo. The "multi_pack_index" struct is actually designed as a singly linked list, and if a MIDX file has been already opened successfully, then the other MIDX files will be skipped and left with a warning "ignoring extra bitmap file." to the output. The discussion link of community: https://public-inbox.org/git/YjzCTLLDCby+kJrZ@nand.local/ Helped-by: Taylor Blau Signed-off-by: Teng Long Signed-off-by: Junio C Hamano