git/builtin/pack-objects.c, branch v2.32.2

Merge branch 'ab/pack-linkage-fix'

2021-05-27T03:36:58Z

"ld" on Solaris fails to link some test helpers, which has been worked around by reshuffling the inline function definitions from a header file to a source file that is the only user of them. * ab/pack-linkage-fix: pack-objects: move static inline from a header to the sole consumer

pack-objects: move static inline from a header to the sole consumer

2021-05-27T03:14:41Z

Move the code that is only used in builtin/pack-objects.c out of pack-objects.h. This fixes an issue where Solaris's SunCC hasn't been able to compile git since 483fa7f42d9 (t/helper/test-bitmap.c: initial commit, 2021-03-31). The real origin of that issue is that in 898eba5e630 (pack-objects: refer to delta objects by index instead of pointer, 2018-04-14) utility functions only needed by builtin/pack-objects.c were added to pack-objects.h. Since then the header has been used in a few other places, but 483fa7f42d9 was the first time it was used by test helper. Since Solaris is stricter about linking and the oe_get_size_slow() function lives in builtin/pack-objects.c the build started failing with: Undefined first referenced symbol in file oe_get_size_slow t/helper/test-bitmap.o ld: fatal: symbol referencing errors. No output written to t/helper/test-tool On other platforms this is presumably OK because the compiler and/or linker detects that the "static inline" functions that reference oe_get_size_slow() aren't used. Let's solve this by moving the relevant code from pack-objects.h to builtin/pack-objects.c. This is almost entirely a code-only move, but because of the early macro definitions in that file referencing some of these inline functions we need to move the definition of "static struct packing_data to_pack" earlier, and declare these inline functions above the macros. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

Merge branch 'jk/pack-objects-negative-options-fix'

2021-05-11T06:27:23Z

Options to "git pack-objects" that take numeric values like --window and --depth should not accept negative values; the input validation has been tightened. * jk/pack-objects-negative-options-fix: pack-objects: clamp negative depth to 0 t5316: check behavior of pack-objects --depth=0 pack-objects: clamp negative window size to 0 t5300: check that we produced expected number of deltas t5300: modernize basic tests

Merge branch 'rs/repack-without-loosening-promised-objects'

2021-05-10T07:59:47Z

"git repack -A -d" in a partial clone unnecessarily loosened objects in promisor pack. * rs/repack-without-loosening-promised-objects: repack: avoid loosening promisor objects in partial clones

Merge branch 'bc/hash-transition-interop-part-1'

2021-05-10T07:59:46Z

SHA-256 transition. * bc/hash-transition-interop-part-1: hex: print objects using the hash algorithm member hex: default to the_hash_algo on zero algorithm value builtin/pack-objects: avoid using struct object_id for pack hash commit-graph: don't store file hashes as struct object_id builtin/show-index: set the algorithm for object IDs hash: provide per-algorithm null OIDs hash: set, copy, and use algo field in struct object_id builtin/pack-redundant: avoid casting buffers to struct object_id Use the final_oid_fn to finalize hashing of object IDs hash: add a function to finalize object IDs http-push: set algorithm when reading object ID Always use oidread to read into struct object_id hash: add an algo member to struct object_id

Merge branch 'ps/rev-list-object-type-filter'

2021-05-07T03:47:41Z

"git rev-list" learns the "--filter=object:type=" option, which can be used to exclude objects of the given kind from the packfile generated by pack-objects. * ps/rev-list-object-type-filter: rev-list: allow filtering of provided items pack-bitmap: implement combined filter pack-bitmap: implement object type filter list-objects: implement object type filter list-objects: support filtering by tag and commit list-objects: move tag processing into its own function revision: mark commit parents as NOT_USER_GIVEN uploadpack.txt: document implication of `uploadpackfilter.allow`

pack-objects: clamp negative depth to 0

2021-05-03T05:30:46Z

A negative delta depth makes no sense, and the code is not prepared to handle it. If passed "--depth=-1" on the command line, then this line from break_delta_chains(): cur->depth = (total_depth--) % (depth + 1); triggers a divide-by-zero. This is undefined behavior according to the C standard, but on POSIX systems results in SIGFPE killing the process. This is certainly one way to inform the use that the command was invalid, but it's a bit friendlier to just treat it as "don't allow any deltas", which we already do for --depth=0. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

pack-objects: clamp negative window size to 0

2021-05-03T05:29:27Z

A negative window size makes no sense, and the code in find_deltas() is not prepared to handle it. If you pass "-1", for example, we end up generate a 0-length array of "struct unpacked", but our loop assumes it has at least one entry in it (and we end up reading garbage memory). We could complain to the user about this, but it's more forgiving to just clamp it to 0, which means "do not find any deltas at all". The 0-case is already tested earlier in the script, so we'll make sure this does the same thing. Reported-by: Yiyuan guo Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'jk/promisor-optim'

2021-04-30T04:50:24Z

Handling of "promisor packs" that allows certain objects to be missing and lazily retrievable has been optimized (a bit). * jk/promisor-optim: revision: avoid parsing with --exclude-promisor-objects lookup_unknown_object(): take a repository argument is_promisor_object(): free tree buffer after parsing

repack: avoid loosening promisor objects in partial clones

2021-04-28T04:36:13Z

When `git repack -A -d` is run in a partial clone, `pack-objects` is invoked twice: once to repack all promisor objects, and once to repack all non-promisor objects. The latter `pack-objects` invocation is with --exclude-promisor-objects and --unpack-unreachable, which loosens all objects unused during this invocation. Unfortunately, this includes promisor objects. Because the -d argument to `git repack` subsequently deletes all loose objects also in packs, these just-loosened promisor objects will be immediately deleted. However, this extra disk churn is unnecessary in the first place. For example, in a newly-cloned partial repo that filters all blob objects (e.g. `--filter=blob:none`), `repack` ends up unpacking all trees and commits into the filesystem because every object, in this particular case, is a promisor object. Depending on the repo size, this increases the disk usage considerably: In my copy of the linux.git, the object directory peaked 26GB of more disk usage. In order to avoid this extra disk churn, pass the names of the promisor packfiles as --keep-pack arguments to the second invocation of `pack-objects`. This informs `pack-objects` that the promisor objects are already in a safe packfile and, therefore, do not need to be loosened. For testing, we need to validate whether any object was loosened. However, the "evidence" (loosened objects) is deleted during the process which prevents us from inspecting the object directory. Instead, let's teach `pack-objects` to count loosened objects and emit via trace2 thus allowing inspecting the debug events after the process is finished. This new event is used on the added regression test. Lastly, add a new perf test to evaluate the performance impact made by this changes (tested on git.git): Test HEAD^ HEAD ---------------------------------------------------------- 5600.3: gc 134.38(41.93+90.95) 7.80(6.72+1.35) -94.2% For a bigger repository, such as linux.git, the improvement is even bigger: Test HEAD^ HEAD ------------------------------------------------------------------- 5600.3: gc 6833.00(918.07+3162.74) 268.79(227.02+39.18) -96.1% These improvements are particular big because every object in the newly-cloned partial repository is a promisor object. Reported-by: SZEDER Gábor Helped-by: Jeff King Helped-by: Jonathan Tan Signed-off-by: Rafael Silva Signed-off-by: Junio C Hamano