git/pack-objects.h, branch v2.20.2

Merge branch 'cc/delta-islands'

2018-11-21T11:39:02Z

A few issues in the implementation of "delta-islands" feature has been corrected. * cc/delta-islands: pack-objects: fix off-by-one in delta-island tree-depth computation pack-objects: zero-initialize tree_depth/layer arrays pack-objects: fix tree_depth and layer invariants

pack-objects: zero-initialize tree_depth/layer arrays

2018-11-21T04:50:27Z

Commit 108f530385 (pack-objects: move tree_depth into 'struct packing_data', 2018-08-16) started maintaining a tree_depth array that matches the "objects" array. We extend the array when: 1. The objects array is extended, in which case we use realloc to extend the tree_depth array. 2. A caller asks to store a tree_depth for object N, and this is the first such request; we create the array from scratch and store the value for N. In the latter case, though, we use regular xmalloc(), and the depth values for any objects besides N is undefined. This happens to not trigger a bug with the current code, but the reasons are quite subtle: - we never ask about the depth for any object with index i < N. This is because we store the depth immediately for all trees and blobs. So any such "i" must be a non-tree, and therefore we will never need to care about its depth (in fact, we really only care about the depth of trees). - there are no objects at this point with index i > N, because we always fill in the depth for a tree immediately after its object entry is created (we may still allocate uninitialized depth entries, but they'll be initialized by packlist_alloc() when it initializes the entry in the "objects" array). So it works, but only by chance. To be defensive, let's zero the array, which matches the "unset" values which would be handed out by oe_tree_depth() already. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

pack-objects: fix tree_depth and layer invariants

2018-11-21T04:50:16Z

Commit 108f530385 (pack-objects: move tree_depth into 'struct packing_data', 2018-08-16) dynamically manages a tree_depth array in packing_data that maintains one of these invariants: 1. tree_depth is NULL (i.e., the requested options don't require us to track tree depths) 2. tree_depth is non-NULL and has as many entries as the "objects" array We maintain (2) by: a. When the objects array grows, grow tree_depth to the same size (unless it's NULL, in which case we can leave it). b. When a caller asks to set a depth via oe_set_tree_depth(), if tree_depth is NULL we allocate it. But in (b), we use the number of stored objects, _not_ the allocated size of the objects array. So we can run into a situation like this: 1. packlist_alloc() needs to store the Nth object, so it grows the objects array to M, where M > N. 2. oe_set_tree_depth() wants to store a depth, so it allocates an array of length N. Now we've violated our invariant. 3. packlist_alloc() needs to store the N+1th object. But it _doesn't_ grow the objects array, since N <= M still holds. We try to assign to tree_depth[N+1], which is out of bounds. That doesn't happen in our test scripts, because the repositories they use are so small, but it's easy to trigger by running: echo HEAD | git pack-objects --revs --delta-islands --stdout >/dev/null in any reasonably-sized repo (like git.git). We can fix it by always growing the array to match pack->nr_alloc, not pack->nr_objects. Likewise for the "layer" array from fe0ac2fb7f (pack-objects: move 'layer' into 'struct packing_data', 2018-08-16), which has the same bug. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'nd/pthreads'

2018-11-18T09:23:52Z

The codebase has been cleaned up to reduce "#ifndef NO_PTHREADS". * nd/pthreads: Clean up pthread_create() error handling read-cache.c: initialize copy_len to shut up gcc 8 read-cache.c: reduce branching based on HAVE_THREADS read-cache.c: remove #ifdef NO_PTHREADS pack-objects: remove #ifdef NO_PTHREADS preload-index.c: remove #ifdef NO_PTHREADS grep: clean up num_threads handling grep: remove #ifdef NO_PTHREADS attr.c: remove #ifdef NO_PTHREADS name-hash.c: remove #ifdef NO_PTHREADS index-pack: remove #ifdef NO_PTHREADS send-pack.c: move async's #ifdef NO_PTHREADS back to run-command.c run-command.h: include thread-utils.h instead of pthread.h thread-utils: macros to unconditionally compile pthreads API

pack-objects: remove #ifdef NO_PTHREADS

2018-11-05T04:42:11Z

Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

Merge branch 'js/pack-objects-mutex-init-fix'

2018-10-30T06:43:43Z

A mutex used in "git pack-objects" were not correctly initialized and this caused "git repack" to dump core on Windows. * js/pack-objects-mutex-init-fix: pack-objects (mingw): initialize `packing_data` mutex in the correct spot pack-objects (mingw): demonstrate a segmentation fault with large deltas pack-objects: fix typo 'detla' -> 'delta'

pack-objects: fix typo 'detla' -> 'delta'

2018-10-19T05:28:15Z

Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

Merge branch 'cc/delta-islands'

2018-09-17T20:53:55Z

Lift code from GitHub to restrict delta computation so that an object that exists in one fork is not made into a delta against another object that does not appear in the same forked repository. * cc/delta-islands: pack-objects: move 'layer' into 'struct packing_data' pack-objects: move tree_depth into 'struct packing_data' t5320: tests for delta islands repack: add delta-islands support pack-objects: add delta-islands support pack-objects: refactor code into compute_layer_order() Add delta-islands.{c,h}

Merge branch 'jk/pack-delta-reuse-with-bitmap'

2018-09-17T20:53:53Z

When creating a thin pack, which allows objects to be made into a delta against another object that is not in the resulting pack but is known to be present on the receiving end, the code learned to take advantage of the reachability bitmap; this allows the server to send a delta against a base beyond the "boundary" commit. * jk/pack-delta-reuse-with-bitmap: pack-objects: reuse on-disk deltas for thin "have" objects pack-bitmap: save "have" bitmap from walk t/perf: add perf tests for fetches from a bitmapped server t/perf: add infrastructure for measuring sizes t/perf: factor out percent calculations t/perf: factor boilerplate out of test_perf

Merge branch 'nd/pack-deltify-regression-fix'

2018-08-22T18:17:05Z

In a recent update in 2.18 era, "git pack-objects" started producing a larger than necessary packfiles by missing opportunities to use large deltas. * nd/pack-deltify-regression-fix: pack-objects: fix performance issues on packing large deltas