git/Documentation/git-pack-objects.txt, branch v2.22.2

list-objects: consume sparse tree walk

2019-01-17T21:44:39Z

When creating a pack-file using 'git pack-objects --revs' we provide a list of interesting and uninteresting commits. For example, a push operation would make the local topic branch be interesting and the known remote refs as uninteresting. We want to discover the set of new objects to send to the server as a thin pack. We walk these commits until we discover a frontier of commits such that every commit walk starting at interesting commits ends in a root commit or unintersting commit. We then need to discover which non-commit objects are reachable from uninteresting commits. This commit walk is not changing during this series. The mark_edges_uninteresting() method in list-objects.c iterates on the commit list and does the following: * If the commit is UNINTERSTING, then mark its root tree and every object it can reach as UNINTERESTING. * If the commit is interesting, then mark the root tree of every UNINTERSTING parent (and all objects that tree can reach) as UNINTERSTING. At the very end, we repeat the process on every commit directly given to the revision walk from stdin. This helps ensure we properly cover shallow commits that otherwise were not included in the frontier. The logic to recursively follow trees is in the mark_tree_uninteresting() method in revision.c. The algorithm avoids duplicate work by not recursing into trees that are already marked UNINTERSTING. Add a new 'sparse' option to the mark_edges_uninteresting() method that performs this logic in a slightly different way. As we iterate over the commits, we add all of the root trees to an oidset. Then, call mark_trees_uninteresting_sparse() on that oidset. Note that we include interesting trees in this process. The current implementation of mark_trees_unintersting_sparse() will walk the same trees as the old logic, but this will be replaced in a later change. Add a '--sparse' flag in 'git pack-objects' to call this new logic. Add a new test script t/t5322-pack-objects-sparse.sh that tests this option. The tests currently demonstrate that the resulting object list is the same as the old algorithm. This includes a case where both algorithms pack an object that is not needed by a remote due to limits on the explored set of trees. When the sparse algorithm is changed in a later commit, we will add a test that demonstrates a change of behavior in some cases. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

pack-objects: add delta-islands support

2018-08-16T17:51:17Z

Implement support for delta islands in git pack-objects and document how delta islands work in "Documentation/git-pack-objects.txt" and Documentation/config.txt. This allows users to setup delta islands in their config and get the benefit of less disk usage while cloning and fetching is still quite fast and not much more CPU intensive. Signed-off-by: Jeff King Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano

Merge branch 'nd/pack-unreachable-objects-doc'

2018-05-23T05:38:24Z

Doc update. * nd/pack-unreachable-objects-doc: pack-objects: validation and documentation about unreachable options

Merge branch 'nd/pack-objects-pack-struct'

2018-05-23T05:38:19Z

"git pack-objects" needs to allocate tons of "struct object_entry" while doing its work, and shrinking its size helps the performance quite a bit. * nd/pack-objects-pack-struct: ci: exercise the whole test suite with uncommon code in pack-objects pack-objects: reorder members to shrink struct object_entry pack-objects: shrink delta_size field in struct object_entry pack-objects: shrink size field in struct object_entry pack-objects: clarify the use of object_entry::size pack-objects: don't check size when the object is bad pack-objects: shrink z_delta_size field in struct object_entry pack-objects: refer to delta objects by index instead of pointer pack-objects: move in_pack out of struct object_entry pack-objects: move in_pack_pos out of struct object_entry pack-objects: use bitfield for object_entry::depth pack-objects: use bitfield for object_entry::dfs_state pack-objects: turn type and in_pack_type to bitfields pack-objects: a bit of document about struct object_entry read-cache.c: make $GIT_TEST_SPLIT_INDEX boolean

pack-objects: validation and documentation about unreachable options

2018-05-06T09:49:32Z

These options are added in [1] [2] [3]. All these depend on running rev-list internally which is normally true since they are always used with "--all --objects" which implies --revs. But let's keep this dependency explicit. While at there, add documentation for them. These are mostly used internally by git-repack. But it's still good to not chase down the right commit message to know how they work. [1] ca11b212eb (let pack-objects do the writing of unreachable objects as loose objects - 2008-05-14) [2] 08cdfb1337 (pack-objects --keep-unreachable - 2007-09-16) [3] e26a8c4721 (repack: extend --keep-unreachable to loose objects - 2016-06-13) Signed-off-by: Nguyễn Thái Ngọc Duy Reviewed-by: Jeff King Signed-off-by: Junio C Hamano

repack: add --keep-pack option

2018-04-16T04:52:29Z

We allow to keep existing packs by having companion .keep files. This is helpful when a pack is permanently kept. In the next patch, git-gc just wants to keep a pack temporarily, for one pack-objects run. git-gc can use --keep-pack for this use case. A note about why the pack_keep field cannot be reused and pack_keep_in_core has to be added. This is about the case when --keep-pack is specified together with either --keep-unreachable or --unpack-unreachable, but --honor-pack-keep is NOT specified. In this case, we want to exclude objects from the packs specified on command line, not from ones with .keep files. If only one bit flag is used, we have to clear pack_keep on pack files with the .keep file. But we can't make any assumption about unreachable objects in .keep packs. If "pack_keep" field is false for .keep packs, we could potentially pull lots of unreachable objects into the new pack, or unpack them loose. The safer approach is ignore all packs with either .keep file or --keep-pack. Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

pack-objects: use bitfield for object_entry::depth

2018-04-16T03:38:58Z

Because of struct packing from now on we can only handle max depth 4095 (or even lower when new booleans are added in this struct). This should be ok since long delta chain will cause significant slow down anyway. Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

gc: do not repack promisor packfiles

2017-12-08T17:52:42Z

Teach gc to stop traversal at promisor objects, and to leave promisor packfiles alone. This has the effect of only repacking non-promisor packfiles, and preserves the distinction between promisor packfiles and non-promisor packfiles. Signed-off-by: Jonathan Tan Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano

list-objects-filter-options: support --no-filter

2017-12-05T17:44:36Z

Teach opt_parse_list_objects_filter() to take --no-filter option and to free the contents of struct filter_options. This command line argument will be automatically inherited by commands using OPT_PARSE_LIST_OBJECTS_FILTER(); this includes pack-objects. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano

pack-objects: add list-objects filtering

2017-11-22T05:11:57Z

Teach pack-objects to use the filtering provided by the traverse_commit_list_filtered() interface to omit unwanted objects from the resulting packfile. Filtering requires the use of the "--stdout" option. Add t5317 test. In the future, we will introduce a "partial clone" mechanism wherein an object in a repo, obtained from a remote, may reference a missing object that can be dynamically fetched from that remote once needed. This "partial clone" mechanism will have a way, sometimes slow, of determining if a missing link is one of the links expected to be produced by this mechanism. This patch introduces handling of missing objects to help debugging and development of the "partial clone" mechanism, and once the mechanism is implemented, for a power user to perform operations that are missing-object aware without incurring the cost of checking if a missing link is expected. Signed-off-by: Jeff Hostetler Reviewed-by: Jonathan Tan Signed-off-by: Junio C Hamano