git/object-store.h, branch v2.18.2

Merge branch 'nd/pack-objects-pack-struct'

2018-05-23T05:38:19Z

"git pack-objects" needs to allocate tons of "struct object_entry" while doing its work, and shrinking its size helps the performance quite a bit. * nd/pack-objects-pack-struct: ci: exercise the whole test suite with uncommon code in pack-objects pack-objects: reorder members to shrink struct object_entry pack-objects: shrink delta_size field in struct object_entry pack-objects: shrink size field in struct object_entry pack-objects: clarify the use of object_entry::size pack-objects: don't check size when the object is bad pack-objects: shrink z_delta_size field in struct object_entry pack-objects: refer to delta objects by index instead of pointer pack-objects: move in_pack out of struct object_entry pack-objects: move in_pack_pos out of struct object_entry pack-objects: use bitfield for object_entry::depth pack-objects: use bitfield for object_entry::dfs_state pack-objects: turn type and in_pack_type to bitfields pack-objects: a bit of document about struct object_entry read-cache.c: make $GIT_TEST_SPLIT_INDEX boolean

Merge branch 'nd/repack-keep-pack'

2018-05-23T05:38:14Z

"git gc" in a large repository takes a lot of time as it considers to repack all objects into one pack by default. The command has been taught to pretend as if the largest existing packfile is marked with ".keep" so that it is left untouched while objects in other packs and loose ones are repacked. * nd/repack-keep-pack: pack-objects: show some progress when counting kept objects gc --auto: exclude base pack if not enough mem to "repack -ad" gc: handle a corner case in gc.bigPackThreshold gc: add gc.bigPackThreshold config gc: add --keep-largest-pack option repack: add --keep-pack option t7700: have closing quote of a test at the beginning of line

repack: add --keep-pack option

2018-04-16T04:52:29Z

We allow to keep existing packs by having companion .keep files. This is helpful when a pack is permanently kept. In the next patch, git-gc just wants to keep a pack temporarily, for one pack-objects run. git-gc can use --keep-pack for this use case. A note about why the pack_keep field cannot be reused and pack_keep_in_core has to be added. This is about the case when --keep-pack is specified together with either --keep-unreachable or --unpack-unreachable, but --honor-pack-keep is NOT specified. In this case, we want to exclude objects from the packs specified on command line, not from ones with .keep files. If only one bit flag is used, we have to clear pack_keep on pack files with the .keep file. But we can't make any assumption about unreachable objects in .keep packs. If "pack_keep" field is false for .keep packs, we could potentially pull lots of unreachable objects into the new pack, or unpack them loose. The safer approach is ignore all packs with either .keep file or --keep-pack. Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

pack-objects: move in_pack out of struct object_entry

2018-04-16T03:38:58Z

Instead of using 8 bytes (on 64 bit arch) to store a pointer to a pack. Use an index instead since the number of packs should be relatively small. This limits the number of packs we can handle to 1k. Since we can't be sure people can never run into the situation where they have more than 1k pack files. Provide a fall back route for it. If we find out they have too many packs, the new in_pack_by_idx[] array (which has at most 1k elements) will not be used. Instead we allocate in_pack[] array that holds nr_objects elements. This is similar to how the optional in_pack_pos field is handled. The new simple test is just to make sure the too-many-packs code path is at least executed. The true test is running make test GIT_TEST_FULL_IN_PACK_ARRAY=1 to take advantage of other special case tests. Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

replace-object: eliminate replace objects prepared flag

2018-04-12T02:38:56Z

Make the oidmap a pointer. That way we eliminate the need for the global boolean variable 'replace_object_prepared' as we can put this information into the pointer being NULL or not. Another advantage of this is that we would more quickly catch code that tries to access replace-map without initializing it. This also allows the '#include "oidmap.h"' introduced in a previous patch to be replaced by the forward declaration of 'struct oidmap;'. Keeping the type opaque discourages circumventing accessor functions; not dragging in other headers avoids some compile time overhead. One disadvantage of this is change is performance as we need to pay the overhead for a malloc. The alternative of moving the global variable into the object store is less modular code. Helped-by: René Scharfe Helped-by: Junio C Hamano Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano

replace-object: move replace_map to object store

2018-04-12T02:38:56Z

The relationship between an object X and another object Y that replaces the object X is defined only within the scope of a single repository. The exception in reachability rule around these replacement objects is also local to a repository (i.e. if traversal from refs reaches X, then both X and Y are reachable and need to be kept from gc). Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano

sha1_file: allow map_sha1_file to handle arbitrary repositories

2018-03-26T17:05:55Z

Signed-off-by: Stefan Beller Signed-off-by: Jonathan Nieder Signed-off-by: Junio C Hamano Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

sha1_file: allow sha1_file_name to handle arbitrary repositories

2018-03-26T17:05:55Z

Signed-off-by: Stefan Beller Signed-off-by: Jonathan Nieder Signed-off-by: Junio C Hamano Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

sha1_file: add repository argument to map_sha1_file

2018-03-26T17:05:55Z

Add a repository argument to allow map_sha1_file callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time. While at it, move the declaration to object-store.h, where it should be easier to find. Signed-off-by: Stefan Beller Signed-off-by: Jonathan Nieder Signed-off-by: Junio C Hamano Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

sha1_file: add repository argument to sha1_file_name

2018-03-26T17:05:55Z

Add a repository argument to allow sha1_file_name callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet. As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time. While at it, move the declaration to object-store.h, where it should be easier to find. Signed-off-by: Stefan Beller Signed-off-by: Jonathan Nieder Signed-off-by: Junio C Hamano Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano