git/object-store.h, branch v2.37.2

Merge branch 'tb/cruft-packs'

2022-06-03T21:30:37Z

A mechanism to pack unreachable objects into a "cruft pack", instead of ejecting them into loose form to be reclaimed later, has been introduced. * tb/cruft-packs: sha1-file.c: don't freshen cruft packs builtin/gc.c: conditionally avoid pruning objects via loose builtin/repack.c: add cruft packs to MIDX during geometric repack builtin/repack.c: use named flags for existing_packs builtin/repack.c: allow configuring cruft pack generation builtin/repack.c: support generating a cruft pack builtin/pack-objects.c: --cruft with expiration reachable: report precise timestamps from objects in cruft packs reachable: add options to add_unseen_recent_objects_to_traversal builtin/pack-objects.c: --cruft without expiration builtin/pack-objects.c: return from create_object_entry() t/helper: add 'pack-mtimes' test-tool pack-mtimes: support writing pack .mtimes files chunk-format.h: extract oid_version() pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles' pack-mtimes: support reading .mtimes files Documentation/technical: add cruft-packs.txt

builtin/pack-objects.c: --cruft without expiration

2022-05-26T22:48:26Z

Teach `pack-objects` how to generate a cruft pack when no objects are dropped (i.e., `--cruft-expiration=never`). Later patches will teach `pack-objects` how to generate a cruft pack that prunes objects. When generating a cruft pack which does not prune objects, we want to collect all unreachable objects into a single pack (noting and updating their mtimes as we accumulate them). Ordinary use will pass the result of a `git repack -A` as a kept pack, so when this patch says "kept pack", readers should think "reachable objects". Generating a non-expiring cruft packs works as follows: - Callers provide a list of every pack they know about, and indicate which packs are about to be removed. - All packs which are going to be removed (we'll call these the redundant ones) are marked as kept in-core. Any packs the caller did not mention (but are known to the `pack-objects` process) are also marked as kept in-core. Packs not mentioned by the caller are assumed to be unknown to them, i.e., they entered the repository after the caller decided which packs should be kept and which should be discarded. Since we do not want to include objects in these "unknown" packs (because we don't know which of their objects are or aren't reachable), these are also marked as kept in-core. - Then, we enumerate all objects in the repository, and add them to our packing list if they do not appear in an in-core kept pack. This results in a new cruft pack which contains all known objects that aren't included in the kept packs. When the kept pack is the result of `git repack -A`, the resulting pack contains all unreachable objects. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-mtimes: support reading .mtimes files

2022-05-26T22:48:26Z

To store the individual mtimes of objects in a cruft pack, introduce a new `.mtimes` format that can optionally accompany a single pack in the repository. The format is defined in Documentation/technical/pack-format.txt, and stores a 4-byte network order timestamp for each object in name (index) order. This patch prepares for cruft packs by defining the `.mtimes` format, and introducing a basic API that callers can use to read out individual mtimes. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

object-store.h: remove unused has_sha1_file*()

2022-04-01T17:16:10Z

These macros were last used in 5d3679ee023 (sha1-file: drop has_sha1_file(), 2019-01-07), so let's remove coccinelle migration rules added 9b45f499818 (object-store: prepare has_{sha1, object}_file to handle any repo, 2018-11-13), along with the compatibility macros themselves. The "These functions.." in the diff context and the general comment about compatibility macros still applies to "NO_THE_REPOSITORY_COMPATIBILITY_MACROS" use just a few lines below this, so let's keep the comment. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

object-file API: have hash_object_file() take "enum object_type"

2022-02-26T01:16:32Z

Change the hash_object_file() function to take an "enum object_type". Since a preceding commit all of its callers are passing either "{commit,tree,blob,tag}_type", or the result of a call to type_name(), the parse_object() caller that would pass NULL is now using stream_object_signature(). Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

object API: rename hash_object_file_literally() to write_*()

2022-02-26T01:16:32Z

Before 0c3db67cc81 (hash-object --literally: fix buffer overrun with extra-long object type, 2015-05-04) the hash-object code being changed here called write_sha1_file() to both hash and write a loose object. Before that we'd use hash_sha1_file() to if "-w" wasn't provided, and otherwise call write_sha1_file(). Now we'll always call the same function for both writing. Let's rename it from hash_*_literally() to write_*_literally(). Even though the write_*() might not actually write if HASH_WRITE_OBJECT isn't in "flags", having it be more similar to write_object_file_flags() than hash_object_file(), but carrying a name that would suggest that it's a variant of the latter is confusing. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

object-file API: have write_object_file() take "enum object_type"

2022-02-26T01:16:31Z

Change the write_object_file() function to take an "enum object_type" instead of a "const char *type". Its callers either passed {commit,tree,blob,tag}_type and can pass the corresponding OBJ_* type instead, or were hardcoding strings like "blob". This avoids the back & forth fragility where the callers of write_object_file() would have the enum type, and convert it themselves via type_name(). We do have to now do that conversion ourselves before calling write_object_file_prepare(), but those codepaths will be similarly adjusted in subsequent commits. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

object-file API: add a format_object_header() function

2022-02-26T01:16:31Z

Add a convenience function to wrap the xsnprintf() command that generates loose object headers. This code was copy/pasted in various parts of the codebase, let's define it in one place and re-use it from there. All except one caller of it had a valid "enum object_type" for us, it's only write_object_file_prepare() which might need to deal with "git hash-object --literally" and a potential garbage type. Let's have the primary API use an "enum object_type", and define a *_literally() function that can take an arbitrary "const char *" for the type. See [1] for the discussion that prompted this patch, i.e. new code in object-file.c that wanted to copy/paste the xsnprintf() invocation. In the case of fast-import.c the callers unfortunately need to cast back & forth between "unsigned char *" and "char *", since format_object_header() ad encode_in_pack_object_header() take different signedness. 1. https://lore.kernel.org/git/211213.86bl1l9bfz.gmgdl@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Jiang Xin Signed-off-by: Junio C Hamano

object-file API: return "void", not "int" from hash_object_file()

2022-02-26T01:16:31Z

The hash_object_file() function added in abdc3fc8421 (Add hash_sha1_file(), 2006-10-14) did not have a meaningful return value, and it never has. One was seemingly added to avoid adding braces to the "ret = " assignments being modified here. Let's instead assign "0" to the "ret" variables at the beginning of the relevant functions, and have them return "void". Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

Merge branch 'ns/tmp-objdir'

2022-01-04T00:24:15Z

New interface into the tmp-objdir API to help in-core use of the quarantine feature. * ns/tmp-objdir: tmp-objdir: disable ref updates when replacing the primary odb tmp-objdir: new API for creating temporary writable databases