git/Documentation/gitformat-pack.txt, branch v2.45.2

midx: implement `BTMP` chunk

2023-12-14T22:38:07Z

When a multi-pack bitmap is used to implement verbatim pack reuse (that is, when verbatim chunks from an on-disk packfile are copied directly[^1]), it does so by using its "preferred pack" as the source for pack-reuse. This allows repositories to pack the majority of their objects into a single (often large) pack, and then use it as the single source for verbatim pack reuse. This increases the amount of objects that are reused verbatim (and consequently, decrease the amount of time it takes to generate many packs). But this performance comes at a cost, which is that the preferred packfile must pace its growth with that of the entire repository in order to maintain the utility of verbatim pack reuse. As repositories grow beyond what we can reasonably store in a single packfile, the utility of verbatim pack reuse diminishes. Or, at the very least, it becomes increasingly more expensive to maintain as the pack grows larger and larger. It would be beneficial to be able to perform this same optimization over multiple packs, provided some modest constraints (most importantly, that the set of packs eligible for verbatim reuse are disjoint with respect to the subset of their objects being sent). If we assume that the packs which we treat as candidates for verbatim reuse are disjoint with respect to any of their objects we may output, we need to make only modest modifications to the verbatim pack-reuse code itself. Most notably, we need to remove the assumption that the bits in the reachability bitmap corresponding to objects from the single reuse pack begin at the first bit position. Future patches will unwind these assumptions and reimplement their existing functionality as special cases of the more general assumptions (e.g. that reuse bits can start anywhere within the bitset, but happen to start at 0 for all existing cases). This patch does not yet relax any of those assumptions. Instead, it implements a foundational data-structure, the "Bitampped Packs" (`BTMP`) chunk of the multi-pack index. The `BTMP` chunk's contents are described in detail here. Importantly, the `BTMP` chunk contains information to map regions of a multi-pack index's reachability bitmap to the packs whose objects they represent. For now, this chunk is only written, not read (outside of the test-tool used in this patch to test the new chunk's behavior). Future patches will begin to make use of this new chunk. [^1]: Modulo patching any `OFS_DELTA`'s that cross over a region of the pack that wasn't used verbatim. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

Merge branch 'tb/format-pack-doc-update'

2023-11-08T02:04:00Z

Doc update. * tb/format-pack-doc-update: Documentation/gitformat-pack.txt: fix incorrect MIDX documentation Documentation/gitformat-pack.txt: fix typo

Documentation/gitformat-pack.txt: fix incorrect MIDX documentation

2023-11-01T04:25:04Z

Back in 32f3c541e3 (multi-pack-index: write pack names in chunk, 2018-07-12) the MIDX's "Packfile Names" (or "PNAM", for short) chunk was described as containing an array of string entries. e0d1bcf825 notes that this is the only chunk in the MIDX format's specification that is not guaranteed to be 4-byte aligned, and so should be placed last. This isn't quite accurate: the entries within the PNAM chunk are not guaranteed to be 4-byte aligned since they are arbitrary strings, but the chunk itself is 4-byte aligned since the ending is padded with NUL bytes. That padding has always been there since 32f3c541e3 via midx.c::write_midx_pack_names(), which ended with: i = MIDX_CHUNK_ALIGNMENT - (written % MIDX_CHUNK_ALIGNMENT) if (i < MIDX_CHUNK_ALIGNMENT) { unsigned char padding[MIDX_CHUNK_ALIGNMENT]; memset(padding, 0, sizeof(padding)) hashwrite(f, padding, i); written += i; } In fact, 32f3c541e3's log message itself describes the chunk in its first paragraph with: Since filenames are not well structured, add padding to keep good alignment in later chunks. So these have always been externally aligned. Correct the corresponding part of our documentation to reflect that. Helped-by: Junio C Hamano Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

Documentation/gitformat-pack.txt: fix typo

2023-11-01T04:25:02Z

e0d1bcf825 (multi-pack-index: add format details, 2018-07-12) describes the MIDX's "PNAM" chunk as having entries which are "null-terminated strings". This is a typo, as strings are terminated with a NUL character, which is a distinct concept from "NULL" or "null", which we typically reserve for the void pointer to address 0. Correct the documentation accordingly. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

documentation: add some commas where they are helpful

2023-10-09T19:06:44Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

documentation: add missing article

2023-10-09T19:06:29Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

documentation: fix verb tense

2023-10-09T19:06:29Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

documentation: fix typos

2023-10-09T19:06:24Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

Documentation/gitformat-pack.txt: drop mixed version section

2023-08-29T18:58:26Z

This section was added in 3d89a8c118 (Documentation/technical: add cruft-packs.txt, 2022-05-20) to highlight a potential pitfall when deploying cruft packs in an environment where multiple versions of Git are GC-ing the same repository. Now that it has been more than a year since 3d89a8c118 was written, let's drop this section as it is no longer relevant. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

Documentation/gitformat-pack.txt: remove multi-cruft packs alternative

2023-08-29T18:58:26Z

This text, originally from 3d89a8c118 (Documentation/technical: add cruft-packs.txt, 2022-05-20) lists multiple cruft packs as a potential alternative to the design of cruft packs. We have always supported multiple cruft packs (i.e. we use the most recent mtime for a given object among all cruft packs which contain it, etc.), but haven't encouraged its use. We still aren't encouraging users to go out and generate multiple cruft packs, but let's take a step in that direction by dropping language that suggests we aren't capable of working with multiple cruft packs. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano