git/Documentation/git-repack.txt, branch v2.48.2

Merge branch 'ps/git-repack-doc-fixes'

2023-10-29T22:09:57Z

Doc updates. * ps/git-repack-doc-fixes: doc/git-repack: don't mention nonexistent "--unpacked" option doc/git-repack: fix syntax for `-g` shorthand option

Merge branch 'en/docfixes'

2023-10-23T20:56:37Z

Documentation typo and grammo fixes. * en/docfixes: (25 commits) documentation: add missing parenthesis documentation: add missing quotes documentation: add missing fullstops documentation: add some commas where they are helpful documentation: fix whitespace issues documentation: fix capitalization documentation: fix punctuation documentation: use clearer prepositions documentation: add missing hyphens documentation: remove unnecessary hyphens documentation: add missing article documentation: fix choice of article documentation: whitespace is already generally plural documentation: fix singular vs. plural documentation: fix verb vs. noun documentation: fix adjective vs. noun documentation: fix verb tense documentation: employ consistent verb tense for a list documentation: fix subject/verb agreement documentation: remove extraneous words ...

Merge branch 'tb/repack-max-cruft-size'

2023-10-18T20:25:41Z

"git repack" learned "--max-cruft-size" to prevent cruft packs from growing without bounds. * tb/repack-max-cruft-size: repack: free existing_cruft array after use builtin/repack.c: avoid making cruft packs preferred builtin/repack.c: implement support for `--max-cruft-size` builtin/repack.c: parse `--max-pack-size` with OPT_MAGNITUDE t7700: split cruft-related tests to t7704

doc/git-repack: don't mention nonexistent "--unpacked" option

2023-10-16T21:21:59Z

The documentation for geometric repacking mentions a "--unpacked" option that supposedly changes how loose objects are rolled up. This option has never existed, and the implied behaviour, namely to include all unpacked objects into the resulting packfile, is in fact the default behaviour. Correct the documentation to not mention this option. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

doc/git-repack: fix syntax for `-g` shorthand option

2023-10-16T21:21:59Z

The `-g` switch is a shorthand for `--geometric=` and allows the user to specify the geometric. The documentation is wrong though and indicates that the syntax for the shorthand is `-g=`. In fact though, the option must be specified without the equals sign via `-g`. Fix the syntax accordingly. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

documentation: fix singular vs. plural

2023-10-09T19:06:29Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

documentation: wording improvements

2023-10-09T19:04:21Z

Diff best viewed with --color-diff. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

builtin/repack.c: implement support for `--max-cruft-size`

2023-10-05T20:26:11Z

Cruft packs are an alternative mechanism for storing a collection of unreachable objects whose mtimes are recent enough to avoid being pruned out of the repository. When cruft packs were first introduced back in b757353676 (builtin/pack-objects.c: --cruft without expiration, 2022-05-20) and a7d493833f (builtin/pack-objects.c: --cruft with expiration, 2022-05-20), the recommended workflow consisted of: - Repacking periodically, either by packing anything loose in the repository (via `git repack -d`) or producing a geometric sequence of packs (via `git repack --geometric= -d`). - Every so often, splitting the repository into two packs, one cruft to store the unreachable objects, and another non-cruft pack to store the reachable objects. Repositories may (out of band with the above) choose periodically to prune out some unreachable objects which have aged out of the grace period by generating a pack with `--cruft-expiration=`. This allowed repositories to maintain relatively few packs on average, and quarantine unreachable objects together in a cruft pack, avoiding the pitfalls of holding unreachable objects as loose while they age out (for more, see some of the details in 3d89a8c118 (Documentation/technical: add cruft-packs.txt, 2022-05-20)). This all works, but can be costly from an I/O-perspective when frequently repacking a repository that has many unreachable objects. This problem is exacerbated when those unreachable objects are rarely (if every) pruned. Since there is at most one cruft pack in the above scheme, each time we update the cruft pack it must be rewritten from scratch. Because much of the pack is reused, this is a relatively inexpensive operation from a CPU-perspective, but is very costly in terms of I/O since we end up rewriting basically the same pack (plus any new unreachable objects that have entered the repository since the last time a cruft pack was generated). At the time, we decided against implementing more robust support for multiple cruft packs. This patch implements that support which we were lacking. Introduce a new option `--max-cruft-size` which allows repositories to accumulate cruft packs up to a given size, after which point a new generation of cruft packs can accumulate until it reaches the maximum size, and so on. To generate a new cruft pack, the process works like so: - Sort a list of any existing cruft packs in ascending order of pack size. - Starting from the beginning of the list, group cruft packs together while the accumulated size is smaller than the maximum specified pack size. - Combine the objects in these cruft packs together into a new cruft pack, along with any other unreachable objects which have since entered the repository. Once a cruft pack grows beyond the size specified via `--max-cruft-size` the pack is effectively frozen. This limits the I/O churn up to a quadratic function of the value specified by the `--max-cruft-size` option, instead of behaving quadratically in the number of total unreachable objects. When pruning unreachable objects, we bypass the new code paths which combine small cruft packs together, and instead start from scratch, passing in the appropriate `--max-pack-size` down to `pack-objects`, putting it in charge of keeping the resulting set of cruft packs sized correctly. This may seem like further I/O churn, but in practice it isn't so bad. We could prune old cruft packs for whom all or most objects are removed, and then generate a new cruft pack with just the remaining set of objects. But this additional complexity buys us relatively little, because most objects end up being pruned anyway, so the I/O churn is well contained. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

repack: implement `--filter-to` for storing filtered out objects

2023-10-02T21:54:31Z

A previous commit has implemented `git repack --filter=` to allow users to filter out some objects from the main pack and move them into a new different pack. It would be nice if this new different pack could be created in a different directory than the regular pack. This would make it possible to move large blobs into a pack on a different kind of storage, for example cheaper storage. Even in a different directory, this pack can be accessible if, for example, the Git alternates mechanism is used to point to it. In fact not using the Git alternates mechanism can corrupt a repo as the generated pack containing the filtered objects might not be accessible from the repo any more. So setting up the Git alternates mechanism should be done before using this feature if the user wants the repo to be fully usable while this feature is used. In some cases, like when a repo has just been cloned or when there is no other activity in the repo, it's Ok to setup the Git alternates mechanism afterwards though. It's also Ok to just inspect the generated packfile containing the filtered objects and then just move it into the '.git/objects/pack/' directory manually. That's why it's not necessary for this command to check that the Git alternates mechanism has been already setup. While at it, as an example to show that `--filter` and `--filter-to` work well with other options, let's also add a test to check that these options work well with `--max-pack-size`. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano

repack: add `--filter=` option

2023-10-02T21:54:30Z

This new option puts the objects specified by `` into a separate packfile. This could be useful if, for example, some blobs take up a lot of precious space on fast storage while they are rarely accessed. It could make sense to move them into a separate cheaper, though slower, storage. It's possible to find which new packfile contains the filtered out objects using one of the following: - `git verify-pack -v ...`, - `test-tool find-pack ...`, which a previous commit added, - `--filter-to=`, which a following commit will add to specify where the pack containing the filtered out objects will be. This feature is implemented by running `git pack-objects` twice in a row. The first command is run with `--filter=`, using the specified filter. It packs objects while omitting the objects specified by the filter. Then another `git pack-objects` command is launched using `--stdin-packs`. We pass it all the previously existing packs into its stdin, so that it will pack all the objects in the previously existing packs. But we also pass into its stdin, the pack created by the previous `git pack-objects --filter=` command as well as the kept packs, all prefixed with '^', so that the objects in these packs will be omitted from the resulting pack. The result is that only the objects filtered out by the first `git pack-objects` command are in the pack resulting from the second `git pack-objects` command. As the interactions with kept packs are a bit tricky, a few related tests are added. Helped-by: Taylor Blau Signed-off-by: John Cai Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano