git/reftable/stack.c, branch v2.44.2

Merge branch 'ps/reftable-styles'

2024-02-12T21:16:10Z

Code clean-up in various reftable code paths. * ps/reftable-styles: reftable/record: improve semantics when initializing records reftable/merged: refactor initialization of iterators reftable/merged: refactor seeking of records reftable/stack: use `size_t` to track stack length reftable/stack: use `size_t` to track stack slices during compaction reftable/stack: index segments with `size_t` reftable/stack: fix parameter validation when compacting range reftable: introduce macros to allocate arrays reftable: introduce macros to grow arrays

Merge branch 'ps/reftable-compacted-tables-permission-fix'

2024-02-06T22:31:20Z

Reftable bugfix. * ps/reftable-compacted-tables-permission-fix: reftable/stack: adjust permissions of compacted tables

Merge branch 'jc/reftable-core-fsync'

2024-02-06T22:31:20Z

The write codepath for the reftable data learned to honor core.fsync configuration. * jc/reftable-core-fsync: reftable/stack: fsync "tables.list" during compaction reftable: honor core.fsync

reftable/stack: use `size_t` to track stack length

2024-02-06T20:10:08Z

While the stack length is already stored as `size_t`, we frequently use `int`s to refer to those stacks throughout the reftable library. Convert those cases to use `size_t` instead to make things consistent. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable/stack: use `size_t` to track stack slices during compaction

2024-02-06T20:10:08Z

We use `int`s to track reftable slices when compacting the reftable stack, which is considered to be a code smell in the Git project. Convert the code to use `size_t` instead. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable/stack: index segments with `size_t`

2024-02-06T20:10:08Z

We use `int`s to index into arrays of segments and track the length of them, which is considered to be a code smell in the Git project. Convert the code to use `size_t` instead. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable/stack: fix parameter validation when compacting range

2024-02-06T20:10:08Z

The `stack_compact_range()` function receives a "first" and "last" index that indicates which tables of the reftable stack should be compacted. Naturally, "first" must be smaller than "last" in order to identify a proper range of tables to compress, which we indeed also assert in the function. But the validations happens after we have already allocated arrays with a size of `last - first + 1`, leading to an underflow and thus an invalid allocation size. Fix this by reordering the array allocations to happen after we have validated parameters. While at it, convert the array allocations to use the newly introduced macros. Note that the relevant variables pointing into arrays should also be converted to use `size_t` instead of `int`. This is left for a later commit in this series. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable: introduce macros to allocate arrays

2024-02-06T20:10:08Z

Similar to the preceding commit, let's carry over macros to allocate arrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This requires us to change the signature of `reftable_calloc()`, which only takes a single argument right now and thus puts the burden on the caller to calculate the final array's size. This is a net improvement though as it means that we can now provide proper overflow checks when multiplying the array size with the member size. Convert callsites of `reftable_calloc()` to the new signature and start using the new macros where possible. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable: introduce macros to grow arrays

2024-02-06T20:10:08Z

Throughout the reftable library we have many cases where we need to grow arrays. In order to avoid too many reallocations, we roughly double the capacity of the array on each iteration. The resulting code pattern is duplicated across many sites. We have similar patterns in our main codebase, which is why we have eventually introduced an `ALLOC_GROW()` macro to abstract it away and avoid some code duplication. We cannot easily reuse this macro here though because `ALLOC_GROW()` uses `REALLOC_ARRAY()`, which in turn will call realloc(3P) to grow the array. The reftable code is structured as a library though (even if the boundaries are fuzzy), and one property this brings with it is that it is possible to plug in your own allocators. So instead of using realloc(3P), we need to use `reftable_realloc()` that knows to use the user-provided implementation. So let's introduce two new macros `REFTABLE_REALLOC_ARRAY()` and `REFTABLE_ALLOC_GROW()` that mirror what we do in our main codebase, with two modifications: - They use `reftable_realloc()`, as explained above. - They use a different growth factor of `2 * cap + 1` instead of `(cap + 16) * 3 / 2`. The second change is because we know a bit more about the allocation patterns in the reftable library. In most cases, we end up only having a handful of items in the array and don't end up growing them. The initial capacity that our normal growth factor uses (which is 24) would thus end up over-allocating in a lot of code paths. This effect is measurable: - Before change: HEAP SUMMARY: in use at exit: 671,983 bytes in 152 blocks total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,402 bytes allocated - After change with a growth factor of `(2 * alloc + 1)`: HEAP SUMMARY: in use at exit: 671,983 bytes in 152 blocks total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,410 bytes allocated - After change with a growth factor of `(alloc + 16)* 2 / 3`: HEAP SUMMARY: in use at exit: 671,983 bytes in 152 blocks total heap usage: 3,833,673 allocs, 3,833,521 frees, 4,728,251,742 bytes allocated While the total heap usage is roughly the same, we do end up allocating significantly more bytes with our usual growth factor (in fact, roughly 21 times as many). Convert the reftable library to use these new macros. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

reftable/stack: fsync "tables.list" during compaction

2024-01-30T19:56:15Z

In 1df18a1c9a (reftable: honor core.fsync, 2024-01-23), we have added code to fsync both newly written reftables as well as "tables.list" to disk. But there are two code paths where "tables.list" is being written: - When appending a new table due to a normal ref update. - When compacting a range of tables during compaction. We have only addressed the former code path, but do not yet sync the new "tables.list" file in the latter. Fix this omission. Note that we are not yet adding any tests. These tests will be added once the "reftable" backend has been upstreamed. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano