git/refs, branch v2.36.2

Revert "Merge branch 'ps/avoid-unnecessary-hook-invocation-with-packed-refs'"

2022-04-13T22:51:33Z

This reverts commit 991b4d47f0accd3955d05927d5ce434e03ffbdb6, reversing changes made to bcd020f88e1e22f38422ac3f73ab06b34ec4bef1.

Merge branch 'ab/refs-various-fixes'

2022-03-29T19:22:02Z

Code clean-up. * ab/refs-various-fixes: refs debug: add a wrapper for "read_symbolic_ref" packed-backend: remove stub BUG(...) functions misc *.c: use designated initializers for struct assignments refs: use designated initializers for "struct ref_iterator_vtable" refs: use designated initializers for "struct ref_storage_be"

Merge branch 'ps/fsync-refs'

2022-03-25T23:38:25Z

Updates to refs traditionally weren't fsync'ed, but we can configure using core.fsync variable to do so. * ps/fsync-refs: core.fsync: new option to harden references

refs debug: add a wrapper for "read_symbolic_ref"

2022-03-17T17:40:14Z

In cd475b3b038 (refs: add ability for backends to special-case reading of symbolic refs, 2022-03-01) when the "read_symbolic_ref" callback was added we'd fall back on "refs_read_raw_ref" if there wasn't any backend implementation of "read_symbolic_ref". As discussed in the preceding commit this would only happen if we were running the "debug" backend, e.g. in the "setup for ref completion" test in t9902-completion.sh with: GIT_TRACE_REFS=1 git fetch --no-tags other Let's improve the trace output, but and also eliminate the now-redundant refs_read_raw_ref() fallback case. As noted in the preceding commit the "packed" backend will never call refs_read_symbolic_ref() (nor is it ever going to). For any future backend such as reftable it's OK to ask that they either implement this (or a wrapper) themselves. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

packed-backend: remove stub BUG(...) functions

2022-03-17T17:38:05Z

Remove the stub BUG(...) functions previously used by the "struct ref_storage_be refs_be_packed" backend. We never call any functions in the packed backend by using it as a "normal" primary ref store, instead we'll always initialize a "files" backend ref-store. It will then via the "packed_ref_store" member of "struct files_ref_store" call selected functions in the "packed" backend, and we'll in addition call others via wrappers in refs.c. So while these would arguably give us *slightly* more meaningful error messages we'll NULL the missing members in the initializer anyway, so we'll reliably get a segfault if we're ever changing the backend and having it call something it doesn't have. So there's no need for this verbose boilerplate, and as shown in a subsequent commit it might even lead to some confusion about the packed backend being a "real" backend. Let's make it clear that it's not. As an aside, this also fixes a warning emitted by SunCC in at least versions 12.5 and 12.6 of Oracle Developer Studio: "refs/packed-backend.c", line 1599: warning: Function has no return statement : packed_create_symref "refs/packed-backend.c", line 1606: warning: Function has no return statement : packed_rename_ref) "refs/packed-backend.c", line 1613: warning: Function has no return statement : packed_copy_ref "refs/packed-backend.c", line 1648: warning: Function has no return statement : packed_create_reflog Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

refs: use designated initializers for "struct ref_iterator_vtable"

2022-03-17T17:36:11Z

Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

refs: use designated initializers for "struct ref_storage_be"

2022-03-17T17:36:04Z

Change the definition of the three refs backends we currently carry to use designated initializers. The "= NULL" assignments being retained here are redundant, and could be removed, but let's keep them for clarity. All of these backends define almost all fields, so we're not saving much in terms of line count by omitting these, but e.g. for "refs_be_debug" it's immediately apparent that we're omitting "init" when comparing its assignment to the others. This is a follow-up to similar work merged in bd4232fac33 (Merge branch 'ab/struct-init', 2021-07-16), a4b9fb6a5cf (Merge branch 'ab/designated-initializers-more', 2021-10-18) and a30321b9eae (Merge branch 'ab/designated-initializers' into ab/designated-initializers-more, 2021-09-27). Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

core.fsync: new option to harden references

2022-03-15T20:30:58Z

When writing both loose and packed references to disk we first create a lockfile, write the updated values into that lockfile, and on commit we rename the file into place. According to filesystem developers, this behaviour is broken because applications should always sync data to disk before doing the final rename to ensure data consistency [1][2][3]. If applications fail to do this correctly, a hard crash of the machine can easily result in corrupted on-disk data. This kind of corruption can in fact be easily observed with Git when the machine hard-resets shortly after writing references to disk. On machines with ext4, this will likely lead to the "empty files" problem: the file has been renamed, but its data has not been synced to disk. The result is that the reference is corrupt, and in the worst case this can lead to data loss. Implement a new option to harden references so that users and admins can avoid this scenario by syncing locked loose and packed references to disk before we rename them into place. [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

refs/files-backend: optimize reading of symbolic refs

2022-03-01T18:13:46Z

When reading references via `files_read_raw_ref()` we always consult both the loose reference, and if that wasn't found, we also consult the packed-refs file. While this makes sense to read a generic reference, it is wasteful in the case where we only care about symbolic references: the packed-refs backend does not support them, and thus it cannot ever return one for us. Special-case reading of symbolic references for the files backend such that we always skip asking the packed-refs backend. We use `refs_read_symbolic_ref()` extensively to determine whether we need to skip updating local symbolic references during a fetch, which is why the change results in a significant speedup when doing fetches in repositories with huge numbers of references. The following benchmark executes a mirror-fetch in a repository with about 2 million references via `git fetch --prune --no-write-fetch-head +refs/*:refs/*`: Benchmark 1: HEAD~ Time (mean ± σ): 68.372 s ± 2.344 s [User: 65.629 s, System: 8.786 s] Range (min … max): 65.745 s … 70.246 s 3 runs Benchmark 2: HEAD Time (mean ± σ): 60.259 s ± 0.343 s [User: 61.019 s, System: 7.245 s] Range (min … max): 60.003 s … 60.649 s 3 runs Summary 'HEAD' ran 1.13 ± 0.04 times faster than 'HEAD~' Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

refs: add ability for backends to special-case reading of symbolic refs

2022-03-01T18:13:46Z

Reading of symbolic and non-symbolic references is currently treated the same in reference backends: we always call `refs_read_raw_ref()` and then decide based on the returned flags what type it is. This has one downside though: symbolic references may be treated different from normal references in a backend from normal references. The packed-refs backend for example doesn't even know about symbolic references, and as a result it is pointless to even ask it for one. There are cases where we really only care about whether a reference is symbolic or not, but don't care about whether it exists at all or may be a non-symbolic reference. But it is not possible to optimize for this case right now, and as a consequence we will always first check for a loose reference to exist, and if it doesn't, we'll query the packed-refs backend for a known-to-not-be-symbolic reference. This is inefficient and requires us to search all packed references even though we know to not care for the result at all. Introduce a new function `refs_read_symbolic_ref()` which allows us to fix this case. This function will only ever return symbolic references and can thus optimize for the scenario layed out above. By default, if the backend doesn't provide an implementation for it, we just use the old code path and fall back to `read_raw_ref()`. But in case the backend provides its own, more efficient implementation, we will use that one instead. Note that this function is explicitly designed to not distinguish between missing references and non-symbolic references. If it did, we'd be forced to always search the packed-refs backend to see whether the symbolic reference the user asked for really doesn't exist, or if it exists as a non-symbolic reference. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano