git/sparse-index.c, branch v2.35.2

sparse-index: add ensure_correct_sparsity function

2021-11-25T00:32:38Z

The `ensure_correct_sparsity` function is intended to provide a means of aligning the in-core index with the sparsity required by the repository settings and other properties of the index. The function first checks whether a sparse index is allowed (per repository & sparse checkout pattern settings). If the sparse index may be used, the index is converted to sparse; otherwise, it is explicitly expanded with `ensure_full_index`. Helped-by: Junio C Hamano Signed-off-by: Victoria Dye Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-index: avoid unnecessary cache tree clearing

2021-11-25T00:32:38Z

When converting a full index to sparse, clear and recreate the cache tree only if the cache tree is not fully valid. The convert_to_sparse operation should exit silently if a cache tree update cannot be successfully completed (e.g., due to a conflicted entry state). However, because this failure scenario only occurs when at least a portion of the cache tree is invalid, we can save ourselves the cost of clearing and recreating the cache tree by skipping the check when the cache tree is fully valid. Helped-by: Derrick Stolee Co-authored-by: Junio C Hamano Signed-off-by: Victoria Dye Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

Merge branch 'ds/sparse-index-ignored-files'

2021-09-20T22:20:44Z

In cone mode, the sparse-index code path learned to remove ignored files (like build artifacts) outside the sparse cone, allowing the entire directory outside the sparse cone to be removed, which is especially useful when the sparse patterns change. * ds/sparse-index-ignored-files: sparse-checkout: clear tracked sparse dirs sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag attr: be careful about sparse directories sparse-checkout: create helper methods sparse-index: use WRITE_TREE_MISSING_OK sparse-index: silently return when cache tree fails unpack-trees: fix nested sparse-dir search sparse-index: silently return when not using cone-mode patterns t7519: rewrite sparse index test

sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag

2021-09-08T05:41:10Z

The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX environment variable or the "index.sparse" config setting before converting the index to a sparse one. This is for ease of use since all current consumers are preparing to compress the index before writing it to disk. If these settings are not enabled, then convert_to_sparse() silently returns without doing anything. We will add a consumer in the next change that wants to use the sparse index as an in-memory data structure, regardless of whether the on-disk format should be sparse. To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip these config checks when enabled. All current consumers are modified to pass '0' in the new 'flags' parameter. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-checkout: create helper methods

2021-09-08T05:41:10Z

As we integrate the sparse index into more builtins, we occasionally need to check the sparse-checkout patterns to see if a path is within the sparse-checkout cone. Create some helper methods that help initialize the patterns and check for pattern matching to make this easier. The existing callers of commands like get_sparse_checkout_patterns() use a custom 'struct pattern_list' that is not necessarily the one in the 'struct index_state', so there are not many previous uses that could adopt these helpers. There are just two in builtin/add.c and sparse-index.c that can use path_in_sparse_checkout(). We add a path_in_cone_mode_sparse_checkout() as well that will only return false if the path is outside of the sparse-checkout definition _and_ the sparse-checkout patterns are in cone mode. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-index: use WRITE_TREE_MISSING_OK

2021-09-08T05:41:09Z

When updating the cache tree in convert_to_sparse(), the WRITE_TREE_MISSING_OK flag indicates that trees might be computed that do not already exist within the object database. This happens in cases such as 'git add' creating new trees that it wants to store in anticipation of a following 'git commit'. If this flag is not specified, then it might trigger a promisor fetch or a failure due to the object not existing locally. Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these possible reasons for the cache_tree_update() to fail. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-index: silently return when cache tree fails

2021-09-08T05:41:09Z

If cache_tree_update() returns a non-zero value, then it could not create the cache tree. This is likely due to a path having a merge conflict. Since we are already returning early, let's return silently to avoid making it seem like we failed to write the index at all. If we remove our dependence on the cache tree within convert_to_sparse(), then we could still recover from this scenario and have a sparse index. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-index: silently return when not using cone-mode patterns

2021-09-08T05:41:09Z

While the sparse-index is only enabled when core.sparseCheckoutCone is also enabled, it is possible for the user to modify the sparse-checkout file manually in a way that does not match cone-mode patterns. In this case, we should refuse to convert an index into a sparse index, since the sparse_checkout_patterns will not be initialized with recursive and parent path hashsets. Also silently return if there are no cache entries, which is a simple case: there are no paths to make sparse! Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

sparse-index: copy dir_hash in ensure_full_index()

2021-08-30T16:24:12Z

Copy the 'index_state->dir_hash' back to the real istate after expanding a sparse index. A crash was observed in 'git status' during some hashmap lookups with corrupted hashmap entries. During an index expansion, new cache-entries are added to the 'index_state->name_hash' and the 'dir_hash' in a temporary 'index_state' variable 'full'. However, only the 'name_hash' hashmap from this temp variable was copied back into the real 'istate' variable. The original copy of the 'dir_hash' was incorrectly preserved. If the table in the 'full->dir_hash' hashmap were realloced, the stale version (in 'istate') would be corrupted. The test suite does not operate on index sizes sufficiently large to trigger this reallocation, so they do not cover this behavior. Increasing the test suite to cover such scale is fragile and likely wasteful. Signed-off-by: Jeff Hostetler Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

sparse-index: recompute cache-tree

2021-07-14T22:05:53Z

When some commands run with command_requires_full_index=1, then the index can get in a state where the in-memory cache tree is actually equal to the sparse index's cache tree instead of the full one. This results in incorrect entry_count values. By clearing the cache tree before converting to sparse, we avoid this issue. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano