git/split-index.c, branch v2.32.2

Always use oidread to read into struct object_id

2021-04-27T07:31:38Z

In the future, we'll want oidread to automatically set the hash algorithm member for an object ID we read into it, so ensure we use oidread instead of hashcpy everywhere we're copying a hash value into a struct object_id. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano

use CALLOC_ARRAY

2021-03-14T00:00:09Z

Add and apply a semantic patch for converting code that open-codes CALLOC_ARRAY to use it instead. It shortens the code and infers the element size automatically. Signed-off-by: René Scharfe Signed-off-by: Junio C Hamano

mem-pool: use more standard initialization and finalization

2020-08-18T19:16:06Z

A typical memory type, such as strbuf, hashmap, or string_list can be stored on the stack or embedded within another structure. mem_pool cannot be, because of how mem_pool_init() and mem_pool_discard() are written. mem_pool_init() does essentially the following (simplified for purposes of explanation here): void mem_pool_init(struct mem_pool **pool...) { *pool = xcalloc(1, sizeof(*pool)); It seems weird to require that mem_pools can only be accessed through a pointer. It also seems slightly dangerous: unlike strbuf_release() or strbuf_reset() or string_list_clear(), all of which put the data structure into a state where it can be re-used after the call, mem_pool_discard(pool) will leave pool pointing at free'd memory. read-cache (and split-index) are the only current users of mem_pools, and they haven't fallen into a use-after-free mistake here, but it seems likely to be problematic for future users especially since several of the current callers of mem_pool_init() will only call it when the mem_pool* is not already allocated (i.e. is NULL). This type of mechanism also prevents finding synchronization points where one can free existing memory and then resume more operations. It would be natural at such points to run something like mem_pool_discard(pool...); and, if necessary, mem_pool_init(&pool...); and then carry on continuing to use the pool. However, this fails badly if several objects had a copy of the value of pool from before these commands; in such a case, those objects won't get the updated value of pool that mem_pool_init() overwrites pool with and they'll all instead be reading and writing from free'd memory. Modify mem_pool_init()/mem_pool_discard() to behave more like strbuf_init()/strbuf_release() or string_list_init()/string_list_clear() In particular: (1) make mem_pool_init() just take a mem_pool* and have it only worry about allocating struct mp_blocks, not the struct mem_pool itself, (2) make mem_pool_discard() free the memory that the pool was responsible for, but leave it in a state where it can be used to allocate more memory afterward (without the need to call mem_pool_init() again). Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

Merge branch 'nd/split-index-null-base-fix'

2019-03-07T00:59:56Z

Split-index fix. * nd/split-index-null-base-fix: read-cache.c: fix writing "link" index ext with null base oid

read-cache.c: fix writing "link" index ext with null base oid

2019-02-13T20:52:48Z

Since commit 7db118303a (unpack_trees: fix breakage when o->src_index != o->dst_index - 2018-04-23) and changes in merge code to use separate index_state for source and destination, when doing a merge with split index activated, we may run into this line in unpack_trees(): o->result.split_index = init_split_index(&o->result); This is by itself not wrong. But this split index information is not fully populated (and it's only so when move_cache_to_base_index() is called, aka force splitting the index, or loading index_state from a file). Both "base_oid" and "base" in this case remain null. So when writing the main index down, we link to this index with null oid (default value after init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh: warning: could not freshen shared index '.../sharedindex.0{40}' This patch makes sure we will not refresh null base_oid (because the file is never there). It also makes sure not to write "link" extension with null base_oid in the first place (no point having it at all). Read code already has protection against null base_oid. There is also another side fix in remove_split_index() that causes a crash when doing "git update-index --no-split-index" when base_oid in the index file is null. In this case we will not load istate->split_index->base but we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code. Reported-by: Luke Diamand Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

read-cache: add invalidate parameter to remove_marked_cache_entries

2019-01-02T23:28:05Z

When marking cache entries for removal, and later removing them all at once using 'remove_marked_cache_entries()', cache entries currently have to be invalidated manually in the cache tree and in the untracked cache. Add an invalidate flag to the function. With the flag set, the function will take care of invalidating the path in the cache tree and in the untracked cache. Note that the current callsites already do the invalidation properly in other places, so we're just passing 0 from there to keep the status quo. This will be useful in a subsequent commit. Signed-off-by: Thomas Gummerer Signed-off-by: Junio C Hamano

split-index: BUG() when cache entry refers to non-existing shared entry

2018-10-11T22:23:29Z

When the split index feature is in use, then a cache entry is: - either only present in the split index, in which case its 'index' field must be 0, - or it should refer to an existing entry in the shared index, i.e. the 'index' field can't be greater than the size of the shared index. If a cache entry were to refer to a non-existing entry in the shared index, then that's a sign of something being wrong in the index state, either as a result of a bug in dealing with the split/shared index entries, or perhaps a (potentially unrelated) memory corruption issue. prepare_to_write_split_index() already has a condition to catch cache entries with such bogus 'index' field, but instead of calling BUG() it just sets cache entry's 'index = 0', and the entry will then be written to the new split index. Don't write a new index file from bogus index state, and call BUG() upon encountering an cache entry referring to a non-existing shared index entry. Running the test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes' doesn't trigger this condition. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

split-index: smudge and add racily clean cache entries to split index

2018-10-11T22:23:29Z

Ever since the split index feature was introduced [1], refreshing a split index is prone to a variant of the classic racy git problem. Consider the following sequence of commands updating the split index when the shared index contains a racily clean cache entry, i.e. an entry whose cached stat data matches with the corresponding file in the worktree and the cached mtime matches that of the index: echo "cached content" >file git update-index --split-index --add file echo "dirty worktree" >file # size stays the same! # ... wait ... git update-index --add other-file Normally, when a non-split index is updated, then do_write_index() (the function responsible for writing all kinds of indexes, "regular", split, and shared) recognizes racily clean cache entries, and writes them with smudged stat data, i.e. with file size set to 0. When subsequent git commands read the index, they will notice that the smudged stat data doesn't match with the file in the worktree, and then go on to check the file's content and notice its dirtiness. In the above example, however, in the second 'git update-index' prepare_to_write_split_index() decides which cache entries stored only in the shared index should be replaced in the new split index. Alas, this function never looks out for racily clean cache entries, and since the file's stat data in the worktree hasn't changed since the shared index was written, it won't be replaced in the new split index. Consequently, do_write_index() doesn't even get this racily clean cache entry, and can't smudge its stat data. Subsequent git commands will then see that the index has more recent mtime than the file and that the (not smudged) cached stat data still matches with the file in the worktree, and, ultimately, will erroneously consider the file clean. Modify prepare_to_write_split_index() to recognize racily clean cache entries, and mark them to be added to the split index. Note that there are two places where it should check raciness: first those cache entries that are only stored in the shared index, and then those that have been copied by unpack_trees() from the shared index while it constructed a new index. This way do_write_index() will get these racily clean cache entries as well, and will then write them with smudged stat data to the new split index. This change makes all tests in 't1701-racy-split-index.sh' pass, so flip the two 'test_expect_failure' tests to success. Also add the '#' (as in nr. of trial) to those tests' description that were omitted when the tests expected failure. Note that after this change if the index is split when it contains a racily clean cache entry, then a smudged cache entry will be written both to the new shared and to the new split indexes. This doesn't affect regular git commands: as far as they are concerned this is just an entry in the split index replacing an outdated entry in the shared index. It did affect a few tests in 't1700-split-index.sh', though, because they actually check which entries are stored in the split index; a previous patch in this series has already made the necessary adjustments in 't1700'. And racily clean cache entries and index splitting are rare enough to not worry about the resulting duplicated smudged cache entries, and the additional complexity required to prevent them is not worth it. Several tests failed occasionally when the test suite was run with 'GIT_TEST_SPLIT_INDEX=yes'. Here are those that I managed to trace back to this racy split index problem, starting with those failing more frequently, with a link to a failing Travis CI build job for each. The highlighted line [2] shows when the racy file was written, which is not always in the failing test but in a preceeding setup test. t3903-stash.sh: https://travis-ci.org/git/git/jobs/385542084#L5858 t4024-diff-optimize-common.sh: https://travis-ci.org/git/git/jobs/386531969#L3174 t4015-diff-whitespace.sh: https://travis-ci.org/git/git/jobs/360797600#L8215 t2200-add-update.sh: https://travis-ci.org/git/git/jobs/382543426#L3051 t0090-cache-tree.sh: https://travis-ci.org/git/git/jobs/416583010#L3679 There might be others, e.g. perhaps 't1000-read-tree-m-3way.sh' and others using 'lib-read-tree-m-3way.sh', but I couldn't confirm yet. [1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge branch 'nd/split-index', 2014-07-16). [2] Note that those highlighted lines are in the 'after failure' fold, and your browser might unhelpfully fold it up before you could take a good look. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

split-index: don't compare cached data of entries already marked for split index

2018-10-11T22:23:29Z

When unpack_trees() constructs a new index, it copies cache entries from the original index [1]. prepare_to_write_split_index() has to deal with this, and it has a dedicated code path for copied entries that are present in the shared index, where it compares the cached data in the corresponding copied and original entries. If the cached data matches, then they are considered the same; if it differs, then the copied entry will be marked for inclusion as a replacement entry in the just about to be written split index by setting the CE_UPDATE_IN_BASE flag. However, a cache entry already has its CE_UPDATE_IN_BASE flag set upon reading the split index, if the entry already has a replacement entry there, or upon refreshing the cached stat data, if the corresponding file was modified. The state of this flag is then preserved when unpack_trees() copies a cache entry from the shared index. So modify prepare_to_write_split_index() to check the copied cache entries' CE_UPDATE_IN_BASE flag first, and skip the thorough comparison of cached data if the flag is already set. Those couple of lines comparing the cached data would then have too many levels of indentation, so extract them into a helper function. Note that comparing the cached data in copied and original entries in the shared index might actually be entirely unnecessary. In theory all code paths refreshing the cached stat data of an entry in the shared index should set the CE_UPDATE_IN_BASE flag in that entry, and unpack_trees() should preserve this flag when copying cache entries. This means that the cached data is only ever changed if the CE_UPDATE_IN_BASE flag is set as well. Our test suite seems to confirm this: instrumenting the conditions in question and running the test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes' showed that the cached data in a copied entry differs from the data in the shared entry only if its CE_UPDATE_IN_BASE flag is indeed set. In practice, however, our test suite doesn't have 100% coverage, GIT_TEST_SPLIT_INDEX is inherently random, and I certainly can't claim to possess complete understanding of what goes on in unpack_trees()... Therefore I kept the comparison of the cached data when CE_UPDATE_IN_BASE is not set, just in case that an unnoticed or future code path were to accidentally miss setting this flag upon refreshing the cached stat data or unpack_trees() were to drop this flag while copying a cache entry. [1] Note that when unpack_trees() constructs the new index and decides that a cache entry should now refer to different content than what was recorded in the original index (e.g. 'git read-tree -m HEAD^'), then that can't really be considered a copy of the original, but rather the creation of a new entry. Notably and pertinent to the split index feature, such a new entry doesn't have a reference to the original's shared index entry anymore, i.e. its 'index' field is set to 0. Consequently, such an entry is treated by prepare_to_write_split_index() as an entry not present in the shared index and it will be added to the new split index, while the original entry will be marked as deleted, and neither the above discussion nor the changes in this patch apply to them. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

split-index: count the number of deleted entries

2018-10-11T22:23:29Z

'struct split_index' contains the field 'nr_deletions', whose name with the 'nr_' prefix suggests that it contains the number of deleted cache entries. However, barring its initialization to 0, this field is only ever set to 1, indicating that there is at least one deleted entry, but not the number of deleted entries. Luckily, this doesn't cause any issues (other than confusing the reader, that is), because the only place reading this field uses it in the same sense, i.e.: 'if (si->nr_deletions)'. To avoid confusion, we could either rename this field to something like 'has_deletions' to make its name match its role, or make it a counter of deleted cache entries to match its name. Let's make it a counter, to keep it in sync with the related field 'nr_replacements', which does contain the number of replaced cache entries. This will also give developers debugging the split index code easy access to the number of deleted cache entries. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano