git/entry.c, branch v2.16.2

Merge branch 'bp/fsmonitor'

2017-11-21T05:07:50Z

We learned to talk to watchman to speed up "git status" and other operations that need to see which paths have been modified. * bp/fsmonitor: fsmonitor: preserve utf8 filenames in fsmonitor-watchman log fsmonitor: read entirety of watchman output fsmonitor: MINGW support for watchman integration fsmonitor: add a performance test fsmonitor: add a sample integration script for Watchman fsmonitor: add test cases for fsmonitor extension split-index: disable the fsmonitor extension when running the split index test fsmonitor: add a test tool to dump the index extension update-index: add fsmonitor support to update-index ls-files: Add support in ls-files to display the fsmonitor valid bit fsmonitor: add documentation for the fsmonitor extension. fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files. update-index: add a new --force-write-index option preload-index: add override to enable testing preload-index bswap: add 64 bit endianness helper get_be64

Merge branch 'ls/filter-process-delayed'

2017-10-11T05:52:24Z

Bugfixes to an already graduated series. * ls/filter-process-delayed: write_entry: untangle symlink and regular-file cases write_entry: avoid reading blobs in CE_RETRY case write_entry: fix leak when retrying delayed filter entry.c: check if file exists after checkout entry.c: update cache entry only for existing files

write_entry: untangle symlink and regular-file cases

2017-10-10T00:03:07Z

The write_entry() function switches on the mode of the entry we're going to write out. The cases for S_IFLNK and S_IFREG are lumped together. In earlier versions of the code, this made some sense. They have a shared preamble (which reads the blob content), a short type-specific body, and a shared conclusion (which writes out the file contents; always for S_IFREG and only sometimes for S_IFLNK). But over time this has grown to make less sense. The preamble now has conditional bits for each type, and the S_IFREG body has grown a lot more complicated. It's hard to follow the logic of which code is running for which mode. Let's give each mode its own case arm. We will still share the conclusion code, which means we now jump to it with a goto. Ideally we'd pull that shared code into its own function, but it touches so much internal state in the write_entry() function that the end result is actually harder to follow than the goto. While we're here, we'll touch up a few bits of whitespace to make the beginning and endings of the cases easier to read. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

write_entry: avoid reading blobs in CE_RETRY case

2017-10-09T23:59:57Z

When retrying a delayed filter-process request, we don't need to send the blob to the filter a second time. However, we read it unconditionally into a buffer, only to later throw away that buffer. We can make this more efficient by skipping the read in the first place when it isn't necessary. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

write_entry: fix leak when retrying delayed filter

2017-10-09T23:59:02Z

When write_entry() retries a delayed filter request, we don't need to send the blob content to the filter again, and set the pointer to NULL. But doing so means we leak the contents we read earlier from read_blob_entry(). Let's make sure to free it before dropping the pointer. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

entry.c: check if file exists after checkout

2017-10-06T05:59:16Z

If we are checking out a file and somebody else racily deletes our file, then we would write garbage to the cache entry. Fix that by checking the result of the lstat() call on that file. Print an error to the user if the file does not exist. Reported-by: Jeff King Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

entry.c: update cache entry only for existing files

2017-10-05T11:01:07Z

In 2841e8f ("convert: add "status=delayed" to filter process protocol", 2017-06-30) we taught the filter process protocol to delay responses. That means an external filter might answer in the first write_entry() call on a file that requires filtering "I got your request, but I can't answer right now. Ask again later!". As Git got no answer, we do not write anything to the filesystem. Consequently, the lstat() call in the finish block of the function writes garbage to the cache entry. The garbage is eventually overwritten when the filter answers with the final file content in a subsequent write_entry() call. Fix the brief time window of garbage in the cache entry by adding a special finish block that does nothing for delayed responses. The cache entry is written properly in a subsequent write_entry() call where the filter responds with the final file content. Reported-by: Jeff King Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.

2017-10-01T08:23:01Z

When the index is read from disk, the fsmonitor index extension is used to flag the last known potentially dirty index entries. The registered core.fsmonitor command is called with the time the index was last updated and returns the list of files changed since that time. This list is used to flag any additional dirty cache entries and untracked cache directories. We can then use this valid state to speed up preload_index(), ie_match_stat(), and refresh_cache_ent() as they do not need to lstat() files to detect potential changes for those entries marked CE_FSMONITOR_VALID. In addition, if the untracked cache is turned on valid_cached_dir() can skip checking directories for new or changed files as fsmonitor will invalidate the cache only for those directories that have been identified as having potential changes. To keep the CE_FSMONITOR_VALID state accurate during git operations; when git updates a cache entry to match the current state on disk, it will now set the CE_FSMONITOR_VALID bit. Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit is cleared and the corresponding untracked cache directory is marked invalid. Signed-off-by: Ben Peart Signed-off-by: Junio C Hamano

Merge branch 'jk/write-in-full-fix'

2017-09-25T06:24:06Z

Many codepaths did not diagnose write failures correctly when disks go full, due to their misuse of write_in_full() helper function, which have been corrected. * jk/write-in-full-fix: read_pack_header: handle signed/unsigned comparison in read result config: flip return value of store_write_*() notes-merge: use ssize_t for write_in_full() return value pkt-line: check write_in_full() errors against "< 0" convert less-trivial versions of "write_in_full() != len" avoid "write_in_full(fd, buf, len) != len" pattern get-tar-commit-id: check write_in_full() return against 0 config: avoid "write_in_full(fd, buf, len) < len" pattern

convert less-trivial versions of "write_in_full() != len"

2017-09-14T06:17:59Z

The prior commit converted many sites to check the return value of write_in_full() for negativity, rather than a mismatch with the input length. This patch covers similar cases, but where the return value is stored in an intermediate variable. These should get the same treatment, but they need to be reviewed more carefully since it would be a bug if the return value is stored in an unsigned type (which indeed, it is in one of the cases). Signed-off-by: Jeff King Reviewed-by: Jonathan Nieder Signed-off-by: Junio C Hamano