git/convert.c, branch v2.34.2

convert: release strbuf to avoid leak

2021-07-26T19:19:20Z

apply_multi_file_filter and async_query_available_blobs both query subprocess output using subprocess_read_status, which writes data into the identically named filter_status strbuf. We add a strbuf_release to avoid leaking their contents. Leak output seen when running t0021 with LSAN: Direct leak of 24 byte(s) in 1 object(s) allocated from: #0 0x49ab49 in realloc ../projects/compiler-rt/lib/asan/asan_malloc_linux.cpp:164:3 #1 0xa8c2b5 in xrealloc wrapper.c:126:8 #2 0x9ff99d in strbuf_grow strbuf.c:98:2 #3 0x9ff99d in strbuf_addbuf strbuf.c:304:2 #4 0xa101d6 in subprocess_read_status sub-process.c:45:5 #5 0x77793c in apply_multi_file_filter convert.c:886:8 #6 0x77793c in apply_filter convert.c:1042:10 #7 0x77a0b5 in convert_to_git_filter_fd convert.c:1492:7 #8 0x8b48cd in index_stream_convert_blob object-file.c:2156:2 #9 0x8b48cd in index_fd object-file.c:2248:9 #10 0x597411 in hash_fd builtin/hash-object.c:43:9 #11 0x596be1 in hash_object builtin/hash-object.c:59:2 #12 0x596be1 in cmd_hash_object builtin/hash-object.c:153:3 #13 0x4ce83e in run_builtin git.c:475:11 #14 0x4ccafe in handle_builtin git.c:729:3 #15 0x4cb01c in run_argv git.c:818:4 #16 0x4cb01c in cmd_main git.c:949:19 #17 0x6bdc2d in main common-main.c:52:11 #18 0x7f42acf79349 in __libc_start_main (/lib64/libc.so.6+0x24349) SUMMARY: AddressSanitizer: 24 byte(s) leaked in 1 allocation(s). Direct leak of 120 byte(s) in 5 object(s) allocated from: #0 0x49ab49 in realloc ../projects/compiler-rt/lib/asan/asan_malloc_linux.cpp:164:3 #1 0xa8c295 in xrealloc wrapper.c:126:8 #2 0x9ff97d in strbuf_grow strbuf.c:98:2 #3 0x9ff97d in strbuf_addbuf strbuf.c:304:2 #4 0xa101b6 in subprocess_read_status sub-process.c:45:5 #5 0x775c73 in async_query_available_blobs convert.c:960:8 #6 0x80029d in finish_delayed_checkout entry.c:183:9 #7 0xa65d1e in check_updates unpack-trees.c:493:10 #8 0xa5f469 in unpack_trees unpack-trees.c:1747:8 #9 0x525971 in checkout builtin/clone.c:815:6 #10 0x525971 in cmd_clone builtin/clone.c:1409:8 #11 0x4ce83e in run_builtin git.c:475:11 #12 0x4ccafe in handle_builtin git.c:729:3 #13 0x4cb01c in run_argv git.c:818:4 #14 0x4cb01c in cmd_main git.c:949:19 #15 0x6bdc2d in main common-main.c:52:11 #16 0x7fa253fce349 in __libc_start_main (/lib64/libc.so.6+0x24349) SUMMARY: AddressSanitizer: 120 byte(s) leaked in 5 allocation(s). Signed-off-by: Andrzej Hunt Signed-off-by: Junio C Hamano

Merge branch 'ds/sparse-index-protections'

2021-04-30T04:50:26Z

Builds on top of the sparse-index infrastructure to mark operations that are not ready to mark with the sparse index, causing them to fall back on fully-populated index that they always have worked with. * ds/sparse-index-protections: (47 commits) name-hash: use expand_to_path() sparse-index: expand_to_path() name-hash: don't add directories to name_hash revision: ensure full index resolve-undo: ensure full index read-cache: ensure full index pathspec: ensure full index merge-recursive: ensure full index entry: ensure full index dir: ensure full index update-index: ensure full index stash: ensure full index rm: ensure full index merge-index: ensure full index ls-files: ensure full index grep: ensure full index fsck: ensure full index difftool: ensure full index commit: ensure full index checkout: ensure full index ...

*: remove 'const' qualifier for struct index_state

2021-04-14T20:46:00Z

Several methods specify that they take a 'struct index_state' pointer with the 'const' qualifier because they intend to only query the data, not change it. However, we will be introducing a step very low in the method stack that might modify a sparse-index to become a full index in the case that our queries venture inside a sparse-directory entry. This change only removes the 'const' qualifiers that are necessary for the following change which will actually modify the implementation of index_name_stage_pos(). Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren Signed-off-by: Junio C Hamano

Merge branch 'jh/simple-ipc'

2021-04-02T21:43:14Z

A simple IPC interface gets introduced to build services like fsmonitor on top. * jh/simple-ipc: t0052: add simple-ipc tests and t/helper/test-simple-ipc tool simple-ipc: add Unix domain socket implementation unix-stream-server: create unix domain socket under lock unix-socket: disallow chdir() when creating unix domain sockets unix-socket: add backlog size option to unix_stream_listen() unix-socket: eliminate static unix_stream_socket() helper function simple-ipc: add win32 implementation simple-ipc: design documentation for new IPC mechanism pkt-line: add options argument to read_packetized_to_strbuf() pkt-line: add PACKET_READ_GENTLE_ON_READ_ERROR option pkt-line: do not issue flush packets in write_packetized_*() pkt-line: eliminate the need for static buffer in packet_write_gently()

Merge branch 'mt/parallel-checkout-part-1'

2021-04-02T21:43:14Z

Preparatory API changes for parallel checkout. * mt/parallel-checkout-part-1: entry: add checkout_entry_ca() taking preloaded conv_attrs entry: move conv_attrs lookup up to checkout_entry() entry: extract update_ce_after_write() from write_entry() entry: make fstat_output() and read_blob_entry() public entry: extract a header file for entry.c functions convert: add classification for conv_attrs struct convert: add get_stream_filter_ca() variant convert: add [async_]convert_to_working_tree_ca() variants convert: make convert_attrs() and convert structs public

Merge branch 'mt/cleanly-die-upon-missing-required-filter'

2021-03-22T21:00:22Z

We had a code to diagnose and die cleanly when a required clean/smudge filter is missing, but an assert before that unnecessarily fired, hiding the end-user facing die() message. * mt/cleanly-die-upon-missing-required-filter: convert: fail gracefully upon missing clean cmd on required filter

convert: add classification for conv_attrs struct

2021-03-18T20:56:40Z

Create `enum conv_attrs_classification` to express the different ways that attributes are handled for a blob during checkout. This will be used in a later commit when deciding whether to add a file to the parallel or delayed queue during checkout. For now, we can also use it in get_stream_filter_ca() to simplify the function (as the classifying logic is the same). Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares Signed-off-by: Junio C Hamano

convert: add get_stream_filter_ca() variant

2021-03-18T20:56:40Z

Like the previous patch, we will also need to call get_stream_filter() with a precomputed `struct conv_attrs`, when we add support for parallel checkout workers. So add the _ca() variant which takes the conversion attributes struct as a parameter. Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares Signed-off-by: Junio C Hamano

convert: add [async_]convert_to_working_tree_ca() variants

2021-03-18T20:56:40Z

Separate the attribute gathering from the actual conversion by adding _ca() variants of the conversion functions. These variants receive a precomputed 'struct conv_attrs', not relying, thus, on an index state. They will be used in a future patch adding parallel checkout support, for two reasons: - We will already load the conversion attributes in checkout_entry(), before conversion, to decide whether a path is eligible for parallel checkout. Therefore, it would be wasteful to load them again later, for the actual conversion. - The parallel workers will be responsible for reading, converting and writing blobs to the working tree. They won't have access to the main process' index state, so they cannot load the attributes. Instead, they will receive the preloaded ones and call the _ca() variant of the conversion functions. Furthermore, the attributes machinery is optimized to handle paths in sequential order, so it's better to leave it for the main process, anyway. Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares Signed-off-by: Junio C Hamano

convert: make convert_attrs() and convert structs public

2021-03-18T20:56:40Z

Move convert_attrs() declaration from convert.c to convert.h, together with the conv_attrs struct and the crlf_action enum. This function and the data structures will be used outside convert.c in the upcoming parallel checkout implementation. Note that crlf_action is renamed to convert_crlf_action, which is more appropriate for the global namespace. Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares Signed-off-by: Junio C Hamano