git/list-objects-filter-options.c, branch v2.40.2

list-objects-filter: remove OPT_PARSE_LIST_OBJECTS_FILTER_INIT()

2022-11-30T01:00:35Z

OPT_PARSE_LIST_OBJECTS_FILTER_INIT() with a non-NULL second argument passes a function pointer via an object pointer, which is undefined. It may work fine on platforms that implement C99 extension J.5.7 (Function pointer casts). Remove the unused macro and avoid the dependency on that extension. Signed-off-by: René Scharfe Signed-off-by: Junio C Hamano

list-objects-filter: initialize sub-filter structs

2022-09-22T19:43:04Z

Since commit c54980ab83 (list-objects-filter: convert filter_spec to a strbuf, 2022-09-11), building with SANITIZE=undefined triggers an error in t5616. The problem is that we end up with a strbuf that has been zero-initialized instead of via STRBUF_INIT. Feeding that strbuf to strbuf_addbuf() in list_objects_filter_copy() means we will call memcpy like: memcpy(some_actual_buffer, NULL, 0); This works on most systems because we're copying zero bytes, but it is technically undefined behavior to ever pass NULL to memcpy. Even though c54980ab83 is where the bug manifests, that is only because we switched away from a string_list, which is OK with being zero-initialized (though it may cause other problems by not duplicating the strings, it happened to be OK in this instance). The actual bug is caused by the commit before that, 2a01bdedf8 (list-objects-filter: add and use initializers, 2022-09-11). There we consistently initialize the top-level filter structs, but we forgot the dynamically allocated ones we stick in filter_options->sub when creating combined filters. Note that we need to fix two spots here: where we parse a "combine:" filter, but also where we transform from a single-filter into a combined one after seeing multiple "--filter" options. In the second spot, we'll do some minor refactoring to avoid repeating our very-long array index. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list-objects-filter: convert filter_spec to a strbuf

2022-09-12T15:38:59Z

Originally, the filter_spec field was just a string pointer. In cf9ceb5a12 (list-objects-filter-options: make filter_spec a string_list, 2019-06-27) it became a string_list, but that commit notes: A strbuf would seem to be a more natural choice for this object, but it unfortunately requires initialization besides just zero'ing out the memory. This results in all container structs, and all containers of those structs, etc., to also require initialization. Initializing them all would be more cumbersome that simply using a string_list, which behaves properly when its contents are zero'd. Now that we've changed the struct to require non-zero initialization anyway (ironically, because string_list also needed non-zero initialization to avoid leaks), we can now convert to that more natural type. This makes the list_objects_filter_spec() function much less awkward, as it had to collapse the string_list to a single-entry list on the fly. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list-objects-filter: add and use initializers

2022-09-12T15:38:59Z

In 7e2619d8ff (list_objects_filter_options: plug leak of filter_spec strings, 2022-09-08), we noted that the filter_spec string_list was inconsistent in how it handled memory ownership of strings stored in the list. The fix there was a bit of a band-aid to set the "strdup_strings" variable right before adding anything. That works OK, and it lets the users of the API continue to zero-initialize the struct. But it makes the code a bit hard to follow and accident-prone, as any other spots appending the filter_spec need to think about whether to set the strdup_strings value, too (there's one such spot in partial_clone_get_default_filter_spec(), which is probably a possible memory leak). So let's do that full cleanup now. We'll introduce a LIST_OBJECTS_FILTER_INIT macro and matching function, and use them as appropriate (though it is for the "_options" struct, this matches the corresponding list_objects_filter_release() function). This is harder than it seems! Many other structs, like git_transport_data, embed the filter struct. So they need to initialize it themselves even if the rest of the enclosing struct is OK with zero-initialization. I found all of the relevant spots by grepping manually for declarations of list_objects_filter_options. And then doing so recursively for structs which embed it, and ones which embed those, and so on. I'm pretty sure I got everything, but there's no change that would alert the compiler if any topics in flight added new declarations. To catch this case, we now double-check in the parsing function that things were initialized as expected and BUG() if appropriate. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list-objects-filter: handle null default filter spec

2022-09-12T15:38:59Z

When we have a remote.*.promisor config variable, we know that we're in a partial clone. Usually there's a matching remote.*.partialclonefilter option, which tells us which filter to use with the remote. If that option is missing, we skip setting up the filter at all. But something funny happens: we stick a NULL entry into the string_list storing the text filter spec. This is a weird state, and could possibly segfault if anybody called called list_objects_filter_spec(), etc. In practice, nobody does, because filter->choice will still be LOFC_DISABLED, so code generally realizes there's no filter to use. And the string_list itself is OK, because it starts in non-dup mode until we actually parse a filter spec. So it blindly stores the NULL without even looking at it. But it's probably worth avoiding this confused state. It's an accident waiting to happen, and it will be a problem if we replace the lazy initialization from 7e2619d8ff (list_objects_filter_options: plug leak of filter_spec strings, 2022-09-08) with a real initialization function. The history is a little interesting here, as the bug was introduced during the merge resolution in 627b826834 (Merge branch 'md/list-objects-filter-combo', 2019-09-18). The original logic comes from cac1137dc4 (list-objects: check if filter is NULL before using, 2018-06-11), where we had a single string via core.partialCloneFilter, and a simple NULL check was sufficient. And it even added a test in t0410 that covers this situation. Later, that was expanded to allow per-remote filters in fa3d1b63e8 (promisor-remote: parse remote.*.partialclonefilter, 2019-06-25). After that commit, we get a promisor struct with a partial_clone_filter string, which could be NULL. The commit checks only that the struct pointer is non-NULL, which is enough. It may pass NULL to gently_parse_list_objects_filter(), but that function is smart enough to consider it a noop. But in parallel, cf9ceb5a12 (list-objects-filter-options: make filter_spec a string_list, 2019-06-27) added a new line of code: before we call gently_parse_list_objets_filter(), we append the filter spec to the string_list. By itself that was OK, since we'd have returned early if the string was NULL. When the two were merged in 627b826834, the result is that we return early only if the struct is NULL, but not the string. And we append to the string_list, meaning we may append NULL. The solution is to return early if either is NULL, as it would mean we don't have a configured filter. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list-objects-filter: don't memset after releasing filter struct

2022-09-12T15:38:58Z

If we see an error while parsing a "combine" filter, we call list_objects_filter_release() to free any allocated memory, and then use memset() to return the struct to a known state. But the release function already does that reinitializing. Doing it again is pointless. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list_objects_filter_options: plug leak of filter_spec strings

2022-09-08T18:08:23Z

The list_objects_filter_options struct contains a string_list to store the filter_spec. Because we allow the options struct to be zero-initialized by callers, the string_list's strdup_strings field is generally not set. Because we don't want to depend on the memory lifetimes of any strings passed in to the list-objects API, everything we add to the string_list is duplicated (either via xstrdup(), or via strbuf_detach()). So far so good, but now we have a problem at cleanup time: when we clear the list, the string_list API doesn't realize that it needs to free all of those strings, and we leak them. One option would be to set strdup_strings right before clearing the list. But this is tricky for two reasons: 1. There's one spot, in partial_clone_get_default_filter_spec(), that fails to duplicate its argument. We could fix that, but... 2. We clear the list in a surprising number of places. As you might expect, we do so in list_objects_filter_release(). But we also clear and rewrite it in expand_list_objects_filter_spec(), list_objects_filter_spec(), and transform_to_combine_type(). We'd have to put the same hack in all of those spots. Instead, let's just set strdup_strings before adding anything. That lets us drop the extra manual xstrdup() calls, fixes the spot mentioned in (1) above that _should_ be duplicating, and future-proofs further calls. We do have to switch the strbuf_detach() calls to use the nodup form, but that's an easy change, and the resulting code more clearly shows the expected ownership transfer. This also resolves a weird inconsistency: when we make a deep copy with list_objects_filter_copy(), it initializes the copy's filter_spec with string_list_init_dup(). So the copy frees its string_list memory correctly, but accidentally leaks the extra manual-xstrdup()'d strings! There is one hiccup, though. In an ideal world, everyone would allocate the list_objects_filter_options struct with an initializer which used STRING_LIST_INIT_DUP under the hood. But there are a bunch of existing callers which think that zero-initializing is good enough. We can leave them as-is by noting that the list is always initially populated via parse_list_objects_filter(). So we can just initialize the strdup_strings flag there. This is arguably a band-aid, but it works reliably. And it doesn't make anything harder if we want to switch all the callers later to a new LIST_OBJECTS_FILTER_INIT. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

list_objects_filter_copy(): deep-copy sparse_oid_name field

2022-09-08T18:05:46Z

The purpose of our copy function is to do a deep copy of each field so that the source and destination structs become independent. We correctly copy the filter_spec string list, but we forgot the sparse_oid_name field. By doing a shallow copy of the pointer, that puts us at risk for a use-after-free if one or both of the structs is cleaned up. I don't think this can be triggered in practice, because we tend to leak the structs rather than actually clean them up. But this should future-proof us for plugging those leaks. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

pack-objects: lazily set up "struct rev_info", don't leak

2022-03-28T16:57:21Z

In the preceding [1] (pack-objects: move revs out of get_object_list(), 2022-03-22) the "repo_init_revisions()" was moved to cmd_pack_objects() so that it unconditionally took place for all invocations of "git pack-objects". We'd thus start leaking memory, which is easily reproduced in e.g. git.git by feeding e83c5163316 (Initial revision of "git", the information manager from hell, 2005-04-07) to "git pack-objects"; $ echo e83c5163316f89bfbde7d9ab23ca2e25604af290 | ./git pack-objects initial [...] ==19130==ERROR: LeakSanitizer: detected memory leaks Direct leak of 7120 byte(s) in 1 object(s) allocated from: #0 0x455308 in __interceptor_malloc (/home/avar/g/git/git+0x455308) #1 0x75b399 in do_xmalloc /home/avar/g/git/wrapper.c:41:8 #2 0x75b356 in xmalloc /home/avar/g/git/wrapper.c:62:9 #3 0x5d7609 in prep_parse_options /home/avar/g/git/diff.c:5647:2 #4 0x5d415a in repo_diff_setup /home/avar/g/git/diff.c:4621:2 #5 0x6dffbb in repo_init_revisions /home/avar/g/git/revision.c:1853:2 #6 0x4f599d in cmd_pack_objects /home/avar/g/git/builtin/pack-objects.c:3980:2 #7 0x4592ca in run_builtin /home/avar/g/git/git.c:465:11 #8 0x457d81 in handle_builtin /home/avar/g/git/git.c:718:3 #9 0x458ca5 in run_argv /home/avar/g/git/git.c:785:4 #10 0x457b40 in cmd_main /home/avar/g/git/git.c:916:19 #11 0x562259 in main /home/avar/g/git/common-main.c:56:11 #12 0x7fce792ac7ec in __libc_start_main csu/../csu/libc-start.c:332:16 #13 0x4300f9 in _start (/home/avar/g/git/git+0x4300f9) SUMMARY: LeakSanitizer: 7120 byte(s) leaked in 1 allocation(s). Aborted Narrowly fixing that commit would have been easy, just add call repo_init_revisions() right before get_object_list(), which is effectively what was done before that commit. But an unstated constraint when setting it up early is that it was needed for the subsequent [2] (pack-objects: parse --filter directly into revs.filter, 2022-03-22), i.e. we might have a --filter command-line option, and need to either have the "struct rev_info" setup when we encounter that option, or later. Let's just change the control flow so that we'll instead set up the "struct rev_info" only when we need it. Doing so leads to a bit more verbosity, but it's a lot clearer what we're doing and why. An earlier version of this commit[3] went behind opt_parse_list_objects_filter()'s back by faking up a "struct option" before calling it. Let's avoid that and instead create a blessed API for this pattern. We could furthermore combine the two get_object_list() invocations here by having repo_init_revisions() invoked on &pfd.revs, but I think clearly separating the two makes the flow clearer. Likewise redundantly but explicitly (i.e. redundant v.s. a "{ 0 }") "0" to "have_revs" early in cmd_pack_objects(). While we're at it add parentheses around the arguments to the OPT_* macros in in list-objects-filter-options.h, as we need to change those lines anyway. It doesn't matter in this case, but is good general practice. 1. https://lore.kernel.org/git/619b757d98465dbc4995bdc11a5282fbfcbd3daa.1647970119.git.gitgitgadget@gmail.com 2. https://lore.kernel.org/git/97de926904988b89b5663bd4c59c011a1723a8f5.1647970119.git.gitgitgadget@gmail.com 3. https://lore.kernel.org/git/patch-1.1-193534b0f07-20220325T121715Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

bundle: parse filter capability

2022-03-09T18:25:27Z

The v3 bundle format has capabilities, allowing newer versions of Git to create bundles with newer features. Older versions that do not understand these new capabilities will fail with a helpful warning. Create a new capability allowing Git to understand that the contained pack-file is filtered according to some object filter. Typically, this filter will be "blob:none" for a blobless partial clone. This change teaches Git to parse this capability, place its value in the bundle header, and demonstrate this understanding by adding a message to 'git bundle verify'. Since we will use gently_parse_list_objects_filter() outside of list-objects-filter-options.c, make it an external method and move its API documentation to before its declaration. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano