git/bundle-uri.c, branch v2.46.4

Sync with 2.45.4

2025-05-28T18:57:08Z

* maint-2.45: Git 2.45.4 Git 2.44.4 Git 2.43.7 wincred: avoid buffer overflow in wcsncat() bundle-uri: fix arbitrary file writes via parameter injection config: quote values containing CR character git-gui: sanitize 'exec' arguments: convert new 'cygpath' calls git-gui: do not mistake command arguments as redirection operators git-gui: introduce function git_redir for git calls with redirections git-gui: pass redirections as separate argument to git_read git-gui: pass redirections as separate argument to _open_stdout_stderr git-gui: convert git_read*, git_write to be non-variadic git-gui: override exec and open only on Windows gitk: sanitize 'open' arguments: revisit recently updated 'open' calls git-gui: use git_read in githook_read git-gui: sanitize $PATH on all platforms git-gui: break out a separate function git_read_nice git-gui: assure PATH has only absolute elements. git-gui: remove option --stderr from git_read git-gui: cleanup git-bash menu item git-gui: sanitize 'exec' arguments: background git-gui: avoid auto_execok in do_windows_shortcut git-gui: sanitize 'exec' arguments: simple cases git-gui: avoid auto_execok for git-bash menu item git-gui: treat file names beginning with "|" as relative paths git-gui: remove unused proc is_shellscript git-gui: remove git config --list handling for git < 1.5.3 git-gui: remove special treatment of Windows from open_cmd_pipe git-gui: remove HEAD detachment implementation for git < 1.5.3 git-gui: use only the configured shell git-gui: remove Tcl 8.4 workaround on 2>@1 redirection git-gui: make _shellpath usable on startup git-gui: use [is_Windows], not bad _shellpath git-gui: _which, only add .exe suffix if not present gitk: encode arguments correctly with "open" gitk: sanitize 'open' arguments: command pipeline gitk: collect construction of blameargs into a single conditional gitk: sanitize 'open' arguments: simple commands, readable and writable gitk: sanitize 'open' arguments: simple commands with redirections gitk: sanitize 'open' arguments: simple commands gitk: sanitize 'exec' arguments: redirect to process gitk: sanitize 'exec' arguments: redirections and background gitk: sanitize 'exec' arguments: redirections gitk: sanitize 'exec' arguments: 'eval exec' gitk: sanitize 'exec' arguments: simple cases gitk: have callers of diffcmd supply pipe symbol when necessary gitk: treat file names beginning with "|" as relative paths Signed-off-by: Taylor Blau

bundle-uri: fix arbitrary file writes via parameter injection

2025-05-23T21:09:48Z

We fetch bundle URIs via `download_https_uri_to_file()`. The logic to fetch those bundles is not handled in-process, but we instead use a separate git-remote-https(1) process that performs the fetch for us. The information about which file should be downloaded and where that file should be put gets communicated via stdin of that process via a "get" request. This "get" request has the form "get $uri $file\n\n". As may be obvious to the reader, this will cause git-remote-https(1) to download the URI "$uri" and put it into "$file". The fact that we are using plain spaces and newlines as separators for the request arguments means that we have to be extra careful with the respective vaules of these arguments: - If "$uri" contained a space we would interpret this as both URI and target location. - If either "$uri" or "$file" contained a newline we would interpret this as a new command. But we neither quote the arguments such that any characters with special meaning would be escaped, nor do we verify that none of these special characters are contained. If either the URI or file contains a newline character, we are open to protocol injection attacks. Likewise, if the URI itself contains a space, then an attacker-controlled URI can lead to partially-controlled file writes. Note that the attacker-controlled URIs do not permit completely arbitrary file writes, but instead allows an attacker to control the path in which we will write a temporary (e.g., "tmp_uri_XXXXXX") file. The result is twofold: - By adding a space in "$uri" we can control where exactly a file will be written to, including out-of-repository writes. The final location is not completely arbitrary, as the injected string will be concatenated with the original "$file" path. Furthermore, the name of the bundle will be "tmp_uri_XXXXXX", further restricting what an adversary would be able to write. Also note that is not possible for the URI to contain a newline because we end up in `credential_from_url_1()` before we try to issue any requests using that URI. As such, it is not possible to inject arbitrary commands via the URI. - By adding a newline to "$file" we can inject arbitrary commands. This gives us full control over where a specific file will be written to. Potential attack vectors would be to overwrite hooks, but if an adversary were to guess where the user's home directory is located they might also easily write e.g. a "~/.profile" file and thus cause arbitrary code execution. This injection can only become possible when the adversary has full control over the target path where a bundle will be downloaded to. While this feels unlikely, it is possible to control this path when users perform a recursive clone with a ".gitmodules" file that is controlled by the adversary. Luckily though, the use of bundle URIs is not enabled by default in Git clients (yet): they have to be enabled by setting the `bundle.heuristic` config key explicitly. As such, the blast radius of this parameter injection should overall be quite contained. Fix the issue by rejecting spaces in the URI and newlines in both the URI and the file. As explained, it shouldn't be required to also restrict the use of newlines in the URI, as we would eventually die anyway in `credential_from_url_1()`. But given that we're only one small step away from arbitrary code execution, let's rather be safe and restrict newlines in URIs, as well. Eventually we should probably refactor the way that Git talks with the git-remote-https(1) subprocess so that it is less fragile. Until then, these two restrictions should plug the issue. Reported-by: David Leadbeater Based-on-patch-by: David Leadbeater Signed-off-by: Patrick Steinhardt Signed-off-by: Taylor Blau

Merge branch 'xx/bundie-uri-fixes'

2024-07-08T21:53:11Z

When bundleURI interface fetches multiple bundles, Git failed to take full advantage of all bundles and ended up slurping duplicated objects. * xx/bundie-uri-fixes: unbundle: extend object verification for fetches fetch-pack: expose fsckObjects configuration logic bundle-uri: verify oid before writing refs

unbundle: extend object verification for fetches

2024-06-20T17:30:08Z

The existing fetch.fsckObjects and transfer.fsckObjects configurations were not fully applied to bundle-involved fetches, including direct bundle fetches and bundle-uri enabled fetches. Furthermore, there was no object verification support for unbundle. This commit extends object verification support in `bundle.c:unbundle` by adding the `VERIFY_BUNDLE_FSCK` option to `verify_bundle_flags`. When this option is enabled, we append the `--fsck-objects` flag to `git-index-pack`. The `VERIFY_BUNDLE_FSCK` option is now used by bundle-involved fetches, where we use `fetch-pack.c:fetch_pack_fsck_objects` to determine whether to enable this option for `bundle.c:unbundle`, specifically in: - `transport.c:fetch_refs_from_bundle` for direct bundle fetches. - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches. This addition ensures a consistent logic for object verification during fetches. Tests have been added to confirm functionality in the scenarios mentioned above. Reviewed-by: Patrick Steinhardt Signed-off-by: Xing Xin Signed-off-by: Junio C Hamano

bundle-uri: verify oid before writing refs

2024-06-20T17:30:07Z

When using the bundle-uri mechanism with a bundle list containing multiple interrelated bundles, we encountered a bug where tips from downloaded bundles were not discovered, thus resulting in rather slow clones. This was particularly problematic when employing the "creationTokens" heuristic. To reproduce this issue, consider a repository with a single branch "main" pointing to commit "A". Firstly, create a base bundle with: git bundle create base.bundle main Then, add a new commit "B" on top of "A", and create an incremental bundle for "main": git bundle create incr.bundle A..main Now, generate a bundle list with the following content: [bundle] version = 1 mode = all heuristic = creationToken [bundle "base"] uri = base.bundle creationToken = 1 [bundle "incr"] uri = incr.bundle creationToken = 2 A fresh clone with the bundle list above should result in a reference "refs/bundles/main" pointing to "B" in the new repository. However, git would still download everything from the server, as if it had fetched nothing locally. So why the "refs/bundles/main" is not discovered? After some digging I found that: 1. Bundles in bundle list are downloaded to local files via `bundle-uri.c:download_bundle_list` or via `bundle-uri.c:fetch_bundles_by_token` for the "creationToken" heuristic. 2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which is called by `bundle-uri.c:unbundle_all_bundles` or called within `bundle-uri.c:fetch_bundles_by_token` for the "creationToken" heuristic. 3. To get all prerequisites of the bundle, the bundle header is read inside `bundle-uri.c:unbundle_from_file` to by calling `bundle.c:read_bundle_header`. 4. Then it calls `bundle.c:unbundle`, which calls `bundle.c:verify_bundle` to ensure the repository contains all the prerequisites. 5. `bundle.c:verify_bundle` calls `parse_object`, which eventually invokes `packfile.c:prepare_packed_git` or `packfile.c:reprepare_packed_git`, filling `raw_object_store->packed_git` and setting `packed_git_initialized`. 6. If `bundle.c:unbundle` succeeds, it writes refs via `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here bundle refs which can target arbitrary objects are written to the repository. 7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions `fetch-pack.c:mark_complete_and_common_ref` and `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to find local tips for negotiation. The `OBJECT_INFO_QUICK` flag prevents `packfile.c:reprepare_packed_git` from being called, resulting in failures to parse OIDs that reside only in the latest bundle. In the example above, when unbunding "incr.bundle", "base.pack" is added to `packed_git` due to prerequisites verification. However, "B" cannot be found for negotiation because it exists in "incr.pack", which is not included in `packed_git`. Fix the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing bundle refs. When `refs.c:refs_update_ref` is called to write the corresponding bundle refs, it triggers `refs.c:ref_transaction_commit`. This, in turn, invokes `refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of the refs storage backend. For files backend, it is `files-backend.c:files_transaction_prepare`, and for reftable backend, it is `reftable-backend.c:reftable_be_transaction_prepare`. Both functions eventually call `object.c:parse_object`, which can invoke `packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures that bundle refs point to valid objects and that all tips from bundle refs are correctly parsed during subsequent negotiations. A set of negotiation-related tests for cloning with bundle-uri has been included to demonstrate that downloaded bundles are utilized to accelerate fetching. Additionally, another test has been added to show that bundles with incorrect headers, where refs point to non-existent objects, do not result in any bundle refs being created in the repository. Reviewed-by: Karthik Nayak Reviewed-by: Patrick Steinhardt Signed-off-by: Xing Xin Signed-off-by: Junio C Hamano

global: introduce `USE_THE_REPOSITORY_VARIABLE` macro

2024-06-14T17:26:33Z

Use of the `the_repository` variable is deprecated nowadays, and we slowly but steadily convert the codebase to not use it anymore. Instead, callers should be passing down the repository to work on via parameters. It is hard though to prove that a given code unit does not use this variable anymore. The most trivial case, merely demonstrating that there is no direct use of `the_repository`, is already a bit of a pain during code reviews as the reviewer needs to manually verify claims made by the patch author. The bigger problem though is that we have many interfaces that implicitly rely on `the_repository`. Introduce a new `USE_THE_REPOSITORY_VARIABLE` macro that allows code units to opt into usage of `the_repository`. The intent of this macro is to demonstrate that a certain code unit does not use this variable anymore, and to keep it from new dependencies on it in future changes, be it explicit or implicit For now, the macro only guards `the_repository` itself as well as `the_hash_algo`. There are many more known interfaces where we have an implicit dependency on `the_repository`, but those are not guarded at the current point in time. Over time though, we should start to add guards as required (or even better, just remove them). Define the macro as required in our code units. As expected, most of our code still relies on the global variable. Nearly all of our builtins rely on the variable as there is no way yet to pass `the_repository` to their entry point. For now, declare the macro in "biultin.h" to keep the required changes at least a little bit more contained. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

cocci: apply rules to rewrite callers of "refs" interfaces

2024-05-07T17:06:59Z

Apply the rules that rewrite callers of "refs" interfaces to explicitly pass `struct ref_store`. The resulting patch has been applied with the `--whitespace=fix` option. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

treewide: remove unnecessary includes in source files

2023-12-26T20:04:31Z

Each of these were checked with gcc -E -I. ${SOURCE_FILE} | grep ${HEADER_FILE} to ensure that removing the direct inclusion of the header actually resulted in that header no longer being included at all (i.e. that no other header pulled it in transitively). ...except for a few cases where we verified that although the header was brought in transitively, nothing from it was directly used in that source file. These cases were: * builtin/credential-cache.c * builtin/pull.c * builtin/send-pack.c Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

bundle-uri: mark unused parameters in callbacks

2023-08-30T00:56:26Z

The first hunk is similar to 02c3c59e62 (hashmap: mark unused callback parameters, 2022-08-19), but was added after that commit. The other two are used with for_all_bundles_in_list(), but don't use their void data pointer. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'gc/config-context'

2023-07-06T18:54:48Z

Reduce reliance on a global state in the config reading API. * gc/config-context: config: pass source to config_parser_event_fn_t config: add kvi.path, use it to evaluate includes config.c: remove config_reader from configsets config: pass kvi to die_bad_number() trace2: plumb config kvi config.c: pass ctx with CLI config config: pass ctx with config files config.c: pass ctx in configsets config: add ctx arg to config_fn_t urlmatch.h: use config_fn_t type config: inline git_color_default_config