git/submodule.c, branch v2.40.2

Sync with 2.39.4

2024-04-19T10:38:37Z

* maint-2.39: (38 commits) Git 2.39.4 fsck: warn about symlink pointing inside a gitdir core.hooksPath: add some protection while cloning init.templateDir: consider this config setting protected clone: prevent hooks from running during a clone Add a helper function to compare file contents init: refactor the template directory discovery into its own function find_hook(): refactor the `STRIP_EXTENSION` logic clone: when symbolic links collide with directories, keep the latter entry: report more colliding paths t5510: verify that D/F confusion cannot lead to an RCE submodule: require the submodule path to contain directories only clone_submodule: avoid using `access()` on directories submodules: submodule paths must not contain symlinks clone: prevent clashing git dirs when cloning submodule in parallel t7423: add tests for symlinked submodule directories has_dir_name(): do not get confused by characters < '/' docs: document security issues around untrusted .git dirs upload-pack: disable lazy-fetching by default fetch/clone: detect dubious ownership of local repositories ...

submodule: require the submodule path to contain directories only

2024-04-17T20:30:04Z

Submodules are stored in subdirectories of their superproject. When these subdirectories have been replaced with symlinks by a malicious actor, all kinds of mayhem can be caused. This _should_ not be possible, but many CVEs in the past showed that _when_ possible, it allows attackers to slip in code that gets executed during, say, a `git clone --recursive` operation. Let's add some defense-in-depth to disallow submodule paths to have anything except directories in them. Signed-off-by: Johannes Schindelin

clone: prevent clashing git dirs when cloning submodule in parallel

2024-04-17T20:30:01Z

While it is expected to have several git dirs within the `.git/modules/` tree, it is important that they do not interfere with each other. For example, if one submodule was called "captain" and another submodule "captain/hooks", their respective git dirs would clash, as they would be located in `.git/modules/captain/` and `.git/modules/captain/hooks/`, respectively, i.e. the latter's files could clash with the actual Git hooks of the former. To prevent these clashes, and in particular to prevent hooks from being written and then executed as part of a recursive clone, we introduced checks as part of the fix for CVE-2019-1387 in a8dee3ca61 (Disallow dubiously-nested submodule git directories, 2019-10-01). It is currently possible to bypass the check for clashing submodule git dirs in two ways: 1. parallel cloning 2. checkout --recurse-submodules Let's check not only before, but also after parallel cloning (and before checking out the submodule), that the git dir is not clashing with another one, otherwise fail. This addresses the parallel cloning issue. As to the parallel checkout issue: It requires quite a few manual steps to create clashing git dirs because Git itself would refuse to initialize the inner one, as demonstrated by the test case. Nevertheless, let's teach the recursive checkout (namely, the `submodule_move_head()` function that is used by the recursive checkout) to be careful to verify that it does not use a clashing git dir, and if it does, disable it (by deleting the `HEAD` file so that subsequent Git calls won't recognize it as a git dir anymore). Note: The parallel cloning test case contains a `cat err` that proved to be highly useful when analyzing the racy nature of the operation (the operation can fail with three different error messages, depending on timing), and was left on purpose to ease future debugging should the need arise. Signed-off-by: Filip Hejsek Signed-off-by: Johannes Schindelin

Merge branch 'ab/no-more-git-global-super-prefix'

2023-01-05T06:07:23Z

Stop using "git --super-prefix" and narrow the scope of its use to the submodule--helper. * ab/no-more-git-global-super-prefix: read-tree: add "--super-prefix" option, eliminate global submodule--helper: convert "{update,clone}" to their own "--super-prefix" submodule--helper: convert "status" to its own "--super-prefix" submodule--helper: convert "sync" to its own "--super-prefix" submodule--helper: convert "foreach" to its own "--super-prefix" submodule--helper: don't use global --super-prefix in "absorbgitdirs" submodule.c & submodule--helper: pass along "super_prefix" param read-tree + fetch tests: test failing "--super-prefix" interaction submodule absorbgitdirs tests: add missing "Migrating git..." tests

read-tree: add "--super-prefix" option, eliminate global

2022-12-26T01:21:44Z

The "--super-prefix" option to "git" was initially added in [1] for use with "ls-files"[2], and shortly thereafter "submodule--helper"[3] and "grep"[4]. It wasn't until [5] that "read-tree" made use of it. At the time [5] made sense, but since then we've made "ls-files" recurse in-process in [6], "grep" in [7], and finally "submodule--helper" in the preceding commits. Let's also remove it from "read-tree", which allows us to remove the option to "git" itself. We can do this because the only remaining user of it is the submodule API, which will now invoke "read-tree" with its new "--super-prefix" option. It will only do so when the "submodule_move_head()" function is called. That "submodule_move_head()" function was then only invoked by "read-tree" itself, but now rather than setting an environment variable to pass "--super-prefix" between cmd_read_tree() we: - Set a new "super_prefix" in "struct unpack_trees_options". The "super_prefixed()" function in "unpack-trees.c" added in [5] will now use this, rather than get_super_prefix() looking up the environment variable we set earlier in the same process. - Add the same field to the "struct checkout", which is only needed to ferry the "super_prefix" in the "struct unpack_trees_options" all the way down to the "entry.c" callers of "submodule_move_head()". Those calls which used the super prefix all originated in "cmd_read_tree()". The only other caller is the "unlink_entry()" caller in "builtin/checkout.c", which now passes a "NULL". 1. 74866d75793 (git: make super-prefix option, 2016-10-07) 2. e77aa336f11 (ls-files: optionally recurse into submodules, 2016-10-07) 3. 89c86265576 (submodule helper: support super prefix, 2016-12-08) 4. 0281e487fd9 (grep: optionally recurse into submodules, 2016-12-16) 5. 3d415425c7b (unpack-trees: support super-prefix option, 2017-01-17) 6. 188dce131fa (ls-files: use repository object, 2017-06-22) 7. f9ee2fcdfa0 (grep: recurse in-process using 'struct repository', 2017-08-02) Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

submodule--helper: don't use global --super-prefix in "absorbgitdirs"

2022-12-26T01:21:43Z

The "--super-prefix" facility was introduced in [1] has always been a transitory hack, which is why we've made it an error to supply it as an option to "git" to commands that don't know about it. That's been a good goal, as it has a global effect we haven't wanted calls to get_super_prefix() from built-ins we didn't expect. But it has meant that when we've had chains of different built-ins using it all of the processes in that "chain" have needed to support it, and worse processes that don't need it have needed to ask for "SUPPORT_SUPER_PREFIX" because their parent process needs it. That's how "fsmonitor--daemon" ended up with it, per [2] it's called from (among other things) "submodule--helper absorbgitdirs", but as we declared "submodule--helper" as "SUPPORT_SUPER_PREFIX" we needed to declare "fsmonitor--daemon" as accepting it too, even though it doesn't care about it. But in the case of "absorbgitdirs" it only needed "--super-prefix" to invoke itself recursively, and we'd never have another "in-between" process in the chain. So we didn't need the bigger hammer of "git --super-prefix", and the "setenv(GIT_SUPER_PREFIX_ENVIRONMENT, ...)" that it entails. Let's instead accept a hidden "--super-prefix" option to "submodule--helper absorbgitdirs" itself. Eventually (as with all other "--super-prefix" users) we'll want to clean this code up so that this all happens in-process. I.e. needing any variant of "--super-prefix" is itself a hack around our various global state, and implicit reliance on "the_repository". This stepping stone makes such an eventual change easier, as we'll need to deal with less global state at that point. The "fsmonitor--daemon" test adjusted here was added in [3]. To assert that it didn't run into the "--super-prefix" message it was asserting the output it didn't have. Let's instead assert the full output that we *do* have, using the same pattern as a preceding change to "t/t7412-submodule-absorbgitdirs.sh" used. We could also remove the test entirely (as [4] did), but even though the initial reason for having it is gone we're still getting some marginal benefit from testing the "fsmonitor" and "submodule absorbgitdirs" interaction, so let's keep it. The change here to have either a NULL or non-"" string as a "super_prefix" instead of the previous arrangement of "" or non-"" is somewhat arbitrary. We could also decide to never have to check for NULL. As we'll be changing the rest of the "git --super-prefix" users to the same pattern, leaving them all consistent makes sense. Why not pick "" over NULL? Because that's how the "prefix" works[5], and having "prefix" and "super_prefix" work the same way will be less confusing. That "prefix" picked NULL instead of "" is itself arbitrary, but as it's easy to make this small bit of our overall API consistent, let's go with that. 1. 74866d75793 (git: make super-prefix option, 2016-10-07) 2. 53fcfbc84f6 (fsmonitor--daemon: allow --super-prefix argument, 2022-05-26) 3. 53fcfbc84f6 (fsmonitor--daemon: allow --super-prefix argument, 2022-05-26) 4. https://lore.kernel.org/git/20221109004708.97668-5-chooglen@google.com/ 5. 9725c8dda20 (built-ins: trust the "prefix" from run_builtin(), 2022-02-16) Signed-off-by: Glen Choo Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

submodule.c & submodule--helper: pass along "super_prefix" param

2022-12-26T01:21:43Z

Start passing the "super_prefix" along as a parameter to get_submodule_displaypath() and absorb_git_dir_into_superproject(), rather than get the value directly as a global. This is in preparation for subsequent commits, where we'll gradually phase out get_super_prefix() for an alternative way of getting the "super_prefix". Most of the users of this get a get_super_prefix() value, either directly or by indirection. The exceptions are: - builtin/rm.c: Doesn't declare SUPPORT_SUPER_PREFIX, so we'd have died if this was provided, so it's safe to pass "NULL". - deinit_submodule(): The "deinit_submodule()" function has never been able to use the "git -super-prefix". It will call "absorb_git_dir_into_superproject()", but it will only do so from the top-level project. If "absorbgitdirs" recurses will use the "path" passed to "absorb_git_dir_into_superproject()" in "deinit_submodule()" as its starting "--super-prefix". So we can safely remove the get_super_prefix() call here, and pass NULL instead. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

diff: mark unused parameters in callbacks

2022-12-13T13:16:23Z

The diff code provides a format_callback interface, but not every callback needs each parameter (e.g., the "opt" and "data" parameters are frequently left unused). Likewise for the output_prefix callback, the low-level change/add_remove interfaces, the callbacks used by xdi_diff(), etc. Mark unused arguments in the callback implementations to quiet -Wunused-parameter. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'jt/submodule-on-demand'

2022-11-23T02:22:25Z

Push all submodules recursively with '--recurse-submodules=on-demand'. * jt/submodule-on-demand: Doc: document push.recurseSubmodules=only

Doc: document push.recurseSubmodules=only

2022-11-14T21:55:50Z

Git learned pushing submodules without pushing the superproject by the user specifying --recurse-submodules=only through 6c656c3fe4 ("submodules: add RECURSE_SUBMODULES_ONLY value", 2016-12-20) and 225e8bf778 ("push: add option to push only submodules", 2016-12-20). For users who use this feature regularly, it is desirable to have an equivalent configuration. It turns out that such a configuration (push.recurseSubmodules=only) is already supported, even though it is neither documented nor mentioned in the commit messages, due to the way the --recurse-submodules=only feature was implemented (a function used to parse --recurse-submodules was updated to support "only", but that same function is used to parse push.recurseSubmodules too). What is left is to document it and test it, which is what this commit does. There is a possible point of confusion when recursing into a submodule that itself has the push.recurseSubmodules=only configuration, because if a repository has only its submodules pushed and not itself, its superproject can never be pushed. Therefore, treat such configurations as being "on-demand", and print a warning message. Signed-off-by: Jonathan Tan Signed-off-by: Taylor Blau