git/ci, branch v2.43.2

Merge branch 'js/ci-discard-prove-state' into maint-2.43

2024-02-09T00:22:00Z

The way CI testing used "prove" could lead to running the test suite twice needlessly, which has been corrected. * js/ci-discard-prove-state: ci: avoid running the test suite _twice_ ci: add support for GitLab CI ci: install test dependencies for linux-musl ci: squelch warnings when testing with unusable Git repo ci: unify setup of some environment variables ci: split out logic to set up failed test artifacts ci: group installation of Docker dependencies ci: make grouping setup more generic ci: reorder definitions for grouping functions

ci: avoid running the test suite _twice_

2023-11-14T00:24:23Z

This is a late amendment of 4a6e4b960263 (CI: remove Travis CI support, 2021-11-23), whereby the `.prove` file (being written by the `prove` command that is used to run the test suite) is no longer retained between CI builds: This feature was only ever used in the Travis CI builds, we tried for a while to do the same in Azure Pipelines CI runs (but I gave up on it after a while), and we never used that feature in GitHub Actions (nor does the new GitLab CI code use it). Retaining the Prove cache has been fragile from the start, even though the idea seemed good at the time, the idea being that the `.prove` file caches information about previous `prove` runs (`save`) and uses them (`slow`) to run the tests in the order from longer-running to shorter ones, making optimal use of the parallelism implied by `--jobs=`. However, using a Prove cache can cause some surprising behavior: When the `prove` caches information about a test script it has run, subsequent `prove` runs (with `--state=slow`) will run the same test script again even if said script is not specified on the `prove` command-line! So far, this bug did not matter. Right until d8f416bbb87c (ci: run unit tests in CI, 2023-11-09) did it not matter. But starting with that commit, we invoke `prove` _twice_ in CI, once to run the regular test suite of regression test scripts, and once to run the unit tests. Due to the bug, the second invocation re-runs all of the tests that were already run as part of the first invocation. This not only wastes build minutes, it also frequently causes the `osx-*` jobs to fail because they already take a long time and now are likely to run into a timeout. The worst part about it is that there is actually no benefit to keep running with `--state=slow,save`, ever since we decided no longer to try to reuse the Prove cache between CI runs. So let's just drop that Prove option and live happily ever after. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

ci: add support for GitLab CI

2023-11-09T09:56:10Z

We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. Results of these pipeline runs will be made available (at least) in GitLab's mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: install test dependencies for linux-musl

2023-11-09T09:56:10Z

The linux-musl CI job executes tests on Alpine Linux, which is based on musl libc instead of glibc. We're missing some test dependencies though, which causes us to skip a subset of tests. Install these test dependencies to increase our test coverage on this platform. There are still some missing test dependecies, but these do not have a corresponding package in the Alpine repositories: - p4 and p4d, both parts of the Perforce version control system. - cvsps, which generates patch sets for CVS. - Subversion and the SVN::Core Perl library, the latter of which is not available in the Alpine repositories. While the tool itself is available, all Subversion-related tests are skipped without the SVN::Core Perl library anyway. The Apache2-based tests require a bit more care though. For one, the module path is different on Alpine Linux, which requires us to add it to the list of known module paths to detect it. But second, the WebDAV module on Alpine Linux is broken because it does not bundle the default database backend [1]. We thus need to skip the WebDAV-based tests on Alpine Linux for now. [1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112 Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: squelch warnings when testing with unusable Git repo

2023-11-09T09:56:10Z

Our CI jobs that run on Docker also use mostly the same architecture to build and test Git via the "ci/run-build-and-tests.sh" script. These scripts also provide some functionality to massage the Git repository we're supposedly operating in. In our Docker-based infrastructure we may not even have a Git repository available though, which leads to warnings when those functions execute. Make the helpers exit gracefully in case either there is no Git in our PATH, or when not running in a Git repository. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: unify setup of some environment variables

2023-11-09T09:56:09Z

Both GitHub Actions and Azure Pipelines set up the environment variables GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are actually the same, the setup is completely duplicate. With the upcoming support for GitLab CI this duplication would only extend even further. Unify the setup of those environment variables so that only the uncommon parts are separated. While at it, we also perform some additional small improvements: - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS. It doesn't hurt on platforms where we don't persist the state, so this further reduces boilerplate. - When running on Windows systems we set `--no-chain-lint` and `--no-bin-wrappers`. Interestingly though, we did so _after_ already having exported the respective environment variables. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: split out logic to set up failed test artifacts

2023-11-09T09:56:09Z

We have some logic in place to create a directory with the output from failed tests, which will then subsequently be uploaded as CI artifacts. We're about to add support for GitLab CI, which will want to reuse the logic. Split the logic into a separate function so that it is reusable. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: group installation of Docker dependencies

2023-11-09T09:56:09Z

The output of CI jobs tends to be quite long-winded and hard to digest. To help with this, many CI systems provide the ability to group output into collapsible sections, and we're also doing this in some of our scripts. One notable omission is the script to install Docker dependencies. Address it to bring more structure to the output for Docker-based jobs. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: make grouping setup more generic

2023-11-09T09:56:09Z

Make the grouping setup more generic by always calling `begin_group ()` and `end_group ()` regardless of whether we have stubbed those functions or not. This ensures we can more readily add support for additional CI platforms. Furthermore, the `group ()` function is made generic so that it is the same for both GitHub Actions and for other platforms. There is a semantic conflict here though: GitHub Actions used to call `set +x` in `group ()` whereas the non-GitHub case unconditionally uses `set -x`. The latter would get overriden if we kept the `set +x` in the generic version of `group ()`. To resolve this conflict, we simply drop the `set +x` in the generic variant of this function. As `begin_group ()` calls `set -x` anyway this is not much of a change though, as the only commands that aren't printed anymore now are the ones between the beginning of `group ()` and the end of `begin_group ()`. Last, this commit changes `end_group ()` to also accept a parameter that indicates _which_ group should end. This will be required by a later commit that introduces support for GitLab CI. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: reorder definitions for grouping functions

2023-11-09T09:56:08Z

We define a set of grouping functions that are used to group together output in our CI, where these groups then end up as collapsible sections in the respective pipeline platform. The way these functions are defined is not easily extensible though as we have an up front check for the CI _not_ being GitHub Actions, where we define the non-stub logic in the else branch. Reorder the conditional branches such that we explicitly handle GitHub Actions. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano