git/ci/lib.sh, branch v2.45.4

ci: fix check for Ubuntu 20.04

2024-06-06T16:59:27Z

In 5ca0c455f1 (ci: fix Python dependency on Ubuntu 24.04, 2024-05-06), we made the use of Python 2 conditional on whether or not the CI job runs Ubuntu 20.04. There was a brown-paper-bag-style bug though, where the condition forgot to invoke the `test` builtin. The result of it is that the check always fails, and thus all of our jobs run with Python 3 by accident. Fix this. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: fix Python dependency on Ubuntu 24.04

2024-05-06T19:26:46Z

Newer versions of Ubuntu have dropped Python 2 starting with Ubuntu 23.04. By default though, our CI setups will try to use that Python version on all Ubuntu-based jobs except for the "linux-gcc" one. We didn't notice this issue due to two reasons: - The "ubuntu:latest" tag always points to the latest LTS release. Until a few weeks ago this was Ubuntu 22.04, which still had Python 2. - Our Docker-based CI jobs had their own script to install dependencies until 9cdeb34b96 (ci: merge scripts which install dependencies, 2024-04-12), where we didn't even try to install Python at all for many of them. Since the CI refactorings have originally been implemented, Ubuntu 24.04 was released, and it being an LTS versions means that the "latest" tag now points to that Python-2-less version. Consequently, those jobs that use "ubuntu:latest" broke. Address this by using Python 2 on Ubuntu 20.04, only, whereas we use Python 3 on all other Ubuntu jobs. Eventually, we should think about dropping support for Python 2 completely. Reported-by: Justin Tobler Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: add jobs to test with the reftable backend

2024-02-07T16:28:37Z

Add CI jobs for both GitHub Workflows and GitLab CI to run Git with the new reftable backend. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: add macOS jobs to GitLab CI

2024-01-18T19:53:17Z

Add a job to GitLab CI which runs tests on macOS, which matches the equivalent "osx-clang" job that we have for GitHub Workflows. One significant difference though is that this new job runs on Apple M1 machines and thus uses the "arm64" architecture. As GCC does not yet support this comparatively new architecture we cannot easily include an equivalent for the "osx-gcc" job that exists in GitHub Workflows. Note that one test marked as `test_must_fail` is surprisingly passing: t7815-grep-binary.sh (Wstat: 0 Tests: 22 Failed: 0) TODO passed: 12 This seems to boil down to an unexpected difference in how regcomp(3P) works when matching NUL bytes. Cross-checking with the respective GitHub job shows that this is not an issue unique to the GitLab CI job as it passes in the same way there. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: make p4 setup on macOS more robust

2024-01-18T19:53:17Z

When setting up Perforce on macOS we put both `p4` and `p4d` into "$HOME/bin". On GitHub CI this directory is indeed contained in the PATH environment variable and thus there is no need for additional setup than to put the binaries there. But GitLab CI does not do this, and thus our Perforce-based tests would be skipped there even though we download the binaries. Refactor the setup code to become more robust by downloading binaries into a separate directory which we then manually append to our PATH. This matches what we do on Linux-based jobs. Note that it may seem like we already did append "$HOME/bin" to PATH because we're actually removing the lines that adapt PATH. But we only ever adapted the PATH variable in "ci/install-dependencies.sh", and didn't adapt it when running "ci/run-build-and-test.sh". Consequently, the required binaries wouldn't be found during the test run unless the CI platform already had the "$HOME/bin" in PATH right from the start. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

Merge branch 'js/ci-discard-prove-state'

2023-12-10T00:37:48Z

The way CI testing used "prove" could lead to running the test suite twice needlessly, which has been corrected. * js/ci-discard-prove-state: ci: avoid running the test suite _twice_

Merge branch 'ps/ci-gitlab'

2023-12-10T00:37:48Z

Add support for GitLab CI. * ps/ci-gitlab: ci: add support for GitLab CI ci: install test dependencies for linux-musl ci: squelch warnings when testing with unusable Git repo ci: unify setup of some environment variables ci: split out logic to set up failed test artifacts ci: group installation of Docker dependencies ci: make grouping setup more generic ci: reorder definitions for grouping functions

ci: avoid running the test suite _twice_

2023-11-14T00:24:23Z

This is a late amendment of 4a6e4b960263 (CI: remove Travis CI support, 2021-11-23), whereby the `.prove` file (being written by the `prove` command that is used to run the test suite) is no longer retained between CI builds: This feature was only ever used in the Travis CI builds, we tried for a while to do the same in Azure Pipelines CI runs (but I gave up on it after a while), and we never used that feature in GitHub Actions (nor does the new GitLab CI code use it). Retaining the Prove cache has been fragile from the start, even though the idea seemed good at the time, the idea being that the `.prove` file caches information about previous `prove` runs (`save`) and uses them (`slow`) to run the tests in the order from longer-running to shorter ones, making optimal use of the parallelism implied by `--jobs=`. However, using a Prove cache can cause some surprising behavior: When the `prove` caches information about a test script it has run, subsequent `prove` runs (with `--state=slow`) will run the same test script again even if said script is not specified on the `prove` command-line! So far, this bug did not matter. Right until d8f416bbb87c (ci: run unit tests in CI, 2023-11-09) did it not matter. But starting with that commit, we invoke `prove` _twice_ in CI, once to run the regular test suite of regression test scripts, and once to run the unit tests. Due to the bug, the second invocation re-runs all of the tests that were already run as part of the first invocation. This not only wastes build minutes, it also frequently causes the `osx-*` jobs to fail because they already take a long time and now are likely to run into a timeout. The worst part about it is that there is actually no benefit to keep running with `--state=slow,save`, ever since we decided no longer to try to reuse the Prove cache between CI runs. So let's just drop that Prove option and live happily ever after. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano

ci: add support for GitLab CI

2023-11-09T09:56:10Z

We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly. Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer. Results of these pipeline runs will be made available (at least) in GitLab's mirror of the Git project at [1]. This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up. [1]: https://gitlab.com/gitlab-org/git Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano

ci: squelch warnings when testing with unusable Git repo

2023-11-09T09:56:10Z

Our CI jobs that run on Docker also use mostly the same architecture to build and test Git via the "ci/run-build-and-tests.sh" script. These scripts also provide some functionality to massage the Git repository we're supposedly operating in. In our Docker-based infrastructure we may not even have a Git repository available though, which leads to warnings when those functions execute. Make the helpers exit gracefully in case either there is no Git in our PATH, or when not running in a Git repository. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano