git/ci, branch v2.20.2

Merge branch 'ab/dynamic-gettext-poison'

2018-11-19T07:24:39Z

Our testing framework uses a special i18n "poisoned localization" feature to find messages that ought to stay constant but are incorrectly marked to be translated. This feature has been made into a runtime option (it used to be a compile-time option). * ab/dynamic-gettext-poison: Makefile: ease dynamic-gettext-poison transition i18n: make GETTEXT_POISON a runtime option

Merge branch 'sg/travis-install-dependencies'

2018-11-13T13:37:27Z

The procedure to install dependencies before testing at Travis CI is getting revamped for both simplicity and flexibility, taking advantage of the recent move to the vm-based environment. * sg/travis-install-dependencies: travis-ci: install packages in 'ci/install-dependencies.sh'

i18n: make GETTEXT_POISON a runtime option

2018-11-09T02:25:19Z

Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON test parameter to only be a GIT_TEST_GETTEXT_POISON= runtime parameter, to be consistent with other parameters documented in "Running tests with special setups" in t/README. When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON to simulate unfriendly translator", 2011-02-22) I was concerned with ensuring that the _() function would get constant folded if NO_GETTEXT was defined, and likewise that GETTEXT_POISON would be compiled out unless it was defined. But as the benchmark in my [1] shows doing a one-off runtime getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was originally added the GIT_TEST_* env variables have become the common idiom for turning on special test setups. So change GETTEXT_POISON to work the same way. Now the GETTEXT_POISON=YesPlease compile-time option is gone, and running the tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off without recompiling. This allows for conditionally amending tests to test with/without poison, similar to what 859fdc0c3c ("commit-graph: define GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do some of that, now we e.g. always run the t0205-gettext-poison.sh test. I did enough there to remove the GETTEXT_POISON prerequisite, but its inverse C_LOCALE_OUTPUT is still around, and surely some tests using it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=. Notes on the implementation: * We still compile a dedicated GETTEXT_POISON build in Travis CI. Perhaps this should be revisited and integrated into the "linux-gcc" build, see ae59a4e44f ("travis: run tests with GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then again maybe not, see [2]. * We now skip a test in t0000-basic.sh under GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This test relies on C locale output, but due to an edge case in how the previous implementation of GETTEXT_POISON worked (reading it from GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does, and needs to be skipped. * The getenv() function is not reentrant, so out of paranoia about code of the form: printf(_("%s"), getenv("some-env")); call use_gettext_poison() in our early setup in git_setup_gettext() so we populate the "poison_requested" variable in a codepath that's won't suffer from that race condition. * We error out in the Makefile if you're still saying GETTEXT_POISON=YesPlease to prompt users to change their invocation. * We should not print out poisoned messages during the test initialization itself to keep it more readable, so the test library hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during setup. See [3]. See also [4] for more on the motivation behind this patch, and the history of the GETTEXT_POISON facility. 1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/ 2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/ 3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/ 4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

travis-ci: install packages in 'ci/install-dependencies.sh'

2018-11-02T02:28:19Z

Ever since we started using Travis CI, we specified the list of packages to install in '.travis.yml' via the APT addon. While running our builds on Travis CI's container-based infrastructure we didn't have another choice, because that environment didn't support 'sudo', and thus we didn't have permission to install packages ourselves. With the switch to the VM-based infrastructure in the previous patch we do get a working 'sudo', so we can install packages by running 'sudo apt-get -y install ...' as well. Let's make use of this and install necessary packages in 'ci/install-dependencies.sh', so all the dependencies (i.e. both packages and "non-packages" (P4 and Git-LFS)) are handled in the same file. Install gcc-8 only in the 'linux-gcc' build job; so far it has been unnecessarily installed in the 'linux-clang' build job as well. Print the versions of P4 and Git-LFS conditionally, i.e. only when they have been installed; with this change even the static analysis and documentation build jobs start using 'ci/install-dependencies.sh' to install packages, and neither of these two build jobs depend on and thus install those. This change will presumably be beneficial for the upcoming Azure Pipelines integration [1]: preliminary versions of that patch series run a couple of 'apt-get' commands to install the necessary packages before running 'ci/install-dependencies.sh', but with this patch it will be sufficient to run only 'ci/install-dependencies.sh'. [1] https://public-inbox.org/git/1a22efe849d6da79f2c639c62a1483361a130238.1539598316.git.gitgitgadget@gmail.com/ Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

ci: add optional test variables

2018-10-19T00:21:28Z

The commit-graph and multi-pack-index features introduce optional data structures that are not required for normal Git operations. It is important to run the normal test suite without them enabled, but it is helpful to also run the test suite using them. Our continuous integration scripts include a second test stage that runs with optional GIT_TEST_* variables enabled. Add the following two variables to that stage: GIT_TEST_COMMIT_GRAPH GIT_TEST_MULTI_PACK_INDEX This will slow down the operation, as we build a commit-graph file after every 'git commit' operation and build a multi-pack-index during every 'git repack' operation. However, it is important that future changes are compatible with these features. Signed-off-by: Derrick Stolee Signed-off-by: Junio C Hamano

Merge branch 'nd/pack-deltify-regression-fix'

2018-08-22T18:17:05Z

In a recent update in 2.18 era, "git pack-objects" started producing a larger than necessary packfiles by missing opportunities to use large deltas. * nd/pack-deltify-regression-fix: pack-objects: fix performance issues on packing large deltas

Merge branch 'sg/travis-retrieve-trash-upon-failure'

2018-08-15T22:08:28Z

The Travis CI scripts were taught to ship back the test data from failed tests. * sg/travis-retrieve-trash-upon-failure: travis-ci: include the trash directories of failed tests in the trace log

travis-ci: include the trash directories of failed tests in the trace log

2018-08-01T16:59:36Z

The trash directory of a failed test might contain invaluable information about the cause of the failure, but we have no access to the trash directories of Travis CI build jobs. The only feedback we get from there is the build job's trace log, so... Modify 'ci/print-test-failures.sh' to create a tar.gz archive of the trash directory of each failed test, encode that archive with base64, and print the resulting block of ASCII text, so it gets embedded in the trace log. Furthermore, run tests with '--immediate' to faithfully preserve the failed state. Extracting the trash directories from the trace log turned out to be a bit of a hassle, partly because of the size of these logs (usually resulting in several hundreds or even thousands of lines of base64-encoded text), and partly because these logs have CRLF, CRCRLF and occasionally even CRCRCRLF line endings, which cause 'base64 -d' from coreutils to complain about "invalid input". For convenience add a small script 'ci/util/extract-trash-dirs.sh', which will extract and unpack all base64-encoded trash directories embedded in the log fed to its standard input, and include an example command to be copy-pasted into a terminal to do it all at the end of the failure report. A few of our tests create sizeable trash directories, so limit the size of each included base64-encoded block, let's say, to 1MB. And just in case something fundamental gets broken and a lot of tests fail at once, don't include trash directories when the combined size of the included base64-encoded blocks would exceed 1MB. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

travis-ci: fail if Coccinelle static analysis found something to transform

2018-07-23T19:08:38Z

Coccinelle's and in turn 'make coccicheck's exit code only indicates that Coccinelle managed to finish its analysis without any errors (e.g. no unknown --options, no missing files, no syntax errors in the semantic patches, etc.), but it doesn't indicate whether it found any undesired code patterns to transform or not. To find out the latter, one has to look closer at 'make coccicheck's standard output and look for lines like: SPATCH result: contrib/coccinelle/.cocci.patch And this only indicates that there is something to transform, but to see what the suggested transformations are one has to actually look into those '*.cocci.patch' files. This makes the automated static analysis build job on Travis CI not particularly useful, because it neither draws our attention to Coccinelle's findings, nor shows the actual findings. Consequently, new topics introducing undesired code patterns graduated to master on several occasions without anyone noticing. The only way to draw attention in such an automated setting is to fail the build job. Therefore, modify the 'ci/run-static-analysis.sh' build script to check all the resulting '*.cocci.patch' files, and fail the build job if any of them turns out to be not empty. Include those files' contents, i.e. Coccinelle's suggested transformations, in the build job's trace log, so we'll know why it failed. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano

travis-ci: run Coccinelle static analysis with two parallel jobs

2018-07-23T19:08:36Z

Currently the static analysis build job runs Coccinelle using a single 'make' job. Using two parallel jobs cuts down the build job's run time from around 10-12mins to 6-7mins, sometimes even under 6mins (there is quite large variation between build job runtimes). More than two parallel jobs don't seem to bring further runtime benefits. Signed-off-by: SZEDER Gábor Signed-off-by: Junio C Hamano