git/t/t0000-basic.sh, branch v2.36.2

test-lib: have --immediate emit valid TAP on failure

2022-03-24T21:47:02Z

Change the "--immediate" option so that it emits valid TAP on failure. Before this it would omit the required plan at the end, e.g. under SANITIZE=leak we'd show a "No plan found in TAP output" error from "prove": $ prove t0006-date.sh :: --immediate t0006-date.sh .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/22 subtests Test Summary Report ------------------- t0006-date.sh (Wstat: 256 Tests: 22 Failed: 1) Failed test: 22 Non-zero exit status: 1 Parse errors: No plan found in TAP output Files=1, Tests=22, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.18 cusr 0.06 csys = 0.27 CPU) Result: FAIL Now we'll emit output that doesn't result in TAP parsing failures: $ prove t0006-date.sh :: --immediate t0006-date.sh .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/22 subtests Test Summary Report ------------------- t0006-date.sh (Wstat: 256 Tests: 22 Failed: 1) Failed test: 22 Non-zero exit status: 1 Files=1, Tests=22, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.19 cusr 0.05 csys = 0.26 CPU) Result: FAIL Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

t0001-t0028: avoid pipes with Git on LHS

2022-03-13T00:22:01Z

Pipes ignore error codes of LHS command and thus we should not use them with Git in tests. As an alternative, use a 'tmp' file to write the Git output so we can test the exit code. Signed-off-by: Shubham Mishra Signed-off-by: Junio C Hamano

Merge branch 'ab/lib-subtest'

2021-10-13T22:15:57Z

Updates to the tests in t0000 to test the test framework. * ab/lib-subtest: test-lib tests: get rid of copy/pasted mock test code test-lib tests: assert 1 exit code, not non-zero test-lib tests: refactor common part of check_sub_test_lib_test*() test-lib tests: avoid subshell for "test_cmp" for readability test-lib tests: don't provide a description for the sub-tests test-lib tests: split up "write and run" into two functions test-lib tests: move "run_sub_test" to a new lib-subtest.sh

test-lib tests: get rid of copy/pasted mock test code

2021-09-22T23:22:41Z

Now that we've split up the write_sub_test_lib_test*() and run_sub_test_lib_test*() functions let's fix those tests in t0000-basic.sh that were verbosely copy/pasting earlier tests. That we caught all of them was asserted with a follow-up change that's not part of this series[1], we might add such a duplication check at some later time, but for now let's just one-off remove the duplicate boilerplate. 1. https://lore.kernel.org/git/patch-v3-6.9-bc79b29f3c-20210805T103237Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

test-lib tests: don't provide a description for the sub-tests

2021-09-22T23:22:41Z

Change the $test_description provided for the generated subtests to be constant, since the only purpose of having it is that test-lib.sh will barf if it isn't supplied. The other purpose of having it was to effectively split up the test description between the argument to test_expect_success and the argument to "write_and_run_sub_test_lib_test". Let's only use one of the two. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

test-lib tests: split up "write and run" into two functions

2021-09-22T23:22:41Z

Refactor the function to write and run tests of the test-lib.sh output into two functions. When this was added back in 565b6fa87bb (tests: refactor mechanics of testing in a sub test-lib, 2012-12-16) there was no reason to do this, but since we started supporting test arguments in 517cd55fd51 (test-lib: self-test that --verbose works, 2013-06-23) we've started to write out duplicate tests simply to test different arguments, now we'll be able to re-use them. This change doesn't consolidate any of those tests yet, it just makes it possible to do so. All the changes in t0000-basic.sh are a simple search-replacement. Since the _run_sub_test_lib_test_common() function doesn't handle running the test anymore we can do away with the sub-shell, which was used to scope an "unset" and "export" shell variables. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

t0000: avoid masking git exit value through pipes

2021-09-16T20:43:42Z

9af0b8dbe2 (t0000-basic: more commit-tree tests., 2006-04-26) adds tests for commit-tree that mask the return exit from git as described in a378fee5b07 (Documentation: add shell guidelines, 2018-10-05). Fix the tests, to avoid pipes by using a temporary file instead. Signed-off-by: Carlo Marcelo Arenas Belón Signed-off-by: Junio C Hamano

test-lib tests: move "run_sub_test" to a new lib-subtest.sh

2021-07-21T23:27:04Z

Move the "check_sub_test_lib_test()" and its sister functions to a new lib-subtest.sh. In the future (not in this series) I'd like to test test-lib's output in a more targeted and smaller test, and I'll need these functions to do that. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

t0000: fix test if run with TEST_OUTPUT_DIRECTORY

2021-07-20T16:19:02Z

Testcases in t0000 are quite special given that they many of them run nested testcases to verify that testing functionality itself works as expected. These nested testcases are realized by writing a new ad-hoc test script which again sources test-lib.sh, where the new script is created in a nested subdirectory located beneath the current trash directory. We then execute the new test script with the nested subdirectory as current working directory and explicitly re-export TEST_OUTPUT_DIRECTORY to point to that directory. While this works as expected in the general case, it falls apart when the developer has TEST_OUTPUT_DIRECTORY explicitly defined either via the environment or via config.mak and runs "make test". In that case, test-lib.sh will clobber the value that we've just carefully set up to instead contain what the developer has defined. As a result, the TEST_OUTPUT_DIRECTORY continues to point at the root output directory, not at the nested one. This issue causes breakage in the 'test_atexit is run' test case: the nested test case writes files into "../../", which is assumed to be the parent's trash directory. But because TEST_OUTPUT_DIRECTORY already points to to the root output directory, we instead end up writing those files outside of the output directory. The parent test case will then try to check whether those files still exist in its own trash directory, which thus must fail now. Fix the issue by adding a new TEST_OUTPUT_DIRECTORY_OVERRIDE variable. If set, then we'll always override the TEST_OUTPUT_DIRECTORY with its value after sourcing GIT-BUILD-OPTIONS. Signed-off-by: Patrick Steinhardt Acked-by: Jeff King Signed-off-by: Junio C Hamano

t0000: clear GIT_SKIP_TESTS before running sub-tests

2021-07-19T20:26:00Z

In t0000, we run several fake "sub-test" suites to verify the behavior of the test suite. But because we don't clear the parent environment completely, the sub-tests can be fooled by variables meant for the parent. For example: GIT_SKIP_TESTS=t1234 ./t0000-basic.sh fails when a sub-test expects its fake t1234 to actually run. This particular pattern is unlikely in practice; we're running a single script, and there is no t1234 in the real test suite anyway (not yet, at least). A more real-world example is: GIT_SKIP_TESTS=t[^0]* make test to run only the t0* tests. The fix is conceptually simple: we should clear the GIT_SKIP_TESTS variable when running the sub-tests, because its contents (if any) will be meant for the main test suite. This is easy to do centrally in our sub-test helper. But there's a catch: some of our tests do set GIT_SKIP_TESTS intentionally to test the feature. We need to allow them to continue to set it, but clear it for all the other tests. And the sub-test helper can't tell if the GIT_SKIP_TESTS it sees is from a test or not. We can handle this by adding a new option to the helper to let callers specify the skip list. I considered adding a more general "--eval" option to let callers set up the env for the sub-test however they like. That would cover this case and possible future ones. But the quoting gets awkward for the callers (since we're now 2 layers deep in evals!), so I went with the simpler more specific solution. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano