linux/tools/testing/selftests/lib.mk, branch v4.14

selftests: lib.mk: print individual test results to console by default

2017-10-31T19:24:00Z

Change run_tests to print individual test results to console by default. Introduce "summary" option to print individual test results to a file /tmp/test_name and just print the summary to the console. This change is necessary to support use-cases where test machines get rebooted once tests are run and the console log should contain the full results. In the following example, individual test results with "summary=1" option are written to /tmp/kcmp_test make --silent TARGETS=kcmp kselftest TAP version 13 selftests: kcmp_test ======================================== pid1: 30126 pid2: 30127 FD: 2 FILES: 2 VM: 1 FS: 2 SIGHAND: 2 IO: 0 SYSVSEM: 0 INV: -1 PASS: 0 returned as expected PASS: 0 returned as expected FAIL: 0 expected but -1 returned (Invalid argument) Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0 1..3 Bail out! Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0 1..3 Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0 1..0 ok 1..1 selftests: kcmp_test [PASS] make --silent TARGETS=kcmp summary=1 kselftest TAP version 13 selftests: kcmp_test ======================================== ok 1..1 selftests: kcmp_test [PASS] Signed-off-by: Shuah Khan

selftests: lib.mk: copy test scripts and test files for make O=dir run

2017-09-21T13:55:40Z

For make O=dir run_tests to work, test scripts, test files, and other dependencies need to be copied over to the object directory. Running tests from the object directory is necessary to avoid making the source tree dirty. Signed-off-by: Shuah Khan

selftests: lib.mk: add TEST_CUSTOM_PROGS to allow custom test run/install

2017-09-21T13:55:39Z

Some tests such as sync can't use generic build rules in lib.mk and require custom rules. Currently there is no provision to allow custom builds and test such as sync use TEST_PROGS which is reserved for test shell scripts. Add TEST_CUSTOM_PROGS variable to lib.mk to run and install custom tests built by individual test make files. Signed-off-by: Shuah Khan

selftests: lib.mk: fix test executable status check to use full path

2017-09-21T13:55:37Z

Fix test executable status check to use full path for make O=dir case,m when tests are relocated to user specified object directory. Without the full path, this check fails to find the file and fails the test. Signed-off-by: Shuah Khan

selftests: lib.mk: kselftest and kselftest-clean fail for make O=dir case

2017-09-21T13:55:36Z

kselftest and kselftest-clean targets fail when object directory is specified to relocate objects. Main Makefile make O= path clears the built-in defines LINK.c, COMPILE.S, LINK.S, and RM that are used in lib.mk to build and clean targets. Define them. Signed-off-by: Shuah Khan

selftests: silence test output by default

2017-09-19T20:08:50Z

Some of the networking tests are very noisy and make it impossible to see if we actually passed the tests as they run. Default to suppressing the output from any tests run in order to make it easier to track what failed. Signed-off-by: Josef Bacik Signed-off-by: Shuah Khan

selftests: lib.mk: change RUN_TESTS to print messages in TAP13 format

2017-08-25T16:17:12Z

Change common RUN_TESTS to print messages in user friendly TAP13 format. This change add TAP13 header at the start of RUN_TESTS target run, and prints the resulting pass/fail messages with test number information in the TAP 13 format for each test in the run tests list. This change covers test scripts as well as test programs. Test programs have an option to use ksft_ API, however test scripts won't be able to. With this change, test scripts can print TAP13 format output without any changes to individual scripts. Test programs can provide TAP13 format output as needed as some tests already do. Tests that haven't been converted will benefit from this change. Tests that are converted benefit from the test counts for all the tests in each test directory. Running firmware tests: make --silent -C tools/testing/selftests/firmware/ run_tests Before the change: modprobe: ERROR: could not insert 'test_firmware': Operation not permitted ./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present You must have the following enabled in your kernel: CONFIG_TEST_FIRMWARE=y selftests: fw_filesystem.sh [FAIL] modprobe: ERROR: could not insert 'test_firmware': Operation not permitted selftests: fw_fallback.sh [FAIL] After the change: TAP version 13 selftests: fw_filesystem.sh ======================================== modprobe: ERROR: could not insert 'test_firmware': Operation not permitted ./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present You must have the following enabled in your kernel: CONFIG_TEST_FIRMWARE=y not ok 1..1 selftests: fw_filesystem.sh [FAIL] selftests: fw_fallback.sh ======================================== modprobe: ERROR: could not insert 'test_firmware': Operation not permitted not ok 1..2 selftests: fw_fallback.sh [FAIL] Signed-off-by: Shuah Khan

selftests: change lib.mk RUN_TESTS to take test list as an argument

2017-08-25T16:15:24Z

Change lib.mk RUN_TESTS to take test list as an argument. This will allow it to be called from individual test makefiles to run additional tests that aren't suitable for a default kselftest run. As an example, timers test includes destructive tests that aren't included in the common run_tests target. Change times/Makefile to use RUN_TESTS call with destructive test list as an argument instead of using its own RUN_TESTS target. Signed-off-by: Shuah Khan

selftests: lib.mk: suppress "cd" output from run_tests target

2017-08-25T16:07:45Z

Suppress "cd" output from run_tests while running tests to declutter the test results. Running efivarfs test: make --silent -C tools/testing/selftests/efivarfs/ run_tests Before the change: skip all tests: must be run as root selftests: efivarfs.sh [PASS] /lkml/linux-kselftest/tools/testing/selftests/efivarfs After the change: skip all tests: must be run as root selftests: efivarfs.sh [PASS] Signed-off-by: Shuah Khan

selftests: warn if failure is due to lack of executable bit

2017-08-09T20:41:20Z

Executing selftests is fragile as if someone forgot to set a secript as executable the test will fail, and you won't know for sure if the failure was caused by the lack of proper permissions or something else. Setting scripts as executable is required, this also enable folks to execute selftests as independent units. Signed-off-by: Luis R. Rodriguez Signed-off-by: Shuah Khan