linux/tools/testing/kunit, branch v5.14

kunit: tool: Assert the version requirement

2021-07-12T20:02:32Z

Commit 87c9c1631788 ("kunit: tool: add support for QEMU") on the 'next' tree adds 'from __future__ import annotations' in 'kunit_kernel.py'. Because it is supported on only >=3.7 Python, people using older Python will get below error: Traceback (most recent call last): File "./tools/testing/kunit/kunit.py", line 20, in import kunit_kernel File "/home/sjpark/linux/tools/testing/kunit/kunit_kernel.py", line 9 from __future__ import annotations ^ SyntaxError: future feature annotations is not defined This commit adds a version assertion in 'kunit.py', so that people get more explicit error message like below: Traceback (most recent call last): File "./tools/testing/kunit/kunit.py", line 15, in assert sys.version_info >= (3, 7), "Python version is too old" AssertionError: Python version is too old Signed-off-by: SeongJae Park Acked-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: remove unnecessary "annotations" import

2021-07-12T19:54:29Z

The import was working around the fact "tuple[T]" was used instead of typing.Tuple[T]. Convert it to use type.Tuple to be consistent with how the rest of the code is anotated. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Tested-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: Fix error messages for cases of no tests and wrong TAP header

2021-07-12T19:53:05Z

This patch addresses misleading error messages reported by kunit_tool in two cases. First, in the case of TAP output having an incorrect header format or missing a header, the parser used to output an error message of 'no tests run!'. Now the parser outputs an error message of 'could not parse test results!'. As an example: Before: $ ./tools/testing/kunit/kunit.py parse /dev/null [ERROR] no tests run! ... After: $ ./tools/testing/kunit/kunit.py parse /dev/null [ERROR] could not parse test results! ... Second, in the case of TAP output with the correct header but no tests, the parser used to output an error message of 'could not parse test results!'. Now the parser outputs an error message of 'no tests run!'. As an example: Before: $ echo -e 'TAP version 14\n1..0' | ./tools/testing/kunit/kunit.py parse [ERROR] could not parse test results! After: $ echo -e 'TAP version 14\n1..0' | ./tools/testing/kunit/kunit.py parse [ERROR] no tests run! Additionally, this patch also corrects the tests in kunit_tool_test.py and adds a test to check the error in the case of TAP output with the correct header but no tests. Signed-off-by: Rae Moar Reviewed-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: Support skipped tests in kunit_tool

2021-06-25T17:31:03Z

Add support for the SKIP directive to kunit_tool's TAP parser. Skipped tests now show up as such in the printed summary. The number of skipped tests is counted, and if all tests in a suite are skipped, the suite is also marked as skipped. Otherwise, skipped tests do affect the suite result. Example output: [00:22:34] ======== [SKIPPED] example_skip ======== [00:22:34] [SKIPPED] example_skip_test # SKIP this test should be skipped [00:22:34] [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped [00:22:34] ============================================================ [00:22:34] Testing complete. 2 tests run. 0 failed. 0 crashed. 2 skipped. Signed-off-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: internal refactor of parser input handling

2021-06-25T17:31:03Z

Note: this does not change the parser behavior at all (except for making one error message more useful). This is just an internal refactor. The TAP output parser currently operates over a List[str]. This works, but we only ever need to be able to "peek" at the current line and the ability to "pop" it off. Also, using a List means we need to wait for all the output before we can start parsing. While this is not an issue for most tests which are really lightweight, we do have some longer (~5 minutes) tests. This patch introduces an LineStream wrapper class that * Exposes a peek()/pop() interface instead of manipulating an array * this allows us to more easily add debugging code [1] * Can consume an input from a generator * we can now parse results as tests are running (the parser code currently doesn't print until the end, so no impact yet). * Tracks the current line number to print better error messages * Would allow us to add additional features more easily, e.g. storing N previous lines so we can print out invalid lines in context, etc. [1] The parsing logic is currently quite fragile. E.g. it'll often say the kernel "CRASHED" if there's something slightly wrong with the output format. When debugging a test that had some memory corruption issues, it resulted in very misleading errors from the parser. Now we could easily add this to trace all the lines consumed and why +import inspect ... def pop(self) -> str: n = self._next + print(f'popping {n[0]}: {n[1].ljust(40, " ")}| caller={inspect.stack()[1].function}') Example output: popping 77: TAP version 14 | caller=parse_tap_header popping 78: 1..1 | caller=parse_test_plan popping 79: # Subtest: kunit_executor_test | caller=parse_subtest_header popping 80: 1..2 | caller=parse_subtest_plan popping 81: ok 1 - parse_filter_test | caller=parse_ok_not_ok_test_case popping 82: ok 2 - filter_subsuite_test | caller=parse_ok_not_ok_test_case popping 83: ok 1 - kunit_executor_test | caller=parse_ok_not_ok_test_suite If we introduce an invalid line, we can see the parser go down the wrong path: popping 77: TAP version 14 | caller=parse_tap_header popping 78: 1..1 | caller=parse_test_plan popping 79: # Subtest: kunit_executor_test | caller=parse_subtest_header popping 80: 1..2 | caller=parse_subtest_plan popping 81: 1..2 # this is invalid! | caller=parse_ok_not_ok_test_case popping 82: ok 1 - parse_filter_test | caller=parse_ok_not_ok_test_case popping 83: ok 2 - filter_subsuite_test | caller=parse_ok_not_ok_test_case popping 84: ok 1 - kunit_executor_test | caller=parse_ok_not_ok_test_case [ERROR] ran out of lines before end token Signed-off-by: Daniel Latypov Reviewed-by: David Gow Acked-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: Remove the unused all_tests.config

2021-06-23T23:49:41Z

This isn't used anywhere. While it's possible that people were manually referencing it, the new default config (in default.config in the same path) provides equivalent functionality. Signed-off-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: Move default config from arch/um -> tools/testing/kunit

2021-06-23T23:49:17Z

The default .kunitconfig file is currently kept in arch/um/configs/kunit_defconfig, but -- with the impending QEMU patch -- will no-longer be exclusively used for UML-based kernels. Move it alongside the other KUnit configs in tools/testing/kunit/configs, and give it a name which matches the existing all_tests.config and broken_on_uml.config files. Also update the Getting Started documentation to point to the new file. Signed-off-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: add support for QEMU

2021-06-11T22:10:23Z

Add basic support to run QEMU via kunit_tool. Add support for i386, x86_64, arm, arm64, and a bunch more. Signed-off-by: Brendan Higgins Tested-by: David Gow Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: Add 'kunit_shutdown' option

2021-06-11T22:04:57Z

Add a new kernel command-line option, 'kunit_shutdown', which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. Signed-off-by: David Gow Signed-off-by: Brendan Higgins Reviewed-by: Stephen Boyd Tested-By: Daniel Latypov Reviewed-by: Daniel Latypov Signed-off-by: Shuah Khan

Merge tag 'linux-kselftest-kunit-5.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest

2021-04-28T01:56:29Z

Pull KUnit updates from Shuah Khan: "Several fixes and a new feature to support failure from dynamic analysis tools such as UBSAN and fake ops for testing. - a fake ops struct for testing a "free" function to complain if it was called with an invalid argument, or caught a double-free. Most return void and have no normal means of signalling failure (e.g. super_operations, iommu_ops, etc.)" * tag 'linux-kselftest-kunit-5.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest: Documentation: kunit: add tips for using current->kunit_test kunit: fix -Wunused-function warning for __kunit_fail_current_test kunit: support failure from dynamic analysis tools kunit: tool: make --kunitconfig accept dirs, add lib/kunit fragment kunit: make KUNIT_EXPECT_STREQ() quote values, don't print literals kunit: Match parenthesis alignment to improve code readability