linux/tools/testing/kunit, branch v6.0

kunit: tool: Enable virtio/PCI by default on UML

2022-07-08T17:22:29Z

There are several tests which depend on PCI, and hence need a bunch of extra options to run under UML. This makes it awkward to give configuration instructions (whether in documentation, or as part of a .kunitconfig file), as two separate, incompatible sets of config options are required for UML and "most other architectures". For non-UML architectures, it's possible to add default kconfig options via the qemu_config python files, but there's no equivalent for UML. Add a new tools/testing/kunit/configs/arch_uml.config file containing extra kconfig options to use on UML. Tested-by: José Expósito Reviewed-by: Daniel Latypov Signed-off-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Daniel Latypov Signed-off-by: Shuah Khan

kunit: tool: make --kunitconfig repeatable, blindly concat

2022-07-08T17:22:02Z

It's come up a few times that it would be useful to have --kunitconfig be repeatable [1][2]. This could be done before with a bit of shell-fu, e.g. $ find fs/ -name '.kunitconfig' -exec cat {} + | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin or equivalently: $ cat fs/ext4/.kunitconfig fs/fat/.kunitconfig | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin But this can be fairly clunky to use in practice. And having explicit support in kunit.py opens the door to having more config fragments of interest, e.g. options for PCI on UML [1], UML coverage [2], variants of tests [3]. There's another argument to be made that users can just use multiple --kconfig_add's, but this gets very clunky very fast (e.g. [2]). Note: there's a big caveat here that some kconfig options might be incompatible. We try to give a clearish error message in the simple case where the same option appears multiple times with conflicting values, but more subtle ones (e.g. mutually exclusive options) will be potentially very confusing for the user. I don't know we can do better. Note 2: if you want to combine a --kunitconfig with the default, you either have to do to specify the current build_dir > --kunitconfig=.kunit --kunitconfig=additional.config or > --kunitconfig=tools/testing/kunit/configs/default.config --kunitconifg=additional.config each of which have their downsides (former depends on --build_dir, doesn't work if you don't have a .kunitconfig yet), etc. Example with conflicting values: > $ ./tools/testing/kunit/kunit.py config --kunitconfig=lib/kunit --kunitconfig=/dev/stdin < CONFIG_KUNIT_TEST=n > CONFIG_KUNIT=m > EOF > ... > kunit_kernel.ConfigError: Multiple values specified for 2 options in kunitconfig: > CONFIG_KUNIT=y > vs from /dev/stdin > CONFIG_KUNIT=m > > CONFIG_KUNIT_TEST=y > vs from /dev/stdin > # CONFIG_KUNIT_TEST is not set [1] https://lists.freedesktop.org/archives/dri-devel/2022-June/357616.html [2] https://lore.kernel.org/linux-kselftest/CAFd5g45f3X3xF2vz2BkTHRqOC4uW6GZxtUUMaP5mwwbK8uNVtA@mail.gmail.com/ [3] https://lore.kernel.org/linux-kselftest/CANpmjNOdSy6DuO6CYZ4UxhGxqhjzx4tn0sJMbRqo2xRFv9kX6Q@mail.gmail.com/ Signed-off-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: add coverage_uml.config to enable GCOV on UML

2022-07-08T00:06:36Z

Now that kunit.py's --kunitconfig is repeatable, let's create a file to hold the various options needed to enable coverage under UML. This can be used like so: $ ./tools/testing/kunit/kunit.py run \ --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \ --make_options=CC=/usr/bin/gcc-6 which on my system is enough to get coverage working [1]. This is still a clunky command, but far better than before. [1] at the time of this commit, I get: Overall coverage rate: lines......: 11.6% (34112 of 295033 lines) functions..: 15.3% (3721 of 24368 functions) Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: refactor internal kconfig handling, allow overriding

2022-07-08T00:03:30Z

Currently, you cannot ovewrwrite what's in your kunitconfig via --kconfig_add. Nor can you override something in a qemu_config via either means. This patch makes it so we have this level of priority * --kconfig_add * kunitconfig file (the default or the one from --kunitconfig) * qemu_config The rationale for this order is that the more "dynamic" sources of kconfig options should take priority. --kconfig_add is obviously the most dynamic. And for kunitconfig, users probably tweak the file manually or specify --kunitconfig more often than they delve into qemu_config python files. And internally, we convert the kconfigs from a python list into a set or dict fairly often. We should just use a dict internally. We exposed the set transform in the past since we didn't define __eq__, so also take the chance to shore up the kunit_kconfig.Kconfig interface. Example ======= Let's consider the unrealistic example where someone would want to disable CONFIG_KUNIT. I.e. they run $ ./tools/testing/kunit/kunit.py config --kconfig_add=CONFIG_KUNIT=n Before ------ We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: # CONFIG_KUNIT is not set After ----- We'd write the following > # CONFIG_KUNIT is not set > CONFIG_KUNIT_TEST=y > CONFIG_KUNIT_ALL_TESTS=y > CONFIG_KUNIT_EXAMPLE_TEST=y And we'd error out with > ERROR:root:Not all Kconfig options selected in kunitconfig were in the generated .config. > This is probably due to unsatisfied dependencies. > Missing: CONFIG_KUNIT_EXAMPLE_TEST=y, CONFIG_KUNIT_TEST=y, CONFIG_KUNIT_ALL_TESTS=y Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: introduce --qemu_args

2022-07-08T00:00:05Z

Example usage: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 \ --kconfig_add=CONFIG_SMP=y --qemu_args='-smp 8' Looking in the test.log, one can see > smp: Bringing up secondary CPUs ... > .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 > smp: Brought up 1 node, 8 CPUs This flag would allow people to make tweaks like this without having to create custom qemu_config files. For consistency with --kernel_args, we allow users to repeat this argument, e.g. you can tack on a --qemu_args='-m 2048', or you could just append it to the first string ('-smp 8 -m 2048'). Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: simplify creating LinuxSourceTreeOperations

2022-07-07T23:59:13Z

Drop get_source_tree_ops() and just call what used to be get_source_tree_ops_from_qemu_config() in both cases. Also rename the functions to have shorter names and add a "_" prefix to note they're not meant to be used outside this function. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: cosmetic: don't specify duplicate kernel cmdline options

2022-07-07T23:51:46Z

Context: When using a non-UML arch, kunit.py will boot the test kernel with options like these by default (this is x86_64): > mem=1G console=tty kunit_shutdown=halt console=ttyS0 kunit_shutdown=reboot The first three options are added unconditionally but are only intended for UML. 1. 'mem=1G' is redundant with the '-m 1024' that we hard-code into the qemu commandline. 2. We specify a 'console' for all tools/testing/kunit/qemu_configs/*.py already, so 'console=tty' gets overwritten. 3. For QEMU, we need to use 'reboot', and for UML we need to use 'halt'. If you switch them, kunit.py will hang until the --timeout expires. This patch: Having these duplicate options is a bit noisy. Switch so we only add UML-specific options for UML. I.e. we now get UML: 'mem=1G console=tty kunit_shutdown=halt' (unchanged) x86_64: 'console=ttyS0 kunit_shutdown=reboot' Side effect: you can't overwrite these options on UML w/ --kernel_arg. But you already couldn't for QEMU (console, kunit_shutdown), and why would you want to? Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: refactoring printing logic into kunit_printer.py

2022-07-07T23:46:25Z

Context: * kunit_kernel.py is importing kunit_parser.py just to use the print_with_timestamp() function * the parser is directly printing to stdout, which will become an issue if we ever try to run multiple kernels in parallel This patch introduces a kunit_printer.py file and migrates callers of kunit_parser.print_with_timestamp() to call kunit_printer.stdout.print_with_timestamp() instead. Future changes: If we want to support showing results for parallel runs, we could then create new Printer's that don't directly write to stdout and refactor the code to pass around these Printer objects. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: redo how we construct and mock LinuxSourceTree

2022-07-07T23:45:59Z

Our main function currently has an optional `linux` argument which is used to by our unit tests to inject a mock. We currently have the same code copy-pasted several times to do if not linux: linux = MakeRealInstance(cli_args.foo, cli_args.bar, ...) But in python, dependency injection isn't necessary or idiomatic when we can just use mock.patch() to mock things out. This change 1. adds a helper to create a LinuxSourceTree from the cli_args 2. drops the `linux` parameter in favor of mocking the __init__ func. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: drop unused load_config argument

2022-07-07T23:45:36Z

It's always set to true except in one test case. And in that test case it can safely be set to true anyways. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan