linux/tools/testing/selftests/tc-testing/plugin-lib, branch v6.7

selftests/tc-testing: implement tdc parallel test run

2023-09-28T07:51:07Z

Use a Python process pool to run the tests in parallel. Not all tests can run in parallel, for instance tests that are not namespaced and tests that use netdevsim, as they can conflict with one another. The code logic will split the tests into serial and parallel. For the parallel tests, we build batches of 32 tests and queue each batch on the process pool. For the serial tests, they are queued as a whole into the process pool, which in turn executes them concurrently with the parallel tests. Even though the tests serialize on rtnl_lock in the kernel, this feature showed results with a ~3x speedup on the wall time for the entire test suite running in a VM: Before - 4m32.502s After - 1m19.202s Examples: In order to run tdc using 4 processes: ./tdc.py -J4 <...> In order to run tdc using 1 process: ./tdc.py -J1 <...> || ./tdc.py <...> Note that the kernel configuration will affect the speed of the tests, especially if such configuration slows down process creation and/or fork(). Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

selftests/tc-testing: localize test resources

2023-09-28T07:51:07Z

As of today, the current tdc architecture creates one netns and uses it to run all tests. This assumption was embedded into the nsPlugin which carried over as how the tests were written. The tdc tests are by definition self contained and can, theoretically, run in parallel. Even though in the kernel they will serialize over the rtnl lock, we should expect a significant speedup of the total wall time for the entire test suite, which is hitting close to 1100 tests at this point. A first step to achieve this goal is to remove sharing of global resources like veth/dummy interfaces and the netns. In this patch we 'localize' these resources on a per test basis. Each test gets it's own netns, VETH/dummy interfaces. The resources are spawned in the pre_suite phase, where tdc will prepare all netns and interfaces for all tests. This is done in order to avoid concurrency issues with netns / interfaces spawning and commands using them. As tdc progresses, the resources are deleted after each test finishes executing. Tests that don't use the nsPlugin still run under the root namespace, but are now required to manage any external resources like interfaces. These cannot be parallelized as their definition doesn't allow it. On the other hand, when using the nsPlugin, tests don't need to create dummy/veth interfaces as these are handled already. Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

tc-testing: add support for sending various scapy packets

2021-06-22T17:52:39Z

It can be worth sending different scapy packets on a given test, as in the last patch of this series. For that, lets listify the scapy attribute and simply iterate over it. Signed-off-by: Marcelo Ricardo Leitner Signed-off-by: David S. Miller

tc-testing: fix list handling

2021-06-22T17:52:39Z

python lists don't have an 'add' method, but 'append'. Fixes: 14e5175e9e04 ("tc-testing: introduce scapyPlugin for basic traffic") Signed-off-by: Marcelo Ricardo Leitner Signed-off-by: David S. Miller

tc-testing: fix eBPF tests failure on linux fresh clones

2020-02-04T08:30:00Z

when the following command is done on a fresh clone of the kernel tree, [root@f31 tc-testing]# ./tdc.py -c bpf test cases that need to build the eBPF sample program fail systematically, because 'buildebpfPlugin' is unable to install the kernel headers (i.e, the 'khdr' target fails). Pass the correct environment to 'make', in place of ENVIR, to allow running these tests. Fixes: 4c2d39bd40c1 ("tc-testing: use a plugin to build eBPF program") Signed-off-by: Davide Caratti Signed-off-by: David S. Miller

tc-testing: don't hardcode 'ip' in nsPlugin.py

2019-09-01T06:47:25Z

the following tdc test fails on Fedora: # ./tdc.py -e 2638 -- ns/SubPlugin.__init__ Test 2638: Add matchall and try to get it -----> prepare stage *** Could not execute: "$TC qdisc add dev $DEV1 clsact" -----> prepare stage *** Error message: "/bin/sh: ip: command not found" returncode 127; expected [0] -----> prepare stage *** Aborting test run. Let nsPlugin.py use the 'IP' variable introduced with commit 92c1a19e2fb9 ("tc-tests: added path to ip command in tdc"), so that the path to 'ip' is correctly resolved to the value we have in tdc_config.py. # ./tdc.py -e 2638 -- ns/SubPlugin.__init__ Test 2638: Add matchall and try to get it All test results: 1..1 ok 1 2638 - Add matchall and try to get it Fixes: 489ce2f42514 ("tc-testing: Restore original behaviour for namespaces in tdc") Reported-by: Hangbin Liu Signed-off-by: Davide Caratti Acked-by: Nicolas Dichtel Signed-off-by: David S. Miller

tc-testing: introduce scapyPlugin for basic traffic

2019-07-09T21:07:20Z

The scapyPlugin allows for simple traffic generation in tdc to test various tc features. It was tested with scapy v2.4.2, but should work with any successive version. In order to use the plugin's functionality, scapy must be installed. This can be done with: pip3 install scapy or to install 2.4.2: pip3 install scapy==2.4.2 If the plugin is unable to import the scapy module, it will terminate the tdc run. The plugin makes use of a new key in the test case data, 'scapy'. This block contains three other elements: 'iface', 'count', and 'packet': "scapy": { "iface": "$DEV0", "count": 1, "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()" }, * iface is the name of the device on the host machine from which the packet(s) will be sent. Values contained within tdc_config.py's NAMES dict can be used here - this is useful if paired with nsPlugin * count is the number of copies of this packet to be sent * packet is a string detailing the different layers of the packet to be sent. If a property isn't explicitly set, scapy will set default values for you. Layers in the packet info are separated by slashes. For info about common TCP and IP properties, see: https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf Caution is advised when running tests using the scapy functionality, since the plugin blindly sends the packet as defined in the test case data. See creating-testcases/scapy-example.json for sample test cases; the first test is intended to pass while the second is intended to fail. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller

tc-testing: Restore original behaviour for namespaces in tdc

2019-06-25T19:57:29Z

This patch restores the original behaviour for tdc prior to the introduction of the plugin system, where the network namespace functionality was split from the main script. It introduces the concept of required plugins for testcases, and will automatically load any plugin that isn't already enabled when said plugin is required by even one testcase. Additionally, the -n option for the nsPlugin is deprecated so the default action is to make use of the namespaces. Instead, we introduce -N to not use them, but still create the veth pair. buildebpfPlugin's -B option is also deprecated. If a test cases requires the features of a specific plugin in order to pass, it should instead include a new key/value pair describing plugin interactions: "plugins": { "requires": "buildebpfPlugin" }, A test case can have more than one required plugin: a list can be inserted as the value for 'requires'. Signed-off-by: Lucas Bates Acked-by: Davide Caratti Tested-by: Nicolas Dichtel Signed-off-by: David S. Miller

tc-testing: Allow test cases to be skipped

2019-03-02T07:05:06Z

By adding a check for an optional key/value pair to the test case data, individual test cases may be skipped to prevent tdc from aborting a test run due to setup or teardown failure. If a test case is skipped, it will still appear in the results output to allow for a consistent number of executed tests in each run. However, the test will be marked as skipped. This support for skipping extends to any plugins that may generate additional results for each executed test. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller

tc-testing: Implement the TdcResults module in tdc

2018-12-08T00:39:03Z

In tdc and the valgrind plugin, begin using the TdcResults module to track executed test cases. Signed-off-by: Lucas Bates Signed-off-by: David S. Miller