Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format

[Mark Brown <broonie@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

On Thu, Dec 14, 2023 at 04:24:34PM +0000, Ryan Roberts wrote:
> When running tests on a CI system (e.g. LAVA) it is useful to output
> test results in TAP format so that the CI can parse the fine-grained
> results to show regressions. Many of the mm selftest binaries already
> output using the TAP format. And the kselftests runner
> (run_kselftest.sh) also uses the format. CI systems such as LAVA can
> already handle nested TAP reports. However, with the mm selftests we
> have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh ->
> individual test binaries) and the middle level did not previously
> support TAP, which breaks the parser.

Reviewed-by: Mark Brown <broonie@kernel.org>

> Let's fix that by teaching run_vmtests.sh to output using the TAP
> format. Ideally this would be opt-in via a command line argument to
> avoid the possibility of breaking anyone's existing scripts that might
> scrape the output. However, it is not possible to pass arguments to
> tests invoked via run_kselftest.sh. So I've implemented an opt-out
> option (-n), which will revert to the existing output format.

What I did for ftrace which had a similar situation was make a wrapper
script which invokes the test runner, make the test runner a
TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure
automatically and make the wrapper a normal TEST_PROGS.  Neither option
is especially lovely.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]