Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format

[Ryan Roberts <ryan.roberts@arm.com>]

On 15/12/2023 13:54, Mark Brown wrote:
> On Thu, Dec 14, 2023 at 04:24:34PM +0000, Ryan Roberts wrote:
>> When running tests on a CI system (e.g. LAVA) it is useful to output
>> test results in TAP format so that the CI can parse the fine-grained
>> results to show regressions. Many of the mm selftest binaries already
>> output using the TAP format. And the kselftests runner
>> (run_kselftest.sh) also uses the format. CI systems such as LAVA can
>> already handle nested TAP reports. However, with the mm selftests we
>> have 3 levels of nesting (run_kselftest.sh -> run_vmtests.sh ->
>> individual test binaries) and the middle level did not previously
>> support TAP, which breaks the parser.
> 
> Reviewed-by: Mark Brown <broonie@kernel.org>

Thanks!

> 
>> Let's fix that by teaching run_vmtests.sh to output using the TAP
>> format. Ideally this would be opt-in via a command line argument to
>> avoid the possibility of breaking anyone's existing scripts that might
>> scrape the output. However, it is not possible to pass arguments to
>> tests invoked via run_kselftest.sh. So I've implemented an opt-out
>> option (-n), which will revert to the existing output format.
> 
> What I did for ftrace which had a similar situation was make a wrapper
> script which invokes the test runner, make the test runner a
> TEST_PROGS_EXTENDED so it's not run by the kselftest infrastructure
> automatically and make the wrapper a normal TEST_PROGS.  Neither option
> is especially lovely.

Yeah that's a good idea... I'll wait and see if anyone shouts that this has
broken something. If nothing is broken, I think it is better to just make TAP
the default rather than adding yet another wrapper.