Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format

[John Hubbard <jhubbard@nvidia.com>]

On 12/18/23 03:32, Ryan Roberts wrote:
...
>> I should also point out that some of the subtests already attempt a TAP
>> output. So now we end up with TAP-within-TAP output for those programs.
> 
> It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)
> 
>>
>> For example:
>>      # -----------------------
>>      # running ./madv_populate
>>      # -----------------------
>>      # TAP version 13
>>      # 1..21
>>      # # [RUN] test_prot_read
>>      # ok 1 MADV_POPULATE_READ with PROT_READ
>>      # ok 2 MADV_POPULATE_WRITE with PROT_READ
>>      # # [RUN] test_prot_write
>>      # ok 3 MADV_POPULATE_READ with PROT_WRITE
>>      ...etc...
>>
>> Note the double level of leading '#' characters.
>>
>> Again, this is still readable enough for humans. But it should probably
>> be removed in subsequent patches to the subtests.
> 
> I personally don't agree with this. It would be difficult to flatten to a single
> TAP instance because the top level doesn't have a clue how many test cases the

That's not quite what I had in mind...

> child is running. Trying to do this will make things more fragile and less
> modular. LAVA can certainly deal with nested test cases and correctly parses
> everything to test case names that contain the test name at each level of
> nesting. The thing I was trying to solve with this patch was that previously the
> top level (run_kselftest.sh) and the bottom level (individual mm test binaries)
> were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was
> confusing the LAVA parser.
> 

I was thinking more along these lines:

a) For the individual programs (binaries), there is actually neither need nor
desire to create TAP output at that level, because frameworks like LAVA only
care about running a lot of tests and parsing the output.

b) Therefore, just stop specifying TAP output at the leaf level, and let
run_vmtests.sh and run_kselftest.sh do it.

Looking at madv_populate.c, I see that it scatters calls to ksft_*() around.
And I was thinking that this is all just redundant, isn't it?


thanks,
-- 
John Hubbard
NVIDIA