Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format

[John Hubbard <jhubbard@nvidia.com>]

On 12/18/23 16:51, John Hubbard wrote:
> On 12/18/23 03:32, Ryan Roberts wrote:
> ...
>>> I should also point out that some of the subtests already attempt a TAP
>>> output. So now we end up with TAP-within-TAP output for those programs.
>>
>> It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)
>>
>>>
>>> For example:
>>>      # -----------------------
>>>      # running ./madv_populate
>>>      # -----------------------
>>>      # TAP version 13
>>>      # 1..21
>>>      # # [RUN] test_prot_read
>>>      # ok 1 MADV_POPULATE_READ with PROT_READ
>>>      # ok 2 MADV_POPULATE_WRITE with PROT_READ
>>>      # # [RUN] test_prot_write
>>>      # ok 3 MADV_POPULATE_READ with PROT_WRITE
>>>      ...etc...
>>>
>>> Note the double level of leading '#' characters.
>>>
>>> Again, this is still readable enough for humans. But it should probably
>>> be removed in subsequent patches to the subtests.
>>
>> I personally don't agree with this. It would be difficult to flatten to a single
>> TAP instance because the top level doesn't have a clue how many test cases the
> 
> That's not quite what I had in mind...
> 
>> child is running. Trying to do this will make things more fragile and less
>> modular. LAVA can certainly deal with nested test cases and correctly parses
>> everything to test case names that contain the test name at each level of
>> nesting. The thing I was trying to solve with this patch was that previously the
>> top level (run_kselftest.sh) and the bottom level (individual mm test binaries)
>> were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was
>> confusing the LAVA parser.
>>
> 
> I was thinking more along these lines:
> 
> a) For the individual programs (binaries), there is actually neither need nor
> desire to create TAP output at that level, because frameworks like LAVA only
> care about running a lot of tests and parsing the output.
> 
> b) Therefore, just stop specifying TAP output at the leaf level, and let
> run_vmtests.sh and run_kselftest.sh do it.
> 
> Looking at madv_populate.c, I see that it scatters calls to ksft_*() around.
> And I was thinking that this is all just redundant, isn't it?
> 

Although I suppose that the counter argument is that the subtests in
madv_populate.c really *do* want to be specifically printed in TAP
format.

arggh, I guess this is just not worth fooling around with after all.
  

thanks,
-- 
John Hubbard
NVIDIA