Re: [PATCH v1] selftests/mm: Log run_vmtests.sh results in TAP format

[Ryan Roberts <ryan.roberts@arm.com>]

On 19/12/2023 00:55, John Hubbard wrote:
> On 12/18/23 16:51, John Hubbard wrote:
>> On 12/18/23 03:32, Ryan Roberts wrote:
>> ...
>>>> I should also point out that some of the subtests already attempt a TAP
>>>> output. So now we end up with TAP-within-TAP output for those programs.
>>>
>>> It's actually TAP-in-TAP-in-TAP if you're running from run_kselftest.sh :)
>>>
>>>>
>>>> For example:
>>>>      # -----------------------
>>>>      # running ./madv_populate
>>>>      # -----------------------
>>>>      # TAP version 13
>>>>      # 1..21
>>>>      # # [RUN] test_prot_read
>>>>      # ok 1 MADV_POPULATE_READ with PROT_READ
>>>>      # ok 2 MADV_POPULATE_WRITE with PROT_READ
>>>>      # # [RUN] test_prot_write
>>>>      # ok 3 MADV_POPULATE_READ with PROT_WRITE
>>>>      ...etc...
>>>>
>>>> Note the double level of leading '#' characters.
>>>>
>>>> Again, this is still readable enough for humans. But it should probably
>>>> be removed in subsequent patches to the subtests.
>>>
>>> I personally don't agree with this. It would be difficult to flatten to a single
>>> TAP instance because the top level doesn't have a clue how many test cases the
>>
>> That's not quite what I had in mind...
>>
>>> child is running. Trying to do this will make things more fragile and less
>>> modular. LAVA can certainly deal with nested test cases and correctly parses
>>> everything to test case names that contain the test name at each level of
>>> nesting. The thing I was trying to solve with this patch was that previously the
>>> top level (run_kselftest.sh) and the bottom level (individual mm test binaries)
>>> were using TAP, but the middle level (run_vmtests.sh) wasn't, and this was
>>> confusing the LAVA parser.
>>>
>>
>> I was thinking more along these lines:
>>
>> a) For the individual programs (binaries), there is actually neither need nor
>> desire to create TAP output at that level, because frameworks like LAVA only
>> care about running a lot of tests and parsing the output.
>>
>> b) Therefore, just stop specifying TAP output at the leaf level, and let
>> run_vmtests.sh and run_kselftest.sh do it.
>>
>> Looking at madv_populate.c, I see that it scatters calls to ksft_*() around.
>> And I was thinking that this is all just redundant, isn't it?
>>
> 
> Although I suppose that the counter argument is that the subtests in
> madv_populate.c really *do* want to be specifically printed in TAP
> format.
> 
> arggh, I guess this is just not worth fooling around with after all.

Yes; I wouldn't want to lose the fine granularity we have currently. For example
cow.c has ~900 test cases now that I've multiplied everything up for mTHP. 16 of
those are known to fail (hugetlb issue) and 1 is skipped. I wouldn't want to
reduce that down to a single cow test case that always fails; that's not helpful
to understand if I've regressed something.

But sounds like we are both on the same page now.


>  
> 
> thanks,