Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests

["Ricardo Cañuelo" <ricardo.canuelo@collabora.com>]

Hi Nikolai,

On mié, nov 15 2023 at 19:43:49, Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com> wrote:
> Introduce a new tag, 'Tested-with:', documented in the
> Documentation/process/submitting-patches.rst file. The tag is expected
> to reference the documented test suites, similarly to the 'V:' field,
> and to certify that the submitter executed the test suite on the change,
> and that it passed.

I think the 'V:' field in MAINTAINERS is a good addition to document
what developers are supposed to test for every subsystem, but in the
case of the per-commit "Tested-with:" tag, I think the real value of it
would be in using it for accountability and traceability purposes
instead, that is, to link to the actual results of the (automatic) tests
that were used to validate a commit.

This would provide two important features:

1. Rather than trusting that the tester did things right and that the
   test environment used was appropriate, we'd now have proof that the
   test results are as expected and a way to reproduce the steps.

2. A history of test results for future reference. When a regression is
   introduced, now we'd have more information about how things worked
   back when the test was still passing.

This is not trivial because tests vary a lot and we'd first need to
define which artifacts to link to, and because whatever is linked (test
commands, output log, results summary) would need to be stored
forever. But since we're doing that already for basically all kernel
mailing lists, I wonder if something like "public-inbox for test
results" could be possible some day.

Cheers,
Ricardo