Re: [PATCH 1/3] MAINTAINERS: Introduce V: field for required tests

["Theodore Ts'o" <tytso@mit.edu>]

On Mon, Nov 20, 2023 at 02:30:49PM +0100, Ricardo Cañuelo wrote:
> 
> This is not trivial because tests vary a lot and we'd first need to
> define which artifacts to link to, and because whatever is linked (test
> commands, output log, results summary) would need to be stored
> forever. But since we're doing that already for basically all kernel
> mailing lists, I wonder if something like "public-inbox for test
> results" could be possible some day.

What we have at work is a way to upload the test results summary
(e.g., just KTAP result lines, or the xfstests junit XML) along with
test run metadata (e.g., what was the kernel commit on which the test
was run, and the test hardware), and this would be stored permanently.
Test artifacts is also preserved but for a limited amount of time
(e.g., some number of months or a year).

The difference in storage lifetimes is because the junit XML file
might be a few kilobytes to tens of kilobytes. but the test artifacts
might be a few megabytes to tens of megabytes.

Of course once you have this data, it becomes possible to detect when
a test may have regressed, or to detect flaky tests, and perhaps to
figure out if certain hardware configurations or kernel configurations
are more likely to trigger a particular test to fail.  So having all
of this data stored centrally would be really cool.  The only question
is who might be able to create such an infrastructure, and be able to
pay for the ongoing development and operational costs....

    	    	    		    		- Ted