Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Pull network and Patch Acceptance Consistency

[<Tim.Bird@sony.com>]

> -----Original Message-----
> From: Mark Brown [mailto:broonie@kernel.org]
> 
> On Mon, Jun 17, 2019 at 08:03:15AM -0300, Mauro Carvalho Chehab wrote:
...
> > Are there any other subsystem currently working to get funding for
> > hosting/automation?
> 
> Not sure if it's specifically what you're looking at but there's stuff
> going on that's at least very adjacent to this, more from the angle of
> providing general infrastructure than subsystem specific things and
> currently mainly foucsed on getting tests run.  To me that sort of
> approach seems good since it avoids duplicated efforts between
> subsystems.
> 
> There's people working on things like KernelCI (people are working on
> expanding to include runtime tests, and there's active efforts on
> securing more funding) and CKI which aren't focused on specific
> subsystems but more on general infrastructure.  Tim Bird (CCed) has been
> pushing on trying to get people working in this area talking to each
> other - there's a mailing list and monthly call:
> 
>     https://elinux.org/Automated_Testing
> 
> and one of the things people are talking about is what sorts of things
> the kernel community would find useful here so it's probably useful at
> least putting ideas for things that'd be useful in the heads of people
> who are interested in working on the infrastructure and automation end
> of things.

Indeed.  Although I haven't piped up, I have been paying close attention to
these discussions, and I know from talking to others who are involved with
automated testing that they are as well.

I'm about to go on vacation, so I'll be incommunicado for about a week, but
just to highlight some of the stuff I'm keen on following up on:
 - if Linus like the syzbot notification mechanism, we should definitely
follow up and try to have more tools and frameworks adopt that mechanism
It's on my to-do list to investigate this further and see how it would integrate
with my particular framework (Fuego).  I think Dmitry said we should avoid
introducing bots with lots of different notification mechanisms, as that will
overload developers and just turn them off.
- I recently saw a very cool system for isolating new warnings (at all levels
of W=[0-3]) introduced by a new patch.  This would be a great thing to 
add to the kernel build system, IMHO, and that's also on my to-do list.
- I was very interested in discussions about the mechanism to check whether
a patch modified the binary or not.   It would be nice to make this part of
the build system as well (something like: "make check-for-binary-change"

Both of the latter items require the ability to set a baseline to compare
against.  So the usage sequence would be:
 - make save-baseline
 - <do commit or pull>
 - make show-new-warnings
or
 - make check-for-binary-change

Just FYI, some of the stuff that the automated testing folk are working on
are:
 - standards for lab equipment management (/board management)
 - standards for test definition
 - standards for test results format and aggregation
 - setting up a multi-framework results aggregation site

Keep the ideas flowing!  While we sometimes don't chime in, I know people
are listening and thinking about the ideas presented.  And many of us will be
at plumbers for live discussions, and at ELC Europe.  A few of us are starting a new event
called Automated Testing Summit that we're co-locating (at least this year) with 
OSSUE/ELCE on October 31, in Lyon France.  Check the Linux Foundation events site
for more information.
-- Tim