Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mel Gorman <mgorman@suse.de>]

On Tue, Jul 07, 2015 at 06:18:20PM +0100, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> > On 07/07/2015 02:24 AM, Mark Brown wrote:
> 
> > >The main things I'm aware of that are happening at the minute are
> > >kselftest development, the 0day tester, plus kernelci.org and the other
> > >build and boot/test bots that are running against various trees.
> 
> > Maybe list all known ones as a start ?
> 
> Off the top of my head the automated ones I'm aware of are Olof's build
> & boot test, Dan running smatch and I think some other static analysis
> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> and I've got a builder too.
> 

There is also Marvin which has existed in some shape or form since November
2013. It checks once a month for new releases and executes a battery of
tests on them across a range of machines. The primary purpose of it is
performance verification and the tests are typically more complex than
what I'd expect to see in kselftest. There are small amounts of overlap
with 0-day but generally it's expected that Marvin runs tests that are more
long-lived. Unlike 0-day, it also does not automatically notify people about
regressions as some verification work is often required and I did not want
it generating noise in any inbox.  Technically, it does support automatic
bisection but it's something I trigger manually when I confirm a problem
is a performance regression and cannot quickly identify the root cause.

It actually has been publishing reports for several months
now but I never mentioned it on the lists. I wrote up
some details after reading this thread and posted it at
http://www.csn.ul.ie/~mel/blog/index.php?/archives/23-Continual-testing-of-mainline-kernels.html

If there was a workshop on testing then I'd be interested in attending and
discussing what Marvin does if there was interest. Right now, performance
tends to my area of interest so I'd be interesting is discussing if there
are areas we are continually getting worse at that are slipping through
the cracks. Chris proposed a topic in this general area that I think would
be useful. I've only started looking at mainline kernel performance again
recently and right now, I'm not aware of a single area where we are getting
consistently worse. More commonly I see cases where we create problems
and then later cover them up by fixing something else in the general
area. Any time I find problems, it's a simple matter of programming and
time to fix them but it'd be useful to hear what other peoples recent
experiences have been.

-- 
Mel Gorman
SUSE Labs