Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Fengguang Wu <fengguang.wu@intel.com>]

On Mon, Jul 13, 2015 at 07:34:41PM +0100, Mark Brown wrote:
> On Sun, Jul 12, 2015 at 07:15:47PM +0800, Fengguang Wu wrote:
> 
> > If 0day failed to catch the error, it might be due to
> 
> > - regressions in 0day itself -- it's still in active development,
> >   in the past year I did 1k patches to 0day build/boot scripts and
> >   2k patches to the follow up LKP (Linux Kernel Performance) tests.
> 
> > - the "report once" logic, it's tricky and can possibly hide issues
> 
> > - failed bisect, rare but possible
> 
> I think some of the issues have been due to bisection getting confused
> by issues appearing in merge commits but ICBW.
> 
> > - machine hang, network/disk fails, etc. maintenance incidents
> 
> > "Loading" may add latency, however it's not the cause to miss errors.
> 
> Latency seems like it might be an issue here, when I say I'm not seeing
> things that's issues coming up in the -next build before they're
> reported by 0day so they might well get fixed before 0day catches up.

Ah linux-next happen to have lower priority and can take many hours to
finish. I'll increase its priority due to its importance.

> > > > Are people ignoring them?
> 
> > > They're not reliably followed through on, no, and one of the things with
> > > 0day is that it just generates a one time report so if things don't get

The real policy is in fact a bit more smart than "report-once".

Build errors will be auto re-reported if it's still not fixed
after 10 days when the branch that introduced the error is updated.

Warnings won't be auto re-reported.

> > > followed up on then that's that.  A regular "these are all the issues"
> > > mail helps chase down those issues.
> 
> > 0day has such report type. It will be sent after each git push (unless
> > you push too quickly) and it looks like this. Just drop me a note and
> > list the git trees/branches you wish to receive such notice emails.
> 
> For me personally it'd be more interesting to be able to get them on
> demand (eg, from a web page) than e-mailed, or e-mailed by a human
> (possibly with fixes!).  The kernelci.org reporting does a lot of this
> but doesn't cover anything except raw compiler warnings.

It should be mostly equivalent if you direct such emails to a local
mbox and check it on demand. :)

For example, setup .procmailrc rule like this:

:0:
* ^Subject: \[.*\] ........................................ BUILD
build-complete-notification

Thanks,
Fengguang