Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug-introducing patches

[Dan Rue <dan.rue@linaro.org>]

On Fri, Sep 07, 2018 at 03:43:46PM -0700, Guenter Roeck wrote:
> On Fri, Sep 07, 2018 at 03:27:01PM -0700, Linus Torvalds wrote:
> > On Fri, Sep 7, 2018 at 2:13 PM Sasha Levin via Ksummit-discuss
> > <ksummit-discuss@lists.linuxfoundation.org> wrote:
> > >
> > > 1. Grab a batch of ~2-3 week old commits from Linus's tree.
> > > 2. Review, basic tests and send stable-rc notification.
> > 
> > Side note: maybe the stable grabbing and testing could be automated?
> > 
> > IOW, right now the stable people intentionally (generally) wait a week
> > before they even start. Maybe there could be an automated queue for
> > "this has been marked for stable" (and the whole "fixes:" magic that
> > you guys already trigger on) that gets applied to the previous stable
> > tree, and starts testing immediately.
> > 
> > Because one of the patterns we *do* obviously see is that something
> > was fine in mainline, but then broke in stable because of an unforseen
> > lack of depdenencies. Sure, it's probably pretty rare (and *many*
> > dependencies willl show up as an actual conflict), but I think the
> > times it does happen it's particularly painful because it can be so
> > non-obvious.
> > 
> > So maybe an automated "linux-next" that starts happening *before* the
> > rc stage would catch some things?
> > 
> 
> And it does, as soon as Greg publishes a set of patches. At the very
> least 0day runs on those, as well as my builders. There is a question
> of scalability, though. I am sure that will improve over time as more
> test resources become available, but six stable releases plus mainline
> plus next plus whatever contributing branches covered by 0day and others
> does take a lot of resources.

We build and test every push to the stable-rc branches, mainline, and
next, too. Branches have a carrying cost (maintenance, triage, etc -
this cost goes down as tooling improves), and pushes have an
infrastructure (build and test) cost. I'll agree with Guenter that I'd
rather see better coverage on fewer branches. It's also important for
tree maintainers to realize the downstream consequences of a push on the
CI/CD environments. Sometimes we see a lot of pushes with little or no
changes, which causes duplicate testing and delays results. On the other
hand, it is nice to have bisection points within a branch for when
problems do emerge.

The fewer branches we track for CI/CD, the more attention they will
receive, the deeper the testing can go, and the better the results will
be. -next is a good example of this principle.

In fact, we recently stopped building and testing the stable release
branches altogether, because it's redundant if you're already testing
every push on the stable-rc branches.

This is also true for test suites/frameworks. I'd prefer to see more
activity around fewer test suites, rather than a proliferation of
different ways of writing and running tests. The cost of adding a test
suite to CI/CD is high, especially if you do it well, and so as we see
efforts centralize around fewer suites, coverage and test quality will
improve in general.

Dan


> 
> Personally I would suggest to further improve test coverage, not to add
> more branches to test. More hardware for sure, but also adding more tests
> such as the network testing suggested by Sasha.


> 
> Guenter
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss