Re: [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Thu, 2024-06-13 at 10:22 +0200, Thorsten Leemhuis wrote:
> Lo! I prepared four proposals for the maintainers summit regarding
> regressions I'll send in reply to this mail. They are somewhat
> related and address different aspects of one scenario I see
> frequently in different variations; so instead of repeating that
> scenario in slightly modified form in each of the proposals, I'm
> putting it out here once:

I think you're missing a piece here about how we actually find
regressions.  A lot, it is true, come from test suites running on the
mainline.  However, for obscure drivers and even some more complex
dependencies, the regression sometimes isn't discovered until it gets
into the hands of the wider pool of testers, often via stable.

This is important, because it emphasizes that zero regressions in
stable is impossible (and thus preventing backporting patches that
cause regressions is also impossible) if stable is the vehicle by which
some regressions are discovered.  Plus it also means that a backport
delay or cadence would actually delay discovery of some regressions
because the patches that cause them won't be seen by the configs that
run into them until they get put into stable.  There's also a longer
delay in discovery of the actual upstream commit because bugs in stable
need to be reproduced or at least identified in mainline before we can
fix them and the discoverers often have a harder time than mainline
users in helping with this.  This stable being both a vehicle for fixed
kernels and a testing platform for regressions is a tension I don't
think we can (or should) resolve.

So what should we do about this?  I think the first thing is to
recognize the important role stable plays in actually finding bugs. 
There already is a -rc tree for stable, but it doesn't actually seem to
be very useful in finding bugs (likely because the pool of testers is
too small), so perhaps we should discuss whether we could expand this,
or whether we really accept that non-rc stable is part of our testing
infrastructure.

The other thing I think would help is better tooling and advice to help
reporters find regressions in stable.  What we do a lot upstream is ask
if they can reproduce it in mainline.  However, not everyone is
equipped to test out mainline kernels, so we could do with helping them
bisect it in stable (note this can be time dependent: older stable
trees more naturally give rise to the question "has this been fixed
upstream" making mainline testing more of an imperative).

Regards,

James