Re: [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Thu, 2024-06-20 at 12:32 +0200, Thorsten Leemhuis wrote:
> On 18.06.24 16:43, James Bottomley wrote:
> > On Thu, 2024-06-13 at 10:22 +0200, Thorsten Leemhuis wrote:
> > > Lo! I prepared four proposals for the maintainers summit
> > > regarding regressions I'll send in reply to this mail. They are
> > > somewhat related and address different aspects of one scenario I
> > > see frequently in different variations; so instead of repeating
> > > that scenario in slightly modified form in each of the proposals,
> > > I'm putting it out here once:
> > 
> > I think you're missing a piece here about how we actually find
> > regressions.  A lot, it is true, come from test suites running on
> > the mainline.
> 
> Sure.
> 
> > However, for obscure drivers and even some more complex
> > dependencies, the regression sometimes isn't discovered until it
> > gets into the hands of the wider pool of testers, often via stable.
> > 
> > This is important, because it emphasizes that zero regressions in
> > stable is impossible (and thus preventing backporting patches that
> > cause regressions is also impossible) if stable is the vehicle by
> > which some regressions are discovered.
> 
> Of course "Zero regressions in stable is impossible" as we are
> dealing with software. ;) And of course even with delayed backport
> for non-urgent fixes some problems would make it through.
> 
> But right now users testing mainline sometimes hardly have a chance
> to test and report problems with mainline in time to prevent a
> backport. Take Linux 6.7.2 (released 2024-01-25 23:58 UTC) with its
> 640 changes for example, where users had only 4 days to do so, as
> almost all of its changes had been merged for 6.8-rc1 (2024-01-21
> 22:23 UTC). FWIW: 200 of those changes were committed to some
> subsystem git tree during January, 363 during December, 70 during
> November, and 7 during October.

I did make this point here:

https://lore.kernel.org/all/7794a2b09ae4fa73ac35fdaec4858145a665efea.camel@HansenPartnership.com/

That merge window fixes should be delayed.  Not because I think a
longer soak in main would allow us to find many more bugs, simply
because it was causing reports in the merge window that weren't handled
because people had other things to do.  The reply was that they're
already doing it and when I looked, they actually started doing it for
the 6.9 merge window (so your 6.7 example is probably out of date).

> So if those 440 fixes could wait some time to be mainlined and were
> not important enough to get into 6.7 (2024-01-07 20:29 UTC) in the
> first place, why the rush backporting them to 6.7.y so quickly after
> the merge window?
> 
> All that leads to the related question "How many of those changes
> maybe should have gone into 6.7?". And maybe even "Should we somehow
> try to motivate more people to try -next?".

Actually, if we got more people to try mainline we could perhaps find
more bugs.  Testing -next is problematic because its instability makes
things like bisection and update to next release difficult.

>  But those are different problems.
> And the situation regarding the first already got somewhat better
> from what I can see -- among others afaics due to me prodding people
> when the queue fixes for recent regression for the -next merge
> window.

Yes, that's why I was asking for stats on 6.9 and 6.10 where this delay
policy was apparently in place.

> >  Plus it also means that a backport
> > delay or cadence would actually delay discovery of some regressions
> > because the patches that cause them won't be seen by the configs
> > that run into them until they get put into stable.
> 
> And why is that a problem?

Because a regression we haven't found yet is still a regression.  If
all we cared about was minimizing the regression stats, we could simply
not look for any of them.  But we do care about this, so we need to
support all our mechanisms for finding them and the point I was making
is that one such mechanism is the early backports to stable.  There is
probably a sweet spot backport delay for regressions we do eventually
find in main, but for regressions that others only find in stable (and
would never have been found in main however log we delayed) arbitrary
delays merely increases the time to finding them.

Perhaps one thing we should track with regressions is time to discovery
and also ask about ones in stable if they could have been found in
mainline?  That would give us more data for tuning the backport delay.

> > [...]
> > 
> > The other thing I think would help is better tooling and advice to
> > help reporters find regressions in stable.  What we do a lot
> > upstream is ask if they can reproduce it in mainline.  However, not
> > everyone is equipped to test out mainline kernels, so we could do
> > with helping them bisect it in stable
> 
> FWIW Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
> /
> https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
> covers this: users that notice a regression in a stable tree will
> bisect that tree. But before...

Some do, but realistically the best others can do is this bug was in
X.Y.Z but not in X.Y.Z-1 because they can't build their own kernels.

> > (note this can be time dependent: older stable
> > trees more naturally give rise to the question "has this been fixed
> > upstream" making mainline testing more of an imperative).
> 
> ...it does so, but tells users to try mainline for two reasons:
> * It might be fixed there already.
> * When Greg receives a regression report for stable he'll usually ask
> "is mainline also affected" anyway to figure out if this is something
> he or somebody else has to look into. And some of the mainline
> developer will ask this, too.

Again not saying that's wrong, just saying we must accept that some
bugs will only be found in stable and thus we could do with improving
our tooling to help stable users pinpoint the backport that caused
them.

James