Re: [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions

[Thorsten Leemhuis <linux@leemhuis.info>]

On 18.06.24 16:43, James Bottomley wrote:
> On Thu, 2024-06-13 at 10:22 +0200, Thorsten Leemhuis wrote:
>> Lo! I prepared four proposals for the maintainers summit regarding
>> regressions I'll send in reply to this mail. They are somewhat
>> related and address different aspects of one scenario I see
>> frequently in different variations; so instead of repeating that
>> scenario in slightly modified form in each of the proposals, I'm
>> putting it out here once:
> 
> I think you're missing a piece here about how we actually find
> regressions.  A lot, it is true, come from test suites running on the
> mainline.

Sure.

> However, for obscure drivers and even some more complex
> dependencies, the regression sometimes isn't discovered until it gets
> into the hands of the wider pool of testers, often via stable.
> 
> This is important, because it emphasizes that zero regressions in
> stable is impossible (and thus preventing backporting patches that
> cause regressions is also impossible) if stable is the vehicle by which
> some regressions are discovered.

Of course "Zero regressions in stable is impossible" as we are dealing
with software. ;) And of course even with delayed backport for
non-urgent fixes some problems would make it through.

But right now users testing mainline sometimes hardly have a chance to
test and report problems with mainline in time to prevent a backport.
Take Linux 6.7.2 (released 2024-01-25 23:58 UTC) with its 640 changes
for example, where users had only 4 days to do so, as almost all of its
changes had been merged for 6.8-rc1 (2024-01-21 22:23 UTC). FWIW: 200 of
those changes were committed to some subsystem git tree during January,
363 during December, 70 during November, and 7 during October.

So if those 440 fixes could wait some time to be mainlined and were not
important enough to get into 6.7 (2024-01-07 20:29 UTC) in the first
place, why the rush backporting them to 6.7.y so quickly after the merge
window?

All that leads to the related question "How many of those changes maybe
should have gone into 6.7?". And maybe even "Should we somehow try to
motivate more people to try -next?". But those are different problems.
And the situation regarding the first already got somewhat better from
what I can see -- among others afaics due to me prodding people when the
queue fixes for recent regression for the -next merge window.

>  Plus it also means that a backport
> delay or cadence would actually delay discovery of some regressions
> because the patches that cause them won't be seen by the configs that
> run into them until they get put into stable.

And why is that a problem?

> [...]
> 
> The other thing I think would help is better tooling and advice to help
> reporters find regressions in stable.  What we do a lot upstream is ask
> if they can reproduce it in mainline.  However, not everyone is
> equipped to test out mainline kernels, so we could do with helping them
> bisect it in stable

FWIW Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst /
https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
covers this: users that notice a regression in a stable tree will bisect
that tree. But before...

> (note this can be time dependent: older stable
> trees more naturally give rise to the question "has this been fixed
> upstream" making mainline testing more of an imperative).

...it does so, but tells users to try mainline for two reasons:
* It might be fixed there already.
* When Greg receives a regression report for stable he'll usually ask
"is mainline also affected" anyway to figure out if this is something he
or somebody else has to look into. And some of the mainline developer
will ask this, too.

Ciao, Thorsten