Re: [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions

[Sasha Levin <sashal@kernel.org>]

On Thu, Jun 13, 2024 at 09:56:56AM -0400, James Bottomley wrote:
>On Thu, 2024-06-13 at 09:06 -0400, Sasha Levin wrote:
>> On Thu, Jun 13, 2024 at 07:58:58AM -0400, James Bottomley wrote:
>> > On Thu, 2024-06-13 at 10:42 +0200, Thorsten Leemhuis wrote:
>> > > The scenario shown at the start of the thread illustrates a
>> > > problem I see frequently: commits with a Fixes: tag end up in new
>> > > to stable series releases just days after being mainlined and
>> > > cause regressions -- just like they do in mainline, which just
>> > > was not known yet at the time of backporting. This happens
>> > > extremely often right after merge windows when huge piles of
>> > > changes are backported to the stable trees each cycle shortly
>> > > after -rc1 is out (which even some kernel developers apparently
>> > > are somewhat afraid to test from what I've
>> > > seen).
>> >
>> > I haven't really observed this for curated fixes.  For most
>> > subsystems, patches with Fixes tags that are cc'd to stable tend to
>> > go steadily outside the merge window.  Obviously a few arrive
>> > within it, but usually at roughly the rate they arrive outside it.
>> >
>> > What I observe in the merge window is huge piles of patches go into
>> > stable *without* a cc:stable tag from the autosel machinery (and
>> > quite a few even without fixes: tags).
>>
>> Could you provide a concrete example? This shouldn't happen.
>
>This one has no fixes or cc stable:
>
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37f1663c91934f664fb850306708094a324c227c
>
>Yet here it is backported:
>
>Message-id: 20240603121056.1837607-1-sashal@kernel.org
>
>(I can't give a lore reference because conveniently it was a personal
>cc with no tracked mailing list).
>
>I picked that one because we discovered a bug with the strlcpy to
>strscpy conversions in SCSI which it looks like this backport has.

In this case, we picked up the commit because it's a dependency for
527e9b704c3d ("scsi: qla2xxx: Use memcpy() and strlcpy() instead of
strcpy() and strncpy()"), it didn't come in via autosel.

>> > So this does beg a couple of questions:
>> >
>> > Since you have the figures, what's the proportion of regressions
>> > caused by backports to stable without cc:stable tags?
>>
>> This question came up two years ago and we had statistics around
>> this. Autosel patches didn't cause more (actually, it was *less*)
>> regressions than stable tagged ones.
>
>OK, so Thorsten's stats should bear this out then ...

Yup, this is an experiment we started about that time. We've extended
autosel to be about 4 weeks behind where Linus is, and wanted to look at
the statistics some time later to see if it improved anything.

I would note here that even two years ago, autosel commits were slightly
"safer" than stable tagged commits (w.r.t the odds of having a follow-up
commit pointing back with a Fixes: tag.).

>> > Could we fix a lot of this by delaying autosel?  It tends to ramp
>> > up in the merge window when everyone is concentrating on other
>> > things, so any regressions it causes naturally get ignored for a
>> > couple of weeks.
>>
>> autosel is currently delayed about 3-4 weeks, sometimes more.
>
>That's fairly recent then.  When I look at 6.8 autosel began its flood
>pretty much the moment the first SCSI pull request went in to the merge
>window.  Checking 6.9 it seems to be about 10 days after ... has that
>made a difference, or is it too early to tell?

The mails may come in during the merge window, but the commits aren't merged
until after 3-4 weeks after (we just present them for review early). In
the 6.8 case, for example, the first autosel commit went into v6.8.6,
which was released about a month after the merge window closed.

-- 
Thanks,
Sasha