[MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions

[Thorsten Leemhuis <linux@leemhuis.info>]

I would like to discuss how to better prevent backports of mainline
commits to stable that turn out to cause regressions.

The scenario shown at the start of the thread illustrates a problem I
see frequently: commits with a Fixes: tag end up in new to stable series
releases just days after being mainlined and cause regressions -- just
like they do in mainline, which just was not known yet at the time of
backporting. This happens extremely often right after merge windows when
huge piles of changes are backported to the stable trees each cycle
shortly after -rc1 is out (which even some kernel developers apparently
are somewhat afraid to test from what I've seen).

I do not want to criticize the stable team for their approach to things,
as I can understand why they are doing this: there is no simple way to
distinguish "this is an urgent (security) fix that should be quickly
backported" from "this should be tested in mainline for a while first"
or "this has a Fixes: tag, but is not backport-worthy at all" -- which
is why they handle changes about equally. I think untangling that aspect
and backporting the non-urgent ones more slowly could help a lot to
prevent many regressions from hitting stable trees.

The thing is: I'm not sure how to achieve that. Here are a few thoughts
my brain came up with:

* For patches that are tagged for backporting it's easy to for
developers to influence the timing, as they can use a stable tag like
`Cc: <stable@vger.kernel.org> # after -rc4` to delay backporting (see
Documentation/process/stable-kernel-rules.rst for details). But for
quite a few developers this is not an option, as such a Cc: implies that
the developer wants the fix to be backported -- and thus should ideally
have tested it, will provide an adjusted patch when needed, and is
willing to handle any fallout. Maybe untangling these aspects from the
stable tag might be wise, so that developers can signal "I think this
should be backported, but I don't want anything to do with it" could
help. If this becomes the norm then maybe the stable team could even
stop taking nearly everything with a Fixes: tag. But I'm not sure if I
like this idea myself, as it has downsides, too.

* We could ask the stable team to only backport changes once they have
been in mainline for a certain time (something like "at the earliest two
weeks after the change was present in a mainline release or
pre-release"?). But to not delay urgent fixes we then would need
developers to mark the urgent ones somehow. That is likely a hard sell,
but maybe less so then what the previous point outlined; untangling
could help here, too.

* Maybe convince the stable team to consider all commits with just a
Fixes: tag as "non urgent", if they were merged during a merge window
with a committer (or author?) date from before the merge window -- and
then only backport them after -rc4 to ensure they got at least three
weeks of mainline testing before they are backported. This is imperfect
and has downsides, but would be relatively simple to realize.

* We could extend the Fixes tag in a fashion similar to the stable tag
(see above) to establish something like `Fixes: cafec0cacafe ("foo: bar:
foobar baz") # after -rc4 if considered backportworthy` -- but some of
these lines will become awfully long (they already are occasionally even
without this add-on note).

Ciao, Thorsten

P.S.: Related things that could be discussed:

* One cause of regressions that happen in stable trees (and not in
mainline) I've seen quite a few times are backports of commits with
Fixes: tags that were part of a patch-series and depend on earlier
patches from the series. The stable-team afaics has no easy way to spot
this, as there is no way to check "was this change part of a series".
Sometimes I wonder if a dedicated tag linking to the submission of a
patch could help -- and is something quite a few maintainers already
really want and add using a "Link" tag despite Linus dislike for that
(IIRC). But following that link for each and every patch slated for
backporting does not scale for the stable team anyway, so it's likely
not worth it.