Re: [MAINTAINERS SUMMIT] [1/4] Create written down guidelines for handling regressions

[Thorsten Leemhuis <linux@leemhuis.info>]

On 13.06.24 10:26, Thorsten Leemhuis wrote:
> Different assumptions about the appropriate handling of regressions
> frequently lead to friction and time consuming discussions during my
> regression tracking and prodding work. That is frustrating, demotivating
> and exhausting for everyone involved and even brought us to situations
> like "then I'm stepping down as maintainer". To avoid things like this,
> I propose we try to pin down guidelines together and ideally make Linus
> bless them.
> 
> The "Expectations and best practices for fixing regressions" in
> Documentation/process/handling-regressions.rst (
> https://docs.kernel.org/process/handling-regressions.html#expectations-and-best-practices-for-fixing-regressions
> ) could be a start for such guidelines -- but I'm obviously biased here,
> as I wrote that text, so feel free to propose something new.
> 
> That text is based on generalized interpretations of statements and
> actions from Linus while keeping practical application and our workflows
> in mind -- including the maintenance of stable trees. I have no idea if
> I went too far somewhere: the submission of that text was addressed to
> Linus, but he did not react; otoh he merged it later after Greg ACKed it
> and it came to his doorstep through the docs tree.
> 
> But in the end it seems most people do not know about this text or do
> not take it for real. [...]

Lo! The discussion here rightfully exposed that the wording regarding
the stable tag was way to strong. Sorry for that, not sure how that
happened, that was not my intend. That and a few other aspects (some
from the discussions here) made me revisit the text regarding
"Expectations and best practices for fixing regressions". See below for
my current draft (the diff view is not really helpful, sorry). Note,
should be easy to add a week or two to any sections regarding the
timing aspects; guess that is best discussed on the summit.

As I said earlier, the text is based on generalized interpretations of
statements and actions from Linus with some interpolation. But in some
areas what I wrote might be not what Linus wants. To sort this out I'm
currently also preparing a few scenarios with related questions for the
maintainers summit audience (incl. Linus) that hopefully will help to
keep the discussion fruitful, targeted, and as short as possible. More
on that on Tuesday.

Ciao, Thorsten
---

Expectations and best practices for fixing regressions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Try to quickly resolve regressions in mainline while applying reasonable care to
prevent additional problems. The appropriate balance depends on the situation;
most regressions should ideally be resolved through a fix or a revert by the
last Sunday within the next two or three weeks after the culprit was identified.

The rules of thumb below outline the appropriate procedure in more detail. The
overall goal is to prevent situations where a regression caused by a recent
change leaves users only three bad options: use a kernel with a regression that
impacts usage, switch to a different kernel series, or run an outdated and thus
potentially insecure kernel for more than two or three weeks. 

In general:

 * Prioritize work on providing, reviewing, and mainlining regression fixes over
   other upstream Linux kernel work, unless the latter concerns severe issues
   (e.g. acute security vulnerabilities, data loss, or bricked hardware).

 * Do not consider fixing regressions from the current development cycle as
   something that can wait till its end: the issue possibly prevents users or
   CI systems from testing, which might drive testers away and mask other bugs.

 * When developing a fix, apply the required care to avoid additional damage. Do
   so even when resolving a regression might take longer than outlined below --
   at least unless a revert could resolve it, as then you should opt for one.

 * Reviewers and maintainers likewise should apply the required care, but at the
   same time should try to route regression fixes quickly through the ranks.

On timing once the change causing the regression became known:

 * If the regression is severe, aim to mainline a fix within two or three work
   days and ideally before the next Sunday; do the same it its is bothering many
   users in general or most people in prevalent environments (say a widespread 
   hardware device, a popular Linux distribution, or a stable/longterm series).

 * Aim to mainline a fix by Sunday after the next, if the culprit made it into
   a kernel deemed for end users during the past three months -- either directly
   through a mainline release or through backports to stable or longterm series.
   If the culprit became known early during a week while being simple to resolve
   using a low-risk patch, try to mainline the fix within the same week instead.

 * For other regressions introduced during the past twelve months, aim to
   mainline a fix before the hindmost Sunday within the next three weeks. One or
   two weeks later are acceptable, if the regression is unlikely to bother more
   than a user or two or is something people can easily live with temporarily.

 * Try your best to mainline a fix before the current development cycle ends,
   unless the culprit was committed more than a year ago: then it is acceptable
   to queue a fix for the next merge window, which definitely should be done in
   case it bear bigger risks.

On patch flow to mainline:

 * Developers, when trying to reach the time periods mentioned above, remember
   to account for the time it will take to test, review, commit, and mainline
   fixes, ideally with them being in linux-next at least briefly. Hence, if
   fixes are urgent, make it obvious to ensure others handle them appropriately.

 * Reviewers, you are kindly asked to assist developers in reaching the time
   periods mentioned above by reviewing regression fixes in a timely manner.

 * Maintainers, you likewise are kindly asked to expedite the handling
   of regression fixes. Thus when beneficial evaluate if skipping linux-next
   might be an option. Also consider sending git pull requests more often than
   usual when appropriate. And try to avoid holding onto regression fixes over
   weekends -- especially when some are marked for backporting to stable series.

On procedure:

 * If a regression seems tangly, precarious or urgent, consider CCing Linus on
   discussions or patch review; do the same if the responsible maintainers
   suspected to be unavailable. 

 * For an urgent regression, consider asking Linus to pick up a fix straight
   from the mailing list: he is totally fine with that for uncontroversial
   fixes. Such requests should ideally come directly from maintainers or happen
   in accordance with them.

 * In case you are unsure if a fix is worth the risk applying just days before
   a new mainline release, send Linus a mail with the usual lists and developers
   in CC; in it, summarize the situation while asking to pick up the fix
   straight from the list. Linus then can make the call and when appropriate
   even postpone the release. Such requests again should ideally come directly
   from maintainers or happen in accordance with them.

On tagging in the patch description:

 * Include the tags Documentation/process/submitting-patches.rst mentions for
   regressions; this usually means a "Reported-by:" tag followed by "Link:" or
   "Closes:" tag pointing to the report as well as a "Fixes:" tag; if it's a
   regression a later change exposed, add a "Fixes:" tag for that one, too.

 * Did the culprit make it into a proper mainline release during the past twelve
   months? Or is it a recent mainline commit backported to stable or longterm
   releases in the past few weeks? Then you are kindly asked to ensure stable
   inclusion as described by Documentation/process/stable-kernel-rules.rst, e.g.
   by adding a "Cc: stable@vger.kernel.org" to the patch description. Note, a
   "Fixes:" tag alone does not guarantee a backport: the stable team sometimes
   silently drop such changes, for example when they do not apply cleanly.

Regarding stable and longterm kernels:

 * When receiving reports about regressions in recent stable or longterm kernel
   series, please consider evaluating at least briefly, if the issue might
   happen in current mainline as well -- and if that seems likely, take hold of
   the report. If in doubt, ask the reporter to check mainline.

 * You are free to leave handling regressions to the stable team, if the problem
   at no point in time occurred with mainline or was fixed there already.

 * Whenever you want to swiftly resolve a mainline regression that recently made
   it into a mainline, stable, or longterm release, fix it quickly in mainline;
   in urgent cases thus involve Linus to fast-track fixes (see above). That's
   required, as the stable team normally does neither revert nor fix any changes
   in their trees as long as those cause the same problem in mainline.

 * In case of urgent fixes for regression affecting stable or longterm kernels,
   you might want to ensure prompt backporting by dropping the stable team a
   note once the fix was mainlined; this is especially advisable during merge
   windows and shortly thereafter, as the fix otherwise might land at the end
   of a huge patch queue.