ksummit.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: James Bottomley <James.Bottomley@hansenpartnership.com>
Cc: Thorsten Leemhuis <linux@leemhuis.info>,
	"ksummit@lists.linux.dev" <ksummit@lists.linux.dev>
Subject: Re: [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions
Date: Thu, 13 Jun 2024 09:06:54 -0400	[thread overview]
Message-ID: <ZmruqWDTG2PK-rbu@sashalap> (raw)
In-Reply-To: <7794a2b09ae4fa73ac35fdaec4858145a665efea.camel@HansenPartnership.com>

On Thu, Jun 13, 2024 at 07:58:58AM -0400, James Bottomley wrote:
>On Thu, 2024-06-13 at 10:42 +0200, Thorsten Leemhuis wrote:
>> The scenario shown at the start of the thread illustrates a problem I
>> see frequently: commits with a Fixes: tag end up in new to stable
>> series releases just days after being mainlined and cause regressions
>> -- just like they do in mainline, which just was not known yet at the
>> time of backporting. This happens extremely often right after merge
>> windows when huge piles of changes are backported to the stable trees
>> each cycle shortly after -rc1 is out (which even some kernel
>> developers apparently are somewhat afraid to test from what I've
>> seen).
>
>I haven't really observed this for curated fixes.  For most subsystems,
>patches with Fixes tags that are cc'd to stable tend to go steadily
>outside the merge window.  Obviously a few arrive within it, but
>usually at roughly the rate they arrive outside it.
>
>What I observe in the merge window is huge piles of patches go into
>stable *without* a cc:stable tag from the autosel machinery (and quite
>a few even without fixes: tags).

Could you provide a concrete example? This shouldn't happen.

>So this does beg a couple of questions:
>
>Since you have the figures, what's the proportion of regressions caused
>by backports to stable without cc:stable tags?

This question came up two years ago and we had statistics around this.
Autosel patches didn't cause more (actually, it was *less*) regressions
than stable tagged ones.

>Could we fix a lot of this by delaying autosel?  It tends to ramp up in
>the merge window when everyone is concentrating on other things, so any
>regressions it causes naturally get ignored for a couple of weeks.

autosel is currently delayed about 3-4 weeks, sometimes more.

-- 
Thanks,
Sasha

  reply	other threads:[~2024-06-13 13:27 UTC|newest]

Thread overview: 107+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-06-13  8:22 [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions Thorsten Leemhuis
2024-06-13  8:26 ` [MAINTAINERS SUMMIT] [1/4] Create written down guidelines for handling regressions Thorsten Leemhuis
2024-09-12 13:33   ` Thorsten Leemhuis
2024-06-13  8:32 ` [MAINTAINERS SUMMIT] [2/4] Ensure recent mainline regression are fixed in latest stable series Thorsten Leemhuis
2024-06-13 11:02   ` Johannes Berg
2024-06-13 11:21     ` Greg KH
2024-06-13 13:18       ` Sasha Levin
2024-06-13 11:17   ` Jiri Kosina
2024-06-13 11:28   ` Laurent Pinchart
2024-06-14  0:50     ` Steven Rostedt
2024-06-14 14:01   ` Mark Brown
2024-06-14 14:32     ` Rafael J. Wysocki
2024-06-13  8:34 ` [MAINTAINERS SUMMIT] [3/4] Elevate handling of regressions that made it to releases deemed for end users Thorsten Leemhuis
2024-06-13 11:34   ` Laurent Pinchart
2024-06-13 11:39     ` Jiri Kosina
2024-06-14 14:10       ` Mark Brown
2024-06-18 12:58         ` Thorsten Leemhuis
2024-06-19 20:25           ` Laurent Pinchart
2024-06-20 10:47             ` Thorsten Leemhuis
2024-06-13 15:56     ` Liam R. Howlett
2024-06-18 12:24     ` Thorsten Leemhuis
2024-06-20 13:20       ` Jani Nikula
2024-06-20 13:35         ` Thorsten Leemhuis
2024-06-20 14:16           ` Mark Brown
2024-06-21  6:47           ` Jiri Kosina
2024-06-21 10:19             ` Thorsten Leemhuis
2024-06-13  8:42 ` [MAINTAINERS SUMMIT] [4/4] Discuss how to better prevent backports of commits that turn out to cause regressions Thorsten Leemhuis
2024-06-13  9:59   ` Jan Kara
2024-06-13 10:18     ` Thorsten Leemhuis
2024-06-13 14:08     ` Konstantin Ryabitsev
2024-06-14  9:19       ` Lee Jones
2024-06-14  9:24         ` Lee Jones
2024-06-14 12:27         ` Konstantin Ryabitsev
2024-06-14 14:26           ` Konstantin Ryabitsev
2024-06-14 14:36             ` Lee Jones
2024-06-14 14:29       ` Michael Ellerman
2024-06-14 14:38         ` Konstantin Ryabitsev
2024-06-14 14:44           ` Rafael J. Wysocki
2024-06-14 15:08           ` Geert Uytterhoeven
2024-06-15 11:29             ` Michael Ellerman
2024-06-17 10:15             ` Jani Nikula
2024-06-17 12:42               ` Geert Uytterhoeven
2024-06-14 15:45           ` Mark Brown
2024-06-14 14:43         ` Mark Brown
2024-06-14 14:51           ` Konstantin Ryabitsev
2024-06-14 15:42             ` Mark Brown
2024-06-14 14:43         ` Steven Rostedt
2024-06-14 14:57           ` Laurent Pinchart
2024-06-16  1:13         ` Linus Torvalds
2024-06-16  3:28           ` Steven Rostedt
2024-06-16  4:59             ` Linus Torvalds
2024-06-16  8:22               ` Paolo Bonzini
2024-06-16  9:05               ` Geert Uytterhoeven
2024-06-16 15:07               ` Steven Rostedt
2024-06-17 13:48               ` Dan Carpenter
2024-06-17 15:23                 ` Dan Carpenter
2024-06-17 14:39               ` Konstantin Ryabitsev
2024-06-17 16:04                 ` Paul E. McKenney
2024-06-17 16:06                   ` Konstantin Ryabitsev
2024-06-17 16:14                     ` Paolo Bonzini
2024-06-17 16:18                       ` Konstantin Ryabitsev
2024-06-17 17:11                         ` Geert Uytterhoeven
2024-06-18 12:05                 ` Michael Ellerman
2024-06-16  7:26           ` Takashi Iwai
2024-06-16  8:10           ` Paolo Bonzini
2024-06-16 11:31             ` Laurent Pinchart
2024-06-16 11:39             ` Takashi Iwai
2024-06-16 16:40             ` Linus Torvalds
2024-06-16  8:31           ` Jiri Kosina
2024-06-16  8:54             ` Geert Uytterhoeven
2024-06-13 19:39     ` Dan Carpenter
2024-06-14  1:00       ` Steven Rostedt
2024-06-13 11:58   ` James Bottomley
2024-06-13 13:06     ` Sasha Levin [this message]
2024-06-13 13:56       ` James Bottomley
2024-06-13 14:02         ` Greg KH
2024-06-13 15:11           ` James Bottomley
2024-06-13 16:27             ` Greg KH
2024-06-14 18:47             ` Sasha Levin
2024-06-17 10:59               ` Vlastimil Babka
2024-06-13 18:08         ` Sasha Levin
2024-06-13 13:45     ` Greg KH
2024-06-13 13:40   ` Sasha Levin
2024-06-18 13:12     ` Thorsten Leemhuis
2024-06-13 14:28   ` Andrew Lunn
2024-06-13 18:14     ` Sasha Levin
2024-06-14 14:41       ` Jan Kara
2024-06-14 15:03         ` Rafael J. Wysocki
2024-06-14 17:46           ` Sasha Levin
2024-06-18 14:43 ` [MAINTAINERS SUMMIT] [0/4] Common scenario for four proposals regarding regressions James Bottomley
2024-06-18 15:50   ` Mark Brown
2024-06-20 10:32   ` Thorsten Leemhuis
2024-06-20 12:57     ` James Bottomley
2024-06-20 13:55       ` Mark Brown
2024-06-20 14:01         ` James Bottomley
2024-06-20 14:42           ` Mark Brown
2024-06-20 16:02             ` James Bottomley
2024-06-20 17:15               ` Mark Brown
2024-06-20 23:25               ` Sasha Levin
2024-06-21  6:33                 ` Thorsten Leemhuis
     [not found]               ` <20240625175131.672d14a4@rorschach.local.home>
2024-06-26  7:36                 ` Greg KH
2024-06-26 18:32                   ` Steven Rostedt
2024-06-26 19:05                     ` James Bottomley
2024-07-25 10:14                   ` Thorsten Leemhuis
2024-07-25 13:14                     ` Greg KH
2024-06-20 16:59       ` Thorsten Leemhuis
2024-06-20 23:18         ` Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ZmruqWDTG2PK-rbu@sashalap \
    --to=sashal@kernel.org \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=ksummit@lists.linux.dev \
    --cc=linux@leemhuis.info \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox