[TECH TOPIC] regressions: our workflows are the biggest enemy

[TECH TOPIC] regressions: our workflows are the biggest enemy

[Thorsten Leemhuis]

Hi! I just submitted the following as a talk/discussion for the kernel
summit track at LPC. Ciao, Thorsten

---

Outline the major pain points why Linux kernel regressions happen and
why resolving some takes a long time as observed by Thorsten during
three years of working as the kernel's regression tracker.

The talk among others will describe why some workflow patterns are what
frequently leads to regressions -- and why they are also a factor why
some subsystems fix regression quickly, while others take weeks or
months to resolve.

While at it, describe what testers, bug reporters, developers, and
maintainers can do to prevent regressions and resolve them more quickly.
Furthermore also cover regzbot, the Regression Tracking Bot Thorsten
uses for his work: what it learned in the recent past, what's on its
roadmap, and where it works poorly or well in practise.

While doing all of the above, keep the audience involved and foster
discussion about the discussed points and anything related.

Assuming Thorsten is invited to the Linux maintainers summit happening
right before the LPC, he'll start this session by quickly recapping what
has been discussed and decided regarding bug/regression reporting,
tracking and handling there.

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Steven Rostedt]

On Wed, 12 Jun 2024 08:34:42 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> The talk among others will describe why some workflow patterns are what
> frequently leads to regressions -- and why they are also a factor why
> some subsystems fix regression quickly, while others take weeks or
> months to resolve.

The above looks more than a TECH TOPIC. If you have real numbers that
point out workflows that are an issue, I think this should be discussed
at Maintainers Summit, and perhaps we should start "encouraging" people
to change their workflows. Especially if we have hard data showing
that it's not working.

/me hopes he's not one of the ones with a bad workflow!

-- Steve

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Guenter Roeck]

On 6/12/24 08:26, Steven Rostedt wrote:
> On Wed, 12 Jun 2024 08:34:42 +0200
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
>> The talk among others will describe why some workflow patterns are what
>> frequently leads to regressions -- and why they are also a factor why
>> some subsystems fix regression quickly, while others take weeks or
>> months to resolve.
> 
> The above looks more than a TECH TOPIC. If you have real numbers that
> point out workflows that are an issue, I think this should be discussed
> at Maintainers Summit, and perhaps we should start "encouraging" people
> to change their workflows. Especially if we have hard data showing
> that it's not working.
> 
> /me hopes he's not one of the ones with a bad workflow!
> 

Same here...

Guenter

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Thorsten Leemhuis]

On 12.06.24 17:50, Guenter Roeck wrote:
> On 6/12/24 08:26, Steven Rostedt wrote:
>> On Wed, 12 Jun 2024 08:34:42 +0200
>> Thorsten Leemhuis <linux@leemhuis.info> wrote:
>>
>>> The talk among others will describe why some workflow patterns are what
>>> frequently leads to regressions -- and why they are also a factor why
>>> some subsystems fix regression quickly, while others take weeks or
>>> months to resolve.
>>
>> The above looks more than a TECH TOPIC.

Stuff for the maintainers summit is still in my editor.

>> If you have real numbers that point out workflows that are an issue,

No, I sadly do not. And they would be flawed anyway, as there are many
regressions I never become aware of -- and it's not really possible to
determine from commits if they were fixing a regression or not. So it
could easily be possible that I only see the bad cases and miss out on a
lot of cases that a subsystem handles well. Not to mention that I
currently miss out on a most DRM bugs, as they use a external bug
tracker. Regzbot can monitor that now, but I have no easy means to
become aware of regressions reported there. The idea was to start vising
a handful of issue trackers every day or two and use a boring search,
but as of now that's still an unrealized plan.

>> I think this should be discussed
>> at Maintainers Summit, and perhaps we should start "encouraging" people
>> to change their workflows. Especially if we have hard data showing
>> that it's not working.
>>
>> /me hopes he's not one of the ones with a bad workflow!
>
> Same here...

Don't worry folks, my plan was to primarily mention the subsystems that
from my point of view are doing a good job and highlight what they are
doing well. Apart from maybe one case -- but I will only do so after
asking the stakeholders for an okay.

Ciao, Thorsten

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Liam R. Howlett]

* Thorsten Leemhuis <linux@leemhuis.info> [240612 12:08]:
> On 12.06.24 17:50, Guenter Roeck wrote:
> > On 6/12/24 08:26, Steven Rostedt wrote:
> >> On Wed, 12 Jun 2024 08:34:42 +0200
> >> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> >>
> >>> The talk among others will describe why some workflow patterns are what
> >>> frequently leads to regressions -- and why they are also a factor why
> >>> some subsystems fix regression quickly, while others take weeks or
> >>> months to resolve.
> >>
> >> The above looks more than a TECH TOPIC.
> 
> Stuff for the maintainers summit is still in my editor.
> 
> >> If you have real numbers that point out workflows that are an issue,
> 
> No, I sadly do not. And they would be flawed anyway, as there are many
> regressions I never become aware of -- and it's not really possible to
> determine from commits if they were fixing a regression or not. So it
> could easily be possible that I only see the bad cases and miss out on a
> lot of cases that a subsystem handles well. Not to mention that I
> currently miss out on a most DRM bugs, as they use a external bug
> tracker. Regzbot can monitor that now, but I have no easy means to
> become aware of regressions reported there. The idea was to start vising
> a handful of issue trackers every day or two and use a boring search,
> but as of now that's still an unrealized plan.
> 
> >> I think this should be discussed
> >> at Maintainers Summit, and perhaps we should start "encouraging" people
> >> to change their workflows. Especially if we have hard data showing
> >> that it's not working.

That would limit the number of people who can see this discussion as not
all maintainers are at the Maintainers Summit.  I suspect that the ones
that are not at the Maintainers Summit will be the same ones who will
benefit most from workflow "encouraging".

Although, I'm a little concerned about what I'm singing up for here...

Thanks,
Liam

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Thorsten Leemhuis]

On 12.06.24 20:35, Liam R. Howlett wrote:
> * Thorsten Leemhuis <linux@leemhuis.info> [240612 12:08]:
>> On 12.06.24 17:50, Guenter Roeck wrote:
>>> On 6/12/24 08:26, Steven Rostedt wrote:
>>>> On Wed, 12 Jun 2024 08:34:42 +0200
>>>> Thorsten Leemhuis <linux@leemhuis.info> wrote:
>
>>>> I think this should be discussed
>>>> at Maintainers Summit, and perhaps we should start "encouraging" people
>>>> to change their workflows. Especially if we have hard data showing
>>>> that it's not working.
> [...]
> Although, I'm a little concerned about what I'm singing up for here...

See the four maintainers summit proposals I just sent for an impression
where I see trouble and how to fix that:

https://lore.kernel.org/all/c6be1b86-f224-417c-a501-6c778999a04f@leemhuis.info/

Ohh, and btw for those that worry that their subsystem is one that from
my point of view is not handling regressions well: if mails from my
other email account (the one dedicated to regressions tracking) rarely
show up in your inbox/your subsystem's mailing list, then things are
most likely fine.

Ciao, Thorsten

Re: [TECH TOPIC] regressions: our workflows are the biggest enemy

[Leon Romanovsky]

On Wed, Jun 12, 2024 at 11:26:22AM -0400, Steven Rostedt wrote:
> On Wed, 12 Jun 2024 08:34:42 +0200
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
> > The talk among others will describe why some workflow patterns are what
> > frequently leads to regressions -- and why they are also a factor why
> > some subsystems fix regression quickly, while others take weeks or
> > months to resolve.
> 
> The above looks more than a TECH TOPIC. If you have real numbers that
> point out workflows that are an issue, I think this should be discussed
> at Maintainers Summit, and perhaps we should start "encouraging" people
> to change their workflows. 

This topic is very interesting to whole Linux kernel community, and I think
it should be presented in the main track and not limited to Maintainers Summit.
Especially if you want "to encourage people to change their workflows".

Thanks