Re: [RFC PATCH v2 1/2] docs: add a document about regression handling

[Thorsten Leemhuis <linux@leemhuis.info>]

On 07.01.22 18:44, Matthias Brugger wrote:
> On 07/01/2022 17:51, Thorsten Leemhuis wrote:
>> On 07.01.22 16:37, Matthias Brugger wrote:
>>> On 07/01/2022 15:21, Thorsten Leemhuis wrote:
>>>> Create a document explaining various aspects around regression handling
>>>> and tracking both for users and developers. Among others describe the
>>>> first rule of Linux kernel development and what it means in practice.
>>>> Also explain what a regression actually is and how to report one
>>>> properly. The text additionally provides a brief introduction to the
>>>> bot
>>>> the kernel's regression tracker uses to facilitate his work. To sum
>>>> things up, provide a few quotes from Linus to show how serious he takes
>>>> regressions.
>>>>
>>>> Signed-off-by: Thorsten Leemhuis <linux@leemhuis.info>
>>> [...]
>>>> +The important bits for people fixing regressions
>>>> +================================================
>>>> +
>>>> +When receiving regression reports by mail, check if the reporter CCed
>>>> `the
>>>> +regression mailing list <https://lore.kernel.org/regressions/>`_
>>>> +(regressions@lists.linux.dev). If not, forward or bounce the report
>>>> to the Linux
>>>> +kernel's regression tracker (regressions@leemhuis.info), unless you
>>>> plan on
>>>
>>> I would have expected it to be the same mailing list
>>> (regressions@lists.linux.dev), is this a typo maybe?
>>
>> Thx for taking a look. Hmm. That's possible, but I (and the grep call I
>> just ran) fail to spot the typo.
>>
>> Maybe the wording is to confusing: regressions@lists.linux.dev is the
>> list, regressions@leemhuis.info is a dedicated email address I (the
>> kernel's regression tracker) use for regression tracking (which reminds
>> me: maybe I should ask for a alias like regressions@kernel.org or
>> regression-tracker@kernel.org).
> 
> Yes it's the wording then :)
> Anyway I just wonder why you we should send the regression to the
> regressions email list, but only to the tracker email address. For me
> that's the confusing part. I'd expect to send it to the list as well and
> the tracker takes it from there. If for any reason someone does not want
> to send a regression to the list, then he can send it to the tracker
> directly. That's my understanding of how this works. If that's correct
> then I'd say we should just explain the difference.

You comments made be revisit the section and made me spot a few other
issues I found less than ideal. So I rewrote it over the weekend (more
than once, to be precise...). I hope this clears things up.

```
The important bits for people fixing regressions
================================================

When submitting fixes for regressions, add "Link:" tags pointing to all
places where the issue was reported, as tools like the Linux kernel
regression bot 'regzbot' heavily rely on these; these pointers are also
of great value for people looking into the issue some time in the
future, as explained in `Documentation/process/submitting-patches.rst`
and :ref:`Documentation/process/5.Posting.rst <development_posting>`::

       Link:
https://lore.kernel.org/r/30th.anniversary.repost@klaava.Helsinki.FI/
       Link: https://bugzilla.kernel.org/show_bug.cgi?id=1234567890

Let the Linux kernel's regression tracker and all other subscribers of
the `regression mailing list <https://lore.kernel.org/regressions/>`_
(regressions@lists.linux.dev) quickly known about newly reported
regressions:

 * When receiving a mailed report that did not CC the list, immediately
send at least a brief "Reply-all" with the list CCed to get it into the
loop; also ensure its CCed on all future replies, in case it got lost.

 * If you receive a report from a bug tracker, forward or bounce the
report to the list, unless the reporter followed
`Documentation/admin-guide/reporting-issues.rst` instructions and did it
already.

[Optional] Ensure the Linux kernel regression bot 'regzbot' tracks the
issue:

 * For mailed reports, check if the reporter included a 'regzbot
command' like the ``#regzbot introduced v5.13..v5.14-rc1`` described
above. If not, send a reply with the regressions list in CC, which
includes a paragraph like the following:

       #regzbot ^introduced v5.13..v5.14-rc1

  Note, in this case there is a caret (^) before the `introduced` to
make regzbot treat the parent mail (the one you reply to) as the report
for the regression you want to see tracked.

  Instead of specifying a version range you can also state a commit-id,
in case the reporter identified the culprit.

 * When receiving a report from a bug tracker and forwarding it to the
regressions list (see above), include a paragraph like this:

       #regzbot introduced: v5.13..v5.14-rc1
       #regzbot from: Some N. Ice Human <some.human@example.com>
       #regzbot monitor:
http://some.bugtracker.example.com/ticket?id=123456789
```

Note, regzbot does not yet support "#regzbot from" and "#regzbot monitor
<bugtracker>", but I wanted to work on that soon anyway -- and this text
will likely take weeks before it hits mailine, so this shouldn't be a
problem.

Ciao, Thorsten