[Ksummit-discuss] [CORE TOPIC] Regression tracking

[Ksummit-discuss] [CORE TOPIC] Regression tracking

[Takashi Iwai]

As this topic doesn't seem posted yet, let me pitch.

Until recently, we've had no regression tracking since Rafael stepped
down from the maintenance of regression list.  Now fortunately,
Thorsten started gathering and posting the regression reports on ML
again, and this helps already a lot.

Though, it immediately raised me a few things to be discussed:

- How can we keep it rolling stably at this time?
- How can each developer / maintainer help or be involved better?
- Can we provide better services than only a plain text list on ML?

Also, since the regression on stable trees was a hot topic:

- Should the regression tracking cover the stable trees?  How well
  does it scale?


thanks,

Takashi

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Rafael J. Wysocki]

On Tuesday, July 26, 2016 04:31:28 PM Takashi Iwai wrote:
> As this topic doesn't seem posted yet, let me pitch.

I'm interested in this one.

> Until recently, we've had no regression tracking since Rafael stepped
> down from the maintenance of regression list.

There were other people trying to do that when I had stopped doing it, but
they gave up eventually.

> Now fortunately, Thorsten started gathering and posting the regression
> reports on ML again, and this helps already a lot.
> 
> Though, it immediately raised me a few things to be discussed:
> 
> - How can we keep it rolling stably at this time?
> - How can each developer / maintainer help or be involved better?
> - Can we provide better services than only a plain text list on ML?

I think it is possible, but it also depends on how much time the person
who tracks regressions can spend on it.

It may be automated somewhat (from experience), but still involves quite a bit
of manual work.

The fact that the list is only sent by email makes it a bit hard to follow
(you always need to look for that particular email in your mailbox) and if you
are a maintainer, it isn't easy to say upfront which bugs may be relevant to you
without checking them one by one.

That's why I was sending bug-specific messages to maintainers too when I was
tracking regressions (it was possible ot opt-out from that, but I'm not sure
whether or not that was a good idea).

> Also, since the regression on stable trees was a hot topic:
> 
> - Should the regression tracking cover the stable trees?  How well
>   does it scale?

>From experience, work needed grows linearly with the number of releases you
are trying to track at the same time.  I've never tracked more than two or three
(maybe) at a time IIRC, because of exactly that.

Thanks,
Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Thorsten Leemhuis]

(@Takashi & @Rafel: repost after subscribing to ksummit-discuss, 
which is moderated for non-members :-/ )

On 26.07.2016 22:36, Rafael J. Wysocki wrote:
> > On Tuesday, July 26, 2016 04:31:28 PM Takashi Iwai wrote:
>> >> As this topic doesn't seem posted yet, let me pitch.
> > I'm interested in this one.
> > 
>> >> Until recently, we've had no regression tracking since Rafael stepped
>> >> down from the maintenance of regression list.
> > There were other people trying to do that when I had stopped doing it, but
> > they gave up eventually.

Yeah, I guess that can easily happen sooner or later, because from what
I can see it's a lot of manual and boring work that only now and then
feels rewarding :-/

>> >> Now fortunately, Thorsten started gathering and posting the regression
>> >> reports on ML again, and this helps already a lot.
>> >>
>> >> Though, it immediately raised me a few things to be discussed:
>> >>
>> >> - How can we keep it rolling stably at this time?
>> >> - How can each developer / maintainer help or be involved better?
>> >> - Can we provide better services than only a plain text list on ML?
> > I think it is possible, but it also depends on how much time the person
> > who tracks regressions can spend on it.
> >
> > It may be automated somewhat (from experience), but still involves quite a bit
> > of manual work.

My rough plan after doing regression tracking for a few weeks now: Look
a bit closer at patchwork and check if it can be adjusted to automate
the boring parts of regression tracking. I have a mostly finished mail
that has some more details on what got me thinking into that direction,
but I'm on holiday till Friday and wanted to send it out once I'm home
again.

>> >> Also, since the regression on stable trees was a hot topic:
>> >> - Should the regression tracking cover the stable trees?  How well
>> >>   does it scale?
> > From experience, work needed grows linearly with the number of releases you
> > are trying to track at the same time.  I've never tracked more than two or three
> > (maybe) at a time IIRC, because of exactly that.

Yeah, without more manpower or some automation (like bitkeeper/git made
Linus scale better) it might lead to a quick burn out if one wants to
track more then two releases.

But OTOH I think it would be really good to track at least the latest
stable tree, as quite a few people only start testing kernel versions
like 4.7 once they got released. Sure, it's not as easy to revert
changes after that point, but it worthwhile nevertheless afaics.

Ciao, Thorsten

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Thorsten Leemhuis]

On 27.07.2016 14:28, Thorsten Leemhuis wrote:
> On 26.07.2016 22:36, Rafael J. Wysocki wrote:
>>> On Tuesday, July 26, 2016 04:31:28 PM Takashi Iwai wrote:
> My rough plan after doing regression tracking for a few weeks now: Look
> a bit closer at patchwork and check if it can be adjusted to automate
> the boring parts of regression tracking. I have a mostly finished mail
> that has some more details on what got me thinking into that direction,
> but I'm on holiday till Friday and wanted to send it out once I'm home
> again.

FWIW, I posted my mail to LKML now:
http://lkml.kernel.org/r/e473a90f-7a20-52e8-0158-94f283638f34@leemhuis.info

Some of the thoughts and ideas mentioned there might be relevant for a
kernel summit discussion about regression tracking; let me know if there
is anything else I could do to help.

Ciao, Thorsten

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Joerg Roedel]

On Wed, Jul 27, 2016 at 02:28:26PM +0200, Thorsten Leemhuis wrote:
> On 26.07.2016 22:36, Rafael J. Wysocki wrote:

> > > There were other people trying to do that when I had stopped doing it, but
> > > they gave up eventually.
> 
> Yeah, I guess that can easily happen sooner or later, because from what
> I can see it's a lot of manual and boring work that only now and then
> feels rewarding :-/

I am also interested in this one. Regression tracking is a very time
consuming task and we should discuss how that can be improved, so that
we get a sustainable solution that works for the kernel.

As Thorsten already pointed out, better tooling would help. But I think
this will also involve teaching our testers/users on how to best report
regressions. Part of the discussion should also be on automation, for
example if we could/should revive the kerneloops tool?

Just my few thoughts, but if this is goind to be discussed, I'd like to
particpate.


	Joerg

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Thorsten Leemhuis]

On 09.08.2016 11:40, Joerg Roedel wrote:
> On Wed, Jul 27, 2016 at 02:28:26PM +0200, Thorsten Leemhuis wrote:
>> On 26.07.2016 22:36, Rafael J. Wysocki wrote:
> 
>>>> There were other people trying to do that when I had stopped doing it, but
>>>> they gave up eventually.
>> Yeah, I guess that can easily happen sooner or later, because from what
>> I can see it's a lot of manual and boring work that only now and then
>> feels rewarding :-/
> […] But I think
> this will also involve teaching our testers/users on how to best report
> regressions.

Yeah, definitely. One afaics important part that needs to be done in
this area: make bugzilla.kernel.org work better. At least some (or maybe
even quite a lot?) of the bugs reported there afaics seem to not get
proper attention from the right people. Due to that I'd assume some
testers will have come to conclusions like "testing for regressions and
bugs is not worth the trouble, as the kernel developers are not
interested in the bugs I file anyway".

> Part of the discussion should also be on automation, for
> example if we could/should revive the kerneloops tool?

Ugh, what's the latest status of it? I just saw
http://www.kerneloops.org/ is kind of alive again as
http://oops.kernel.org/ (I had missed that) and has data from recent
kernel versions. But it seems the code behind it hasn't seen much
changes in the last two years:
https://github.com/oops-kernel-org/kerneloops/commits/master

Side note: kernelopps and error reporting was talked about on KS2013:
https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these
days by default installs abrt-addon-kerneloops (see
https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
for a very brief intro). Things it picks up can be seen on
https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package
maintainers use the data afaik, but I don't know any more details.
Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not
something other distros can easily integrate like it happened with the
original kerneloops tool years ago.

>[...]

Ciao, Thorsten

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Takashi Iwai]

On Tue, 09 Aug 2016 21:02:22 +0200,
Thorsten Leemhuis wrote:
> 
> On 09.08.2016 11:40, Joerg Roedel wrote:
> > On Wed, Jul 27, 2016 at 02:28:26PM +0200, Thorsten Leemhuis wrote:
> >> On 26.07.2016 22:36, Rafael J. Wysocki wrote:
> > 
> >>>> There were other people trying to do that when I had stopped doing it, but
> >>>> they gave up eventually.
> >> Yeah, I guess that can easily happen sooner or later, because from what
> >> I can see it's a lot of manual and boring work that only now and then
> >> feels rewarding :-/
> > […] But I think
> > this will also involve teaching our testers/users on how to best report
> > regressions.
> 
> Yeah, definitely. One afaics important part that needs to be done in
> this area: make bugzilla.kernel.org work better. At least some (or maybe
> even quite a lot?) of the bugs reported there afaics seem to not get
> proper attention from the right people. Due to that I'd assume some
> testers will have come to conclusions like "testing for regressions and
> bugs is not worth the trouble, as the kernel developers are not
> interested in the bugs I file anyway".

Right, unfortunately Bugzilla is often left uncared although we've
already got a regression report.

Meanwhile there is a regression flag already in bugzilla.kernel.org,
and this could be automatically sending a notification to you.  Then
you can ping the corresponding maintainer.

> > Part of the discussion should also be on automation, for
> > example if we could/should revive the kerneloops tool?
> 
> Ugh, what's the latest status of it? I just saw
> http://www.kerneloops.org/ is kind of alive again as
> http://oops.kernel.org/ (I had missed that) and has data from recent
> kernel versions. But it seems the code behind it hasn't seen much
> changes in the last two years:
> https://github.com/oops-kernel-org/kerneloops/commits/master
> 
> Side note: kernelopps and error reporting was talked about on KS2013:
> https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these
> days by default installs abrt-addon-kerneloops (see
> https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
> for a very brief intro). Things it picks up can be seen on
> https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package
> maintainers use the data afaik, but I don't know any more details.
> Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not
> something other distros can easily integrate like it happened with the
> original kerneloops tool years ago.

Yeah, I'm afraid that it's too heavy-weight.  And overall, I guess
you'd get way too many noises by that, more than the really useful
data.

IMO, the possible tactics is to educate subsystem maintainers to
properly report regressions.  Ideally, you wouldn't have to handle any
bugs by yourself, but just let your slaves debugging, and report back
to you the status :)


Takashi

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Joerg Roedel]

On Tue, Aug 09, 2016 at 09:02:22PM +0200, Thorsten Leemhuis wrote:
> Yeah, definitely. One afaics important part that needs to be done in
> this area: make bugzilla.kernel.org work better. At least some (or maybe
> even quite a lot?) of the bugs reported there afaics seem to not get
> proper attention from the right people. Due to that I'd assume some
> testers will have come to conclusions like "testing for regressions and
> bugs is not worth the trouble, as the kernel developers are not
> interested in the bugs I file anyway".

In my case, I am only looking into the kernel.org bugzilla if someone
pings me about it. Usually I get the upstream bug reports via email. And
this is part of the problem, that we have multiple ways that we teach
users how to report bugs/regressions. That really increases the effort
needed for you to compile the regression list and keep track of
everything in there.

So the right step here is to discuss how we want to get bugs reported
and settle on a single way we tell the users and testers. From there it
is much easier to get the reports dispatched to the right people.

> Ugh, what's the latest status of it? I just saw
> http://www.kerneloops.org/ is kind of alive again as
> http://oops.kernel.org/ (I had missed that) and has data from recent
> kernel versions. But it seems the code behind it hasn't seen much
> changes in the last two years:
> https://github.com/oops-kernel-org/kerneloops/commits/master

Interesting, I wasn't aware of that either. From a first look it seems
that the reporting tool is probably not widely used, at least not like
the Fedora tool.

> Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not
> something other distros can easily integrate like it happened with the
> original kerneloops tool years ago.

Well, but maybe parts of that software could be re-used for our
regression tracking. The oops-parser is really cool, as it extracts the
stack-traces and finds the source-files the symbols belong to.

You can extract the same information from LKML bug-reports and use the
file-names together with the MAINTAINERS file to automatically compile
an initial list of potential developers who might look at the report.

To further automate this, it could be put into a mail-filter which is
called by a procmail rule against LKML mails and that spits out
potential bug/regression reports.

Certainly a few heuristics are needed too to improve the signal-to-noise
ratio, but over time this might become a helpful automation step.


	Joerg

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Rafael J. Wysocki]

On Wednesday, August 10, 2016 12:31:18 AM Joerg Roedel wrote:
> On Tue, Aug 09, 2016 at 09:02:22PM +0200, Thorsten Leemhuis wrote:
> > Yeah, definitely. One afaics important part that needs to be done in
> > this area: make bugzilla.kernel.org work better. At least some (or maybe
> > even quite a lot?) of the bugs reported there afaics seem to not get
> > proper attention from the right people. Due to that I'd assume some
> > testers will have come to conclusions like "testing for regressions and
> > bugs is not worth the trouble, as the kernel developers are not
> > interested in the bugs I file anyway".
> 
> In my case, I am only looking into the kernel.org bugzilla if someone
> pings me about it. Usually I get the upstream bug reports via email. And
> this is part of the problem, that we have multiple ways that we teach
> users how to report bugs/regressions. That really increases the effort
> needed for you to compile the regression list and keep track of
> everything in there.
> 
> So the right step here is to discuss how we want to get bugs reported
> and settle on a single way we tell the users and testers. From there it
> is much easier to get the reports dispatched to the right people.

I need to mention here that PM and ACPI regressions in bugzilla.kernel.org
are actually tracked by a team at Intel.  We track all of the PM and ACPI
bugs filed via the BZ, including regressions, and we're going to be doing
that.

Thanks,
Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[James Bottomley]

On Wed, 2016-08-10 at 00:31 +0200, Joerg Roedel wrote:
> On Tue, Aug 09, 2016 at 09:02:22PM +0200, Thorsten Leemhuis wrote:
> > Yeah, definitely. One afaics important part that needs to be done 
> > in this area: make bugzilla.kernel.org work better. At least some 
> > (or maybe even quite a lot?) of the bugs reported there afaics seem 
> > to not get proper attention from the right people. Due to that I'd 
> > assume some testers will have come to conclusions like "testing for 
> > regressions and bugs is not worth the trouble, as the kernel 
> > developers are not interested in the bugs I file anyway".
> 
> In my case, I am only looking into the kernel.org bugzilla if someone
> pings me about it. Usually I get the upstream bug reports via email.


You can have bugzilla.kernel.org configured to send your list email
when someone reports a new bug.

It used to actually be useful: you could reply and it would capture
that in the bugzilla entry.  However, since the migration to bugzilla
version 3, this has started to fail, so you need to add the reporter's
cc directly as well now.

Once you get this configuration set up it means that bugzilla and email
can be directly equivalent, it's just that only email will have the
complete bug text (because bugzilla is lossy on the replies).

James

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Jani Nikula]

On Wed, 10 Aug 2016, Joerg Roedel <joro@8bytes.org> wrote:
> In my case, I am only looking into the kernel.org bugzilla if someone
> pings me about it. Usually I get the upstream bug reports via email. And
> this is part of the problem, that we have multiple ways that we teach
> users how to report bugs/regressions. That really increases the effort
> needed for you to compile the regression list and keep track of
> everything in there.

How about adding an entry, say "B:", to MAINTAINERS to let each
subsystem/driver specify the preferred method of reporting bugs? For
example, most drivers/gpu/drm bugs would get more attention at
https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Joerg Roedel]

On Wed, Aug 10, 2016 at 03:03:42PM +0300, Jani Nikula wrote:
> How about adding an entry, say "B:", to MAINTAINERS to let each
> subsystem/driver specify the preferred method of reporting bugs? For
> example, most drivers/gpu/drm bugs would get more attention at
> https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.

I thought about that too, it might make sense if we can't agree on
anything better. In the end it is more hassle for the users and requires
more knowledge on their side. They have to know at least in which part of
the kernel the problem occured and then look that part up in the
MAINTAINERS file to find out how to report the bug. There are probably
quite a few users who give up somewhere on that path.


	Joerg

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Joerg Roedel]

On Tue, Aug 09, 2016 at 04:20:02PM -0700, James Bottomley wrote:
> You can have bugzilla.kernel.org configured to send your list email
> when someone reports a new bug.
> 
> It used to actually be useful: you could reply and it would capture
> that in the bugzilla entry.  However, since the migration to bugzilla
> version 3, this has started to fail, so you need to add the reporter's
> cc directly as well now.
> 
> Once you get this configuration set up it means that bugzilla and email
> can be directly equivalent, it's just that only email will have the
> complete bug text (because bugzilla is lossy on the replies).

That would be start, at least. For my case, there is not even an IOMMU
category yet, I'll try to open a bug to change that and see what
happens :)


	Joerg

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Daniel Vetter]

On Wed, Aug 10, 2016 at 5:01 PM, Joerg Roedel <joro@8bytes.org> wrote:
> On Wed, Aug 10, 2016 at 03:03:42PM +0300, Jani Nikula wrote:
>> How about adding an entry, say "B:", to MAINTAINERS to let each
>> subsystem/driver specify the preferred method of reporting bugs? For
>> example, most drivers/gpu/drm bugs would get more attention at
>> https://bugs.freedesktop.org/ than https://bugzilla.kernel.org/.
>
> I thought about that too, it might make sense if we can't agree on
> anything better. In the end it is more hassle for the users and requires
> more knowledge on their side. They have to know at least in which part of
> the kernel the problem occured and then look that part up in the
> MAINTAINERS file to find out how to report the bug. There are probably
> quite a few users who give up somewhere on that path.

We already have this trouble on the gfx side, just worse: Users
generally have no idea which one of kernel driver, X driver, mesa/gl,
libva/video or whatever exactly blew up. Which is why we're
redirecting everyone to bugs.freedesktop.org, there we can at elast
reassign between the different parts.

Of course that then means we can't easily reassing to other kernel
components, but that seems to happen far less often (for us).

Anyway, +1 from me for B: in MAINTAINERS to make it at least possible
for users to figure out where to file a bug. Default would be mailo:
<mailing list from same maintainer entry>.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [CORE TOPIC] Regression tracking

[Laura Abbott]

On 08/09/2016 12:02 PM, Thorsten Leemhuis wrote:
> On 09.08.2016 11:40, Joerg Roedel wrote:
>> On Wed, Jul 27, 2016 at 02:28:26PM +0200, Thorsten Leemhuis wrote:
>>> On 26.07.2016 22:36, Rafael J. Wysocki wrote:
>>
>>>>> There were other people trying to do that when I had stopped doing it, but
>>>>> they gave up eventually.
>>> Yeah, I guess that can easily happen sooner or later, because from what
>>> I can see it's a lot of manual and boring work that only now and then
>>> feels rewarding :-/
>> […] But I think
>> this will also involve teaching our testers/users on how to best report
>> regressions.
>
> Yeah, definitely. One afaics important part that needs to be done in
> this area: make bugzilla.kernel.org work better. At least some (or maybe
> even quite a lot?) of the bugs reported there afaics seem to not get
> proper attention from the right people. Due to that I'd assume some
> testers will have come to conclusions like "testing for regressions and
> bugs is not worth the trouble, as the kernel developers are not
> interested in the bugs I file anyway".
>
>> Part of the discussion should also be on automation, for
>> example if we could/should revive the kerneloops tool?
>
> Ugh, what's the latest status of it? I just saw
> http://www.kerneloops.org/ is kind of alive again as
> http://oops.kernel.org/ (I had missed that) and has data from recent
> kernel versions. But it seems the code behind it hasn't seen much
> changes in the last two years:
> https://github.com/oops-kernel-org/kerneloops/commits/master
>
> Side note: kernelopps and error reporting was talked about on KS2013:
> https://lwn.net/Articles/571997/ ABRT was mentioned there. Fedora these
> days by default installs abrt-addon-kerneloops (see
> https://abrt.readthedocs.io/en/latest/supported_langs.html#linux-kernel
> for a very brief intro). Things it picks up can be seen on
> https://retrace.fedoraproject.org/faf/summary/ The Fedora kernel package
> maintainers use the data afaik, but I don't know any more details.
> Anyway: ABRT is quite Fedora/RH specific and quite heavy. IOW: it is not
> something other distros can easily integrate like it happened with the
> original kerneloops tool years ago.

Fedora uses ABRT but the data tends to be fairly noisy for kernel regression
tracking from my experience. It gets flooded with reports from old kernel
versions and issues that have been there for a while. It's certainly
possible to use the data and I'm glad it's collected but it still requires
more manual thinking and analysis than I would like. What's much more
useful with ABRT is the one click "file a bugzilla" and duplication detection.


Thanks,
Laura