Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt <rostedt@goodmis.org>]

On Tue, 4 Jul 2017 21:03:22 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> On 03.07.2017 18:30, Steven Rostedt wrote:
> > On Sun, 2 Jul 2017 19:51:43 +0200
> > Thorsten Leemhuis <linux@leemhuis.info> wrote:  
> >>  * How to get subsystems maintainer involved more in regression tracking
> >> to better make sure that reported regressions are tracked and not
> >> forgotten accidentally.  
> > We should push harder for all reproducer tests to be put into
> > selftests. I try to do that myself [...]
> > [...]
> > By adding reproducing tests to selftests, we can easily see what
> > regressions are still there.
> > [...]
> > What is selftests?  (Jeopardy answer for all of the above ;-)  
> 
> Sure, writing and running selftests is a good idea. But as you said
> yourself in the later part of your mail: it won't help much in
> situations where the kernel (or a selftest) needs to run on a certain
> hardware or a specific (and maybe rare or complex) configuration. Sadly
> a lot of the regressions in my recent reports were of this kind afaics :-/
> 
> In fact I got the impression that most of the regressions that might get
> caught by selftests were directly handled by the subsystem maintainer
> and never made it to me or my reports -- and thus I can't ask
> maintainers to write selftests. *If* I got better aware of those
> problems I (a) could make sure they are not forgotten and (b) sooner or
> later could publicly state something like "hey, you had ten regressions
> recently in your subsystem where writing a selftest might have been a
> good idea, but you didn't even write one -- why?" (if we want something
> like that).

I'm betting there's a lot of reproducer code that never makes it into a
test. How do we solve that? Perhaps we need people looking at LKML for
any signs "I did this, and it caused a bug" or "Here's a test case
which can trigger the bug". Each of these instances should end up in
selftests, and I'm sure they are not.

We can't do much for special hardware, even though those tests should
still be in the selftests for those that have the hardware, but we can
do something about special configs. Perhaps selfttests should have a
"config test" section. I have that in my own tests, but I use ktest to
build them.


> 
> > […]  
> >>  * how to make the Linux kernel development so good that the mainstream
> >> distros stop their kernel forks and do what they do with Firefox: Ship
> >> the latest stable version (users get a new version with new features
> >> every few weeks) or a longterm branch (makes a big version jump about
> >> once a year; see Firefox ESR).  
> 
> Hehe, I maybe left the field "regression tracking" to much here and
> wandered too far into QA territory.
> 
> > This wont ever happen (famous last words). Distros want "stable
> > kernels" with new features.   
> 
> Ha, yes, it's a long shot (and maybe more a vague idea to work towards
> to). And maybe Debian stable and RHEL will always use the model they use
> today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and
> some others are updating to the latest Linux kernel release every few
> weeks already and it works fine for them. Maybe we can get Ubuntu and
> others to follow sooner or later.
> 
> Sure, for some people a version jump to a major new kernel release will
> sound crazy, but when Linus introduced the current development scheme  a
> lot of people also said "that will never fly" -- that was 13 years ago
> now and it works quite well. The situation was similar with Firefox as well.
> 
> > That's not what stable is about.  
> 
> That afaics (disclaimer: English is not my mother tongue) depends on the
> interpretation of the word, as it can mean "nothing changes" or "rock
> solid/reliable" (even when two people have a "stable relationship" it
> does not mean that nothing changes between them...).

Nothing to do with what language your mother tongue is ;-)

When the stable releases were created, there was some pretty strict
requirements for what should go into stable. Of course the requirements
have changed throughout the years. But there are big differences in
what Red Hat considers something "stable" and what the Linux stable
releases consider to be stable. That is where I meant that things wont
change.

-- Steve