Re: [Ksummit-discuss] [CORE TOPIC] stable workflow

[Olof Johansson <olof@lixom.net>]

[-- Attachment #1: Type: text/plain, Size: 5329 bytes --]

On Sun, Jul 10, 2016 at 3:39 PM, Theodore Ts'o <tytso@mit.edu> wrote:

> On Sun, Jul 10, 2016 at 11:28:21AM -0700, Guenter Roeck wrote:
> > > There are **eleven** stable or longterm trees listed on kernel.org.
> >
> > I think this is one of the problems we are having: There are way too many
> > stable / longterm trees.
>
> Part of this is because it's too easy for someone to say, "I want to
> support [34].XX as a stable kernel".  Maybe it will only be for one
> architecture and only used for one platform (e.g. Yacto, or some other
> random distribution), but it's not immediately obvious (a) who is
> going to be using the stable kernel, and (b) what sort of testing it is
> actually getting.
>
> This is fine if stable kernels are advertised as being "best efforts
> only; whatever an individual stable kernel maintainer feels like
> putting into the project".  Which is fine, but then it's also no
> surprise if device kernel maintainers and BSP kernel maintainers
> aren't aren't taking the -stable kernel series.  And it also becomes
> surprising if other people are expecting that stable trees are
> supposed to be more stable than that, and then get indignant when
> there are regressions, bug fixes that aren't backported, bug fixes
> that work fine on the tip but which break after getting backported,
> etc.
>
> To be clear, though: That's the way things are right now, and someone
> who wants to change it is going to have to propose a procedure which
> ends up taking less work on maintainers and individual patch
> submiters, and/or volunteers to do the extra work, or realistically,
> it's not going to happen.
>
> > I think we are having kind of a circular problem: Device/BSP kernels
> > don't track stable because stable branches are considered to be not
> stable
> > enough, and stable branches are not tested well enough because they are
> not
> > picked up anyway. The only means to break that circle is to improve
> > stable testing to the point where people do feel comfortable picking it
> up.
> >
> > The key to solving that problem might be automation. There are lots of
> tools
> > available nowadays which could be used for that purpose (gerrit,
> buildbot, ...).
> > Patch submissions to stable releases could be run through an automated
> test
> > system and only be applied to stable release candidates after all tests
> passed.
> > This is widely done with vendor kernels today, and should be possible for
> > stable kernels as well. Such a system could even pick up patches tagged
> > with Fixes: or with Cc: stable from mainline automatically.
>
> Testing works fine for core kernel features and for things like file
> systems.  But it really doesn't work with real hardware, and Olaf
> described a couple of scenarios where fixes to device drivers broke
> older hardware supported by the same driver.  If what we are most
> worried about is "no regressions", one really extreme approach would
> be for a particular stable kernel series, to have a branch which
> *only* has patches for which reliable and comprehensive tests exist.
> This branch would at least get all of the security fixes and other bug
> fixes which are applicable to the core kernel, and but it would filter
> out, at least initially, all or most device driver patches.
>
> We could have another branch which includes the device driver fixes,
> and perhaps over time we could figure out some scheme by which if the
> significant device kernel and BSP kernel users could be convinced to
> contribute hardware and some test engineer resources, maybe some of
> the device driver fixes could go into the "tested" stable branch as
> well.
>
> Or maybe we just leave a clean separation between "core" and "device
> driver" stable branches, since in practice the answer seems to be that
> once an embedded device kernel maintainer gets things working, they
> **really** don't want to touch the device drivers ever again, since if
> there are any hardware or software issues, they want users buying an
> upgraded device every 12-18 months anyway.  :-)    At least that way
> maybe the users will get the core security and stability fixes....
>
> Or maybe we have a different policy for x86-specific device drivers
> than we do for the embedded architectures, since in practice we have
> more end users testing the x86 stable kernels, where as the embedded
> architectures tend to get things like OTA updates, and so it's not
> surprising that those maintainers are much more paranoid about driver
> changes which might brick their devices.
>
> (Yes, I know that some drivers are shared between x86 and ARM; and I
> suspect that's one of the places where we could easily have a problem
> where a bugfix that fixes things for an device on an x86 base might
> accidentally cause a regression for the same device hanging off of a
> different bus in a SOC configuration....  and no amount of test
> automation has any *hope* of catching thoes sorts of problems.)
>

Just to clarify, my commentary was NOT for ARM SoC support. It was for
drivers frequently used on x86 laptops. So it's not an "embedded only"
problem.

That being said, this was several years ago, and it's not necessarily worth
focusing all that much on -- I just wanted to give an example of a case
where using -stable in a product tree got pushback and why.


-Olof

[-- Attachment #2: Type: text/html, Size: 6266 bytes --]