Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] stable kernel process automation and improvement

[Mark Brown <broonie@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 4113 bytes --]

On Fri, Jul 05, 2019 at 04:12:31PM -0400, Sasha Levin wrote:
> On Fri, Jul 05, 2019 at 05:41:42PM +0100, Mark Brown wrote:

> > I'm a bit worried about these, especially pushed together - one
> > of the things the AUTOSEL stuff does quite often is pull in
> > driver changes and our coverage of drivers is especially weak.

> Our driver coverage is indeed weak, but I don't think that the solution
> is to leave drivers/ alone. On the contrary, I think that making
> drivers/ move quickly together with the rest of the kernel will
> encourage vendors to up their testing game.

I'm not saying leave it alone, it's more a question of how
aggressive we are about picking up things we think might be
relevant fixes but haven't had some sort of domain specific
analysis of.  Testing is a good way to mitigate the potential
risks here.

> This came up in the last MS, and the agreement there was that we expect
> stable kernel users to test their workloads before throwing it into
> production.

That's kind of the problem - if people are doing testing and end
up finding problems coming back in the stable kernel that's the
sort of thing that encourages them to not just take stable en
masse as we say they should.  Part of the deal with stable is
that it is conservative, people can trust it to be a low risk
update.  That's not happening now as far as I'm aware but it does
worry me that it might happen.

> If we were to start avoiding driver updates, it would act as an
> incentive for people not to upgrade their kernel.

I'm not sure I follow the logic here?

> Right now I'm working with a certain hardware vendor who does a crappy
> job at tagging fixes for stable, and it's horribly painful. I end up
> spending time triaging a bug, reporting it to the vendor, only to be
> told "oh grab this fix from upstream".

> This user experience is just bad, and I can't imagine how difficult it
> is for users who are less familiar with the kerenl.

Well, the advice from the upstream community has always been that
you should track upstream and I'm sure people will be praising
this vendor's upstream focus but obviously that's not always
terribly helpful or realistic for production systems.  In my
(mostly embedded and consumer electronics based) experience
support for older kernel versions is generally part of the
commercial discussion with the hardware vendor, there's an
understanding that the hardware will only get bought if it works
on kernel versions that are useful to the customer or (depending
on the power relationships) that the customer will use kernel
versions that the vendor supports.  Sometimes, especially for
smaller customers, that doesn't work out but those are usually
the people who are more likely to track upstream and/or do
considerable testing before fixing a version and generally are on
their own.

This is where the out of tree patch stacks from vendors come from
- everyone agrees that they'll use one or more given kernel
versions, enterprise distros or whatever and then the vendor
commits to supporting what's agreed but often that doesn't just
include bug fixing but also new features (or entirely new bits of
hardware).  As a result those vendors are shipping their patch
stacks out of tree, users are getting their bug fixes from there
and those vendors are not finding much user demand for vanilla
LTS as a separate thing.  They may even find conflicts with it an
annoying hassle.  Frankly for them upstream support is often a
bit of an investment in reducing the cost of future out of tree
patch stacks and giving a longer general market life to products
rather than something customers directly demand.  None of this is
ideal from an upstream point of view of course but it does
function for people.

It sounds like somewhere along the line this process has come
unstuck for you and you have a vendor that's not aligned with
what you need but I don't think that's quite the same question
as the issues with pulling patches into stable without either
testing coverage or direct identification of an issue by someone
with domain knowledge which is what I'm worrying about.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]