Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] stable kernel process automation and improvement

[Mark Brown <broonie@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 4321 bytes --]

On Mon, Jul 08, 2019 at 07:02:08AM -0400, Sasha Levin wrote:
> On Sat, Jul 06, 2019 at 01:32:14AM +0100, Mark Brown wrote:

> > I'm not saying leave it alone, it's more a question of how
> > aggressive we are about picking up things we think might be
> > relevant fixes but haven't had some sort of domain specific
> > analysis of.  Testing is a good way to mitigate the potential
> > risks here.

> I agree, and for various subsystems and drivers where the maintainers
> volunteer their domain specific expertise to send backports to stable, I
> have "blacklisted" it from AUTOSEL since indeed it's a much better
> option.

Hrm, it's definitely getting a bunch of stuff for my subsystems
where I do tag things for stable...

> > > This came up in the last MS, and the agreement there was that we expect
> > > stable kernel users to test their workloads before throwing it into
> > > production.

> > That's kind of the problem - if people are doing testing and end
> > up finding problems coming back in the stable kernel that's the
> > sort of thing that encourages them to not just take stable en
> > masse as we say they should.  Part of the deal with stable is
> > that it is conservative, people can trust it to be a low risk
> > update.  That's not happening now as far as I'm aware but it does
> > worry me that it might happen.

> Right, and the rate at which AUTOSEL commits are reverted is lower than
> commits that are actually tagged for stable. If AUTOSEL commits on their
> own were being reverted left and right I'd agree we need to tone it
> down, but I don't see it happening now.

I'm not sure how many people will actually report problems they
experience upstream rather than just fixing things locally and
just moving on.  The more code is the more likely it is that one
of the users will report things.

> > > If we were to start avoiding driver updates, it would act as an
> > > incentive for people not to upgrade their kernel.

> > I'm not sure I follow the logic here?

> The way I see it, the lower your "effective delta" is between to
> kernels, the easier it is to move forward. For example, if I have a
> product that runs on 4.19 and uses all our core kernel code + 10
> drivers, and I know that those drivers had most of the fixes backported
> to my LTS tree, I'd feel much more confident going to 5.4 knowning that
> I already have most of the patches that come with 5.4.

I see, that's definitely a new one to me.  The concerns people
usually have about upgrading are more around the core kernel
changing performance characteristics or something in a way that
disrupts important workloads.  I'm not quite sure I follow the
logic there TBH, it seems to be discounting new development
rather too much - even if the drivers have been very static
there's all the integration with the rest of the kernel to think
about.

> For me it's a matter of how one would budget a move from a kernel X LTS
> to kernel Y LTS, and I think that as that budget requirement grows it's
> actually harder to actually do it (and convince management), acting as a
> negative incentive to stay with whatever works now.

If the drivers are static enough to only be getting bug fixes
surely the rest of the kernel is a massively more substantial
concern?

> I have a different experience with this. I'd like to think that we're a
> bigger customer and this process wasn't working too well for us. My
> thinking was that if it's broken for us I can only imagine how bad it is
> for the smaller customers.

...

> This is where our story is different, which might explain my experience
> being different: we usually require vendors to upstream everything, and
> so they do. This means we don't have much of a out-of-tree patch
> stacks/fixes from the vendor directly, and we expect to pick up patches
> via the regular stable process, and that didn't happen all too well so
> far.

That sounds like they didn't pick up on the bit about getting
things through LTS.  This sounds like a pretty unusual request
for a vendor to be getting, it doesn't 100% surprise me that
it might take a few goes for them to understand what you're
looking for, or that you're having a worse time than most users.
For enterprise type stuff AFAICT people are expecting people to
get their stable versions from distros rather than raw LTS.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]