Re: [Ksummit-discuss] [LTSI-dev] [Stable kernel] feature backporting collaboration

[Tsugikazu Shibata <tshibata@ab.jp.nec.com>]

On Tue, Sep 13, 2016 at 10:13PM, Greg KH wrote:
>On Tue, Sep 13, 2016 at 08:20:59AM -0400, Josh Boyer wrote:
>> On Tue, Sep 13, 2016 at 8:09 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> > On Tue, Sep 13, 2016 at 11:38:14AM +0100, Mark Brown wrote:
>> >> On Tue, Sep 13, 2016 at 08:19:31AM +0200, Greg KH wrote:
>> >> > what the next kernel will be.  I work with them to show that it
>> >> > doesn't really matter _what_ kernel is picked, if their code is
>> >> > merged upstream, or ready to be merged, the specific kernel number doesn't
>matter.
>> >>
>> >> In the cases I'm aware of it's more about knowing when the kernel
>> >> will appear so people can commit to integration activities than the
>> >> version number itself - I've never really heard "I need version X",
>> >> it's always been "when will we know which version Greg has chosen?".
>> >
>> > Yes, I hear that a lot, so you need to follow up with, "why does it
>> > matter what version Greg picks?", and then their response to me
>> > always is, "so we know what kernel to start to rebase our huge
>> > patchsets to earlier", which again, is the thing we want to keep them from doing!
>> >
>> > I got a few emails when I stopped 3.14.y this week along the lines
>> > of, "oops, we were using that kernel, what are we supposed to do
>> > now!"  Each time I asked if 4.4 or 4.8 worked for them.  And each
>> > time I got back a response a day later along the lines of, "oh wow,
>> > yes, it does, we'll use that."
>> >
>> > I have half-a-mind to just skip a LTS kernel for a whole year and
>> > see if anyone even notices, I feel it is being used for all the
>> > wrong reasons...
>>
>> What are the right reasons?  I've always found the LTS kernels to be a
>> weird thing.  They don't serve the same purpose the normal stable
>> kernels do for users (fixes after initial release, before the next
>> release).  They don't serve the same purpose as a vendor kernel
>> (hardened, stable, tuned, supported).  Developers tend to abuse them
>> as you've descried.  So I guess I've forgotten the original intention
>> of the LTS kernels to begin with.  Could you remind me (us)?
>
>Originally it was to make my "day job" of maintaining an enterprise kernel (SLES)
>easier (didn't have to have a bunch of bugfixes listed in the spec file, took advantage
>of more testers, got to do stable kernel work on company time, etc.)
>
>But then, others used it for their "products", especially the embedded space, as they
>wanted to support a single kernel for the lifetime of the product.  So they asked for
>a kernel a year for this.
>
>But, it turned out that they would only use the kernel series for a while during the
>development phase, and then stop after they "shipped"
>the device.  Look at all of the Android phones sitting on old obsolete versions of 3.4
>and 3.10 stable kernels.  They aren't even updated to newer ones, and so, it didn't
>really help all that much.  Even though I am fixing security bugs for these kernels, no
>one pushes them to the users.  I have an example of a security bug that a Google
>researcher found in a 3.10 kernel (but not mainline) I fixed and pushed out an update,
>but never got picked up in Nexus phones until 6 months later when I found the right
>person/group to poke within Google.
>
>That was a 6 month window where anyone could have gotten root on your phone,
>easily.
>
>People say "look, we are using an LTS kernel in our product, all must be good!" but if
>they don't update it, it's broken and insecure, and really no better than if they were
>using 3.10.0 in a way.
>
>But if we didn't provide an LTS, would companies constantly update their kernels to
>newer releases to keep up with the security and bugfixes?
>That goes against everything those managers/PMs have ever been used to in the past,
>yet it's actually the best thing they could do.  It's a long road of education and doing
>work on their part to get test frameworks set up to be able to qualify "larger" upgrades.
>It also requires that their chip vendor not add 1.5 million lines of code to their kernel,
>rewrite the scheduler, and duplicate all existing drivers with a "-2" suffix.
>
>Ok, this is rambling, and something I've been mulling over for a while now.  I'm
>working with some people at some of the chip companies to see about how we can do
>this better, hopefully the work of education, testing, and other assurances can help
>everyone out in the end, and start to resolve these issues, but it's going to be slow
>going.
>
>Yes, I'll have an LTS this year, but next year?  Maybe not, my dream would be that it
>wouldn't be needed.  And one has to dream :)

In this thread, We LTSI team could recognized there is still huge gap between community and industry (especially that have no Enterprise distribution).
Some years ago, I think Enterprise distros got number of requests to include specific in-house patches from companies but distros were never accepted because of upstream first policy and as the result, companies were going to send patch to upstream. 
And then, It looks better forming now.
OTH, Industry that have no such distros, companies should build their own distribution by their own but they have no enough resources to maintain the kernel long term. So, LTS is only the choice for such industry. 
In the case, companies should understand LTS is a community work, so they should not only consume LTS but also work with upstream.
By the discussion here, large outstanding patches waiting for next LTS without considering upstream. Also, others are trying to push unready patches for next LTS candidates. These are the example of the gap.
We LTSI were trying to do some activities to reduce the gap but It seems still not enough.
As for a next step, I think industry should discuss this problem to find better way otherwise we will lose only a choice. We would try to set up a chance of this discussion at coming Embedded Linux Conference Europe. I hope related people would join there and exchange their own opinion.

Thanks,
Tsugikazu Shibata

>thanks,
>
>greg k-h