[MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

Hi,

I believe that reporters of embargoed security issues have always been 
confused about reporting to security@kernel.org vs. reporting to 
linux-distros@, as those two lists have completely different ways of 
dealing with the report (different purpose, different deadlines, different 
obligations imposed on the reporters, etc).

Our documentation originally suggested reporting to both in some "hybrid" 
mode, and the results were not great, quite often leading to confusion 
left and right.

This led to a slight update to our documentation [1], where reporters are 
discouraged from reporting kernel issues to linux-distros@ ever.

While I generally agree with the change *now*, given the current 
conditions, I would like to bring this up for discussion on how to deal 
with this longer term.

With my distro hat on, I really want the kernel security bugs to be 
*eventually* processed through linux-distros@ somehow, for one sole 
reason: it means that our distro security people will be aware of it, 
track it internally, and keep an eye on the fix being ported to all of our 
codestreams. This is exactly how this is done for all other packages.

I would be curious to hear about this from other distros, but I sort of 
expect that they would agree.

If this process doesn't happen, many kernel security (with CVE potentially 
eventually assigned retroactively) fixes will go by unnoticed, as distro 
kernel people will never be explicitly made aware of them, and distros 
will be missing many of the patches. Which I don't think is a desired end 
effect.

I have been discussing this with Greg already some time ago, and I know 
that his response to this is "then use -stable, and you'll get everything 
automatically" (which is obviously true, because stable is represented at 
security@kernel.org), but:

- Neither us (nor RedHat, nor Ubuntu, as far as I am aware) are picking 
  stable as a primary base for distro kernels. There are many reasons for 
  this (lifecycles not matching, stable picking up way too many things for 
  taste of some of us, etc), but that's probably slightly off-topic for 
  this discussion

- For several varying reasons, our security people really struggle with 
  ensuring that whenever CVE is published, we as a distro can guarantee,
  that fix for that particular CVE is included. linux-distros@ provides
  that connection between bugfix and CVE report, and that is lost if the 
  fix goes only through security@kernel.org

  And yes, I hate the whole CVE thing with passion, but it unfortunately 
  exists and is seen as an industry standard by many. And with us not 
  being able to systematically / automatically guarantee that fix for 
  particulart CVE is included, Linux will be not allowed into many places.

I am currently not sure what exactly would be the solution to this.

One thing to try would of course be to discuss with linux-distros@ people 
whether they'd be willing to adjust their rules to fit our needs better; 
but before that happens, we should be ourselves clear on what our needs 
towards them actually are.

Another option might be to ensure representation of distros at 
security@kernel.org, but that would completely change the purpose of it, 
and I don't think that's desired.

... ?

[1] https://git.kernel.org/linus/4fee0915e649b

Thanks,

-- 
Jiri Kosina
Director, SUSE Labs Core

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Vegard Nossum]

(Sending this with a few extra people in Bcc so they'll see it without
getting spammed with replies if they don't want them.)

On 8/15/23 11:28, Jiri Kosina wrote:
> Hi,
> 
> I believe that reporters of embargoed security issues have always been
> confused about reporting to security@kernel.org vs. reporting to
> linux-distros@, as those two lists have completely different ways of
> dealing with the report (different purpose, different deadlines, different
> obligations imposed on the reporters, etc).
> 
> Our documentation originally suggested reporting to both in some "hybrid"
> mode, and the results were not great, quite often leading to confusion
> left and right.
> 
> This led to a slight update to our documentation [1], where reporters are
> discouraged from reporting kernel issues to linux-distros@ ever.
> 
> While I generally agree with the change *now*, given the current
> conditions, I would like to bring this up for discussion on how to deal
> with this longer term.

Sorry to toot this particular horn all the time, but before this update
to the security documentation I had also submitted patches to update it
to be much clearer and more explicit about the s@k.o and linux-distros
distinction:

https://lore.kernel.org/all/20230305220010.20895-1-vegard.nossum@oracle.com/

The main objection was probably this response from Greg KH, but I had
the impression he was (at the time) not totally against wording things
in this direction at some point:

https://lore.kernel.org/all/ZAWB5kwcG9IpWvE%2F@kroah.com/

I think it's worth pointing out that the latest update to the
documentation (where reporters are discouraged from reporting kernel
issues to linux-distros@ ever) came just after a private discussion (on
both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where
Linus in particular came out hard against the linux-distros policy, in
particular the requirement to disclose PoCs if those were shared with
the list in the first place. I therefore think that Linus himself needs
to be involved in this discussion and that his arguments need to be made
in public instead of just paraphrased by me.

> With my distro hat on, I really want the kernel security bugs to be
> *eventually* processed through linux-distros@ somehow, for one sole
> reason: it means that our distro security people will be aware of it,
> track it internally, and keep an eye on the fix being ported to all of our
> codestreams. This is exactly how this is done for all other packages.
> 
> I would be curious to hear about this from other distros, but I sort of
> expect that they would agree.

+1 from me; the distros list has been an extremely important resource
for distros in order to get fixes shipped out with minimal delay.

I keep saying this as well: almost all users get their kernels through
distros. Very few end users build their own kernels from source; in
other words, it's not enough that a fix has been committed to
mainline/stable git to consider it fixed. The connection between
upstream git and end users is clearly the distros, so distros should be
in the loop _at some point_.

> If this process doesn't happen, many kernel security (with CVE potentially
> eventually assigned retroactively) fixes will go by unnoticed, as distro
> kernel people will never be explicitly made aware of them, and distros
> will be missing many of the patches. Which I don't think is a desired end
> effect.
> 
> I have been discussing this with Greg already some time ago, and I know
> that his response to this is "then use -stable, and you'll get everything
> automatically" (which is obviously true, because stable is represented at
> security@kernel.org), but:
> 
> - Neither us (nor RedHat, nor Ubuntu, as far as I am aware) are picking
>    stable as a primary base for distro kernels. There are many reasons for
>    this (lifecycles not matching, stable picking up way too many things for
>    taste of some of us, etc), but that's probably slightly off-topic for
>    this discussion
> 
> - For several varying reasons, our security people really struggle with
>    ensuring that whenever CVE is published, we as a distro can guarantee,
>    that fix for that particular CVE is included. linux-distros@ provides
>    that connection between bugfix and CVE report, and that is lost if the
>    fix goes only through security@kernel.org
> 
>    And yes, I hate the whole CVE thing with passion, but it unfortunately
>    exists and is seen as an industry standard by many. And with us not
>    being able to systematically / automatically guarantee that fix for
>    particulart CVE is included, Linux will be not allowed into many places.
> 
> I am currently not sure what exactly would be the solution to this.
> 
> One thing to try would of course be to discuss with linux-distros@ people
> whether they'd be willing to adjust their rules to fit our needs better;
> but before that happens, we should be ourselves clear on what our needs
> towards them actually are.
> 
> Another option might be to ensure representation of distros at
> security@kernel.org, but that would completely change the purpose of it,
> and I don't think that's desired.
> 
> ... ?

I'll throw in another idea: distros@kernel.org.

A closed list which will be notified by security@kernel.org once they
feel patches for a particular issue are ready for testing/consumption by
distros (and hopefully before the issue is disclosed publicly, if the
reporter still wishes to do that).

The members and list rules would be totally up to the security team to
decide.

> 
> [1] https://git.kernel.org/linus/4fee0915e649b
> 
> Thanks,
> 


Vegard

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Tue, 15 Aug 2023, Vegard Nossum wrote:

> I think it's worth pointing out that the latest update to the 
> documentation (where reporters are discouraged from reporting kernel 
> issues to linux-distros@ ever) came just after a private discussion (on 
> both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where 
> Linus in particular came out hard against the linux-distros policy, in 
> particular the requirement to disclose PoCs if those were shared with 
> the list in the first place. I therefore think that Linus himself needs 
> to be involved in this discussion and that his arguments need to be made 
> in public instead of just paraphrased by me.

And I acutally agree with Linus there -- I see no huge value with POCs 
being shared there.

I know the linux-distros@ people would probably argue that POCs are 
required so that they can integrate them into their QA / CI / regression 
testing pipeline, but, quite honestly, I don't believe this is happening 
in reality anyway.

> > With my distro hat on, I really want the kernel security bugs to be
> > *eventually* processed through linux-distros@ somehow, for one sole
> > reason: it means that our distro security people will be aware of it,
> > track it internally, and keep an eye on the fix being ported to all of our
> > codestreams. This is exactly how this is done for all other packages.
> > 
> > I would be curious to hear about this from other distros, but I sort of
> > expect that they would agree.
> 
> +1 from me; the distros list has been an extremely important resource
> for distros in order to get fixes shipped out with minimal delay.
> 
> I keep saying this as well: almost all users get their kernels through
> distros. Very few end users build their own kernels from source; in
> other words, it's not enough that a fix has been committed to
> mainline/stable git to consider it fixed. The connection between
> upstream git and end users is clearly the distros, so distros should be
> in the loop _at some point_.

Thanks for underlying exactly my point here.

> I'll throw in another idea: distros@kernel.org.
> 
> A closed list which will be notified by security@kernel.org once they
> feel patches for a particular issue are ready for testing/consumption by
> distros (and hopefully before the issue is disclosed publicly, if the
> reporter still wishes to do that).
> 
> The members and list rules would be totally up to the security team to
> decide.

That might be a viable option as well. It'll be a little bit inconvenient 
for the distro security people to follow different sets of rules, but 
still better than current situation. They'll probably also want the 
"assign a CVE" process that currently works on linux-distros@, but that's 
doable. I can ask SUSE security people what they'd think about such an 
idea, input from others would also be welcome.

(and maybe, if the distros@ process really gets established, 
linux-distros@ will finally realize that they'd better adjust their 
process to accomodate kernel :) ).

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
> I'll throw in another idea: distros@kernel.org.
> 
> A closed list which will be notified by security@kernel.org once they
> feel patches for a particular issue are ready for testing/consumption by
> distros (and hopefully before the issue is disclosed publicly, if the
> reporter still wishes to do that).
> 
> The members and list rules would be totally up to the security team to
> decide.

As per the lawyers, and government officials we have worked with in the
past, having a closed list for preannouncements like this will be
either:

  - deemed illegal in some countries
  - made to have all "major"[1] Linux users on it.

Neither of which actually will work out at all, the whole
"preannouncement" stuff just is not possible, sorry.  I'm amazed that
other projects have been able to "get away with it" for as long as they
have without either being infiltrated by "the powers that be" or
shutdown yet.

Politics is a rough game, the only way to survive is to not play it for
stuff like this.

So no, "distros@k.o" isn't going to be possible for the LF to host, and
any other group that wants to run such a thing will quickly have these
issues as well, it's amazing that linux-distros has been able to survive
for as long as it has.

greg k-h

[1] "Major" includes most government agencies in most countries.

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Steven Rostedt]

On Tue, 15 Aug 2023 13:23:36 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
> > I'll throw in another idea: distros@kernel.org.
> > 
> > A closed list which will be notified by security@kernel.org once they
> > feel patches for a particular issue are ready for testing/consumption by
> > distros (and hopefully before the issue is disclosed publicly, if the
> > reporter still wishes to do that).
> > 
> > The members and list rules would be totally up to the security team to
> > decide.  
> 
> As per the lawyers, and government officials we have worked with in the
> past, having a closed list for preannouncements like this will be
> either:

I guess the question is, what "preannouncements" are the lawyers worried about?

It looks like Jiri's concern is just to make sure that distros have
security patches in. Would just a list of "here's the commits that fix
security issues" be considered a preannouncement, without having any
reference to *what* security issue they fix? It would basically be a subset
of the stable tree. That is, anything that came from security@k.o, and does
not include all the AUTOSEL and other minor fixes that stable pulls in.

> 
>   - deemed illegal in some countries
>   - made to have all "major"[1] Linux users on it.

And if this list only has a list of patches that are already fixed, how
dangerous is it to not be 100% closed?

I mean, it was pretty obvious that the huge churn with spectre/meltdown
patches that were being added to the kernel at the late stage of an -rc
release was fixing "something big".

> 
> Neither of which actually will work out at all, the whole
> "preannouncement" stuff just is not possible, sorry.  I'm amazed that
> other projects have been able to "get away with it" for as long as they
> have without either being infiltrated by "the powers that be" or
> shutdown yet.

Have there been lists shutdown by the powers that be already? Or is this
just a threat made by the power that be?

> 
> Politics is a rough game, the only way to survive is to not play it for
> stuff like this.
> 
> So no, "distros@k.o" isn't going to be possible for the LF to host, and
> any other group that wants to run such a thing will quickly have these
> issues as well, it's amazing that linux-distros has been able to survive
> for as long as it has.

I'll have to talk with some laywers, as I'm curious to what would be
considered illegal about linux-distros. Are you aware of any government
specific laws I could go read? I'm not a lawyer, but I've read quite a bit
of laws that I can usually get an idea of the problem for my own
references (and my experience is that lawyers don't even know until
something is settled in court).

Note, linux-distros is not about "Linux users" but "Linux distributors".
They are not end users but are distributing a product (and having to follow
all the rules of the GPL and such to do so). They are not users, they are
part of a supply chain.

How is security@kernel.org different? If the bug is in the kernel, then the
bug is in the distro. Perhaps if we find a bug in the kernel, we should
send it to the distro we are using, and not to the Linux community? As Jiri
said, most people use a distro kernel, and not the mainline or even the
stable one. Really, if you think about it. The closest product to the user
is the distro. If someone finds a bug in the kernel, they can see if they
can exploit a distro with it. If they can, perhaps they should send it to
the security folks of the distro first. Then the distro can send it to
security@kernel.org. Maybe this already happens?

-- Steve

> 
> greg k-h
> 
> [1] "Major" includes most government agencies in most countries.

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Daniel Borkmann]

On 8/15/23 2:42 PM, Steven Rostedt wrote:
> On Tue, 15 Aug 2023 13:23:36 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
>> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
>>> I'll throw in another idea: distros@kernel.org.
>>>
>>> A closed list which will be notified by security@kernel.org once they
>>> feel patches for a particular issue are ready for testing/consumption by
>>> distros (and hopefully before the issue is disclosed publicly, if the
>>> reporter still wishes to do that).
>>>
>>> The members and list rules would be totally up to the security team to
>>> decide.
>>
>> As per the lawyers, and government officials we have worked with in the
>> past, having a closed list for preannouncements like this will be
>> either:
> 
> I guess the question is, what "preannouncements" are the lawyers worried about?
> 
> It looks like Jiri's concern is just to make sure that distros have
> security patches in. Would just a list of "here's the commits that fix
> security issues" be considered a preannouncement, without having any
> reference to *what* security issue they fix? It would basically be a subset
> of the stable tree. That is, anything that came from security@k.o, and does
> not include all the AUTOSEL and other minor fixes that stable pulls in.

Not really answering your question, but the above is also a somewhat misleading
"assurance" for distros. Some security fixes might potentially have come in via
AUTOSEL, and some others might not have been discussed at security@k.o in the
first place. How would you classify which ones of the whole set are relevant
for distros and which ones are not?

Imho, if distros decide to maintain their own trees outside of stable it would
still require manual triaging of the set of stable patches that went into a minor
release (and if in doubt, just cherry-pick the patch) - that's just the cost to
pay for maintaining non-stable base. It's the same issue as discussed in [0].

   [0] https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/

>>    - deemed illegal in some countries
>>    - made to have all "major"[1] Linux users on it.
> 
> And if this list only has a list of patches that are already fixed, how
> dangerous is it to not be 100% closed?
> 
> I mean, it was pretty obvious that the huge churn with spectre/meltdown
> patches that were being added to the kernel at the late stage of an -rc
> release was fixing "something big".
> 
>> Neither of which actually will work out at all, the whole
>> "preannouncement" stuff just is not possible, sorry.  I'm amazed that
>> other projects have been able to "get away with it" for as long as they
>> have without either being infiltrated by "the powers that be" or
>> shutdown yet.
> 
> Have there been lists shutdown by the powers that be already? Or is this
> just a threat made by the power that be?
> 
>> Politics is a rough game, the only way to survive is to not play it for
>> stuff like this.
>>
>> So no, "distros@k.o" isn't going to be possible for the LF to host, and
>> any other group that wants to run such a thing will quickly have these
>> issues as well, it's amazing that linux-distros has been able to survive
>> for as long as it has.
> 
> I'll have to talk with some laywers, as I'm curious to what would be
> considered illegal about linux-distros. Are you aware of any government
> specific laws I could go read? I'm not a lawyer, but I've read quite a bit
> of laws that I can usually get an idea of the problem for my own
> references (and my experience is that lawyers don't even know until
> something is settled in court).
> 
> Note, linux-distros is not about "Linux users" but "Linux distributors".
> They are not end users but are distributing a product (and having to follow
> all the rules of the GPL and such to do so). They are not users, they are
> part of a supply chain.
> 
> How is security@kernel.org different? If the bug is in the kernel, then the
> bug is in the distro. Perhaps if we find a bug in the kernel, we should
> send it to the distro we are using, and not to the Linux community? As Jiri
> said, most people use a distro kernel, and not the mainline or even the
> stable one. Really, if you think about it. The closest product to the user
> is the distro. If someone finds a bug in the kernel, they can see if they
> can exploit a distro with it. If they can, perhaps they should send it to
> the security folks of the distro first. Then the distro can send it to
> security@kernel.org. Maybe this already happens?

I mainly just wanted to chime in on this one and mention that from my past
experience (at least I've seen it couple of times from RH/Canonical) this
will not work overly well.

We had seen users reporting kernel security bugs there and they were stuck
in the security team's triage/bug queue for 3+ months before someone got in
contact with upstream. :-(

Presumably the teams were overloaded when it happened, or the bugs were
misclassified due to lack of domain specific knowledge or they wanted to fix
it themselves but didn't get to it yet.

Had they been reported to s@k.org, then the relevant maintainers/developers
could have been pulled in and it would have been fixed upstream possibly the
same day it got reported.

So my biggest concern with reporting to distro first is that "things get stuck
in the process", unfortunately. The less additional/artificial hops, the better,
imho.

Cheers,
Daniel

> -- Steve
> 
>>
>> greg k-h
>>
>> [1] "Major" includes most government agencies in most countries.
> 
> 

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Laurent Pinchart]

On Tue, Aug 15, 2023 at 03:17:00PM +0200, Daniel Borkmann wrote:
> On 8/15/23 2:42 PM, Steven Rostedt wrote:
> > On Tue, 15 Aug 2023 13:23:36 +0200 Greg KH wrote:
> >> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
> >>> I'll throw in another idea: distros@kernel.org.
> >>>
> >>> A closed list which will be notified by security@kernel.org once they
> >>> feel patches for a particular issue are ready for testing/consumption by
> >>> distros (and hopefully before the issue is disclosed publicly, if the
> >>> reporter still wishes to do that).
> >>>
> >>> The members and list rules would be totally up to the security team to
> >>> decide.
> >>
> >> As per the lawyers, and government officials we have worked with in the
> >> past, having a closed list for preannouncements like this will be
> >> either:
> > 
> > I guess the question is, what "preannouncements" are the lawyers worried about?
> > 
> > It looks like Jiri's concern is just to make sure that distros have
> > security patches in. Would just a list of "here's the commits that fix
> > security issues" be considered a preannouncement, without having any
> > reference to *what* security issue they fix? It would basically be a subset
> > of the stable tree. That is, anything that came from security@k.o, and does
> > not include all the AUTOSEL and other minor fixes that stable pulls in.
> 
> Not really answering your question, but the above is also a somewhat misleading
> "assurance" for distros. Some security fixes might potentially have come in via
> AUTOSEL, and some others might not have been discussed at security@k.o in the
> first place. How would you classify which ones of the whole set are relevant
> for distros and which ones are not?

*LOTS* of fixes for driver bugs that can cause security-related issues
are never reported to security@k.o. The author of the bug fix may not
even realize that a security issue is fixed. Those issues may have a
narrower area of impact individually as one would need to run Linux on
specific devices, but collectively they should be considered as
important as other security problems.

> Imho, if distros decide to maintain their own trees outside of stable it would
> still require manual triaging of the set of stable patches that went into a minor
> release (and if in doubt, just cherry-pick the patch) - that's just the cost to
> pay for maintaining non-stable base. It's the same issue as discussed in [0].
> 
>    [0] https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/
> 
> >>    - deemed illegal in some countries
> >>    - made to have all "major"[1] Linux users on it.
> > 
> > And if this list only has a list of patches that are already fixed, how
> > dangerous is it to not be 100% closed?
> > 
> > I mean, it was pretty obvious that the huge churn with spectre/meltdown
> > patches that were being added to the kernel at the late stage of an -rc
> > release was fixing "something big".
> > 
> >> Neither of which actually will work out at all, the whole
> >> "preannouncement" stuff just is not possible, sorry.  I'm amazed that
> >> other projects have been able to "get away with it" for as long as they
> >> have without either being infiltrated by "the powers that be" or
> >> shutdown yet.
> > 
> > Have there been lists shutdown by the powers that be already? Or is this
> > just a threat made by the power that be?
> > 
> >> Politics is a rough game, the only way to survive is to not play it for
> >> stuff like this.
> >>
> >> So no, "distros@k.o" isn't going to be possible for the LF to host, and
> >> any other group that wants to run such a thing will quickly have these
> >> issues as well, it's amazing that linux-distros has been able to survive
> >> for as long as it has.
> > 
> > I'll have to talk with some laywers, as I'm curious to what would be
> > considered illegal about linux-distros. Are you aware of any government
> > specific laws I could go read? I'm not a lawyer, but I've read quite a bit
> > of laws that I can usually get an idea of the problem for my own
> > references (and my experience is that lawyers don't even know until
> > something is settled in court).
> > 
> > Note, linux-distros is not about "Linux users" but "Linux distributors".
> > They are not end users but are distributing a product (and having to follow
> > all the rules of the GPL and such to do so). They are not users, they are
> > part of a supply chain.
> > 
> > How is security@kernel.org different? If the bug is in the kernel, then the
> > bug is in the distro. Perhaps if we find a bug in the kernel, we should
> > send it to the distro we are using, and not to the Linux community? As Jiri
> > said, most people use a distro kernel, and not the mainline or even the
> > stable one. Really, if you think about it. The closest product to the user
> > is the distro. If someone finds a bug in the kernel, they can see if they
> > can exploit a distro with it. If they can, perhaps they should send it to
> > the security folks of the distro first. Then the distro can send it to
> > security@kernel.org. Maybe this already happens?
> 
> I mainly just wanted to chime in on this one and mention that from my past
> experience (at least I've seen it couple of times from RH/Canonical) this
> will not work overly well.
> 
> We had seen users reporting kernel security bugs there and they were stuck
> in the security team's triage/bug queue for 3+ months before someone got in
> contact with upstream. :-(
> 
> Presumably the teams were overloaded when it happened, or the bugs were
> misclassified due to lack of domain specific knowledge or they wanted to fix
> it themselves but didn't get to it yet.
> 
> Had they been reported to s@k.org, then the relevant maintainers/developers
> could have been pulled in and it would have been fixed upstream possibly the
> same day it got reported.
> 
> So my biggest concern with reporting to distro first is that "things get stuck
> in the process", unfortunately. The less additional/artificial hops, the better,
> imho.
> 
> >> [1] "Major" includes most government agencies in most countries.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Catalin Marinas]

On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote:
> On Tue, 15 Aug 2023 13:23:36 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> > Politics is a rough game, the only way to survive is to not play it for
> > stuff like this.
> > 
> > So no, "distros@k.o" isn't going to be possible for the LF to host, and
> > any other group that wants to run such a thing will quickly have these
> > issues as well, it's amazing that linux-distros has been able to survive
> > for as long as it has.
> 
> I'll have to talk with some laywers, as I'm curious to what would be
> considered illegal about linux-distros. Are you aware of any government
> specific laws I could go read? I'm not a lawyer, but I've read quite a bit
> of laws that I can usually get an idea of the problem for my own
> references (and my experience is that lawyers don't even know until
> something is settled in court).

One thing that comes to mind is the hosting location of openwall.org.
For some reason my employer decided to block access to it, though
apparently one-way emails to linux-distros still work. I can see some
politicians wondering why one sends security pre-announcements to a
sanctioned country. For this reason, I'd much prefer an equivalent
hosted by the LF but the foundation may not want to be in a position to
police who's on that list, what qualifies as a Linux distro, which
country they come from, potentially removing them based on the
geopolitical situation of the day.

I guess security@kernel.org can be easier to justify as a closed forum
for fixing the actual bug with the aim to release the fix to the world
as soon as possible. But yeah, from the disclosure perspective, I don't
see much difference other than fewer people (well, the LF could ask for
US gov security clearance to be on this list ;)).

FWIW, Arm does want some official pre-announcement mechanism for
kernel bug-fixes with severe security implications. Some of our partners
(especially those with large cloud deployments or distro vendors
providing them with software) need a bit of time to roll out a fix and
consider that the public disclosure via the kernel stable/linus trees is
too late, pretty much a zero-day for them. So far we pointed them at
linux-distros but there is a growing pressure for private disclosure
mechanisms (only for reports originating from Arm). Maybe your idea of
first reporting to distro security teams is not that bad.

-- 
Catalin

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 03:20:00PM +0100, Catalin Marinas wrote:
> On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote:
> > On Tue, 15 Aug 2023 13:23:36 +0200
> > Greg KH <gregkh@linuxfoundation.org> wrote:
> > > Politics is a rough game, the only way to survive is to not play it for
> > > stuff like this.
> > > 
> > > So no, "distros@k.o" isn't going to be possible for the LF to host, and
> > > any other group that wants to run such a thing will quickly have these
> > > issues as well, it's amazing that linux-distros has been able to survive
> > > for as long as it has.
> > 
> > I'll have to talk with some laywers, as I'm curious to what would be
> > considered illegal about linux-distros. Are you aware of any government
> > specific laws I could go read? I'm not a lawyer, but I've read quite a bit
> > of laws that I can usually get an idea of the problem for my own
> > references (and my experience is that lawyers don't even know until
> > something is settled in court).
> 
> One thing that comes to mind is the hosting location of openwall.org.
> For some reason my employer decided to block access to it, though
> apparently one-way emails to linux-distros still work. I can see some
> politicians wondering why one sends security pre-announcements to a
> sanctioned country. For this reason, I'd much prefer an equivalent
> hosted by the LF but the foundation may not want to be in a position to
> police who's on that list, what qualifies as a Linux distro, which
> country they come from, potentially removing them based on the
> geopolitical situation of the day.

Lots of people want the LF to do such a thing, and last I heard, there
were some people potentially working on this but I do not know the
status of it, sorry.  Try asking the owners of openwall.org if you are
curious, they should know the current state.

Note, this is independant of the kernel, hence my not wanting to get
involved in it at all.

> I guess security@kernel.org can be easier to justify as a closed forum
> for fixing the actual bug with the aim to release the fix to the world
> as soon as possible. But yeah, from the disclosure perspective, I don't
> see much difference other than fewer people (well, the LF could ask for
> US gov security clearance to be on this list ;)).
> 
> FWIW, Arm does want some official pre-announcement mechanism for
> kernel bug-fixes with severe security implications. Some of our partners
> (especially those with large cloud deployments or distro vendors
> providing them with software) need a bit of time to roll out a fix and
> consider that the public disclosure via the kernel stable/linus trees is
> too late, pretty much a zero-day for them. So far we pointed them at
> linux-distros but there is a growing pressure for private disclosure
> mechanisms (only for reports originating from Arm). Maybe your idea of
> first reporting to distro security teams is not that bad.

Loads of companies/governments have been pestering us for access to
security@k.o for decades now, this isn't going to change for the obvious
reason that having such groups on the list is not going to help us fix
any problem, but instead, just give everyone early access to known
security problems.

Same thing would happen for any potential distro@k.o list, remember who
some of the largest users of Linux is (i.e. governments) and many of
them have their own custom "distros" for their systems for valid
reasons.

So no, we can't do that if you care about security overall, this would
make things insecure.

sorry,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Steven Rostedt]

On Tue, 15 Aug 2023 16:41:37 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> Loads of companies/governments have been pestering us for access to
> security@k.o for decades now, this isn't going to change for the obvious
> reason that having such groups on the list is not going to help us fix
> any problem, but instead, just give everyone early access to known
> security problems.
> 
> Same thing would happen for any potential distro@k.o list, remember who
> some of the largest users of Linux is (i.e. governments) and many of
> them have their own custom "distros" for their systems for valid
> reasons.
> 
> So no, we can't do that if you care about security overall, this would
> make things insecure.

Even if the only thing that is shown is a commit sha that should be taken?

-- Steve

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 08:42:53AM -0400, Steven Rostedt wrote:
> On Tue, 15 Aug 2023 13:23:36 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
> > > I'll throw in another idea: distros@kernel.org.
> > > 
> > > A closed list which will be notified by security@kernel.org once they
> > > feel patches for a particular issue are ready for testing/consumption by
> > > distros (and hopefully before the issue is disclosed publicly, if the
> > > reporter still wishes to do that).
> > > 
> > > The members and list rules would be totally up to the security team to
> > > decide.  
> > 
> > As per the lawyers, and government officials we have worked with in the
> > past, having a closed list for preannouncements like this will be
> > either:
> 
> I guess the question is, what "preannouncements" are the lawyers worried about?

Access to known security problems before they are fixed for everyone.

> It looks like Jiri's concern is just to make sure that distros have
> security patches in. Would just a list of "here's the commits that fix
> security issues" be considered a preannouncement, without having any
> reference to *what* security issue they fix? It would basically be a subset
> of the stable tree. That is, anything that came from security@k.o, and does
> not include all the AUTOSEL and other minor fixes that stable pulls in.

As Laurent said, and as many who have analyzed the data have proven, the
HUGE majority of "security fixes" flow through the normal stable release
process.  And yet, many distros/companies don't even keep up with that,
why would stuff sent to security@k.o be any different?

We used to have someone on security@k.o that would notify linux-distros
about problems when we had fixed them, but I think they got tired of
constantly keeping on top of this and stopped doing it.  It's a
thankless job, just like being on the security@k.o alias is, and I don't
blame them for stopping.

> >   - deemed illegal in some countries
> >   - made to have all "major"[1] Linux users on it.
> 
> And if this list only has a list of patches that are already fixed, how
> dangerous is it to not be 100% closed?
> 
> I mean, it was pretty obvious that the huge churn with spectre/meltdown
> patches that were being added to the kernel at the late stage of an -rc
> release was fixing "something big".

Hardware embargoes have different "rules" that we have agreed to follow,
based on the lawyers for all companies involved agreeing to those rules,
that's independent of security@k.o which doesn't get involved in this
process at all (despite some of us overlapping both lists, but that's
just an individual person thing, not a "whole group" thing.)

And I would honestly say that everyone involved in the hw-embargo
process currently hates it and I don't think it works well but it's the
best that we have at the moment until it can be re-negotiated OR
hardware companies stop doing stupid things with their CPUs.

> > Neither of which actually will work out at all, the whole
> > "preannouncement" stuff just is not possible, sorry.  I'm amazed that
> > other projects have been able to "get away with it" for as long as they
> > have without either being infiltrated by "the powers that be" or
> > shutdown yet.
> 
> Have there been lists shutdown by the powers that be already? Or is this
> just a threat made by the power that be?

No lists have been stopped that I know of, because we have been told by
governments that our current process is ok and in fact is "the only
reasonable way it can be done that we know of."  See my US Senate
testimony/interview about Spectre/Meltdown for details about this.  I
think it's online somewhere, but don't know where...

> > Politics is a rough game, the only way to survive is to not play it for
> > stuff like this.
> > 
> > So no, "distros@k.o" isn't going to be possible for the LF to host, and
> > any other group that wants to run such a thing will quickly have these
> > issues as well, it's amazing that linux-distros has been able to survive
> > for as long as it has.
> 
> I'll have to talk with some laywers, as I'm curious to what would be
> considered illegal about linux-distros. Are you aware of any government
> specific laws I could go read? I'm not a lawyer, but I've read quite a bit
> of laws that I can usually get an idea of the problem for my own
> references (and my experience is that lawyers don't even know until
> something is settled in court).
> 
> Note, linux-distros is not about "Linux users" but "Linux distributors".
> They are not end users but are distributing a product (and having to follow
> all the rules of the GPL and such to do so). They are not users, they are
> part of a supply chain.
> 
> How is security@kernel.org different? If the bug is in the kernel, then the
> bug is in the distro. Perhaps if we find a bug in the kernel, we should
> send it to the distro we are using, and not to the Linux community? As Jiri
> said, most people use a distro kernel, and not the mainline or even the
> stable one.

security@k.o is very different in that our job is to triage reported
bugs and fix them as soon as possible and get the fix out to the world.
We almost never have embargoes (there is a limited, very very short,
potential embargo we will agree to in some limited situations), but for
the huge majority of what we do, all we are here for is to fix problems.

That's it, no notifications, no delays, nothing else except fix the
issue as soon as we can.  Yes, sometimes this takes a long time, but
once we have a working fix, we get it merged quickly and move on to the
next issue.

> Really, if you think about it. The closest product to the user
> is the distro. If someone finds a bug in the kernel, they can see if they
> can exploit a distro with it. If they can, perhaps they should send it to
> the security folks of the distro first. Then the distro can send it to
> security@kernel.org. Maybe this already happens?

The huge majority of Linux use in the world is Android, everything else
is a rounding error.  Android does now have projects where security bugs
reported to them are required for the reporter to submit the problem to
security@k.o as well.  We fix the issue, get it pushed out, and the
reporter gets the credit.  Sometime in the future Android picks the
fixes up and pushed them out to their users (the delay there is much
argued about, I've worked with many companies over the years about this
and I understand the issues involved, but really, it should be sooner
than the 4-6 months it currently takes, but this is on those companies,
not us.  Governments now know this and are pushing laws to resolve this
delay (i.e. the CRA)).

After Android, Debian is by far the largest Linux user, and the Debian
kernel developers do an awesome job of tracking the stable kernel
releases which have been documented to fix 99% of known security issues
_BEFORE_ they are known (data produced by Google security team for 2
years straight).

After that, the use of Linux tapers off to "roll your own kernel.org
releases" (still a huge number of absolute boxes thanks to many
government instances and corporate clouds) to the various "enterprise"
distros, down to the other community distros (Fedora/openSUSE/Arch/etc.)

So the top end (Android and Debian and kernel.org) are covered today
with stable/LTS releases.  Same for the bottom end
(Fedora/openSUSE/Arch/etc.)  It's that corporate sponsored "middle tier"
(which is still quite small overall) that likes complaining about this
stuff because they don't want to take the stable kernel updates for
various (in my opinion stupid) reasons.

So who would such a "distros@" list help?  Only that middle-tier, small
group of very well financed companies that want to go their own way with
their kernel releases because they want to.

Why are they not just doing what the huge majority of Linux users doing
and taking the "feed of known issues that resolve problems before they
are public knowledge" that we provide today for free to them?  Because
they somehow think that knowing a specific single bugfix is more special
than all of those other bugfixes, which honestly, is just loony.

Anyway, sorry for the long rant, that's my take on the kernel ecosystem
as someone who has been seeing this for a very long time and working
with loads of different companies using the kernel.

As for the legal issues involved in such a preannouncement list, think
about all of those huge government agencies in many many different
countries who use Linux and would insist that they get access to that
feed for their own security reasons.  I would say that they honestly
have a better reason than the smaller "enterprise" distros out there,
and they can back up their request for access with legal measures.[1]

If we do not even have such a list, there's no legal measures that can
be used, which is why kernel.org should not host such a list.

thanks,

greg k-h

[1] Someone who is involved in EU governments told me once that if the
EU were to be setting up a MITRE-like organization to get
pre-announcements of security problems as the CRA recommends, we would
instantly solve the "bugs are kept secret" problem because one thing the
EU governments are really good at is making secret things public :)

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 11:04:22AM -0400, Steven Rostedt wrote:
> On Tue, 15 Aug 2023 16:41:37 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > Loads of companies/governments have been pestering us for access to
> > security@k.o for decades now, this isn't going to change for the obvious
> > reason that having such groups on the list is not going to help us fix
> > any problem, but instead, just give everyone early access to known
> > security problems.
> > 
> > Same thing would happen for any potential distro@k.o list, remember who
> > some of the largest users of Linux is (i.e. governments) and many of
> > them have their own custom "distros" for their systems for valid
> > reasons.
> > 
> > So no, we can't do that if you care about security overall, this would
> > make things insecure.
> 
> Even if the only thing that is shown is a commit sha that should be taken?

I provide a list of all of those commit sha in every release today.

You sound like someone in a meeting I had many years ago at a "big chip
company", the conversation went something like:

  Security Manager - "Just tell us which releases in every stable
  release you make that fix a security bug."
  Me - "I can't do that as that would imply we have audited every commit
  to determine if they are, or are not, a security fix, and so you would
  have a false sense of security if you don't take all of them."
  SM - "But I don't care about any unknown security fixes, I only want
  to know the known security fixes!"
  Me - "You will then have insecure systems when it is determined later
  that those other fixes were security fixes."
  SM - "I don't care about future security issues, I only want to know
  about known ones now!"
  Me - "..."

Luckily a VP stepped in then and said, "The community is giving you a
set of known bugfixes for all issues that they know of at the moment,
for free, why are you refusing to take all of them and insisting that
they do more work for you for free so that you can have a more insecure
system?"

And yes, this company is not using a distro at all, they have their own
"releases" based on the kernel.org tree.  And they are HUGE, one might
argue much bigger users of Linux than many of the "enterprise" distros
are on quantity of Linux systems in use.

So again, what's preventing you from just taking all published fixes?
Android does this, are your systems somehow more special than Android,
the largest user of Linux by far in the world?  :)

Also, to further drive home this issue, a "big car manufacturer" a few
years ago told me, "we now have a team of developers that are going to
audit every stable kernel commit to determine if it is, or is not, a
security fix for our systems before we will take it."  They did this for
about a year and came back saying "we could not keep up with the flow at
all, and everytime we guessed wrong, we put the security of our systems
at risk.  It ended up being cheaper, and safer, to just take all of the
fixes."  Which they now do.

So, why are you refusing to do the cheaper (from a business point of
view), and safer (from a security point of view) and not just take all
of the fixes we provide?

Again, the Google security team proved that taking the stable kernel
releases fixes 90%+ of all known security problems in their systems
BEFORE they are known!  They analyzed all of the data on this for 2
years straight and published their results before they just gave in and
said "just take them all, to not do so is crazy".

(the 10% not fixed were attributed to out-of-tree code that was not
upstream yet, nothing we can do about that...)

And yes, I'm arguing strongly that the old model of the "enterprise
kernel releases" is wrong.  So much so that I fail to understand that if
Android can provide a secure, up-to-date, internal ABI STABLE, kernel
based on the LTS releases, why can't these enterprise distros do the
same for their customers?

And I think Oracle does do this for their enterprise kernel releases,
which is nice to see.  Yes, I'm saying more people need to be like
Oracle and Android, what is the world coming to... :)

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Konrad Rzeszutek Wilk]

..snip..
> 
> We used to have someone on security@k.o that would notify linux-distros
> about problems when we had fixed them, but I think they got tired of
> constantly keeping on top of this and stopped doing it.  It's a
> thankless job, just like being on the security@k.o alias is, and I don't
> blame them for stopping.

Hi Greg,

Oracle will happily pay someone this "thankless job" (actually I think it
is a pretty exciting job as you get to learn and try your hand on
everything) to do this and also help with the security fixes.

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 02:46:27PM -0400, Konrad Rzeszutek Wilk wrote:
> ..snip..
> > 
> > We used to have someone on security@k.o that would notify linux-distros
> > about problems when we had fixed them, but I think they got tired of
> > constantly keeping on top of this and stopped doing it.  It's a
> > thankless job, just like being on the security@k.o alias is, and I don't
> > blame them for stopping.
> 
> Hi Greg,
> 
> Oracle will happily pay someone this "thankless job" (actually I think it
> is a pretty exciting job as you get to learn and try your hand on
> everything) to do this and also help with the security fixes.

The thing is, people on security@k.o are there on their own recognition,
and can not represent, nor notify, their employer of things discussed
there (otherwise the group can't really be called independent.)  We have
had to remove members in the past who were only using access there for
their employer so I'm a bit hesitant to only add someone for the single
reason to funnel stuff from the list elsewhere for obvious reasons.

Others in the group are free to disagree with me about this, it's run as
a "collective" by those doing the work there, not by fiat.

Note, the people on security@k.o almost always do NOT fix the issue
reported, they are there to triage and drag in the correct maintainers
and then help review proposed changes.  If you as a maintainer are drug
into the list enough times, you're asked if you want to join to save the
round-trip emails.  So when people are added, it's because of problems
in their kernel area, or because they have done lots of reviews of
subsystems in ways relating to security issues, not because they are
there to fix issues in other parts of the kernel.

And again, if only the issues that are reported to security@k.o are sent
to linux-distros, the distros will only get a tiny tiny subset of the
actual bugs being fixed in the kernel on a weekly basis.  Trying to get
access to this tiny feed does not solve the real issue of distros not
properly updating to get all of the needed fixes.

Also remember that some subsystems refuse to participate in
security@k.o, their fixes come in through the "normal" stable releases,
with work done on mailing lists.  So again, if you only see the
security@k.o issues, you will miss major problems being resolved.

Work on solving the root problem here for your distro please, don't
fixate on the CVE nonsense dance, that provides a false sense of
security and not actual security at all.

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Tue, 15 Aug 2023, Daniel Borkmann wrote:

> Not really answering your question, but the above is also a somewhat 
> misleading "assurance" for distros. Some security fixes might 
> potentially have come in via AUTOSEL, and some others might not have 
> been discussed at security@k.o in the first place. How would you 
> classify which ones of the whole set are relevant for distros and which 
> ones are not?
> 
> Imho, if distros decide to maintain their own trees outside of stable it 
> would still require manual triaging of the set of stable patches that 
> went into a minor release (and if in doubt, just cherry-pick the patch) 
> - that's just the cost to pay for maintaining non-stable base. It's the 
> same issue as discussed in [0].
> 
>   [0]
> https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/

Whith my kernel developer hat on, I fully agree. The whole CVE handling 
has been a disaster for many reasons, and it's sad that people still 
believe in it. Fixes are fixes, and you ideally want them all, right?

With my distro kernel person hat on, the linux-distros@ process helps us a 
lot with a first iteration of identifying the big set of things that we 
(well, our security team) should take a look into immediately, because 
they have been explicitly reported (and verified) to be security relevant.

It has been the case with pretty much every opensource/Linux-relevant 
project out there. Kernel has now removed itself from it. Again, from very 
good and understandable reasons for now, but I don't think that should 
mark the ultimate end of the story.

That doesn't mean that we don't have a process for evaluating what to 
merge/cherry-pick from other trees (including stable, of course). We 
absolutely do. We just can't relistically base on -stable, for multitude 
of reasons.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Tue, 15 Aug 2023, Greg KH wrote:

> > Really, if you think about it. The closest product to the user
> > is the distro. If someone finds a bug in the kernel, they can see if they
> > can exploit a distro with it. If they can, perhaps they should send it to
> > the security folks of the distro first. Then the distro can send it to
> > security@kernel.org. Maybe this already happens?
> 
> The huge majority of Linux use in the world is Android, everything else
> is a rounding error. 

Sorry, but in my view this is a slight oversimplification.

Sure, Android is ultra-huge, with userbase being the metric.

But you very well aware of where Linux as an server/enterprise distro is 
running, and that's big as well. And you can't just count that by "number 
of deployments/users" metric. Stock exchanges, air traffic control, NASA, 
govermental insitutions of all sorts, you name it.

And, at the end of the day, all those huge deployments then directly 
contribute back to kernel development, by allowing companies like Red Hat, 
SUSE, Canonical, ... to put kernel engineers on their payroll.

So I'd like to (with both my community and distro hats on now :) ) make 
sure they/we can proceed with that without unnecessary hurdles.

[ .. paragraps about how enterprise/server distributions are irrelevant 
     stripped here :) .. ]

> So the top end (Android and Debian and kernel.org) are covered today 
> with stable/LTS releases.  Same for the bottom end 
> (Fedora/openSUSE/Arch/etc.)  

Are they? I don't think Fedora is picking LTS as a base for the kernel. 
openSUSE definitely is not. So how are they covered?

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Tue, 15 Aug 2023, Greg KH wrote:

> Why are they not just doing what the huge majority of Linux users doing
> and taking the "feed of known issues that resolve problems before they
> are public knowledge" that we provide today for free to them?  Because
> they somehow think that knowing a specific single bugfix is more special
> than all of those other bugfixes, which honestly, is just loony.

If you'd like me to turn this proposal into "What can we do to make sure 
that most major distros are eventually basing their kernels on -stable" 
discussion, I'd be happy to do that, but I believe this has been discussed 
quite extensively already.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Steven Rostedt]

On Wed, 16 Aug 2023 00:13:56 +0200 (CEST)
Jiri Kosina <jikos@kernel.org> wrote:

> > The huge majority of Linux use in the world is Android, everything else
> > is a rounding error.   
> 
> Sorry, but in my view this is a slight oversimplification.

I agree. And that's just taking in "numbers". What about impact. If there's
a security flaw in an android phone, it opens up each individual for an
attack, but it usually requires an attacker to hit them individually.

But if an enterprise is hit. All it takes is to go after one server, and
you have access to thousands of users and their private data.

It's not just the number of installations that count.

-- Steve

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Tue, Aug 15, 2023 at 06:31:20PM -0400, Steven Rostedt wrote:
> On Wed, 16 Aug 2023 00:13:56 +0200 (CEST)
> Jiri Kosina <jikos@kernel.org> wrote:
> 
> > > The huge majority of Linux use in the world is Android, everything else
> > > is a rounding error.   
> > 
> > Sorry, but in my view this is a slight oversimplification.
> 
> I agree. And that's just taking in "numbers". What about impact. If there's
> a security flaw in an android phone, it opens up each individual for an
> attack, but it usually requires an attacker to hit them individually.
> 
> But if an enterprise is hit. All it takes is to go after one server, and
> you have access to thousands of users and their private data.
> 
> It's not just the number of installations that count.

Very true, but as an individual, we regard the private data in our
phones usually more important than the data stored in corporate systems :)

Anyway, my whole point here is that there are huge swaths of users of
Linux, the majority in quantity (not judging quality), that are now
doing the right thing here by taking all known fixes into their kernel
trees.  It's only a small (in absolute numbers, not importance, I'm not
judging importance as everyone is more important than everyone else,
like always), number that are not doing this and asking for us to give
them access to a tiny feed of fixes instead so that they can have a
false sense of security.

Please, let's work on fixing that false sense of security.  It's wrong,
and our users deserve better as we (the community) ARE doing all of the
fixing the best we can, but not all of our users seem to be able to take
advantage of this due to the "policy decisions" of others outside of our
community.

And note, those "policy decisions of companies" are now known by
governments to be incorrect, and soon will be made illegal in many
countries, so we are on the right side here.

Together we were able to solve the "impossible" problem of Android
kernel security.  Now that that windmill is semi-successfully conquered
despite many who thought it never could be, why can't we help other
users of our product to do the same?  If not, we run the risk of having
Linux be blamed for bad security, when in reality, it's the policy of
companies not taking advantage of what we are providing them, a nuance
that Linux users will never really understand, nor should they have to.

Let's fix this properly please, access to the security@k.o reports will
do nothing to solve the root issue, and only paper it over for a few
more years.

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Wed, Aug 16, 2023 at 12:17:36AM +0200, Jiri Kosina wrote:
> On Tue, 15 Aug 2023, Greg KH wrote:
> 
> > Why are they not just doing what the huge majority of Linux users doing
> > and taking the "feed of known issues that resolve problems before they
> > are public knowledge" that we provide today for free to them?  Because
> > they somehow think that knowing a specific single bugfix is more special
> > than all of those other bugfixes, which honestly, is just loony.
> 
> If you'd like me to turn this proposal into "What can we do to make sure 
> that most major distros are eventually basing their kernels on -stable" 
> discussion, I'd be happy to do that, but I believe this has been discussed 
> quite extensively already.

It has, and note that no one in the room would be in a real position to
solve the problem (Project Managers and VPs are not usually invited to
the kernel summit.)  Now that many laws are being passed to mandate this
soon, I think the governments might solve this issue for us
automatically :)

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Solar Designer]

On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
> (Sending this with a few extra people in Bcc so they'll see it without
> getting spammed with replies if they don't want them.)

Thank you, Vegard!

I've also now skimmed this thread at
https://lore.kernel.org/ksummit/nycvar.YFH.7.76.2308160014330.14207@cbobk.fhfr.pm/T/#t

I also found interesting the adjacent thread on "Quality standards for
embargoed code".

I notice these are tagged "MAINTAINERS SUMMIT", I assume in reference to
the one in Richmond, VA on November 16th, 2023?  I cannot easily get to
the US, but I'd consider getting to the summit in Bilbao, Spain on
September 19-21 if the relevant people would be there as well and my
attendance would be expected to help?  I don't think there's anything we
can't discuss over e-mail (and in fact e-mail is more specific), but
meeting in person is a gesture that might help establish an atmosphere
of trust and assurance of good intent.

> I think it's worth pointing out that the latest update to the
> documentation (where reporters are discouraged from reporting kernel
> issues to linux-distros@ ever)

It isn't "ever" - rather, it is "NEVER contact the "linux-distros"
mailing list until AFTER discussing it with the kernel security team."
So the kernel security team can, perhaps after having arrived at a fix,
choose to direct the reporter to also contact linux-distros.  Now, I
don't know whether this would actually be happening or not.  Maybe some
friendly dialogue and agreeing on things could affect that.

> came just after a private discussion (on
> both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where
> Linus in particular came out hard against the linux-distros policy, in
> particular the requirement to disclose PoCs if those were shared with
> the list in the first place. I therefore think that Linus himself needs
> to be involved in this discussion and that his arguments need to be made
> in public instead of just paraphrased by me.

In that private thread, two linux-distros policy aspects came up as
being inconsistent with s@k.o preferences:

1. For s@k.o, public disclosure is typically in up to 7 days since fix
is ready, but for linux-distros the deadline is 14 days max since
linux-distros is notified even if the fix is not ready.  Apparently,
this one makes Greg particularly unhappy about linux-distros.

2. _If_ the reporter shares PoCs/exploits with linux-distros, then per
current policy those should also be made public (within up to 7 days
more after the vulnerability is publicly disclosed as such).  Linus said
that this must be up to the reporter only, and we should not be imposing
such policy.  In a sense, this is already up to the reporter - if they
disagree, they just shouldn't post PoCs/exploits to linux-distros (can
instead post an offer to share with individual distros) - however, this
is problematic in practice because not everyone reads the rules before
posting and sometimes people change their mind during the embargo time
(or are required to "change their mind" by their employer, etc.)

I agree both of these are problems, and I suggest discussing them on
oss-security.  Maybe we can arrive at satisfactory solutions/exceptions.

While we're at it, I'd like to point out that while the wording (in the
commit above) that "the linux-distros rules do not contribute to
actually fixing any potential security problems" is technically correct
(yes, the _rules_ do not contribute to _upstream_ fixes), it may be
misleading.  It implies that linux-distros members would not help fix
bugs, whereas in that very StackRot/CVE-2023-3269 thread Vegard,
receiving the messages as part of Oracle Linux's linux-distros
membership, has actually contributed to fixing the bug upstream.  Vegard
is probably too humble to bring this up, so I do.  Also, even when not
contributing to upstream fixes, linux-distros members will generally
"contribute to actually fixing any potential security problems" in their
respective distros (even if e.g. by back-porting upstream fixes when
those are ready).  So I would omit this wording if it were up to me.

> On 8/15/23 11:28, Jiri Kosina wrote:
> >With my distro hat on, I really want the kernel security bugs to be
> >*eventually* processed through linux-distros@ somehow, for one sole
> >reason: it means that our distro security people will be aware of it,
> >track it internally, and keep an eye on the fix being ported to all of our
> >codestreams. This is exactly how this is done for all other packages.
> >
> >I would be curious to hear about this from other distros, but I sort of
> >expect that they would agree.
> 
> +1 from me; the distros list has been an extremely important resource
> for distros in order to get fixes shipped out with minimal delay.

Yes, I'd like this to be happening as well, and the current kernel
documentation does not preclude that from happening.  So now it's up to
the people on s@k.o.

> >[1] https://git.kernel.org/linus/4fee0915e649b

Alexander

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Wed, 16 Aug 2023, Greg KH wrote:

> > > Why are they not just doing what the huge majority of Linux users doing
> > > and taking the "feed of known issues that resolve problems before they
> > > are public knowledge" that we provide today for free to them?  Because
> > > they somehow think that knowing a specific single bugfix is more special
> > > than all of those other bugfixes, which honestly, is just loony.
> > 
> > If you'd like me to turn this proposal into "What can we do to make sure 
> > that most major distros are eventually basing their kernels on -stable" 
> > discussion, I'd be happy to do that, but I believe this has been discussed 
> > quite extensively already.
> 
> It has, and note that no one in the room would be in a real position to
> solve the problem (Project Managers and VPs are not usually invited to
> the kernel summit.)  

Not sure about other distros, but at least at SUSE, this decision powers 
are held by the kernel developers / maintainers of the branches in 
question.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Vegard Nossum]

On 8/16/23 00:17, Jiri Kosina wrote:
> On Tue, 15 Aug 2023, Greg KH wrote:
> 
>> Why are they not just doing what the huge majority of Linux users doing
>> and taking the "feed of known issues that resolve problems before they
>> are public knowledge" that we provide today for free to them?  Because
>> they somehow think that knowing a specific single bugfix is more special
>> than all of those other bugfixes, which honestly, is just loony.
> 
> If you'd like me to turn this proposal into "What can we do to make sure
> that most major distros are eventually basing their kernels on -stable"
> discussion, I'd be happy to do that, but I believe this has been discussed
> quite extensively already.

I really think these are two different and orthogonal topics and we
ought to treat them separately: tracking stable vs. having advance
notice for some vulnerabilities.

(Regarding tracking stable, I fully agree that all distros should be
keeping up with stable and not just cherry-picking specific fixes or
whatever commits happen to have a CVE assigned. No contest there.)

This proposal is about embargoed security issues, issues that were
reported because somebody knows (or has good reason to suspect) that
an issue is exploitable and is planning to disclose that fact publicly.

I posit that there is objectively a difference between:

a) vulnerabilities that have been discovered and reported by a
security researcher and where a detailed analysis and/or PoCs/exploits
are certain to be made published publicly; and

b) patches that appear in mainline/stable with no vulnerability
analysis and no known exploit vector.

I agree that both categories of issues may be equally serious or
equally exploitable, so the difference is in the fact that somebody
has done a thorough analysis of specific bugs and/or is planning to
disclose that publicly.

If distros are only made aware of a vulnerability when it is
disclosed publicly by a security researcher, that leaves a window of
opportunity for attackers -- patches that appear in stable do not
immediately make it onto end-user systems for a variety of reasons:
the time it takes to backport the patch (as most distros carry at
least some out-of-tree patches), testing/validation (especially if
there were conflicts or other issues with the patch), and then of
course the time it takes end-user to upgrade.

I think several distros at this point are making use of live patching
technology -- for really critical vulnerabilities, this has the
potential to reduce that window to very close to zero: live patching
updates that are available and can be applied from the moment the
vulnerability is disclosed publicly.

So it really isn't a choice between one or the other -- the best
security for end users is when we do both.

(I should add that I don't think anybody is really asking for a private
feed of s@k.o -- IMHO the best solution would be for security
researchers to notify the distros list when s@k.o has a tentative or
final patch, before disclosing the vulnerability publicly. The best path
I see for that is: 1) relax the linux-distros policy to not require
posting more than what the reporter is comfortable disclosing, 2) fix
the kernel.org documentation to be clearer for reporters, 3) have a
person on s@k.o who can guide the reporter towards linux-distros if they
are planning to do a public disclosure anyway.)

Thanks,


Vegard

Re: Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Donald Buczek]

Hi, Alexander and all,

I was more or less waiting for a public discussion to start on oss-security or somewhere else, because I'd like to express how important the announcements of Linux kernel issues via linux-distros is to us.

We operate the IT infrastructure of a scientific research institute (456 users, 144 Linux servers, 98 Linux workstations, about 6 PB used storage). We are not using any distro, everything is build locally and we maintain the systems by continuous rolling upgrades. 

All systems from workstations to cluster servers run the same (set of) kernels and identical userspace. The setting is rather unusual these days, all shared storage including homes is on software raid and available to the other systems via automounted nfs. We have a few public servers and a rather big computer cluster with a job scheduler. Everything is multi-user. We need to maintain separation between users and to defend the systems against attacks from the (possible abused) unprivileged user accounts. 

Our scientist run software on cluster-servers which often take many, many days to complete because of the size of the input data. Other might, for example, experiment with Jupyter Notebooks and have a lot of in-memory state. We have long running remote login sessions round the clock. For efficiency, most things run natively on hardware and not in virtual machines. System reboots have a lot of negative impact for our users.

Even nowadays not everything is single-user or microservices. "All users of the X:X kernel series must upgrade" or "just run mainline" is not a helpful strategy for us, because there is the operational issue of booting into a new kernel. Also, because we have a rather unusual settings, it is not unusual that we run into bugs others haven't discovered before. That's why we rather test new releases and roll them out slowly. That may be a corner case, but these corner cases exist, too.

We heavily rely on the information about kernel security issues published to linux-distros, which we, of course, can only receive via oss-security after the embargo. We analyze each and every new topic on oss-security to decide, whether it is relevant to us and what we can do about it. Nearly all of the userspace issues are of no relevance to us, but many of the kernel issues are, if we happen to run affected kernel versions. 

We go a long way to avoid rebooting. This might be as easy as disabling unused dynamic modules by just removing the .ko files from userspace, but sometimes we even convert an upstream fix into a loadable module which uses ftrace to replace or wrap the buggy functions in the running systems. A "reboot party" would only be a measure of last resort. 

oss-security gives us the notifications and enough information or references so that we can deal with the problems. We can not just follow mainline (or a stable series) and analyze every patch ourself to identify, whether a security bug was silently fixed. We can not boot into a new kernel every few days, just in case.

Exploits are sometimes helpful, too, for example to quickly identify whether we run affected versions, which sometimes isn't that trivial to find out otherwise, or to verify that our mitigations are working as intended. 

I would very much appreciate if a policy could be found with the kernel community that encourages to send information to linux-distros. If that requires that the rather strict procedures and deadlines of linux-distros are relaxed, I hope there is room for compromise.

Best

  Donald

On 8/16/23 5:26 PM, Solar Designer wrote:
> On Tue, Aug 15, 2023 at 12:17:03PM +0200, Vegard Nossum wrote:
>> (Sending this with a few extra people in Bcc so they'll see it without
>> getting spammed with replies if they don't want them.)
> 
> Thank you, Vegard!
> 
> I've also now skimmed this thread at
> https://lore.kernel.org/ksummit/nycvar.YFH.7.76.2308160014330.14207@cbobk.fhfr.pm/T/#t
> 
> I also found interesting the adjacent thread on "Quality standards for
> embargoed code".
> 
> I notice these are tagged "MAINTAINERS SUMMIT", I assume in reference to
> the one in Richmond, VA on November 16th, 2023?  I cannot easily get to
> the US, but I'd consider getting to the summit in Bilbao, Spain on
> September 19-21 if the relevant people would be there as well and my
> attendance would be expected to help?  I don't think there's anything we
> can't discuss over e-mail (and in fact e-mail is more specific), but
> meeting in person is a gesture that might help establish an atmosphere
> of trust and assurance of good intent.
> 
>> I think it's worth pointing out that the latest update to the
>> documentation (where reporters are discouraged from reporting kernel
>> issues to linux-distros@ ever)
> 
> It isn't "ever" - rather, it is "NEVER contact the "linux-distros"
> mailing list until AFTER discussing it with the kernel security team."
> So the kernel security team can, perhaps after having arrived at a fix,
> choose to direct the reporter to also contact linux-distros.  Now, I
> don't know whether this would actually be happening or not.  Maybe some
> friendly dialogue and agreeing on things could affect that.
> 
>> came just after a private discussion (on
>> both mailing lists) about the StackRot/CVE-2023-3269 vulnerability where
>> Linus in particular came out hard against the linux-distros policy, in
>> particular the requirement to disclose PoCs if those were shared with
>> the list in the first place. I therefore think that Linus himself needs
>> to be involved in this discussion and that his arguments need to be made
>> in public instead of just paraphrased by me.
> 
> In that private thread, two linux-distros policy aspects came up as
> being inconsistent with s@k.o preferences:
> 
> 1. For s@k.o, public disclosure is typically in up to 7 days since fix
> is ready, but for linux-distros the deadline is 14 days max since
> linux-distros is notified even if the fix is not ready.  Apparently,
> this one makes Greg particularly unhappy about linux-distros.
> 
> 2. _If_ the reporter shares PoCs/exploits with linux-distros, then per
> current policy those should also be made public (within up to 7 days
> more after the vulnerability is publicly disclosed as such).  Linus said
> that this must be up to the reporter only, and we should not be imposing
> such policy.  In a sense, this is already up to the reporter - if they
> disagree, they just shouldn't post PoCs/exploits to linux-distros (can
> instead post an offer to share with individual distros) - however, this
> is problematic in practice because not everyone reads the rules before
> posting and sometimes people change their mind during the embargo time
> (or are required to "change their mind" by their employer, etc.)
> 
> I agree both of these are problems, and I suggest discussing them on
> oss-security.  Maybe we can arrive at satisfactory solutions/exceptions.
> 
> While we're at it, I'd like to point out that while the wording (in the
> commit above) that "the linux-distros rules do not contribute to
> actually fixing any potential security problems" is technically correct
> (yes, the _rules_ do not contribute to _upstream_ fixes), it may be
> misleading.  It implies that linux-distros members would not help fix
> bugs, whereas in that very StackRot/CVE-2023-3269 thread Vegard,
> receiving the messages as part of Oracle Linux's linux-distros
> membership, has actually contributed to fixing the bug upstream.  Vegard
> is probably too humble to bring this up, so I do.  Also, even when not
> contributing to upstream fixes, linux-distros members will generally
> "contribute to actually fixing any potential security problems" in their
> respective distros (even if e.g. by back-porting upstream fixes when
> those are ready).  So I would omit this wording if it were up to me.
> 
>> On 8/15/23 11:28, Jiri Kosina wrote:
>>> With my distro hat on, I really want the kernel security bugs to be
>>> *eventually* processed through linux-distros@ somehow, for one sole
>>> reason: it means that our distro security people will be aware of it,
>>> track it internally, and keep an eye on the fix being ported to all of our
>>> codestreams. This is exactly how this is done for all other packages.
>>>
>>> I would be curious to hear about this from other distros, but I sort of
>>> expect that they would agree.
>>
>> +1 from me; the distros list has been an extremely important resource
>> for distros in order to get fixes shipped out with minimal delay.
> 
> Yes, I'd like this to be happening as well, and the current kernel
> documentation does not preclude that from happening.  So now it's up to
> the people on s@k.o.
> 
>>> [1] https://git.kernel.org/linus/4fee0915e649b
> 
> Alexander
> 


-- 
Donald Buczek
buczek@molgen.mpg.de
Tel: +49 30 8413 1433

Re: Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Miroslav Benes]

[ apologies for a slight off topic ]

Hi,

On Fri, 25 Aug 2023, Donald Buczek wrote:

> We go a long way to avoid rebooting. This might be as easy as disabling 
> unused dynamic modules by just removing the .ko files from userspace, 
> but sometimes we even convert an upstream fix into a loadable module 
> which uses ftrace to replace or wrap the buggy functions in the running 
> systems. A "reboot party" would only be a measure of last resort.

the kernel live patching infrastructure might help you with this. See 
Documentation/livepatch/ and samples/livepatch/ in the kernel tree.

Regards,
Miroslav

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Michal Suchánek]

Hello,

On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote:
> On Tue, Aug 15, 2023 at 06:31:20PM -0400, Steven Rostedt wrote:
> > On Wed, 16 Aug 2023 00:13:56 +0200 (CEST)
> > Jiri Kosina <jikos@kernel.org> wrote:
> > 
> > > > The huge majority of Linux use in the world is Android, everything else
> > > > is a rounding error.   
> > > 
> > > Sorry, but in my view this is a slight oversimplification.
> > 
> > I agree. And that's just taking in "numbers". What about impact. If there's
> > a security flaw in an android phone, it opens up each individual for an
> > attack, but it usually requires an attacker to hit them individually.
> > 
> > But if an enterprise is hit. All it takes is to go after one server, and
> > you have access to thousands of users and their private data.
> > 
> > It's not just the number of installations that count.
> 
> Very true, but as an individual, we regard the private data in our
> phones usually more important than the data stored in corporate systems :)

If my data happens to be on a corporate system I do care about their
security. With everything moving to the cloud today most of the data
users can see on their Android phones is in fact on such a corporate
system.

> Together we were able to solve the "impossible" problem of Android
> kernel security.  Now that that windmill is semi-successfully conquered
> despite many who thought it never could be, why can't we help other
> users of our product to do the same?  If not, we run the risk of having
> Linux be blamed for bad security, when in reality, it's the policy of
> companies not taking advantage of what we are providing them, a nuance
> that Linux users will never really understand, nor should they have to.

The real impossible problem of Android security is that those Android
systems aren't really opensource, and it's (next to) impossible to get
updataes when the device vendor does not provide one.

So many Android device users are running long-obsolete systems because
there is nothing more recent that runs on their device.

That won't change no matter what stable does or does not.

Thanks

Michal

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Greg KH]

On Fri, Feb 16, 2024 at 06:14:27PM +0100, Michal Suchánek wrote:
> On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote:

That was a very old thread, why dig it up now?

> > Together we were able to solve the "impossible" problem of Android
> > kernel security.  Now that that windmill is semi-successfully conquered
> > despite many who thought it never could be, why can't we help other
> > users of our product to do the same?  If not, we run the risk of having
> > Linux be blamed for bad security, when in reality, it's the policy of
> > companies not taking advantage of what we are providing them, a nuance
> > that Linux users will never really understand, nor should they have to.
> 
> The real impossible problem of Android security is that those Android
> systems aren't really opensource, and it's (next to) impossible to get
> updataes when the device vendor does not provide one.
> 
> So many Android device users are running long-obsolete systems because
> there is nothing more recent that runs on their device.
> 
> That won't change no matter what stable does or does not.

That is why governments are making laws to require this to happen.  The
EU just did this for all mobile devices, and I expect other governments
to do the same over time.

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Michal Suchánek]

On Fri, Feb 16, 2024 at 06:34:37PM +0100, Greg KH wrote:
> On Fri, Feb 16, 2024 at 06:14:27PM +0100, Michal Suchánek wrote:
> > On Wed, Aug 16, 2023 at 04:55:39PM +0200, Greg KH wrote:
> 
> That was a very old thread, why dig it up now?

It was linked to recently and I did not realize how old it is.

> > > Together we were able to solve the "impossible" problem of Android
> > > kernel security.  Now that that windmill is semi-successfully conquered
> > > despite many who thought it never could be, why can't we help other
> > > users of our product to do the same?  If not, we run the risk of having
> > > Linux be blamed for bad security, when in reality, it's the policy of
> > > companies not taking advantage of what we are providing them, a nuance
> > > that Linux users will never really understand, nor should they have to.
> > 
> > The real impossible problem of Android security is that those Android
> > systems aren't really opensource, and it's (next to) impossible to get
> > updataes when the device vendor does not provide one.
> > 
> > So many Android device users are running long-obsolete systems because
> > there is nothing more recent that runs on their device.
> > 
> > That won't change no matter what stable does or does not.
> 
> That is why governments are making laws to require this to happen.  The
> EU just did this for all mobile devices, and I expect other governments
> to do the same over time.

Indeed, with the requirement to ship updates for several years, and the
CRA requirement to not ship broken updates things may improve, at least
it looks so on paper.

Thanks

Michal

Re: [MAINTAINERS SUMMIT] Handling of embargoed security issues -- security@korg vs. linux-distros@

[Jiri Kosina]

On Fri, 16 Feb 2024, Michal Suchánek wrote:

> > That was a very old thread, why dig it up now?
> 
> It was linked to recently and I did not realize how old it is.

Yeah, I take the "blame" for that. I referenced that thread on our 
internal IRC in order to point out another case where the role of 
(commercial) distros was downplayed.

-- 
Jiri Kosina
SUSE Labs