[MAINTAINERS SUMMIT] Quality standards for embargoed code

[MAINTAINERS SUMMIT] Quality standards for embargoed code

[Sasha Levin]

Hi folks,

I'd like to have a discussion about how the community handles code drops
to address embargoed security issues: my concern is that we sidestap our
regular development workflow (post patches, review, test, bots, etc...)
that gives us a good quality baseline, and end up taking largely
untested code that causes pain.

In my opinion, there's no benefit in promptly releasing kernels
containing fixes for such issues if these kernels are not usable by
(some) users.

Hardware issues are here to stay, we see an increase in embargoed
security issues, but we're still treating them as one-offs. We should
start to adapt our workflows to these, and a good starting point (IMO)
is assuring/improving the quality of what goes through the pipeline.

Some of the initial thoughts I had around these:

1. Ask (require) organizations that repeatedly go through this mechanism
to create a test environment that can demonstrate how the embargoed code
passes different build/validation tests. We should set a minimal bar to
the demonstrated quality of code that we'll "sneak" behind the backs of
community members.

2. Create a group of trusted "testers" who can test embargoed code with
different (ideally "real") workloads and environments. I think that
we're overly focused on keeping the circle of people in the know small.

3. Work with KernelCI/OpenSSF on setting up a (small) environment
similar to the public one that we could run embargoed code through.

-- 
Thanks,
Sasha

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

On Tue, Aug 15, 2023 at 12:58:37PM -0400, Sasha Levin wrote:

> 1. Ask (require) organizations that repeatedly go through this mechanism
> to create a test environment that can demonstrate how the embargoed code
> passes different build/validation tests. We should set a minimal bar to
> the demonstrated quality of code that we'll "sneak" behind the backs of
> community members.

This would be great, it's especially frustrating when the issues people
find are readily visible either in build testing or with virtual
environments and therefore even if people want to keep things secret
they should be able to do the testing themselves.  I'm not sure what the
consequences would be for messing up other than a bit of yelling but
perhaps that's enough.

> 2. Create a group of trusted "testers" who can test embargoed code with
> different (ideally "real") workloads and environments. I think that
> we're overly focused on keeping the circle of people in the know small.

> 3. Work with KernelCI/OpenSSF on setting up a (small) environment
> similar to the public one that we could run embargoed code through.

If these environments are documented and based on available code (they
should be) that could be a good way of setting the requirements for
people who want to do everything in house.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Sasha Levin]

On Tue, Aug 15, 2023 at 06:18:16PM +0100, Mark Brown wrote:
>On Tue, Aug 15, 2023 at 12:58:37PM -0400, Sasha Levin wrote:
>
>> 1. Ask (require) organizations that repeatedly go through this mechanism
>> to create a test environment that can demonstrate how the embargoed code
>> passes different build/validation tests. We should set a minimal bar to
>> the demonstrated quality of code that we'll "sneak" behind the backs of
>> community members.
>
>This would be great, it's especially frustrating when the issues people
>find are readily visible either in build testing or with virtual
>environments and therefore even if people want to keep things secret
>they should be able to do the testing themselves.  I'm not sure what the
>consequences would be for messing up other than a bit of yelling but
>perhaps that's enough.

My thinking was that the community could define a set of requirements
that we expect to be tested before we're willing to let code sneak in,
something along the lines of:

1. Built with GCC v1, v2, v3 for platforms x86, arm64, ... using
allmodconfig/allyesconfig/allnoconfig/...

2. Built with Clang v1, v2, v3 for platforms x86, arm64, ... using
allmodconfig/allyesconfig/allnoconfig/...

3. Run through LTP released in the past month.

4. Custom community provided tests, etc...

The consequence of letting something sneak through will then be on us,
and will probably trigger the addition of a test to the list of tests
above.

-- 
Thanks,
Sasha

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On Tue, Aug 15, 2023 at 02:10:40PM -0400, Sasha Levin wrote:

> My thinking was that the community could define a set of requirements
> that we expect to be tested before we're willing to let code sneak in,
> something along the lines of:

> 1. Built with GCC v1, v2, v3 for platforms x86, arm64, ... using
> allmodconfig/allyesconfig/allnoconfig/...

> 2. Built with Clang v1, v2, v3 for platforms x86, arm64, ... using
> allmodconfig/allyesconfig/allnoconfig/...

> 3. Run through LTP released in the past month.

> 4. Custom community provided tests, etc...

...and these tests pass!  :P

I'd also add booting on some set of qemu platform/firmware combinations
and the higher quality kselftest suites.  Guenter's tests would be good
too.

> The consequence of letting something sneak through will then be on us,
> and will probably trigger the addition of a test to the list of tests
> above.

Right.  I think the core thing is at least smoke testing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Dave Hansen]

On 8/15/23 09:58, Sasha Levin wrote:
> I'd like to have a discussion about how the community handles code
> drops to address embargoed security issues: my concern is that we
> sidestap our regular development workflow (post patches, review,
> test, bots, etc...)

I couldn't agree more.  Working on these issues feels like you're
hacking with one arm tied behind your back.  Things are _way_ better
than they used to be, but the closer the folks working behind closed
doors get to the "regular" workflows, the better off everyone is.

> 1. Ask (require) organizations that repeatedly go through this mechanism
> to create a test environment that can demonstrate how the embargoed code
> passes different build/validation tests. We should set a minimal bar to
> the demonstrated quality of code that we'll "sneak" behind the backs of
> community members.

Intel does send things through 0day internally, with a few minor
differences from how public stuff gets tested.  But, I don't think any
information about that internal testing ever makes it into the material
that get merged.  We'll fix that.

> 2. Create a group of trusted "testers" who can test embargoed code with
> different (ideally "real") workloads and environments. I think that
> we're overly focused on keeping the circle of people in the know small.

The docs:

> https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html

_should_ allow the "hardware security team" to add testers today:

> The hardware security team identifies the developers (domain experts)
> who will form the initial response team for a particular issue. The
> initial response team can bring in further developers (domain
> experts) to address the issue in the best technical way.
Do we need to make this more explicit that some of those developers
might be focused on testing?

> 3. Work with KernelCI/OpenSSF on setting up a (small) environment
> similar to the public one that we could run embargoed code through.

That would be really nice.

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Sasha Levin]

On Tue, Aug 15, 2023 at 10:19:21AM -0700, Dave Hansen wrote:
>On 8/15/23 09:58, Sasha Levin wrote:
>> I'd like to have a discussion about how the community handles code
>> drops to address embargoed security issues: my concern is that we
>> sidestap our regular development workflow (post patches, review,
>> test, bots, etc...)
>
>I couldn't agree more.  Working on these issues feels like you're
>hacking with one arm tied behind your back.  Things are _way_ better
>than they used to be, but the closer the folks working behind closed
>doors get to the "regular" workflows, the better off everyone is.
>
>> 1. Ask (require) organizations that repeatedly go through this mechanism
>> to create a test environment that can demonstrate how the embargoed code
>> passes different build/validation tests. We should set a minimal bar to
>> the demonstrated quality of code that we'll "sneak" behind the backs of
>> community members.
>
>Intel does send things through 0day internally, with a few minor
>differences from how public stuff gets tested.  But, I don't think any
>information about that internal testing ever makes it into the material
>that get merged.  We'll fix that.

Beyond running tests, it would also be great to standardize on what we
need to run, and if Intel wants to start the discussion by openning up
it's tests for embargoed code then it'll e a great start!

>> 2. Create a group of trusted "testers" who can test embargoed code with
>> different (ideally "real") workloads and environments. I think that
>> we're overly focused on keeping the circle of people in the know small.
>
>The docs:
>
>> https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html
>
>_should_ allow the "hardware security team" to add testers today:

It probably does, but the way it's written now you'd need a lawyer to
confirm that.

It also definitely happened before where developers were pulled in just
to test something specific (like the Itanium fixes not long ago), but
this just feels like an one-off rather than something that happens for
every embargoed issue.

>> The hardware security team identifies the developers (domain experts)
>> who will form the initial response team for a particular issue. The
>> initial response team can bring in further developers (domain
>> experts) to address the issue in the best technical way.
>Do we need to make this more explicit that some of those developers
>might be focused on testing?

Not only that, but I think that we should have a standard set of testers
(let it be bots of humans), rather than developers that are pulled in
ad-hoc. The current process has a lot of friction around involving
parties that are not strictly needed for the development of the fix.

To me this is an attempt to get as much real world testing as we can get
with the smallest group possible.

-- 
Thanks,
Sasha

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

On 8/15/23 11:19, Sasha Levin wrote:
> On Tue, Aug 15, 2023 at 10:19:21AM -0700, Dave Hansen wrote:
>> On 8/15/23 09:58, Sasha Levin wrote:
...>>> 1. Ask (require) organizations that repeatedly go through this
mechanism
>>> to create a test environment that can demonstrate how the embargoed code
>>> passes different build/validation tests. We should set a minimal bar to
>>> the demonstrated quality of code that we'll "sneak" behind the backs of
>>> community members.
>>
>> Intel does send things through 0day internally, with a few minor
>> differences from how public stuff gets tested.  But, I don't think any
>> information about that internal testing ever makes it into the material
>> that get merged.  We'll fix that.
> 
> Beyond running tests, it would also be great to standardize on what we
> need to run, and if Intel wants to start the discussion by openning up
> it's tests for embargoed code then it'll e a great start!

I'll go rattle some cages.  It might be boring old 0day, but I'll find out.

>>> 2. Create a group of trusted "testers" who can test embargoed code with
>>> different (ideally "real") workloads and environments. I think that
>>> we're overly focused on keeping the circle of people in the know small.
>>
>> The docs:
>>
>>> https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html
>>
>> _should_ allow the "hardware security team" to add testers today:
> 
> It probably does, but the way it's written now you'd need a lawyer to
> confirm that.

How about something like the attached patch for that doc?  Does that
help ensure we leave the lawyers alone? :)

[-- Attachment #2: embargoed-hardware-issues.patch --]
[-- Type: text/x-patch, Size: 1650 bytes --]

diff --git a/Documentation/process/embargoed-hardware-issues.rst b/Documentation/process/embargoed-hardware-issues.rst
index cb686238f21d..43f96ed709ba 100644
--- a/Documentation/process/embargoed-hardware-issues.rst
+++ b/Documentation/process/embargoed-hardware-issues.rst
@@ -95,13 +95,15 @@ The Linux kernel community has a dedicated hardware security team for
 initial contact, which oversees the process of handling such issues under
 embargo rules.
 
-The hardware security team identifies the developers (domain experts) who
-will form the initial response team for a particular issue. The initial
-response team can bring in further developers (domain experts) to address
-the issue in the best technical way.
-
-All involved developers pledge to adhere to the embargo rules and to keep
-the received information confidential. Violation of the pledge will lead to
+The hardware security team identifies the domain experts who will form the
+initial response team for a particular issue. The initial response team can
+bring in further members to address the issue in the best technical way.
+These members can be experts in a particular area of the kernel like a
+subsystem (PCI, scheduler, memory management, etc...) or an aspect of
+kernel development like testing.
+
+All those involved pledge to adhere to the embargo rules and to keep the
+received information confidential. Violation of the pledge will lead to
 immediate exclusion from the current issue and removal from all related
 mailing-lists. In addition, the hardware security team will also exclude
 the offender from future issues. The impact of this consequence is a highly

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Greg KH]

On Tue, Aug 15, 2023 at 11:34:37AM -0700, Dave Hansen wrote:
> On 8/15/23 11:19, Sasha Levin wrote:
> > On Tue, Aug 15, 2023 at 10:19:21AM -0700, Dave Hansen wrote:
> >> On 8/15/23 09:58, Sasha Levin wrote:
> ...>>> 1. Ask (require) organizations that repeatedly go through this
> mechanism
> >>> to create a test environment that can demonstrate how the embargoed code
> >>> passes different build/validation tests. We should set a minimal bar to
> >>> the demonstrated quality of code that we'll "sneak" behind the backs of
> >>> community members.
> >>
> >> Intel does send things through 0day internally, with a few minor
> >> differences from how public stuff gets tested.  But, I don't think any
> >> information about that internal testing ever makes it into the material
> >> that get merged.  We'll fix that.
> > 
> > Beyond running tests, it would also be great to standardize on what we
> > need to run, and if Intel wants to start the discussion by openning up
> > it's tests for embargoed code then it'll e a great start!
> 
> I'll go rattle some cages.  It might be boring old 0day, but I'll find out.
> 
> >>> 2. Create a group of trusted "testers" who can test embargoed code with
> >>> different (ideally "real") workloads and environments. I think that
> >>> we're overly focused on keeping the circle of people in the know small.
> >>
> >> The docs:
> >>
> >>> https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html
> >>
> >> _should_ allow the "hardware security team" to add testers today:
> > 
> > It probably does, but the way it's written now you'd need a lawyer to
> > confirm that.
> 
> How about something like the attached patch for that doc?  Does that
> help ensure we leave the lawyers alone? :)

The "trick" here is that lawyers agreed with the original wording,
changing it without also getting their approval might be tough :(

Anyway, the main reason we have NOT added testers (nor allowed
developers to use the test systems from their employer) is that those
test systems are able to be accessed by a huge/unknown number of other
people, none of who should have access to the potential changes under
development.

If that can be solved, with a "private" kernelci/lkft/openssf/whatever
test instance, that would be wonderful.  Ideally it should be the
responsibility of the hardware vendor for which we are fixing their
broken hardware with kernel changes to provide this for us.

I know that Linaro has made some lkft access available to some of us in
the past with "private" test systems, but that was a long time ago and I
don't think I have access to that anymore with their most recent rewrite
of their backend.  Oh, and their systems primarily test ARM cpus, of
which we generally do NOT use the embargoed-hw system because those CPUs
usually don't have these types of problems :)

So if we do get access to private testing systems, that do not leak
information to non-members of the group, be it however, I see no reason
why the current wording of the document would not be applicable as we
are allowed to do so:

	The initial response team can bring in further developers
	(domain experts) to address the issue in the best technical way.

And testing is the best technical way to ensure fixes are correct :)

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]

On Tue, Aug 15, 2023 at 09:57:46PM +0200, Greg KH wrote:

> Anyway, the main reason we have NOT added testers (nor allowed
> developers to use the test systems from their employer) is that those
> test systems are able to be accessed by a huge/unknown number of other
> people, none of who should have access to the potential changes under
> development.

> If that can be solved, with a "private" kernelci/lkft/openssf/whatever
> test instance, that would be wonderful.  Ideally it should be the
> responsibility of the hardware vendor for which we are fixing their
> broken hardware with kernel changes to provide this for us.

I think we could usefully have such systems or scripts available which
people could use at their option as part of setting the baseline,
ideally something based on free software so people can stand the stack
up themselves if they want.  Probably there will be occasions when it
gets used, if only by upstream people, and it's less stop energy to
point people at something they can concretely use rather than a list of
tests which people might not already know how to run.  If it's just a
list of requirements there's more chance people might mess up running in
ways that non-obviously don't actually test the thing.

> I know that Linaro has made some lkft access available to some of us in
> the past with "private" test systems, but that was a long time ago and I
> don't think I have access to that anymore with their most recent rewrite
> of their backend.  Oh, and their systems primarily test ARM cpus, of
> which we generally do NOT use the embargoed-hw system because those CPUs
> usually don't have these types of problems :)

They do have a bunch of qemu stuff (though it's not super comprehensive
in terms of things like firmware combos since they're more focused on
runtime testing) and while that's not the world emulation would catch
some of the wider spread issues we see.  OTOH the infra software isn't
published.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Greg KH]

On Tue, Aug 15, 2023 at 09:47:43PM +0100, Mark Brown wrote:
> On Tue, Aug 15, 2023 at 09:57:46PM +0200, Greg KH wrote:
> 
> > Anyway, the main reason we have NOT added testers (nor allowed
> > developers to use the test systems from their employer) is that those
> > test systems are able to be accessed by a huge/unknown number of other
> > people, none of who should have access to the potential changes under
> > development.
> 
> > If that can be solved, with a "private" kernelci/lkft/openssf/whatever
> > test instance, that would be wonderful.  Ideally it should be the
> > responsibility of the hardware vendor for which we are fixing their
> > broken hardware with kernel changes to provide this for us.
> 
> I think we could usefully have such systems or scripts available which
> people could use at their option as part of setting the baseline,
> ideally something based on free software so people can stand the stack
> up themselves if they want.  Probably there will be occasions when it
> gets used, if only by upstream people, and it's less stop energy to
> point people at something they can concretely use rather than a list of
> tests which people might not already know how to run.  If it's just a
> list of requirements there's more chance people might mess up running in
> ways that non-obviously don't actually test the thing.

I would _love_ a "stand this stack up myself" type of thing to have,
just that alone might solve most of this issue by allowing the
developers doing this work to "do it themselves" before even offering
the change up to the small group for review.

So consider me someone who would gladly consume this type of thing not
only for hw-embargoed issues, or for "normal" security issues, but also
for my normal development/maintainer workflow as really, they all have
the same need here.

thanks,

greg k-h