Re: [MAINTAINERS SUMMIT] Quality standards for embargoed code

[Sasha Levin <sashal@kernel.org>]

On Tue, Aug 15, 2023 at 10:19:21AM -0700, Dave Hansen wrote:
>On 8/15/23 09:58, Sasha Levin wrote:
>> I'd like to have a discussion about how the community handles code
>> drops to address embargoed security issues: my concern is that we
>> sidestap our regular development workflow (post patches, review,
>> test, bots, etc...)
>
>I couldn't agree more.  Working on these issues feels like you're
>hacking with one arm tied behind your back.  Things are _way_ better
>than they used to be, but the closer the folks working behind closed
>doors get to the "regular" workflows, the better off everyone is.
>
>> 1. Ask (require) organizations that repeatedly go through this mechanism
>> to create a test environment that can demonstrate how the embargoed code
>> passes different build/validation tests. We should set a minimal bar to
>> the demonstrated quality of code that we'll "sneak" behind the backs of
>> community members.
>
>Intel does send things through 0day internally, with a few minor
>differences from how public stuff gets tested.  But, I don't think any
>information about that internal testing ever makes it into the material
>that get merged.  We'll fix that.

Beyond running tests, it would also be great to standardize on what we
need to run, and if Intel wants to start the discussion by openning up
it's tests for embargoed code then it'll e a great start!

>> 2. Create a group of trusted "testers" who can test embargoed code with
>> different (ideally "real") workloads and environments. I think that
>> we're overly focused on keeping the circle of people in the know small.
>
>The docs:
>
>> https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html
>
>_should_ allow the "hardware security team" to add testers today:

It probably does, but the way it's written now you'd need a lawyer to
confirm that.

It also definitely happened before where developers were pulled in just
to test something specific (like the Itanium fixes not long ago), but
this just feels like an one-off rather than something that happens for
every embargoed issue.

>> The hardware security team identifies the developers (domain experts)
>> who will form the initial response team for a particular issue. The
>> initial response team can bring in further developers (domain
>> experts) to address the issue in the best technical way.
>Do we need to make this more explicit that some of those developers
>might be focused on testing?

Not only that, but I think that we should have a standard set of testers
(let it be bots of humans), rather than developers that are pulled in
ad-hoc. The current process has a lot of friction around involving
parties that are not strictly needed for the development of the fix.

To me this is an attempt to get as much real world testing as we can get
with the smallest group possible.

-- 
Thanks,
Sasha