[MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Dan Williams]

This topic shares some aspects with other proposals around maintainer /
contributor stress management, but with a particular focus on current
hardware enabling dynamics.

The observation is that silicon complexity continues an inexorable climb
as functional and performance enhancements literally push the boundaries
our current kernel-user API contracts. Proposals like "offload this to
hardware...", "route that security concern through this mechanism...",
"migrate that application to this resource..." are increasingly less
isolated to self-contained drivers and more likely to have
cross-subsystem implications.

Standardization helps, but there is often a "system-software
implementation-specific" gap that a standard leaves for an operating
system to resolve. Linux is nowadays a first mover, and a primary
deployment target. In that role it is unable to benefit from other
operating system vendors to close that implementation-specific gap and
put a bounding box around hardware vendor differentiation. As always the
only tool Linux has at its disposal to manage those concerns is upstream
code acceptance.

When expectations are mishandled a contributor can find themselves
squeezed between program management and upstream maintainer skepticism.
That friction burns community resources in multiple directions. It is
also a false choice. A contributor's role is to partner with the
maintainer and other hardware vendors of similar functionality to arrive
at a solution that maximizes maintainability. Schedule is important, but
second to maintainability / cross-vendor-scalability.

I perceive a trap where upstream design decisions start to bias towards
expediency rather than maintainability. The theme of the discussion for
maintainer summit is questions like, but not limited to:

- When do vendors need to share a common ABI?
- How well is our "community consensus" protocol working to give
  contributors actionable feedback?
- Is there more we can do to enable contributors to steer the right path
  out of the expediency vs maintainability trap?

"Confidential Computing" is an example of an area with several
cross-silicon-vendor implementations that continue to add features with
a steady stream of upstream design decisions that cross multiple
subsystems.

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Greg KH]

On Mon, Aug 21, 2023 at 05:43:21PM -0700, Dan Williams wrote:
> - When do vendors need to share a common ABI?

When they do the "same type of thing" :)

> - How well is our "community consensus" protocol working to give
>   contributors actionable feedback?
> - Is there more we can do to enable contributors to steer the right path
>   out of the expediency vs maintainability trap?
> 
> "Confidential Computing" is an example of an area with several
> cross-silicon-vendor implementations that continue to add features with
> a steady stream of upstream design decisions that cross multiple
> subsystems.

And the normal "you all need to get together and agree on an api
yourself, otherwise we can't take any of this" should work here, right?

Well, except for the groups that snuck stuff in before we realized what
it really was, I guess we are stuck with them.

Why not have the community "fight it out" among themselves first, before
we have to worry about it?

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Linus Walleij]

Dan wrote:

> Standardization helps, but there is often a "system-software
> implementation-specific" gap that a standard leaves for an operating
> system to resolve. Linux is nowadays a first mover, and a primary
> deployment target.

When we designed pin control, I first made an iteration which we had
to throw away, keeping the old quote from "the mythical man-month"
in mind, "always count on throwing one version away". The same
happened with the GPIO character device. v2 looks good. We need
to realize at least from what I've seen, that Fred Brooks was right
about this.

Always having to throw the first version away is the most obvious
pattern that we should always expect, yet fail repeatedly to account
for and always get annoyed at. Why? It is expected.

Greg wrote:

 Why not have the community "fight it out" among themselves first, before
> we have to worry about it?

We can be a bit proactive with this, but it's more of elusive
entrepreneurship or magical coincidence when we do, and the
work happens seemingly unorganized.

The v2 pin control was designed in a conference room at the Embedded
Linux Conference in San Diego 2012 I think. nVidia, Texas Instruments and
Renesas were there, possibly more. We had a whiteboard. That
ended up with with the lines hooking it up in drivers/base/dd.c and that
is still the core behaviour.

It wasn't a fight, more like a good version of committee work, like a
committee without any prestige.

Collect the key people in the same geographical location and talk
it over, magic happens. Sometimes, at least. It's worth a try.

Yours,
Linus Walleij

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Laurent Pinchart]

On Tue, Aug 22, 2023 at 10:55:13AM +0200, Greg KH wrote:
> On Mon, Aug 21, 2023 at 05:43:21PM -0700, Dan Williams wrote:
> > - When do vendors need to share a common ABI?
> 
> When they do the "same type of thing" :)
> 
> > - How well is our "community consensus" protocol working to give
> >   contributors actionable feedback?
> > - Is there more we can do to enable contributors to steer the right path
> >   out of the expediency vs maintainability trap?
> > 
> > "Confidential Computing" is an example of an area with several
> > cross-silicon-vendor implementations that continue to add features with
> > a steady stream of upstream design decisions that cross multiple
> > subsystems.
> 
> And the normal "you all need to get together and agree on an api
> yourself, otherwise we can't take any of this" should work here, right?
> 
> Well, except for the groups that snuck stuff in before we realized what
> it really was, I guess we are stuck with them.
> 
> Why not have the community "fight it out" among themselves first, before
> we have to worry about it?

In some (many ?) cases, the lowest effort path is to try and sneak it in
without us noticing rather than "fighting it out" or "designing it out"
among themselves. There are cases where this behaviour is even the
consensus among vendors, as they collectively prefer keeping the design
effort low and get drivers and whole new subsystems upstream without
taking the community interests into account at all. The drivers/accel/
story is a fairly good example of the conflict between vendors who want
to disclose as little as possible and ship closed-source userpace, and a
community that insists on having fully open stacks. As long as vendors
will believe they can get away with it, they will keep trying, and we'll
spend lots of energy fighting back.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Jason Gunthorpe]

On Tue, Aug 22, 2023 at 05:29:13PM +0300, Laurent Pinchart wrote:
> On Tue, Aug 22, 2023 at 10:55:13AM +0200, Greg KH wrote:
> > On Mon, Aug 21, 2023 at 05:43:21PM -0700, Dan Williams wrote:
> > > - When do vendors need to share a common ABI?
> > 
> > When they do the "same type of thing" :)
> > 
> > > - How well is our "community consensus" protocol working to give
> > >   contributors actionable feedback?
> > > - Is there more we can do to enable contributors to steer the right path
> > >   out of the expediency vs maintainability trap?
> > > 
> > > "Confidential Computing" is an example of an area with several
> > > cross-silicon-vendor implementations that continue to add features with
> > > a steady stream of upstream design decisions that cross multiple
> > > subsystems.
> > 
> > And the normal "you all need to get together and agree on an api
> > yourself, otherwise we can't take any of this" should work here, right?
> > 
> > Well, except for the groups that snuck stuff in before we realized what
> > it really was, I guess we are stuck with them.
> > 
> > Why not have the community "fight it out" among themselves first, before
> > we have to worry about it?
> 
> In some (many ?) cases, the lowest effort path is to try and sneak it in
> without us noticing rather than "fighting it out" or "designing it out"
> among themselves. There are cases where this behaviour is even the
> consensus among vendors, as they collectively prefer keeping the design
> effort low and get drivers and whole new subsystems upstream without
> taking the community interests into account at all. 

I've begun to have the opinion that the old incentive structure in the
industry has fallen apart.

It used to be that we had Customers, OEMs, OS vendors, and Chip
builders as separate entities with a strong need to work
together. Community consensus and industry standards was a good way to
make that necessary collaboration work. You could even see a buisness
alignment that participating was good for all sides.

Now we have single entities that are OEM/(Largest) Customer/OS vendor
and sometimes even Chip designer all rolled into one opaque box. They
run, and only care about, a completely proprietary stack in userspace.

Now the business alignment is gone. Replaced by a perspective that the
utility of open source and standards is not to foster collaboration
and grow the industry but to drive down cost.

This is not just a Linux problem, I'm seeing signs of similar stresses
in the standards area as well.

As a vendor, if your biggest customers are not interested in
standards, you are not going to try to make them. As a huge customer
you don't want to make standards because that would only help your
competition. Just enough commonality to lower the cost and grind the
vendors.

So, instead of nice formal standards, we get defacto standards by who
can get stuff merged into Linux. The unfortunate truth is that in many
cases Linux is not well setup to manage the inevitable conflict that
results.

> The drivers/accel/ story is a fairly good example of the conflict
> between vendors who want to disclose as little as possible and ship
> closed-source userpace, and a community that insists on having fully
> open stacks.

We've never had a clear consensus on what userspace threshold exists
for getting something into the kernel.

In practice I think we've made it far too hard to get rid of stuff
that didn't work out for whatever reason. Once you are in, you are in
forever. That puts a lot of pressure on the maintainers to get it
right the first time, whatever "right" means. Also alot of incentive
to get "in" at any cost because once you are "in" you are golden.

For instance, I recently tried to remove some totally dead and
unusable code exposing a uAPI. It clearly was snuck in only to support
some kind of out of tree patchset and proprietary userspace. I got
pushback, and left it alone.

Or if you assert drivers/accel should not have happened, why can't we
delete that directory? Otherwise we've delivered a grandfathered and
permanent competitive advantage to one device.

It is a tough problem.

Jason

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Linus Walleij]

On Thu, Aug 24, 2023 at 2:47 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> On Tue, Aug 22, 2023 at 05:29:13PM +0300, Laurent Pinchart wrote:

> > In some (many ?) cases, the lowest effort path is to try and sneak it in
> > without us noticing rather than "fighting it out" or "designing it out"
> > among themselves. There are cases where this behaviour is even the
> > consensus among vendors, as they collectively prefer keeping the design
> > effort low and get drivers and whole new subsystems upstream without
> > taking the community interests into account at all.
>
> I've begun to have the opinion that the old incentive structure in the
> industry has fallen apart.
(...)
> Now we have single entities that are OEM/(Largest) Customer/OS vendor
> and sometimes even Chip designer all rolled into one opaque box. They
> run, and only care about, a completely proprietary stack in userspace.

I have a more optimistic view.

Maybe it depends where you look.

When Android appeared in 2007 and especially after Nexus One in 2010
the Linux handset industry was running to catch up with Apple iPhone,
and they didn't focus much about working upstream, at times implying
the Linux kernel was even interchangeable to them.

Since then they come a long way, after a few generations of lost
hardware that the kernel never supported properly, Android is
pushing the Generic Kernel Image and being more restrictive about
proprietary extensions every day. It's going the right way.
Todd Kjos at Google and Greg Kroah-Hartman from the community
have been instrumental here.

For deeply embedded silicon even in datacenters, companies like
Red Hat have pushed the vendors to work upstream because they
don't want to carry any custom patches. Jon Masters has been
instrumental here, requiring upstream drivers and ACPI support for
server silicon.

Qualcomm is working with a strong upstream focus. I think this is driven
by both handset Android and laptop ChromeOS work. They do a great
job minimizing their upstream gap.

There are problematic areas, I would point to any firmware that talks
directly to userspace such as sensor hubs (Android still isn't using the
IIO subsystem...)

> As a vendor, if your biggest customers are not interested in
> standards, you are not going to try to make them. As a huge customer
> you don't want to make standards because that would only help your
> competition. Just enough commonality to lower the cost and grind the
> vendors.

But Google and Red Hat acted as responsible "big customers" in a
sense, those put requirements on hardware vendors wanting to run
their userspace with full support from the vendor.

> > The drivers/accel/ story is a fairly good example of the conflict
> > between vendors who want to disclose as little as possible and ship
> > closed-source userpace, and a community that insists on having fully
> > open stacks.
>
> We've never had a clear consensus on what userspace threshold exists
> for getting something into the kernel.

If you want to push it, Alyssa showed only two days ago what *can* be
done: fully conformant OpenGL ES 3.1 with open drivers is *possible*
to do using just open tools. If Alyssa can do it for her driver, why can't
nVidia, Intel, AMD...
https://rosenzweig.io/blog/first-conformant-m1-gpu-driver.html

For drivers/accel I was under the impression that since LF is backing
PyTorch that would be the default userspace, but I don't know how they
stand with that as it seems CUDA-centric for accelerators, and
admittedly I don't know what conformance would mean in that case.
What is even the backend API for an accelerator userspace?
CUDA and OpenCL?
Or something new I haven't even heard of yet?

> In practice I think we've made it far too hard to get rid of stuff
> that didn't work out for whatever reason. Once you are in, you are in
> forever. That puts a lot of pressure on the maintainers to get it
> right the first time, whatever "right" means. Also alot of incentive
> to get "in" at any cost because once you are "in" you are golden.

I guess you are referring to the uAPI and userspace exercise tools.

For GPIO (admittedly a smaller problem than GPU) we simply made
a new uAPI to supercede the old one when it didn't work out.
It's the usual: always plan to throw the first version away.
The first one wasn't useless, it was useful to see what was needed
for the second version. We will get rid of the first version eventually,
just like Video4Linux is now replaced with V4L2. Maybe it is a bit
un-elegant to carry two uAPIs for a time, but I think it is simply
normal for our business to end up with that.

For in-kernel APIs we are doing this so often it isn't even funny,
such as network devices "napi" (new API), or the block layer mq
(multiqueue) that gradually but eventually completely replace the
old APIs.

> For instance, I recently tried to remove some totally dead and
> unusable code exposing a uAPI. It clearly was snuck in only to support
> some kind of out of tree patchset and proprietary userspace. I got
> pushback, and left it alone.

That's depressing. I recently ran into that as well, but it didn't seem
impossible, instead I was asked to begin with marking it broken:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8f0adae1cb1a3cf83e38dd435831baa38dd84b4c

Can you try that approach? When it's been broken for a while and
nobody cared to fix it, certainly it can be deleted.

Yours,
Linus Walleij

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Jason Gunthorpe]

On Thu, Aug 24, 2023 at 10:16:31AM +0200, Linus Walleij wrote:
> On Thu, Aug 24, 2023 at 2:47 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
> > On Tue, Aug 22, 2023 at 05:29:13PM +0300, Laurent Pinchart wrote:
> 
> > > In some (many ?) cases, the lowest effort path is to try and sneak it in
> > > without us noticing rather than "fighting it out" or "designing it out"
> > > among themselves. There are cases where this behaviour is even the
> > > consensus among vendors, as they collectively prefer keeping the design
> > > effort low and get drivers and whole new subsystems upstream without
> > > taking the community interests into account at all.
> >
> > I've begun to have the opinion that the old incentive structure in the
> > industry has fallen apart.
> (...)
> > Now we have single entities that are OEM/(Largest) Customer/OS vendor
> > and sometimes even Chip designer all rolled into one opaque box. They
> > run, and only care about, a completely proprietary stack in userspace.
> 
> I have a more optimistic view.
> 
> Maybe it depends where you look.

Yes, I didn't say, but I'm specifically looking at the
Datacenter/Cloud/Enterprise area - not Android. Arguably Android is
more classical with Google acting principally as an OS Vendor.

> For deeply embedded silicon even in datacenters, companies like
> Red Hat have pushed the vendors to work upstream because they
> don't want to carry any custom patches. Jon Masters has been
> instrumental here, requiring upstream drivers and ACPI support for
> server silicon.

The influence of the OS vendor in this space has declined
considerably. No hyperscale cloud uses Red Hat as the hypervisor
OS. Many now even provide their own in-house preferred OS for the VMs
to use.

This is what I mean, take Google Cloud as example. Their cloud side
has a propritary closed hypervisor environment. They are their own
OEM, manufacturing their own systems. They have their own hypervisor
OS and VM OS that they control. They even make some of their own
chips, and have vendors make customized off-the-shelf chips just for
them.

IMHO there are a number of surprising and new motivations that come
from this consolidation - this is not the familiar dynamic.

It pus Linux in the role of de-facto standards body without the tools
of a standards body to manage the situation.

> For drivers/accel I was under the impression that since LF is backing
> PyTorch that would be the default userspace, but I don't know how they
> stand with that as it seems CUDA-centric for accelerators, and
> admittedly I don't know what conformance would mean in that case.
> What is even the backend API for an accelerator userspace?
> CUDA and OpenCL?

Yeah, there is a big glue layer between pytorch and the actual HW.

I feel the industry settled on things like pytorch as the agreed
interop layer, and left alone, the driver layer below. So we have
CUDA/ROCm/OneAPI as all different HW stacks leading up to pytorch.

This is not even new, in HPC networking for the least 30 years we've
had MPI as the agreed interop layer and under MPI are fullly parallel
per-device stacks - with varying levels of openness. At least
classical HPC had several well funded actors who had a strong
incentive to follow open source methodologies.

AI HPC hasn't reached an incentive yet, it is too new and exciting.

> For GPIO (admittedly a smaller problem than GPU) we simply made
> a new uAPI to supercede the old one when it didn't work out.

To use Dan's example, if we have to call a mullagan on Confidential
Compute uAPIs it would be a total disaster as we would be unable to
remove the old ones, ever. Some cloud operator would build their
proprietary software stack on the old API and simply refuse to ever
change. This stuff is huge and super complicated, so the cost of
keeping two would be impractical.

I think your point of try-and-fail-fast only works if we have the
flexability to wipe out the failure. If the failure lasts forever the
cost of failing can become too great.

Jason

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

On Thu, Aug 24, 2023 at 10:16:31AM +0200, Linus Walleij wrote:

> Since then they come a long way, after a few generations of lost
> hardware that the kernel never supported properly, Android is
> pushing the Generic Kernel Image and being more restrictive about
> proprietary extensions every day. It's going the right way.
> Todd Kjos at Google and Greg Kroah-Hartman from the community
> have been instrumental here.

I do have some concerns about the effects of GKI on framework
improvements, by making it difficult for vendors to update any of the
shared APIs it pushes them into implementing anything they can't get
from the frameworks in their drivers instead.  This means if they're
going to upstream things and do the best job possible they'll end up
working on multiple versions which provides a bunch of stop energy.
Life will be much easier for vendors if they just worry about their
specific devices as much as possible.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Bird, Tim]

> -----Original Message-----
> From: Mark Brown <broonie@kernel.org>
> On Thu, Aug 24, 2023 at 10:16:31AM +0200, Linus Walleij wrote:
> 
> > Since then they come a long way, after a few generations of lost
> > hardware that the kernel never supported properly, Android is
> > pushing the Generic Kernel Image and being more restrictive about
> > proprietary extensions every day. It's going the right way.
> > Todd Kjos at Google and Greg Kroah-Hartman from the community
> > have been instrumental here.
> 
> I do have some concerns about the effects of GKI on framework
> improvements, by making it difficult for vendors to update any of the
> shared APIs it pushes them into implementing anything they can't get
> from the frameworks in their drivers instead.  This means if they're
> going to upstream things and do the best job possible they'll end up
> working on multiple versions which provides a bunch of stop energy.
> Life will be much easier for vendors if they just worry about their
> specific devices as much as possible.

For an alternative perspective on GKI, I'm not a big fan of it.  The contractual
nature of the agreement with Google to use it is arguably a GPL violation.
Sony has had experiences where GKI prevented us from making changes
to the kernel to address issues raised in carrier certification, causing delays
and extra (very funky) workarounds, since we couldn't change kernel code
directly.  Google is not very responsive to non-top-tier phone vendors, and
using GKI you are essentially at their mercy.  GKI, for us, removed some of the
value of open source (ie, the ability to modify the source to suit our needs).

 -- Tim

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Bart Van Assche]

On 8/24/23 10:20, Bird, Tim wrote:
> Sony has had experiences where GKI prevented us from making changes 
> to the kernel to address issues raised in carrier certification, 
> causing delays and extra (very funky) workarounds, since we couldn't
> change kernel code directly.  Google is not very responsive to 
> non-top-tier phone vendors, and using GKI you are essentially at 
> their mercy.  GKI, for us, removed some of the value of open source 
> (ie, the ability to modify the source to suit our needs).

Since I'm a Google employee: can you give an example of Google not being
very responsive, e.g. a bug number or patch URL? All the phone vendor
requests I have seen are processed in a reasonable time.

The quality of the Android kernel patches that I have seen and that are 
sent directly to Google is often low. Sometimes there is no other way to 
describe these as a nightmare from the point of view of code maintenance.

The GKI indeed makes changes to the core kernel harder after its API has 
been frozen. Does carrier certification really require changes to the 
core kernel?

Thanks,

Bart.

Re: [MAINTAINERS SUMMIT] Between a rock and a hard place, managing expectations...

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

On Thu, Aug 24, 2023 at 12:29:34PM -0700, Bart Van Assche wrote:

> The GKI indeed makes changes to the core kernel harder after its API has
> been frozen. Does carrier certification really require changes to the core
> kernel?

I don't know anything about the issue Tim was facing but I've definitely
run into issues during testing which were best fixed by extending the
core, sometimes it's not a straight bug in the core (which I'd guess
should be relatively easy to get a backport for, you can probably just
go via upstream stable) but rather that this is the appropriate layer to
handle whatever the issue is.  Teach the core about a concept, then tell
the core that your device does the thing which you've discovered needs
to be handled.

Note that I've got no experience working with GKI here, but unless
things have changed a lot there's not really any phase of the process
where there isn't substantial time pressure.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]