[MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Early in my Linux career there was palpable concern around Linux being
locked out of future computing platforms by hardware vendors who did not
provide open drivers, or even documentation for their hardware. For the
hardware vendors that did participate upstream, maintainers used code
acceptance to influence them towards common Linux commands and
cross-vendor cooperation.

The internalized lesson from those days was: "Be wary of vendors pushing
'do anything you want and get away with it' passthrough tunnels. Demand
open documentation of all interfaces."

Present day realities and discussions merit revisiting that lesson:

1/ The truth of the matter is that until the Kernel Lockdown facility
   arrived, device vendors *had* an unfettered passthrough tunnel via
   userspace driver mechanisms like /dev/mem and pci-sysfs. The presence of
   those facilities did not appear to injure the ascension of Linux.

2/ Device passthrough, kernel passing opaque payloads, is already taken
   for granted in many subsystems. USB and HID have "raw" interfaces, EFI
   variables provide platform-specific configuration, and the oft-cited
   examples of SCSI and NVME that provide facilities to marshal any command
   payload whether mainline maintainers think the functionality is a good
   idea or not. In the case of NVME, the specification continues to evolve
   despite this Linux bypass.

3/ The practice of requiring Linux commands to wrap all device commands
   does not appear to have accelerated upstream participation in the CXL
   subsystem. I.e. CXL, in contrast to NVME, relegates passthrough to a
   build-time debug option. Some vendors are even shipping vendor
   specific firmware update facilities even though mainline has support for
   the CXL standard firmware update mechanism.

   With the impending arrival of CXL switch devices wanting to share
   mailbox handling code with the CXL core, the prohibition of
   device-specific commands is going to generate significant upstream work
   to wrap all that in Linux commands with little perceivable long term
   benefit to the subsystem.

CXL and RDMA are also foreshadowing conflicts across subsystems. It is
not difficult to imagine a future CXL or RDMA device that supports mem,
block, net, and drm/accel functionality. Which subsystem's
device-command policy applies to such a thing?

Enter the fwctl proposal [1]. From the CXL subsystem perspective it
looks like a long-term solution to the problem of managing expectations
between hardware vendors and mainline subsystems. It disclaims support
for the fast-path (data-plane) and is targeted at the long tail of
slow-path (config/debug plane) device-specific operations that are often
uninteresting to mainline. It sets expectations that the device must
advertise the effect of all commands so that the kernel can deploy
reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
for commands that may affect user-data. It sets common expectations for
device designers, distribution maintainers, and kernel developers. It is
complimentary to the Linux-command path for operations that need deeper
kernel coordination.

The upstream discussion has yielded the full spectrum of positions on
device specific functionality, and it is a topic that needs cross-kernel
consensus as hardware increasingly spans cross-subsystem concerns.
Please consider it for a Maintainers Summit discussion.

[1]: http://lore.kernel.org/0-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Christoph Hellwig]

Passthrough in general is useful, but the problem is that protocols
aren't designed with it in mind.  Look at the list of command that
affect too much state and we have to block in SCSI and NVMe.  And
in NVMe I'm fighting a constant fight in the technical working group
to not add new command that instantly disable the access control we've
built into NVMe passthrough.   So IMHO passthrough can be good idea,
but only if the protocol is designed for it, and protocol designer
generally have a hard time how software works at all, never mind
the futuristic concepts of layering, abstraction and access control.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Greg KH]

On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:
> Enter the fwctl proposal [1]. From the CXL subsystem perspective it
> looks like a long-term solution to the problem of managing expectations
> between hardware vendors and mainline subsystems. It disclaims support
> for the fast-path (data-plane) and is targeted at the long tail of
> slow-path (config/debug plane) device-specific operations that are often
> uninteresting to mainline.

That's not true at all, device-specific operations are very interesting
to mainline, and I would argue that the "slow-path" is the most
important thing for the kernel to manage as that is where the security
and unification layers can be properly enforced.

Vendors that think the control plane should just be allowed to be
accessed by userspace "blindly" are not saying outloud that they just
want to circumvent the security layer entirely like they previously were
doing by directly accessing /dev/mem which is one of the strongest
reasons to keep enforcing this through the kernel as Christoph points
out.

It's the cumulation of multiple vendors of semi-alike config paths that
allow us to standardize them to provide the most important thing of all,
a unified view to userspace where a user does not care about what type
of hardware is running, which is really the goal of Linux as well, but
directly goes against what a vendor wants to have happen.

> It sets expectations that the device must
> advertise the effect of all commands so that the kernel can deploy
> reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
> for commands that may affect user-data.

Yes, this is a good start, but it might still be too "vendor-specific"
at this point in time.

> It sets common expectations for
> device designers, distribution maintainers, and kernel developers. It is
> complimentary to the Linux-command path for operations that need deeper
> kernel coordination.

Yes, it's a good start, BUT by circumventing the network control plane,
the network driver maintainers rightfully are worried about this as
their review comments seem to be ignored here.  The rest of us
maintainers can't ignore that objection, sorry.

> The upstream discussion has yielded the full spectrum of positions on
> device specific functionality, and it is a topic that needs cross-kernel
> consensus as hardware increasingly spans cross-subsystem concerns.
> Please consider it for a Maintainers Summit discussion.

SHould be a fun discussion, thanks for proposing this.

greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:
> On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:

<...>

> > It sets common expectations for
> > device designers, distribution maintainers, and kernel developers. It is
> > complimentary to the Linux-command path for operations that need deeper
> > kernel coordination.
> 
> Yes, it's a good start, BUT by circumventing the network control plane,
> the network driver maintainers rightfully are worried about this as
> their review comments seem to be ignored here.  The rest of us
> maintainers can't ignore that objection, sorry.

Can you please point to the TECHNICAL review comments that were
presented and later ignored?

I don't see any, but I and probably Jonathan who posted in-depth
articles in LWN might have missed them.

Thanks

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Greg KH]

On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:
> On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:
> > On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:
> 
> <...>
> 
> > > It sets common expectations for
> > > device designers, distribution maintainers, and kernel developers. It is
> > > complimentary to the Linux-command path for operations that need deeper
> > > kernel coordination.
> > 
> > Yes, it's a good start, BUT by circumventing the network control plane,
> > the network driver maintainers rightfully are worried about this as
> > their review comments seem to be ignored here.  The rest of us
> > maintainers can't ignore that objection, sorry.
> 
> Can you please point to the TECHNICAL review comments that were
> presented and later ignored?

I can't remember review comments that were made yesterday, let alone
months ago, sorry.

greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Tue, Jul 09, 2024 at 02:33:17PM +0200, Greg KH wrote:
> On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:
> > On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:
> > > On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:
> > 
> > <...>
> > 
> > > > It sets common expectations for
> > > > device designers, distribution maintainers, and kernel developers. It is
> > > > complimentary to the Linux-command path for operations that need deeper
> > > > kernel coordination.
> > > 
> > > Yes, it's a good start, BUT by circumventing the network control plane,
> > > the network driver maintainers rightfully are worried about this as
> > > their review comments seem to be ignored here.  The rest of us
> > > maintainers can't ignore that objection, sorry.
> > 
> > Can you please point to the TECHNICAL review comments that were
> > presented and later ignored?
> 
> I can't remember review comments that were made yesterday, let alone
> months ago, sorry.

So I will summarize the situation for you. There are NO technical review
comments from netdev maintainer (not plural maintainers). The difference
is philosophical and not technical.

And yes, "rest of us maintainers" can ignore philosophical objections.
At the end, Linux kernel is distributed open-source project with
different people who have different opinions.

Thanks

> 
> greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Mon, 2024-07-08 at 15:26 -0700, Dan Williams wrote:
> Early in my Linux career there was palpable concern around Linux
> being locked out of future computing platforms by hardware vendors
> who did not provide open drivers, or even documentation for their
> hardware. For the hardware vendors that did participate upstream,
> maintainers used code acceptance to influence them towards common
> Linux commands and cross-vendor cooperation.

Firstly could I say "passthrough" seems to be the wrong word here. 
When I see it I think of device pass through from host to VM (SRIOV and
the like), which is becoming the bedrock of the virtualization world. 
However, this proposal seems to be more about user space device drivers
and fat firmware cards.

> The internalized lesson from those days was: "Be wary of vendors
> pushing 'do anything you want and get away with it' passthrough
> tunnels. Demand open documentation of all interfaces."
> 
> Present day realities and discussions merit revisiting that lesson:
> 
> 1/ The truth of the matter is that until the Kernel Lockdown facility
>    arrived, device vendors *had* an unfettered passthrough tunnel via
>    userspace driver mechanisms like /dev/mem and pci-sysfs. The
> presence of
>    those facilities did not appear to injure the ascension of Linux.
> 
> 2/ Device passthrough, kernel passing opaque payloads, is already
> taken
>    for granted in many subsystems. USB and HID have "raw" interfaces,
> EFI
>    variables provide platform-specific configuration, and the oft-
> cited
>    examples of SCSI and NVME that provide facilities to marshal any
> command
>    payload whether mainline maintainers think the functionality is a
> good
>    idea or not. In the case of NVME, the specification continues to
> evolve
>    despite this Linux bypass.

Time was decades ago Oracle demanded raw access to the SCSI device
because they claimed it was easier for customers and faster for them if
they just talked to devices in their native protocol and got all of the
annoying kernel filesystems and page cache out of the way.  Fast
fowards to today and database vendors largely use filesystems thanks to
the evolution of interfaces (direct I/O) that support what they want to
do and the huge annoyance for customers of having to manage huge
numbers of unidentifiable raw devices.

For NVMe and net we do have SPDK and DPDK.  What I find is that people
tend to use them for niche use cases (like the NVMe KV command set) or
obscure network routers.  Even though the claim they both make is to
get the kernel out of the way and do stuff "way faster" the difficulty
they create by bypassing everything is quite a high burden.

For USB security tokens in the early days we had the huge problem of
everyone inventing their own interface, then they realised this was
unsustainable and came up with CTAP, but it's just a unified way for
user space applications to talk to FIDO tokens over raw USB ... is this
a problem?

> 
> 3/ The practice of requiring Linux commands to wrap all device
> commands
>    does not appear to have accelerated upstream participation in the
> CXL
>    subsystem. I.e. CXL, in contrast to NVME, relegates passthrough to
> a
>    build-time debug option. Some vendors are even shipping vendor
>    specific firmware update facilities even though mainline has
> support for
>    the CXL standard firmware update mechanism.
> 
>    With the impending arrival of CXL switch devices wanting to share
>    mailbox handling code with the CXL core, the prohibition of
>    device-specific commands is going to generate significant upstream
> work
>    to wrap all that in Linux commands with little perceivable long
> term
>    benefit to the subsystem.
> 
> CXL and RDMA are also foreshadowing conflicts across subsystems. It
> is not difficult to imagine a future CXL or RDMA device that supports
> mem, block, net, and drm/accel functionality. Which subsystem's
> device-command policy applies to such a thing?

We already have that today: pretty much every device protocol looks a
bit network like and has an Over Ethernet or Over RDMA equivalent.

What all of the prior pass through's taught us is that if the use case
is big enough it will get pulled into the kernel and the kernel will
usually manage it better (DB users).  If it remains a niche use case it
will likely remain out of the kernel, but we won't be hurt by it (NVME
KV protocol) and sometimes it doesn't really matter and the device
manufacturers will sort it out on their own (USB tokens).

> Enter the fwctl proposal [1]. From the CXL subsystem perspective it
> looks like a long-term solution to the problem of managing
> expectations between hardware vendors and mainline subsystems. It
> disclaims support for the fast-path (data-plane) and is targeted at
> the long tail of slow-path (config/debug plane) device-specific
> operations that are often uninteresting to mainline. It sets
> expectations that the device must advertise the effect of all
> commands so that the kernel can deploy reasonable Kernel Lockdown
> policy, or otherwise require CAP_SYS_RAWIO for commands that may
> affect user-data. It sets common expectations for device designers,
> distribution maintainers, and kernel developers. It is complimentary
> to the Linux-command path for operations that need
> deeper kernel coordination.

This proposal does look to me more like a tool for configuring highly
malleable fat firmware (or really mini-os) offload type devices (like
intelligent network cards) to interact correctly and be easier to
debug.  Every cloud vendor effectively has their own one of these, so I
think the problem isn't going away, so trying to bring some order to it
looks like a potentially good idea.

> The upstream discussion has yielded the full spectrum of positions on
> device specific functionality, and it is a topic that needs cross-
> kernel consensus as hardware increasingly spans cross-subsystem
> concerns. Please consider it for a Maintainers Summit discussion.

I'm with Greg on this ... can you point to some of the contrary
positions?

Regards,

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Christoph Hellwig wrote:
> Passthrough in general is useful, but the problem is that protocols
> aren't designed with it in mind.  Look at the list of command that
> affect too much state and we have to block in SCSI and NVMe.  And
> in NVMe I'm fighting a constant fight in the technical working group
> to not add new command that instantly disable the access control we've
> built into NVMe passthrough.   So IMHO passthrough can be good idea,
> but only if the protocol is designed for it, and protocol designer
> generally have a hard time how software works at all, never mind
> the futuristic concepts of layering, abstraction and access control.
> 

I think this protocol feedback from the kernel community to hardware
designers is top on the list of benefits for this discussion.

How to draw that "needs kernel mediation" line takes
kernel-developer-practitioner expertise.

The positive takeaway from the NVME experience in my mind is that no
doubt the NVME subsystem has blocked commands that hardware developers
prefer that Linux not filter. That filter is likely trivially bypassed
by repurposing some other non-filtered command to have the desired side
effect. Something like "write to a magic LBA == run filtered command".
The fact that those cynical hacks are not deployed and the technical
working group is still holding discussions on the proper protocol means
that hardware designers have a higher incentive to get it right than to
get it deployed.

The conversation seems to break down when it comes to "trust us this
passthrough is only for device-specific functionality with low cross
vendor or kernel appeal." How can we design protocols that have a
high-level of trust?

A "Command Effects Log" seems like that starting point, with trust that
cynical abuses of that contract have a higher cost than benefit, and
trust that the protocol limits the potential damage of such abuse.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Greg KH wrote:
> On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:
> > Enter the fwctl proposal [1]. From the CXL subsystem perspective it
> > looks like a long-term solution to the problem of managing expectations
> > between hardware vendors and mainline subsystems. It disclaims support
> > for the fast-path (data-plane) and is targeted at the long tail of
> > slow-path (config/debug plane) device-specific operations that are often
> > uninteresting to mainline.
> 
> That's not true at all, device-specific operations are very interesting
> to mainline, and I would argue that the "slow-path" is the most
> important thing for the kernel to manage as that is where the security
> and unification layers can be properly enforced.
> 
> Vendors that think the control plane should just be allowed to be
> accessed by userspace "blindly" are not saying outloud that they just
> want to circumvent the security layer entirely like they previously were
> doing by directly accessing /dev/mem which is one of the strongest
> reasons to keep enforcing this through the kernel as Christoph points
> out.
> 
> It's the cumulation of multiple vendors of semi-alike config paths that
> allow us to standardize them to provide the most important thing of all,
> a unified view to userspace where a user does not care about what type
> of hardware is running, which is really the goal of Linux as well, but
> directly goes against what a vendor wants to have happen.

100% agree and this is the argument I made for creating
tsm-report-configfs. What I meant by "config-plane" is something more
along the lines of an FPGA bitstream redefining device parameters that
get realized after a reset, less the in-band live-config-plane of a
device that is likely similar across a class of devices where the kernel
has a clear interest in mediating.

A fuzzier example is that CXL has commands like "read vendor debug log"
and I can see a device having specific control toggles to modify the
data the device dumps into that log. The expectation is the device
designer will put anything of general/kernel interest into other formal
telemetry commands. That "vendor debug log" command is trusted to not
have other kernel relevant side effects, it just facilitates debug
between end users and device manufacturers.

> > It sets expectations that the device must
> > advertise the effect of all commands so that the kernel can deploy
> > reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
> > for commands that may affect user-data.
> 
> Yes, this is a good start, but it might still be too "vendor-specific"
> at this point in time.

This gets back to the trusted protocols point that Christoph raised. The
community has examples where this tension is resolved with negotiated
protocol concessions. The fwctl discussion has not progressed to the
protocol concessions phase.

> > It sets common expectations for
> > device designers, distribution maintainers, and kernel developers. It is
> > complimentary to the Linux-command path for operations that need deeper
> > kernel coordination.
> 
> Yes, it's a good start, BUT by circumventing the network control plane,
> the network driver maintainers rightfully are worried about this as
> their review comments seem to be ignored here.  The rest of us
> maintainers can't ignore that objection, sorry.

Hence a mediated discussion proposal because the on list debate has a
visible deficit of trust that email is unlikely to overcome.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

James Bottomley wrote:
> > The upstream discussion has yielded the full spectrum of positions on
> > device specific functionality, and it is a topic that needs cross-
> > kernel consensus as hardware increasingly spans cross-subsystem
> > concerns. Please consider it for a Maintainers Summit discussion.
> 
> I'm with Greg on this ... can you point to some of the contrary
> positions?

This thread has that discussion:

http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com

I do not want to speak for others on the saliency of their points, all I
can say is that the contrary positions have so far not moved me to drop
consideration of fwctl for CXL.

Where CXL has a Command Effects Log that is a reasonable protocol for
making decisions about opaque command codes, and that CXL already has a
few years of experience with the commands that *do* need a Linux-command
wrapper.

Some open questions from that thread are: what does it mean for the fate
of a proposal if one subsystem Acks the ABI and another Naks it for a
device that crosses subsystem functionality? Would a cynical hardware
response just lead to plumbing an NVME admin queue, or CXL mailbox to
get device-specific commands past another subsystem's objection?

My reconsideration of the "debug-build only" policy for CXL
device-specific commands was influenced by a conversation with a distro
developer where they asserted, paraphrasing: "at what point is a device
vendor incentivized to ship an out-of-tree module just to restore their
passthrough functionality?. At that point upstream has lost out on
collaboration and distro kernel ABI has gained another out-of-tree
consumer."

So the tension is healthy, but it has diminishing returns past a certain
point.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Jul 09, 2024 at 12:43:50PM -0700, Dan Williams wrote:

> A "Command Effects Log" seems like that starting point, with trust that
> cynical abuses of that contract have a higher cost than benefit, and
> trust that the protocol limits the potential damage of such abuse.

I've taken the view that companies are now very vigilant about
security and often have their own internal incentives and procedures
to do secure things.

If someone does a cynical security breaking thing and deploys it to a
wide user base they are likely to be caught by a security researcher
and embarassed with a CVE and a web site with a snappy name.

Not 100% of course, but it is certainly not a wild west of people just
doing whatever they want.

The other half of this bargin is we have to be much clearer about what
the security model is and what is security breaking. Like Christoph I
often have conversations with people who don't understand the basics
of how the Linux security models should work and are doing device-side
work that has to fit into it.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

On Tue, 9 Jul 2024 15:15:13 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> James Bottomley wrote:
> > > The upstream discussion has yielded the full spectrum of positions on
> > > device specific functionality, and it is a topic that needs cross-
> > > kernel consensus as hardware increasingly spans cross-subsystem
> > > concerns. Please consider it for a Maintainers Summit discussion.  
> > 
> > I'm with Greg on this ... can you point to some of the contrary
> > positions?  
> 
> This thread has that discussion:
> 
> http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> 
> I do not want to speak for others on the saliency of their points, all I
> can say is that the contrary positions have so far not moved me to drop
> consideration of fwctl for CXL.

I was resisting rat holing. Oh well...

For a 'subset' of CXL.  There are a wide range of controls that are highly
destructive, potentially to other hosts (simplest one is a command that
will surprise remove someone else's memory). For those I'm not sure
fwctl gets us anywhere - but we still need a solution (Subject to
config gates etc as typically this is BMCs not hosts).
Maybe fwctl eventually ends up with levels of 'safety' (beyond the
current read vs write vs write_full, or maybe those are enough).

Complexities such as message tunneling to multiple components are also
going to be fun, but we want the non destructive bits of those to work
as part of the safe set, so we can get telemetry from downstream devices.

Good to cover the debug and telemetry usecase, but it still leaves us with
gaping holes were we need to solve the permissions problem, perhaps that
is layered on top of fwctl, perhaps something else is needed.

So if fwctl is adopted, I do want the means to use it for the highly
destructive stuff as well!  Maybe that's a future discussion.


> 
> Where CXL has a Command Effects Log that is a reasonable protocol for
> making decisions about opaque command codes, and that CXL already has a
> few years of experience with the commands that *do* need a Linux-command
> wrapper.

Worth asking if this will incorporate unknown but not vendor defined
commands.  There is a long tail of stuff in the spec we haven't caught up
with yet.  Or you thinking keep this for the strictly vendor defined stuff?

> 
> Some open questions from that thread are: what does it mean for the fate
> of a proposal if one subsystem Acks the ABI and another Naks it for a
> device that crosses subsystem functionality? Would a cynical hardware
> response just lead to plumbing an NVME admin queue, or CXL mailbox to
> get device-specific commands past another subsystem's objection?
> 
> My reconsideration of the "debug-build only" policy for CXL
> device-specific commands was influenced by a conversation with a distro
> developer where they asserted, paraphrasing: "at what point is a device
> vendor incentivized to ship an out-of-tree module just to restore their
> passthrough functionality?. At that point upstream has lost out on
> collaboration and distro kernel ABI has gained another out-of-tree
> consumer."
> 
> So the tension is healthy, but it has diminishing returns past a certain
> point.
> 

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

On Wed, 10 Jul 2024 14:22:38 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Tue, 9 Jul 2024 15:15:13 -0700
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > James Bottomley wrote:  
> > > > The upstream discussion has yielded the full spectrum of positions on
> > > > device specific functionality, and it is a topic that needs cross-
> > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > concerns. Please consider it for a Maintainers Summit discussion.    
> > > 
> > > I'm with Greg on this ... can you point to some of the contrary
> > > positions?    
> > 
> > This thread has that discussion:
> > 
> > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > 
> > I do not want to speak for others on the saliency of their points, all I
> > can say is that the contrary positions have so far not moved me to drop
> > consideration of fwctl for CXL.  
> 
> I was resisting rat holing. Oh well...

To throw another 'fun' one in there.  For anything integrated with the host
there is a proposal to provide a MCTP via PCC (ACPI described mailbox). [1]
I don't think it makes sense to rule that out as it's logically no
different from MCTP in general (e.g. a host controller for PCI VDM, or
I2C etc)

Anyone who has a suitable firmware can do whatever they like with that
and the interfaces is exposed directly to userspace. Adam, perhaps you can
describe your use case a little?  Is it applicable to general server distros?

We might suggest distributions don't enable MCTP but does that
actually get us anywhere?  Anyhow, I suspect there are other similar routes, but
this one happens to be under review at the moment.

[1] https://lore.kernel.org/all/20240702225845.322234-1-admiyo@os.amperecomputing.com/

> 
> For a 'subset' of CXL.  There are a wide range of controls that are highly
> destructive, potentially to other hosts (simplest one is a command that
> will surprise remove someone else's memory). For those I'm not sure
> fwctl gets us anywhere - but we still need a solution (Subject to
> config gates etc as typically this is BMCs not hosts).
> Maybe fwctl eventually ends up with levels of 'safety' (beyond the
> current read vs write vs write_full, or maybe those are enough).
> 
> Complexities such as message tunneling to multiple components are also
> going to be fun, but we want the non destructive bits of those to work
> as part of the safe set, so we can get telemetry from downstream devices.
> 
> Good to cover the debug and telemetry usecase, but it still leaves us with
> gaping holes were we need to solve the permissions problem, perhaps that
> is layered on top of fwctl, perhaps something else is needed.
> 
> So if fwctl is adopted, I do want the means to use it for the highly
> destructive stuff as well!  Maybe that's a future discussion.
> 
> 
> > 
> > Where CXL has a Command Effects Log that is a reasonable protocol for
> > making decisions about opaque command codes, and that CXL already has a
> > few years of experience with the commands that *do* need a Linux-command
> > wrapper.  
> 
> Worth asking if this will incorporate unknown but not vendor defined
> commands.  There is a long tail of stuff in the spec we haven't caught up
> with yet.  Or you thinking keep this for the strictly vendor defined stuff?
> 
> > 
> > Some open questions from that thread are: what does it mean for the fate
> > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > device that crosses subsystem functionality? Would a cynical hardware
> > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > get device-specific commands past another subsystem's objection?
> > 
> > My reconsideration of the "debug-build only" policy for CXL
> > device-specific commands was influenced by a conversation with a distro
> > developer where they asserted, paraphrasing: "at what point is a device
> > vendor incentivized to ship an out-of-tree module just to restore their
> > passthrough functionality?. At that point upstream has lost out on
> > collaboration and distro kernel ABI has gained another out-of-tree
> > consumer."
> > 
> > So the tension is healthy, but it has diminishing returns past a certain
> > point.
> >   
> 
> 
> 

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Jul 09, 2024 at 09:02:25AM -0700, James Bottomley wrote:

> For NVMe and net we do have SPDK and DPDK.  What I find is that people
> tend to use them for niche use cases (like the NVMe KV command set) or
> obscure network routers.  Even though the claim they both make is to
> get the kernel out of the way and do stuff "way faster" the difficulty
> they create by bypassing everything is quite a high burden.

[..]
 
> What all of the prior pass through's taught us is that if the use case
> is big enough it will get pulled into the kernel and the kernel will
> usually manage it better (DB users).  If it remains a niche use case it
> will likely remain out of the kernel, but we won't be hurt by it (NVME
> KV protocol) and sometimes it doesn't really matter and the device
> manufacturers will sort it out on their own (USB tokens).

I don't see it as being linked to big enough use case at all.

The kernel gets involved if there are good technical reasons to do
so. Databases running over real filesystems with O_DIRECT is really
technically better than raw block devices.

While DPDK shows the opposite, userspace is the technically better
option. This is now shown at scale. DPDK is not some niche. A big
chunk of internet traffic is going through DPDKs, especially for
mobile. Many ORAN solutions include DPDK on Linux.

What has been improved kernel-side is the intergation. DPDK
deployments now often use RDMA raw queue pairs instead of VFIO, which
laregly eliminates the "high burden".

There are many other cases, like DPDK, where the right answer is to
reduce the kernel involvement. It is not so simple that things always
get pulled into the kernel.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Jul 09, 2024 at 03:47:23PM +0300, Leon Romanovsky wrote:
> On Tue, Jul 09, 2024 at 02:33:17PM +0200, Greg KH wrote:
> > On Tue, Jul 09, 2024 at 03:25:47PM +0300, Leon Romanovsky wrote:
> > > On Tue, Jul 09, 2024 at 12:01:06PM +0200, Greg KH wrote:
> > > > On Mon, Jul 08, 2024 at 03:26:43PM -0700, Dan Williams wrote:
> > > 
> > > <...>
> > > 
> > > > > It sets common expectations for
> > > > > device designers, distribution maintainers, and kernel developers. It is
> > > > > complimentary to the Linux-command path for operations that need deeper
> > > > > kernel coordination.
> > > > 
> > > > Yes, it's a good start, BUT by circumventing the network control plane,
> > > > the network driver maintainers rightfully are worried about this as
> > > > their review comments seem to be ignored here.  The rest of us
> > > > maintainers can't ignore that objection, sorry.
> > > 
> > > Can you please point to the TECHNICAL review comments that were
> > > presented and later ignored?
> > 
> > I can't remember review comments that were made yesterday, let alone
> > months ago, sorry.
> 
> So I will summarize the situation for you. There are NO technical review
> comments from netdev maintainer (not plural maintainers). The difference
> is philosophical and not technical.

Yes, to my knowledge no technical comments where given against fwctl
that have not been addressed.

Also, please, can we understand that networking in Linux has diverse
maintainership?

These days a huge amount of networking hardware is being deployed
where the software netdev networking stack is not the primary software
driving the HW. Many sites principally use RDMA, DPDK, and other
stacks on some devices.

For instance I previously shared a paper from Azure indicating that
over 70% of traffic on some of their networks was RDMA focused. [1]

We need to *share* the responsibility to support this complex HW. I'm
certain we won't always agree on the right way to do that.

If there is no real technical harm, let's not leap to vetoing ideas
from other networking-related subsystem maintainers please.

Jason

[1] https://www.usenix.org/system/files/nsdi23-bai.pdf

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Wed, Jul 10, 2024 at 02:22:38PM +0100, Jonathan Cameron wrote:
> On Tue, 9 Jul 2024 15:15:13 -0700
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > James Bottomley wrote:
> > > > The upstream discussion has yielded the full spectrum of positions on
> > > > device specific functionality, and it is a topic that needs cross-
> > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > concerns. Please consider it for a Maintainers Summit discussion.  
> > > 
> > > I'm with Greg on this ... can you point to some of the contrary
> > > positions?  
> > 
> > This thread has that discussion:
> > 
> > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > 
> > I do not want to speak for others on the saliency of their points, all I
> > can say is that the contrary positions have so far not moved me to drop
> > consideration of fwctl for CXL.
> 
> I was resisting rat holing. Oh well...
> 
> For a 'subset' of CXL.  There are a wide range of controls that are highly
> destructive, potentially to other hosts (simplest one is a command that
> will surprise remove someone else's memory).

I don't know alot of CXL, but from talking with Dan and reading these
posts it seems to me that CXL turn into a network, with switches and
multi-node and then somehow hid some kind of 'raw packet' interface to
communicate node-to-node. But never added any kind of node level
authorization? ie trust the nodes not to hurt each other?

Sounds sketchy to me :)

> So if fwctl is adopted, I do want the means to use it for the highly
> destructive stuff as well!  Maybe that's a future discussion.

With that kind of security model you probably have to trust the
userspace, even in a lockdown kernel.

ie can userspace replace the CXL HW that has the command interface
with VFIO and then do anything with nothing more than CAP_SYS_ADMIN
and root?

If so it is not unreasonable that a fwctl interface has a similar
level of protection.

> > Where CXL has a Command Effects Log that is a reasonable protocol for
> > making decisions about opaque command codes, and that CXL already has a
> > few years of experience with the commands that *do* need a Linux-command
> > wrapper.
> 
> Worth asking if this will incorporate unknown but not vendor defined
> commands.  There is a long tail of stuff in the spec we haven't caught up
> with yet.  Or you thinking keep this for the strictly vendor defined stuff?

I would allow as much as possible in fwctl that meets the defined
functional limitations and security model.

There is security merit in saying userspace will run, parse and
convert to output complex commands if it can safely do so. From an end
user perspective running a common tool to view the output is generally
always preferred anyhow, and the typical user doesn't really care if
the tool trundles through sysfs or does something else.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Thu, 2024-07-11 at 10:50 -0300, Jason Gunthorpe wrote:
> On Tue, Jul 09, 2024 at 09:02:25AM -0700, James Bottomley wrote:
> 
> > For NVMe and net we do have SPDK and DPDK.  What I find is that
> > people tend to use them for niche use cases (like the NVMe KV
> > command set) or obscure network routers.  Even though the claim
> > they both make is to get the kernel out of the way and do stuff
> > "way faster" the difficulty they create by bypassing everything is
> > quite a high burden.
> 
> [..]
>  
> > What all of the prior pass through's taught us is that if the use
> > case is big enough it will get pulled into the kernel and the
> > kernel will usually manage it better (DB users).  If it remains a
> > niche use case it will likely remain out of the kernel, but we
> > won't be hurt by it (NVME KV protocol) and sometimes it doesn't
> > really matter and the device manufacturers will sort it out on
> > their own (USB tokens).
> 
> I don't see it as being linked to big enough use case at all.

'it' being fwctl?  I'm happy to take a wait and see approach with that.
I'm in the camp that doesn't squash a novel proposal because the kernel
should be controlling it.  I'm confident that if the use case becomes
big enough the kernel will likely do it in the end.

> The kernel gets involved if there are good technical reasons to do
> so. Databases running over real filesystems with O_DIRECT is really
> technically better than raw block devices.

Exactly for a whole host of performance and more particularly
management reasons.

> While DPDK shows the opposite, userspace is the technically better
> option. This is now shown at scale. DPDK is not some niche. A big
> chunk of internet traffic is going through DPDKs, especially for
> mobile. Many ORAN solutions include DPDK on Linux.

ORAN being Open Radio Access Network?  But that's a case in point: the
kernel doesn't have a LTE stack or APN handling for networking.  RAN
hardware is not very widespread outside of cell providers, meaning it
doesn't get a lot of widespread exposure.  However, I believe Osmocom
is trying to change this (giving linux a native stack instead of DPDK)
and they're on record as referring to DPDK as the "Rabbit Hole".

To look at a counter example: how many linux based router boxes (i.e.
hardware based not cloud based) actually use DPDK?  I have a huge list
of cloud projects that overran and got cancelled because they decided
to use DPDK to replace a function the kernel was already doing (because
faster) and then found that if you replace function X in the kernel
generally the rest of the alphabet needs replacing as well, which blows
your project deadlines.

That's not to say there aren't a whole host of uses for DPDK: novel
protocols, traffic classification experiments, etc.  It's just that
it's like this honeytrap for the unwary and any project that comes
along with DPDK somewhere in its spec immediately gets extra scrutiny
(it's not that we don't do them, just that we make sure there's a
genuine use case that isn't reinventing what the kernel already does).

> What has been improved kernel-side is the intergation. DPDK
> deployments now often use RDMA raw queue pairs instead of VFIO, which
> laregly eliminates the "high burden".
> 
> There are many other cases, like DPDK, where the right answer is to
> reduce the kernel involvement. It is not so simple that things always
> get pulled into the kernel.

I don't disagree: there are many novel protocols and other use cases
that will never make it into the kernel simply because they won't get
the adoption; they're all ideal candidates for DPDK.  However, I do
take issue with "reduce kernel involvement" that's what gives rise to
projects that start by rewriting a piece of kernel networking in DPDK
and get sucked down the Rabbit Hole and never come out the other side.

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Thu, Jul 11, 2024 at 08:16:55AM -0700, James Bottomley wrote:

> > > What all of the prior pass through's taught us is that if the use
> > > case is big enough it will get pulled into the kernel and the
> > > kernel will usually manage it better (DB users).  If it remains a
> > > niche use case it will likely remain out of the kernel, but we
> > > won't be hurt by it (NVME KV protocol) and sometimes it doesn't
> > > really matter and the device manufacturers will sort it out on
> > > their own (USB tokens).
> > 
> > I don't see it as being linked to big enough use case at all.
> 
> 'it' being fwctl?  

Sorry I ment "it" as "if the use case is big enough it will get pulled
into the kernel"

> I'm in the camp that doesn't squash a novel proposal because the kernel
> should be controlling it.  I'm confident that if the use case becomes
> big enough the kernel will likely do it in the end.

Yes, I agree as well. If there is technical excellence to draw
someting into the kernel then it will definitely happen regardless if
other options are available.

> > While DPDK shows the opposite, userspace is the technically better
> > option. This is now shown at scale. DPDK is not some niche. A big
> > chunk of internet traffic is going through DPDKs, especially for
> > mobile. Many ORAN solutions include DPDK on Linux.
> 
> ORAN being Open Radio Access Network?  

Yes

> But that's a case in point: the kernel doesn't have a LTE stack or
> APN handling for networking.

That isn't really the main point of ORAN, a native LTE stack is
interesting separately but is a different problem space than ORAN.

ORAN is about doing all the complex maths for RF signal processing in
software disaggregated from the antennae hardware, in realtime. It is
some bonkers combination of >= 100G networking and hard realtime
packet processing with incredible math using dedicated accelerator HW.

> I don't disagree: there are many novel protocols and other use cases
> that will never make it into the kernel simply because they won't get
> the adoption;

My point is that prefering userspace/kernel isn't about NOVEL, it is
about technical excellence.

DPDK running a >= 100G NIC with extensive HW packet processing
offloads is technically excellent at doing proprietary traffic
processing work in mobile and cloud networks.

We are already running these DPDK things at near perfect performance
with minimal integration hurdles, there is not a lot of room to
actually be *better* with an in-kernel alternative.

This is what I mean, technically good displaces technically bad, it
has nothing to do with kernel=good userspace=bad.

The design of DPDK, with reduced kernel involvment, is a technically
correct approach to achiving it's goal, and it's goal is legitimate.

To expand a bit, Mellanox has supported both paths for a long time, a
lot of work has been done to improve the netstack and mlx5 to give
matching performance compared to DPDK in some use cases. Yet, I think
the market has spoken and DPDK seems to have much more popularity in
some very big verticals.

Yes, using DPDK carries a heavy development cost and people should
avoid it if they don't see a 5-10x improvment, IMHO.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Jonathan Cameron wrote:
> On Tue, 9 Jul 2024 15:15:13 -0700
> Dan Williams <dan.j.williams@intel.com> wrote:
> 
> > James Bottomley wrote:
> > > > The upstream discussion has yielded the full spectrum of positions on
> > > > device specific functionality, and it is a topic that needs cross-
> > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > concerns. Please consider it for a Maintainers Summit discussion.  
> > > 
> > > I'm with Greg on this ... can you point to some of the contrary
> > > positions?  
> > 
> > This thread has that discussion:
> > 
> > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > 
> > I do not want to speak for others on the saliency of their points, all I
> > can say is that the contrary positions have so far not moved me to drop
> > consideration of fwctl for CXL.
> 
> I was resisting rat holing. Oh well...
> 
> For a 'subset' of CXL.  There are a wide range of controls that are highly
> destructive, potentially to other hosts (simplest one is a command that
> will surprise remove someone else's memory). For those I'm not sure
> fwctl gets us anywhere - but we still need a solution (Subject to
> config gates etc as typically this is BMCs not hosts).
> Maybe fwctl eventually ends up with levels of 'safety' (beyond the
> current read vs write vs write_full, or maybe those are enough).

It is not clear to me that fwctl needs more levels of safety vs the
local subsystem config options controlling what can and can not be sent
over the channel. The CXL backend for fwctl adds the local "command
effects" level of safety.

For the "Linux as BMC" case the security model is external to the
kernel, right? Which means it does not present a protocol that the
kernel can reason about.

Unless and until someone develops an authorization model for BMC nodes
to join a network topology I think that use case is orthogonal to the
primary in-band use case for fwctl.

It is still useful there to avoid defining yet another transport, but a
node that has unfettered access to wreak havoc on the network is not the
kernel's problem.

> Complexities such as message tunneling to multiple components are also
> going to be fun, but we want the non destructive bits of those to work
> as part of the safe set, so we can get telemetry from downstream devices.
> 
> Good to cover the debug and telemetry usecase, but it still leaves us with
> gaping holes were we need to solve the permissions problem, perhaps that
> is layered on top of fwctl, perhaps something else is needed.

But that's more a CXL switch-management command security protocol
problem than fwctl, right? In other words, as far as I understand, there
is no spec provided permission model for switch management that Linux
could enforce, so it's more in the category of build a kernel that can
pass any payload and hope someone else has solved the problem of
limiting what damage that node can inflict.

> So if fwctl is adopted, I do want the means to use it for the highly
> destructive stuff as well!  Maybe that's a future discussion.
> 
> > Where CXL has a Command Effects Log that is a reasonable protocol for
> > making decisions about opaque command codes, and that CXL already has a
> > few years of experience with the commands that *do* need a Linux-command
> > wrapper.
> 
> Worth asking if this will incorporate unknown but not vendor defined
> commands.  There is a long tail of stuff in the spec we haven't caught up
> with yet.  Or you thinking keep this for the strictly vendor defined stuff?

Long term, yes, it should be able to expand to any command code family.
Short term, to get started, the CXL "Feature" facility at least conveys
whether opcodes are reads or writes, independent of their side effects,
and are scoped to be "settings".

There is still the matter of background commands need to support
cancellation to avoid indefinite background-command-slot monopolization,
and there are still commands that need kernel coordination. So, I see
fwctl command support arriving in stages.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

On Thu, 11 Jul 2024 12:05:59 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Jul 10, 2024 at 02:22:38PM +0100, Jonathan Cameron wrote:
> > On Tue, 9 Jul 2024 15:15:13 -0700
> > Dan Williams <dan.j.williams@intel.com> wrote:
> >   
> > > James Bottomley wrote:  
> > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > device specific functionality, and it is a topic that needs cross-
> > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > concerns. Please consider it for a Maintainers Summit discussion.    
> > > > 
> > > > I'm with Greg on this ... can you point to some of the contrary
> > > > positions?    
> > > 
> > > This thread has that discussion:
> > > 
> > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > 
> > > I do not want to speak for others on the saliency of their points, all I
> > > can say is that the contrary positions have so far not moved me to drop
> > > consideration of fwctl for CXL.  
> > 
> > I was resisting rat holing. Oh well...
> > 
> > For a 'subset' of CXL.  There are a wide range of controls that are highly
> > destructive, potentially to other hosts (simplest one is a command that
> > will surprise remove someone else's memory).  
> 
> I don't know alot of CXL, but from talking with Dan and reading these
> posts it seems to me that CXL turn into a network, with switches and
> multi-node and then somehow hid some kind of 'raw packet' interface to
> communicate node-to-node. But never added any kind of node level
> authorization? ie trust the nodes not to hurt each other?

You can't actually communicate node to node in the sense of host
to host.  You can just unplug stuff on another other host
(I guess that's a low bandwith comms channel...)
Data access should not be possible in general (ignoring shared
memory which is unrelated to the control path).
Control plane for the nasty stuff should all be in control
of one entity in the system - termed a fabric manager.

> 
> Sounds sketchy to me :)

Yes. The model is with the intent that this is only exposed by
hardware to a BMC / Fabric Manager - so security is by wiring.
The reason it's exposed on a PCI upstream port on a switch is that
there are designs where the Fabric manager is 'just another host'.
It's probably not running general software but that Fabric Manager
is running Linux too.  This isn't hugely different to not wiring
your MCTP management interface directly to the host such that the
OS can mess with it.

> 
> > So if fwctl is adopted, I do want the means to use it for the highly
> > destructive stuff as well!  Maybe that's a future discussion.  
> 
> With that kind of security model you probably have to trust the
> userspace, even in a lockdown kernel.

Agreed - but if we put infrastructure in place I want it to support
this as well.

> 
> ie can userspace replace the CXL HW that has the command interface
> with VFIO and then do anything with nothing more than CAP_SYS_ADMIN
> and root?

Yes if the wiring put that special PCI function on your PCI hierarchy.

> 
> If so it is not unreasonable that a fwctl interface has a similar
> level of protection.
> 
> > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > making decisions about opaque command codes, and that CXL already has a
> > > few years of experience with the commands that *do* need a Linux-command
> > > wrapper.  
> > 
> > Worth asking if this will incorporate unknown but not vendor defined
> > commands.  There is a long tail of stuff in the spec we haven't caught up
> > with yet.  Or you thinking keep this for the strictly vendor defined stuff?  
> 
> I would allow as much as possible in fwctl that meets the defined
> functional limitations and security model.
> 
> There is security merit in saying userspace will run, parse and
> convert to output complex commands if it can safely do so. From an end
> user perspective running a common tool to view the output is generally
> always preferred anyhow, and the typical user doesn't really care if
> the tool trundles through sysfs or does something else.

Fair enough.  A bit of potential duplication won't be too painful
(fwctl vs stuff that we know is safe enough for a 'normal' interface).

Thanks,

Jonathan
> 
> Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Thu, Jul 11, 2024 at 06:01:00PM +0100, Jonathan Cameron wrote:

> Control plane for the nasty stuff should all be in control
> of one entity in the system - termed a fabric manager.
> 
> > 
> > Sounds sketchy to me :)
> 
> Yes. The model is with the intent that this is only exposed by
> hardware to a BMC / Fabric Manager - so security is by wiring.

If you rely on physical seperation then I'd say that the Linux who
gets access to that physical HW should have rights to operate it, even
from userspace. You may say only root/user/label should have those
special rights, but it is kind of baked into the model that the
special physical connection lets you harm other nodes too.

It is important to think what things should be in fwctl, if I were to
take a similar situation for IB, the fabric manager plugs into
/dev/infiniband/umad* and that is the special interface for exchanging
packets between nodes to do fabric management.

But IB has a well defined container for fabric management and it is
easy to steer things into proper places.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

On Thu, 11 Jul 2024 09:36:27 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> Jonathan Cameron wrote:
> > On Tue, 9 Jul 2024 15:15:13 -0700
> > Dan Williams <dan.j.williams@intel.com> wrote:
> >   
> > > James Bottomley wrote:  
> > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > device specific functionality, and it is a topic that needs cross-
> > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > concerns. Please consider it for a Maintainers Summit discussion.    
> > > > 
> > > > I'm with Greg on this ... can you point to some of the contrary
> > > > positions?    
> > > 
> > > This thread has that discussion:
> > > 
> > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > 
> > > I do not want to speak for others on the saliency of their points, all I
> > > can say is that the contrary positions have so far not moved me to drop
> > > consideration of fwctl for CXL.  
> > 
> > I was resisting rat holing. Oh well...
> > 
> > For a 'subset' of CXL.  There are a wide range of controls that are highly
> > destructive, potentially to other hosts (simplest one is a command that
> > will surprise remove someone else's memory). For those I'm not sure
> > fwctl gets us anywhere - but we still need a solution (Subject to
> > config gates etc as typically this is BMCs not hosts).
> > Maybe fwctl eventually ends up with levels of 'safety' (beyond the
> > current read vs write vs write_full, or maybe those are enough).  
> 
> It is not clear to me that fwctl needs more levels of safety vs the
> local subsystem config options controlling what can and can not be sent
> over the channel. The CXL backend for fwctl adds the local "command
> effects" level of safety.
> 
> For the "Linux as BMC" case the security model is external to the
> kernel, right? Which means it does not present a protocol that the
> kernel can reason about.

The security model is indeed external, but I'd like a Linux BMC
config to allow turning off the protections but still using the
same fundamental interfaces as we normally use for the safe stuff.
I don't want
1) The CXL IOCTLs
2) FWCTL
3) Yet another interface.

> 
> Unless and until someone develops an authorization model for BMC nodes
> to join a network topology I think that use case is orthogonal to the
> primary in-band use case for fwctl.

Use case wise I agree this isn't the current primary in-band use case
for fwctl, hence the rat hole introductory comment.

> 
> It is still useful there to avoid defining yet another transport, but a
> node that has unfettered access to wreak havoc on the network is not the
> kernel's problem.

As long as I can enable it via a sensible interface (and don't need to
spin another) that is fine by me.

> 
> > Complexities such as message tunneling to multiple components are also
> > going to be fun, but we want the non destructive bits of those to work
> > as part of the safe set, so we can get telemetry from downstream devices.
> > 
> > Good to cover the debug and telemetry usecase, but it still leaves us with
> > gaping holes were we need to solve the permissions problem, perhaps that
> > is layered on top of fwctl, perhaps something else is needed.  
> 
> But that's more a CXL switch-management command security protocol
> problem than fwctl, right? In other words, as far as I understand, there
> is no spec provided permission model for switch management that Linux
> could enforce, so it's more in the category of build a kernel that can
> pass any payload and hope someone else has solved the problem of
> limiting what damage that node can inflict.

Two separate things here.

For tunneling, there is plenty that will map to fwctl because it's just
a transport question.  The tunnel command itself has a CEL that says
it might eat babies so we'd need to check the relevant CEL for the
destination to make sure they were just as safe as non tunneled version.
So it's just an implementation detail, be it a fiddly one.

For destructive options sure it's a config problem. But I do want
to be able to lock down the kernel on the BMC but still allow the
discructive command. Lock down is protecting and restricting the BMC
not the other hosts in this use case. 

> 
> > So if fwctl is adopted, I do want the means to use it for the highly
> > destructive stuff as well!  Maybe that's a future discussion.
> >   
> > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > making decisions about opaque command codes, and that CXL already has a
> > > few years of experience with the commands that *do* need a Linux-command
> > > wrapper.  
> > 
> > Worth asking if this will incorporate unknown but not vendor defined
> > commands.  There is a long tail of stuff in the spec we haven't caught up
> > with yet.  Or you thinking keep this for the strictly vendor defined stuff?  
> 
> Long term, yes, it should be able to expand to any command code family.
> Short term, to get started, the CXL "Feature" facility at least conveys
> whether opcodes are reads or writes, independent of their side effects,
> and are scoped to be "settings".
> 
> There is still the matter of background commands need to support
> cancellation to avoid indefinite background-command-slot monopolization,
> and there are still commands that need kernel coordination. So, I see
> fwctl command support arriving in stages.

Makes sense.  Tunneled access to CXL features should be an a good explorative
feature to do reasonably soon.

Jonathan

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Wed, Jul 10, 2024 at 10:05:14AM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 09, 2024 at 12:43:50PM -0700, Dan Williams wrote:
> 
> > A "Command Effects Log" seems like that starting point, with trust that
> > cynical abuses of that contract have a higher cost than benefit, and
> > trust that the protocol limits the potential damage of such abuse.
> 
> I've taken the view that companies are now very vigilant about
> security and often have their own internal incentives and procedures
> to do secure things.
> 
> If someone does a cynical security breaking thing and deploys it to a
> wide user base they are likely to be caught by a security researcher
> and embarassed with a CVE and a web site with a snappy name.

That may be the case in the server world, and for protocols such as
NVMe. My experience in the media world differs. I've seen too many
horrors to list them all here, so I'll only mention one of the worst
examples coming to my mind, of an (BSP) driver taking a physical address
from unpriviledged userspace and giving it to a DMA engine without any
filtering. I think this was mostly to be blamed on the developer not
knowing better, there was no malicious intent.

In general, can we trust closed-source firmwares when they document the
side effects of pass-through commands ? Again, I think the answer
differs between different classes of devices, the security culture is
not uniform across the whole IT industry.

> Not 100% of course, but it is certainly not a wild west of people just
> doing whatever they want.
> 
> The other half of this bargin is we have to be much clearer about what
> the security model is and what is security breaking. Like Christoph I
> often have conversations with people who don't understand the basics
> of how the Linux security models should work and are doing device-side
> work that has to fit into it.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> James Bottomley wrote:
> > > The upstream discussion has yielded the full spectrum of positions on
> > > device specific functionality, and it is a topic that needs cross-
> > > kernel consensus as hardware increasingly spans cross-subsystem
> > > concerns. Please consider it for a Maintainers Summit discussion.
> > 
> > I'm with Greg on this ... can you point to some of the contrary
> > positions?
> 
> This thread has that discussion:
> 
> http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> 
> I do not want to speak for others on the saliency of their points, all I
> can say is that the contrary positions have so far not moved me to drop
> consideration of fwctl for CXL.
> 
> Where CXL has a Command Effects Log that is a reasonable protocol for
> making decisions about opaque command codes, and that CXL already has a
> few years of experience with the commands that *do* need a Linux-command
> wrapper.
> 
> Some open questions from that thread are: what does it mean for the fate
> of a proposal if one subsystem Acks the ABI and another Naks it for a
> device that crosses subsystem functionality? Would a cynical hardware
> response just lead to plumbing an NVME admin queue, or CXL mailbox to
> get device-specific commands past another subsystem's objection?

My default answer would be to trust the maintainers of the relevant
subsystems (or try to convince them when you disagree :-)). Not only
should they know the technical implications best, they should also have
a good view of the whole vertical stack, and the implications of
pass-through for their ecosystem. This may result in a single NAK
overriding ACKs, but we could also try to find technical solutions when
we'll face such issues, to enforce different sets of rules for the
different functions of a device.

Subsystem hopping is something we're recently noticed for camera ISPs,
where a vendor wanted to move from V4L2 to DRM. Technical reasons for
doing so were given, and they were (in my opinion) rather excuses. The
unspoken real (again in my opinion) reason was to avoid documenting the
firmware interface and ship userspace binary blobs with no way for free
software to use all the device's features. That's something we have been
fighting against for years, trying to convince vendors that they can
provide better and more open camera support without the world
collapsing, with increasing success recently. Saying amen to
pass-through in this case would be a huge step back that would hurt
users and the whole ecosystem in the short and long term.

> My reconsideration of the "debug-build only" policy for CXL
> device-specific commands was influenced by a conversation with a distro
> developer where they asserted, paraphrasing: "at what point is a device
> vendor incentivized to ship an out-of-tree module just to restore their
> passthrough functionality?. At that point upstream has lost out on
> collaboration and distro kernel ABI has gained another out-of-tree
> consumer."
> 
> So the tension is healthy, but it has diminishing returns past a certain
> point.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > James Bottomley wrote:
> > > > The upstream discussion has yielded the full spectrum of positions on
> > > > device specific functionality, and it is a topic that needs cross-
> > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > 
> > > I'm with Greg on this ... can you point to some of the contrary
> > > positions?
> > 
> > This thread has that discussion:
> > 
> > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > 
> > I do not want to speak for others on the saliency of their points, all I
> > can say is that the contrary positions have so far not moved me to drop
> > consideration of fwctl for CXL.
> > 
> > Where CXL has a Command Effects Log that is a reasonable protocol for
> > making decisions about opaque command codes, and that CXL already has a
> > few years of experience with the commands that *do* need a Linux-command
> > wrapper.
> > 
> > Some open questions from that thread are: what does it mean for the fate
> > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > device that crosses subsystem functionality? Would a cynical hardware
> > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > get device-specific commands past another subsystem's objection?
> 
> My default answer would be to trust the maintainers of the relevant
> subsystems (or try to convince them when you disagree :-)).

You know, trust is a two-way street. If you want to trust maintainers,
they need to trust others as well. The situation where one maintainer
says "I don't trust you, so I will not allow you and other X maintainers
to do Y" is not a healthy situation.

> Not only should they know the technical implications best, they should also have
> a good view of the whole vertical stack, and the implications of
> pass-through for their ecosystem. 

It is wishful thinking. It is clearly not true for large subsystems
and/or complex devices.

Thanks

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:
> On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > James Bottomley wrote:
> > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > device specific functionality, and it is a topic that needs cross-
> > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > 
> > > > I'm with Greg on this ... can you point to some of the contrary
> > > > positions?
> > > 
> > > This thread has that discussion:
> > > 
> > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > 
> > > I do not want to speak for others on the saliency of their points, all I
> > > can say is that the contrary positions have so far not moved me to drop
> > > consideration of fwctl for CXL.
> > > 
> > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > making decisions about opaque command codes, and that CXL already has a
> > > few years of experience with the commands that *do* need a Linux-command
> > > wrapper.
> > > 
> > > Some open questions from that thread are: what does it mean for the fate
> > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > device that crosses subsystem functionality? Would a cynical hardware
> > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > get device-specific commands past another subsystem's objection?
> > 
> > My default answer would be to trust the maintainers of the relevant
> > subsystems (or try to convince them when you disagree :-)).
> 
> You know, trust is a two-way street. If you want to trust maintainers,
> they need to trust others as well. The situation where one maintainer
> says "I don't trust you, so I will not allow you and other X maintainers
> to do Y" is not a healthy situation.
> 
> > Not only should they know the technical implications best, they should also have
> > a good view of the whole vertical stack, and the implications of
> > pass-through for their ecosystem. 
> 
> It is wishful thinking. It is clearly not true for large subsystems
> and/or complex devices.

Are you saying that kernel communities behind large subsystems for
complex devices generally have no idea about what they're doing ? Or
that in a small number of particular cases those communities are
clueless ? Or does that apply to just the maintainer, not the whole
subsystem core developers ? I'd like to better understand the scale of
your claim here.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > James Bottomley wrote:
> > > > The upstream discussion has yielded the full spectrum of positions on
> > > > device specific functionality, and it is a topic that needs cross-
> > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > concerns. Please consider it for a Maintainers Summit discussion.
> > >
> > > I'm with Greg on this ... can you point to some of the contrary
> > > positions?
> >
> > This thread has that discussion:
> >
> > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> >
> > I do not want to speak for others on the saliency of their points, all I
> > can say is that the contrary positions have so far not moved me to drop
> > consideration of fwctl for CXL.
> >
> > Where CXL has a Command Effects Log that is a reasonable protocol for
> > making decisions about opaque command codes, and that CXL already has a
> > few years of experience with the commands that *do* need a Linux-command
> > wrapper.
> >
> > Some open questions from that thread are: what does it mean for the fate
> > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > device that crosses subsystem functionality? Would a cynical hardware
> > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > get device-specific commands past another subsystem's objection?
>
> My default answer would be to trust the maintainers of the relevant
> subsystems (or try to convince them when you disagree :-)). Not only
> should they know the technical implications best, they should also have
> a good view of the whole vertical stack, and the implications of
> pass-through for their ecosystem. This may result in a single NAK
> overriding ACKs, but we could also try to find technical solutions when
> we'll face such issues, to enforce different sets of rules for the
> different functions of a device.
>
> Subsystem hopping is something we're recently noticed for camera ISPs,
> where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> doing so were given, and they were (in my opinion) rather excuses. The
> unspoken real (again in my opinion) reason was to avoid documenting the
> firmware interface and ship userspace binary blobs with no way for free
> software to use all the device's features. That's something we have been
> fighting against for years, trying to convince vendors that they can
> provide better and more open camera support without the world
> collapsing, with increasing success recently. Saying amen to
> pass-through in this case would be a huge step back that would hurt
> users and the whole ecosystem in the short and long term.

In my view, DRM is a more suitable model for complex ISPs than V4L2:

- Userspace Complexity: ISPs demand a highly complex and evolving API,
similar to Vulkan or OpenGL. Applications typically need a framework
like libcamera to utilize ISPs effectively, much like Mesa for
graphics cards.

- Lack of Standardization: There's no universal standard for ISPs;
each vendor implements unique features and usage patterns. DRM
addresses this through vendor-specific IOCTLs

- Proprietary Architectures: Vendors often don't fully disclose their
hardware architectures. DRM cleverly only necessitates a Mesa
implementation, not comprehensive documentation.


Our current approach of pushing back against vendors, instead of
seeking compromise, has resulted in the vast majority of the market
(99% if not more) relying on out-of-tree drivers. This leaves users
with no options for utilizing their cameras outside of Android.

DRM allows a hybrid model, where:
- Open Source Foundation: Standard use cases are covered by a fully
open-source stack.
- Vendor Differentiation: Vendors retain the freedom to implement
proprietary features (e.g., automatic makeup) as closed source.

This approach would allow billions of users to access their hardware
more securely and with in-tree driver support. Our current stubborn
pursuit of an idealistic goal has already negatively impacted both
users and the ecosystem.

The late wins, in my opinion, cannot scale to the consumer market, and
Linux will remain a niche market for ISPs.


If such a hybrid model goes against Linux goals, this is something
that should be agreed upon by the whole community, so we have the same
criteria for all subsystems.



--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:
> On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:
> > On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > James Bottomley wrote:
> > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > 
> > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > positions?
> > > > 
> > > > This thread has that discussion:
> > > > 
> > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > 
> > > > I do not want to speak for others on the saliency of their points, all I
> > > > can say is that the contrary positions have so far not moved me to drop
> > > > consideration of fwctl for CXL.
> > > > 
> > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > making decisions about opaque command codes, and that CXL already has a
> > > > few years of experience with the commands that *do* need a Linux-command
> > > > wrapper.
> > > > 
> > > > Some open questions from that thread are: what does it mean for the fate
> > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > get device-specific commands past another subsystem's objection?
> > > 
> > > My default answer would be to trust the maintainers of the relevant
> > > subsystems (or try to convince them when you disagree :-)).
> > 
> > You know, trust is a two-way street. If you want to trust maintainers,
> > they need to trust others as well. The situation where one maintainer
> > says "I don't trust you, so I will not allow you and other X maintainers
> > to do Y" is not a healthy situation.
> > 
> > > Not only should they know the technical implications best, they should also have
> > > a good view of the whole vertical stack, and the implications of
> > > pass-through for their ecosystem. 
> > 
> > It is wishful thinking. It is clearly not true for large subsystems
> > and/or complex devices.
> 
> Are you saying that kernel communities behind large subsystems for
> complex devices generally have no idea about what they're doing ? Or
> that in a small number of particular cases those communities are
> clueless ? Or does that apply to just the maintainer, not the whole
> subsystem core developers ? I'd like to better understand the scale of
> your claim here.

I don't know how you jumped from saying "the maintainers of the relevant
subsystems" to "kernel communities". I'm talking about maintainers, not
communities.

There is no way to know everything about everything. In large subsystems,
the stack above kernel is so vast, which makes it impossible to know all
use cases. This is why some words (... good ... whole ...) in your sentence
are not accurate.

So the idea that one maintainer somehow equal to the whole community and
this person can block something for other members of the larger community
is overreaching.

Thanks

> 
> -- 
> Regards,
> 
> Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Leon,

On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:
> On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:
> > On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:
> > > On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > James Bottomley wrote:
> > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > 
> > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > positions?
> > > > > 
> > > > > This thread has that discussion:
> > > > > 
> > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > 
> > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > consideration of fwctl for CXL.
> > > > > 
> > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > wrapper.
> > > > > 
> > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > get device-specific commands past another subsystem's objection?
> > > > 
> > > > My default answer would be to trust the maintainers of the relevant
> > > > subsystems (or try to convince them when you disagree :-)).
> > > 
> > > You know, trust is a two-way street. If you want to trust maintainers,
> > > they need to trust others as well. The situation where one maintainer
> > > says "I don't trust you, so I will not allow you and other X maintainers
> > > to do Y" is not a healthy situation.
> > > 
> > > > Not only should they know the technical implications best, they should also have
> > > > a good view of the whole vertical stack, and the implications of
> > > > pass-through for their ecosystem. 
> > > 
> > > It is wishful thinking. It is clearly not true for large subsystems
> > > and/or complex devices.
> > 
> > Are you saying that kernel communities behind large subsystems for
> > complex devices generally have no idea about what they're doing ? Or
> > that in a small number of particular cases those communities are
> > clueless ? Or does that apply to just the maintainer, not the whole
> > subsystem core developers ? I'd like to better understand the scale of
> > your claim here.
> 
> I don't know how you jumped from saying "the maintainers of the relevant
> subsystems" to "kernel communities". I'm talking about maintainers, not
> communities.

I wasn't too sure, so that's why I asked. I have also not been very
precise in my previous e-mails. When I mentioned trusting maintainers, I
meant trusting the combined knowledge of the relevant maintainer(s) and
core developer(s) for a subsystem. The number of people that this
covers, and how they collectively reach agreements, very much depends on
subsystems.

> There is no way to know everything about everything. In large subsystems,
> the stack above kernel is so vast, which makes it impossible to know all
> use cases. This is why some words (... good ... whole ...) in your sentence
> are not accurate.
> 
> So the idea that one maintainer somehow equal to the whole community and
> this person can block something for other members of the larger community
> is overreaching.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Ricardo,

On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > James Bottomley wrote:
> > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > device specific functionality, and it is a topic that needs cross-
> > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > >
> > > > I'm with Greg on this ... can you point to some of the contrary
> > > > positions?
> > >
> > > This thread has that discussion:
> > >
> > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > >
> > > I do not want to speak for others on the saliency of their points, all I
> > > can say is that the contrary positions have so far not moved me to drop
> > > consideration of fwctl for CXL.
> > >
> > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > making decisions about opaque command codes, and that CXL already has a
> > > few years of experience with the commands that *do* need a Linux-command
> > > wrapper.
> > >
> > > Some open questions from that thread are: what does it mean for the fate
> > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > device that crosses subsystem functionality? Would a cynical hardware
> > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > get device-specific commands past another subsystem's objection?
> >
> > My default answer would be to trust the maintainers of the relevant
> > subsystems (or try to convince them when you disagree :-)). Not only
> > should they know the technical implications best, they should also have
> > a good view of the whole vertical stack, and the implications of
> > pass-through for their ecosystem. This may result in a single NAK
> > overriding ACKs, but we could also try to find technical solutions when
> > we'll face such issues, to enforce different sets of rules for the
> > different functions of a device.
> >
> > Subsystem hopping is something we're recently noticed for camera ISPs,
> > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > doing so were given, and they were (in my opinion) rather excuses. The
> > unspoken real (again in my opinion) reason was to avoid documenting the
> > firmware interface and ship userspace binary blobs with no way for free
> > software to use all the device's features. That's something we have been
> > fighting against for years, trying to convince vendors that they can
> > provide better and more open camera support without the world
> > collapsing, with increasing success recently. Saying amen to
> > pass-through in this case would be a huge step back that would hurt
> > users and the whole ecosystem in the short and long term.
> 
> In my view, DRM is a more suitable model for complex ISPs than V4L2:

I know we disagree on this topic :-) I'm sure we'll continue the
conversation, but I think the technical discussion likely belongs to a
different mail thread.

> - Userspace Complexity: ISPs demand a highly complex and evolving API,
> similar to Vulkan or OpenGL. Applications typically need a framework
> like libcamera to utilize ISPs effectively, much like Mesa for
> graphics cards.
> 
> - Lack of Standardization: There's no universal standard for ISPs;
> each vendor implements unique features and usage patterns. DRM
> addresses this through vendor-specific IOCTLs
> 
> - Proprietary Architectures: Vendors often don't fully disclose their
> hardware architectures. DRM cleverly only necessitates a Mesa
> implementation, not comprehensive documentation.

This point isn't technical and is more on-topic for this mail thread.

V4L2 doesn't require hundreds of pages of comprehensive documentation in
text form. An open-source userspace implementation that covers the
feature set exposed by the driver is acceptable in place of
documentation (provided, of course, that the userspace code wouldn't be
deliberately obfuscated). This is similar in spirit to the rule for GPU
DRM drivers.

> Our current approach of pushing back against vendors, instead of
> seeking compromise, has resulted in the vast majority of the market
> (99% if not more) relying on out-of-tree drivers. This leaves users
> with no options for utilizing their cameras outside of Android.
> 
> DRM allows a hybrid model, where:
> - Open Source Foundation: Standard use cases are covered by a fully
> open-source stack.
> - Vendor Differentiation: Vendors retain the freedom to implement
> proprietary features (e.g., automatic makeup) as closed source.

V4L2 does as well, you can implement all kind of closed-source ISP
control algorithms in userspace, as long as there's an open-source
implementation that exercises the same hardware features. A good analogy
for people less familiar with ISPs is shader compilers, GPU vendors are
free to ship closed-source implementations that include more
optimizations, as long as the open-source, less optimized implementation
covers the same GPU ISA, so that open-source developers can also work on
optimizing it.

Thinking that DRM would offer a free pass-through path compared to V4L2
doesn't seem realistic to me. Both subsystems will have similar rules.

> This approach would allow billions of users to access their hardware
> more securely and with in-tree driver support. Our current stubborn
> pursuit of an idealistic goal has already negatively impacted both
> users and the ecosystem.
> 
> The late wins, in my opinion, cannot scale to the consumer market, and
> Linux will remain a niche market for ISPs.
> 
> If such a hybrid model goes against Linux goals, this is something
> that should be agreed upon by the whole community, so we have the same
> criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

Hi Laurent

On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Ricardo,
>
> On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > James Bottomley wrote:
> > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > >
> > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > positions?
> > > >
> > > > This thread has that discussion:
> > > >
> > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > >
> > > > I do not want to speak for others on the saliency of their points, all I
> > > > can say is that the contrary positions have so far not moved me to drop
> > > > consideration of fwctl for CXL.
> > > >
> > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > making decisions about opaque command codes, and that CXL already has a
> > > > few years of experience with the commands that *do* need a Linux-command
> > > > wrapper.
> > > >
> > > > Some open questions from that thread are: what does it mean for the fate
> > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > get device-specific commands past another subsystem's objection?
> > >
> > > My default answer would be to trust the maintainers of the relevant
> > > subsystems (or try to convince them when you disagree :-)). Not only
> > > should they know the technical implications best, they should also have
> > > a good view of the whole vertical stack, and the implications of
> > > pass-through for their ecosystem. This may result in a single NAK
> > > overriding ACKs, but we could also try to find technical solutions when
> > > we'll face such issues, to enforce different sets of rules for the
> > > different functions of a device.
> > >
> > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > doing so were given, and they were (in my opinion) rather excuses. The
> > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > firmware interface and ship userspace binary blobs with no way for free
> > > software to use all the device's features. That's something we have been
> > > fighting against for years, trying to convince vendors that they can
> > > provide better and more open camera support without the world
> > > collapsing, with increasing success recently. Saying amen to
> > > pass-through in this case would be a huge step back that would hurt
> > > users and the whole ecosystem in the short and long term.
> >
> > In my view, DRM is a more suitable model for complex ISPs than V4L2:
>
> I know we disagree on this topic :-) I'm sure we'll continue the
> conversation, but I think the technical discussion likely belongs to a
> different mail thread.
>
> > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > similar to Vulkan or OpenGL. Applications typically need a framework
> > like libcamera to utilize ISPs effectively, much like Mesa for
> > graphics cards.
> >
> > - Lack of Standardization: There's no universal standard for ISPs;
> > each vendor implements unique features and usage patterns. DRM
> > addresses this through vendor-specific IOCTLs
> >
> > - Proprietary Architectures: Vendors often don't fully disclose their
> > hardware architectures. DRM cleverly only necessitates a Mesa
> > implementation, not comprehensive documentation.
>
> This point isn't technical and is more on-topic for this mail thread.
>
> V4L2 doesn't require hundreds of pages of comprehensive documentation in
> text form. An open-source userspace implementation that covers the
> feature set exposed by the driver is acceptable in place of
> documentation (provided, of course, that the userspace code wouldn't be
> deliberately obfuscated). This is similar in spirit to the rule for GPU
> DRM drivers.

In DRM vendors typically define a custom IOCTL per driver to pass
command buffers.
Only the command buffer structure, and a mesa implementation using
that command buffer to support the standard features is required.

In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
be passed from userspace, they are typically formed in the driver from
a strictly checked struct.

>
> > Our current approach of pushing back against vendors, instead of
> > seeking compromise, has resulted in the vast majority of the market
> > (99% if not more) relying on out-of-tree drivers. This leaves users
> > with no options for utilizing their cameras outside of Android.
> >
> > DRM allows a hybrid model, where:
> > - Open Source Foundation: Standard use cases are covered by a fully
> > open-source stack.
> > - Vendor Differentiation: Vendors retain the freedom to implement
> > proprietary features (e.g., automatic makeup) as closed source.
>
> V4L2 does as well, you can implement all kind of closed-source ISP
> control algorithms in userspace, as long as there's an open-source
> implementation that exercises the same hardware features. A good analogy

Is it really mandatory to have an open-source 3A algorithm? I thought
defining the input and output from the algorithm was good enough.
AFAIK for some time there was no ipu3 open source algorithm, and the
driver has been upstream.

> for people less familiar with ISPs is shader compilers, GPU vendors are
> free to ship closed-source implementations that include more
> optimizations, as long as the open-source, less optimized implementation
> covers the same GPU ISA, so that open-source developers can also work on
> optimizing it.

I believe a more accurate description is that in v4l2 is that we
expect that all the registers, device architecture and behaviour to be
documented and accessed with standard IOCTLs. Anything not documented
cannot be accessed by userspace.

In DRM their concern is that there is a fully open source
implementation that the user can use. Vendors have custom IOCTLs and
they can offer proprietary software for some use cases.

>
> Thinking that DRM would offer a free pass-through path compared to V4L2
> doesn't seem realistic to me. Both subsystems will have similar rules.

DRM does indeed allow vendors to pass random command buffers and they
will be sent to the hardware. We cannot do that in v4l2.

I might be wrong, but GPU drivers do not deeply inspect the command
buffers to make sure that they do not use any feature not covered by
mesa.

>
> > This approach would allow billions of users to access their hardware
> > more securely and with in-tree driver support. Our current stubborn
> > pursuit of an idealistic goal has already negatively impacted both
> > users and the ecosystem.
> >
> > The late wins, in my opinion, cannot scale to the consumer market, and
> > Linux will remain a niche market for ISPs.
> >
> > If such a hybrid model goes against Linux goals, this is something
> > that should be agreed upon by the whole community, so we have the same
> > criteria for all subsystems.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Sun, Jul 21, 2024 at 09:51:05PM +0300, Laurent Pinchart wrote:

> That may be the case in the server world, and for protocols such as
> NVMe. My experience in the media world differs. I've seen too many
> horrors to list them all here, so I'll only mention one of the worst
> examples coming to my mind, of an (BSP) driver taking a physical address
> from unpriviledged userspace and giving it to a DMA engine without any
> filtering. I think this was mostly to be blamed on the developer not
> knowing better, there was no malicious intent.
> 
> In general, can we trust closed-source firmwares when they document the
> side effects of pass-through commands ? Again, I think the answer
> differs between different classes of devices, the security culture is
> not uniform across the whole IT industry.

That does make sense to me, and I certainly don't feel the same
comfort when looking at embedded or consumer HW that has a
historically much weaker security story.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Mon, Jul 22, 2024 at 02:10:04PM +0300, Laurent Pinchart wrote:
> Hi Leon,
> 
> On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:
> > On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:
> > > On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:
> > > > On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > James Bottomley wrote:
> > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > 
> > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > positions?
> > > > > > 
> > > > > > This thread has that discussion:
> > > > > > 
> > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > 
> > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > consideration of fwctl for CXL.
> > > > > > 
> > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > wrapper.
> > > > > > 
> > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > get device-specific commands past another subsystem's objection?
> > > > > 
> > > > > My default answer would be to trust the maintainers of the relevant
> > > > > subsystems (or try to convince them when you disagree :-)).
> > > > 
> > > > You know, trust is a two-way street. If you want to trust maintainers,
> > > > they need to trust others as well. The situation where one maintainer
> > > > says "I don't trust you, so I will not allow you and other X maintainers
> > > > to do Y" is not a healthy situation.
> > > > 
> > > > > Not only should they know the technical implications best, they should also have
> > > > > a good view of the whole vertical stack, and the implications of
> > > > > pass-through for their ecosystem. 
> > > > 
> > > > It is wishful thinking. It is clearly not true for large subsystems
> > > > and/or complex devices.
> > > 
> > > Are you saying that kernel communities behind large subsystems for
> > > complex devices generally have no idea about what they're doing ? Or
> > > that in a small number of particular cases those communities are
> > > clueless ? Or does that apply to just the maintainer, not the whole
> > > subsystem core developers ? I'd like to better understand the scale of
> > > your claim here.
> > 
> > I don't know how you jumped from saying "the maintainers of the relevant
> > subsystems" to "kernel communities". I'm talking about maintainers, not
> > communities.
> 
> I wasn't too sure, so that's why I asked. I have also not been very
> precise in my previous e-mails. When I mentioned trusting maintainers, I
> meant trusting the combined knowledge of the relevant maintainer(s) and
> core developer(s) for a subsystema

Unfortunately, the reason for this topic proposed for Maintainer's summit
is that the maintainer and core developers are disagree and there is no way
to resolve it, because it is not technical difference, but a philosophical one.


> The number of people that this covers, and how they collectively reach
> agreements, very much depends on subsystems.
> 
> > There is no way to know everything about everything. In large subsystems,
> > the stack above kernel is so vast, which makes it impossible to know all
> > use cases. This is why some words (... good ... whole ...) in your sentence
> > are not accurate.
> > 
> > So the idea that one maintainer somehow equal to the whole community and
> > this person can block something for other members of the larger community
> > is overreaching.
> 
> -- 
> Regards,
> 
> Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Mon, Jul 22, 2024 at 04:28:28PM +0300, Leon Romanovsky wrote:
> On Mon, Jul 22, 2024 at 02:10:04PM +0300, Laurent Pinchart wrote:
> > On Mon, Jul 22, 2024 at 01:44:07PM +0300, Leon Romanovsky wrote:
> > > On Mon, Jul 22, 2024 at 11:53:17AM +0300, Laurent Pinchart wrote:
> > > > On Mon, Jul 22, 2024 at 10:31:19AM +0300, Leon Romanovsky wrote:
> > > > > On Sun, Jul 21, 2024 at 10:25:30PM +0300, Laurent Pinchart wrote:
> > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > James Bottomley wrote:
> > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > 
> > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > positions?
> > > > > > > 
> > > > > > > This thread has that discussion:
> > > > > > > 
> > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > 
> > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > consideration of fwctl for CXL.
> > > > > > > 
> > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > wrapper.
> > > > > > > 
> > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > 
> > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > subsystems (or try to convince them when you disagree :-)).
> > > > > 
> > > > > You know, trust is a two-way street. If you want to trust maintainers,
> > > > > they need to trust others as well. The situation where one maintainer
> > > > > says "I don't trust you, so I will not allow you and other X maintainers
> > > > > to do Y" is not a healthy situation.
> > > > > 
> > > > > > Not only should they know the technical implications best, they should also have
> > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > pass-through for their ecosystem. 
> > > > > 
> > > > > It is wishful thinking. It is clearly not true for large subsystems
> > > > > and/or complex devices.
> > > > 
> > > > Are you saying that kernel communities behind large subsystems for
> > > > complex devices generally have no idea about what they're doing ? Or
> > > > that in a small number of particular cases those communities are
> > > > clueless ? Or does that apply to just the maintainer, not the whole
> > > > subsystem core developers ? I'd like to better understand the scale of
> > > > your claim here.
> > > 
> > > I don't know how you jumped from saying "the maintainers of the relevant
> > > subsystems" to "kernel communities". I'm talking about maintainers, not
> > > communities.
> > 
> > I wasn't too sure, so that's why I asked. I have also not been very
> > precise in my previous e-mails. When I mentioned trusting maintainers, I
> > meant trusting the combined knowledge of the relevant maintainer(s) and
> > core developer(s) for a subsystema
> 
> Unfortunately, the reason for this topic proposed for Maintainer's summit
> is that the maintainer and core developers are disagree and there is no way
> to resolve it, because it is not technical difference, but a philosophical one.

Having been involved in a similar disagreement, I'm not sure
"philosophical" is the right term. I can't talk about the fwctl issue in
particular as I have only vaguely followed the saga, and I will
therefore not take a side there, but in general I tend to use
"political" instead of "philosophical". The issues of market control,
competition and vendor lock-in vs. empowerment also play important
roles. This makes it even more difficult to discuss the disagreements
openly.

> > The number of people that this covers, and how they collectively reach
> > agreements, very much depends on subsystems.
> > 
> > > There is no way to know everything about everything. In large subsystems,
> > > the stack above kernel is so vast, which makes it impossible to know all
> > > use cases. This is why some words (... good ... whole ...) in your sentence
> > > are not accurate.
> > > 
> > > So the idea that one maintainer somehow equal to the whole community and
> > > this person can block something for other members of the larger community
> > > is overreaching.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Mon, Jul 22, 2024 at 05:13:43PM +0300, Laurent Pinchart wrote:

> > Unfortunately, the reason for this topic proposed for Maintainer's summit
> > is that the maintainer and core developers are disagree and there is no way
> > to resolve it, because it is not technical difference, but a philosophical one.
> 
> Having been involved in a similar disagreement, I'm not sure
> "philosophical" is the right term. I can't talk about the fwctl issue in
> particular as I have only vaguely followed the saga, and I will
> therefore not take a side there, but in general I tend to use
> "political" instead of "philosophical". The issues of market control,
> competition and vendor lock-in vs. empowerment also play important
> roles. This makes it even more difficult to discuss the disagreements
> openly.

I've prefered philosophical in this case because I don't view any ill
will on any side here, while I would tend to think the word political
carries a more negative tone.

There isn't really any lock in discussion here, there is some
philosophical disagreement on how much "generalism" should be enforced
by the kernel in some topics and a fear that if things are more
permissive through the kernel then some will loose power to force
their views.

As we don't even agree on how much generalism the kernel should be
enforcing, and what value that even brings to the ecosystem, it is
very much philosophical.

This is a little different than the open shader compiler sorts of
arguments where GPU compilers really underpin a lot of value in the
software stack. We are talking about configuring flash on a device,
and debugging the FW running on the device. There is minimal value to
open source communities with these activities in the first place.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jiri Kosina]

On Mon, 8 Jul 2024, Dan Williams wrote:

> 2/ Device passthrough, kernel passing opaque payloads, is already taken
>    for granted in many subsystems. USB and HID have "raw" interfaces

Just as a completely random datapoint here: after I implemented hidraw 
inteface long time ago, I was a little bit hesitant about really merging 
it, because there was a general fear that this would shatter the HID 
driver ecosystem, making it difficult for people to find proper drivers 
for their devices, etc.

Turns out that that didn't happen. Drivers for generic devices are still 
implemented properly in the kernel, and hidraw is mostly used for rather 
specific, one-off solutions, where the vendor's business plan is "ship 
this one appliance and forget forever", which doesn't really cause any 
harm to the whole ecosystem.

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Tue, 2024-07-23 at 13:20 +0200, Jiri Kosina wrote:
> On Mon, 8 Jul 2024, Dan Williams wrote:
> 
> > 2/ Device passthrough, kernel passing opaque payloads, is already
> > taken for granted in many subsystems. USB and HID have "raw"
> > interfaces
> 
> Just as a completely random datapoint here: after I implemented
> hidraw inteface long time ago, I was a little bit hesitant about
> really merging it, because there was a general fear that this would
> shatter the HID driver ecosystem, making it difficult for people to
> find proper drivers  for their devices, etc.

The problem with hidraw is that userspace has to understand the device
to use it, but a lot of HID devices (keyboards, mice, serial ports,
etc.) want to fit into an existing ecosystem so they have to have a
kernel driver to avoid having to update all the user applications. 
However, entirely new devices don't have the existing ecosystem
problem.

> Turns out that that didn't happen. Drivers for generic devices are
> still implemented properly in the kernel, and hidraw is mostly used
> for rather specific, one-off solutions, where the vendor's business
> plan is "ship this one appliance and forget forever", which doesn't
> really cause any harm to the whole ecosystem.

That's not entirely true.  FIDO tokens (the ones Konstantin is
recommending for kernel.org access) are an entire class of devices that
use hidraw and don't have a kernel driver.  There's an array of
manufacturers producing them, but the CTAP specification and its
conformance is what keeps a single user mode driver (which is now
present as a separate implementation in all web browsers and the
userspace libfido2) for all of them.  Fido is definitely not a one off,
but on the other hand, not having a kernel driver doesn't seem to harm
the ecosystem and they can get away with it because there was no
existing device type for them to fit into (except, as you say, an array
of incompatible and short lived USB key tokens which annoyed everyone
by having usability limits due to the oneoffness).

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jiri Kosina]

On Tue, 23 Jul 2024, James Bottomley wrote:

> That's not entirely true.  FIDO tokens (the ones Konstantin is
> recommending for kernel.org access) are an entire class of devices that
> use hidraw and don't have a kernel driver.  There's an array of
> manufacturers producing them, but the CTAP specification and its
> conformance is what keeps a single user mode driver (which is now
> present as a separate implementation in all web browsers and the
> userspace libfido2) for all of them.  

Agreed, but that pretty much underlines my point though.

The ecosystem didn't get shattered as a result of me having created 
hidraw.

libfido2 is on pretty much everyone's machine now (at least for those who 
need it), and people are using that all the time to authenticate to 
kernel.org/Google/Okta/whatnot. No workflow got broken in the process.

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Tue, Jul 23, 2024 at 07:36:24AM -0400, James Bottomley wrote:
> On Tue, 2024-07-23 at 13:20 +0200, Jiri Kosina wrote:
> > On Mon, 8 Jul 2024, Dan Williams wrote:
> > 
> > > 2/ Device passthrough, kernel passing opaque payloads, is already
> > > taken for granted in many subsystems. USB and HID have "raw"
> > > interfaces
> > 
> > Just as a completely random datapoint here: after I implemented
> > hidraw inteface long time ago, I was a little bit hesitant about
> > really merging it, because there was a general fear that this would
> > shatter the HID driver ecosystem, making it difficult for people to
> > find proper drivers  for their devices, etc.
> 
> The problem with hidraw is that userspace has to understand the device
> to use it, but a lot of HID devices (keyboards, mice, serial ports,
> etc.) want to fit into an existing ecosystem so they have to have a
> kernel driver to avoid having to update all the user applications. 
> However, entirely new devices don't have the existing ecosystem
> problem.
> 
> > Turns out that that didn't happen. Drivers for generic devices are
> > still implemented properly in the kernel, and hidraw is mostly used
> > for rather specific, one-off solutions, where the vendor's business
> > plan is "ship this one appliance and forget forever", which doesn't
> > really cause any harm to the whole ecosystem.
> 
> That's not entirely true.  FIDO tokens (the ones Konstantin is
> recommending for kernel.org access) are an entire class of devices that
> use hidraw and don't have a kernel driver.  There's an array of
> manufacturers producing them, but the CTAP specification and its
> conformance is what keeps a single user mode driver (which is now
> present as a separate implementation in all web browsers and the
> userspace libfido2) for all of them.  Fido is definitely not a one off,
> but on the other hand, not having a kernel driver doesn't seem to harm
> the ecosystem and they can get away with it because there was no
> existing device type for them to fit into (except, as you say, an array
> of incompatible and short lived USB key tokens which annoyed everyone
> by having usability limits due to the oneoffness).

While "userspace drivers" often cause allergic reactions, I think I
won't cause a controversy if I say that we are all used to them in
certain areas. My heart rate will increase if someone proposes replacing
a USB webcam driver with a libusb-based solution, but I don't lose sleep
over the fact that my GPU is mostly controlled by code in Mesa.

What I get from the discussions I've followed or partcipated in over the
years is that the main worry of free software communities is being
forced to use closed-source userspace components, whether that would be
to make the device usable at all, or to achieve decent level of
performance or full feature set. We've been through years of mostly
closed-source GPU support, of printer "windrivers", and quite a few
other horrors. The good news is that we've so far overcome lots (most)
of those challenges. Reverse engineering projects paid off, and so did
working hand-in-hand with industry actors in multiple ways (both openly
and behind the scenes). One could then legitimately ask why we're still
scared.

I can't fully answer that question, but there are two points that I
think are relevant. Note that due to my background and experience, this
will be heavily biased towards consumer and embedded hardware, not data
centre-grade devices. Some technologies from the latter however have a
tendency to migrate to the former over time, so the distinction isn't
necessarily as relevant as one may consider.

The first point is that hardware gets more complicated over time, and in
some markets there's also an increase in the number of vendors and
devices. There's a perceived (whether true or not) danger that we won't
be able to keep up with just reverse engineering and a development model
relying on hobyists. Getting vendors involved is important if we want to
scale.

Second, I think there's a fear of regression. For some categories of
devices, we have made slow but real progress to try and convince the
industry to be more open. This sometimes took a decade of work,
patiently building bridges and creating ecosystems brick by brick. Some
of those ecosystems are sturdy, some not so. Giving pass-through a blank
check will likely have very different effects in different areas. I
don't personally believe it will shatter everything, but I'm convinced
it carries risk in areas where cooperation with vendors is in its
infancy or is fragile for any other reason.

Finally, let's not forget that pass-through APIs are not an all or
nothing option. To cite that example only, DRM requires GPU drivers to
have an open-source userspace implementation to merge the kernel driver,
and the same subsystems strongly pushes for API standardization for
display controllers. We can set different rules for different cases.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Wed, 2024-07-24 at 01:22 +0200, Jiri Kosina wrote:
> On Tue, 23 Jul 2024, James Bottomley wrote:
> 
> > That's not entirely true.  FIDO tokens (the ones Konstantin is
> > recommending for kernel.org access) are an entire class of devices
> > that
> > use hidraw and don't have a kernel driver.  There's an array of
> > manufacturers producing them, but the CTAP specification and its
> > conformance is what keeps a single user mode driver (which is now
> > present as a separate implementation in all web browsers and the
> > userspace libfido2) for all of them.  
> 
> Agreed, but that pretty much underlines my point though.
> 
> The ecosystem didn't get shattered as a result of me having created 
> hidraw.

Yes, we're in agreement on this.  I was just extrapolating to not every
bypass is inherently evil.

> libfido2 is on pretty much everyone's machine now (at least for those
> who need it), and people are using that all the time to authenticate
> to kernel.org/Google/Okta/whatnot. No workflow got broken in the
> process.

Well, there is one use case I can think of that would have the kernel
talking to a fido token (i.e. us having a kernel driver): using it as
the root for trusted and encrypted keys.  It might be very useful for
security features like encrypted device tree or kernel command line
files, or even passing in a private X.509 key to add to the kernel
trusted keyrings or for module signing.   The rush to bypass the kernel
deprived us of thinking about this as an application, but, since the
spec is open, if anyone cares enough, I'm sure it will eventually get
written.

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
[...]
> What I get from the discussions I've followed or partcipated in over
> the years is that the main worry of free software communities is
> being forced to use closed-source userspace components, whether that
> would be to make the device usable at all, or to achieve decent level
> of performance or full feature set. We've been through years of
> mostly closed-source GPU support, of printer "windrivers", and quite
> a few other horrors. The good news is that we've so far overcome lots
> (most) of those challenges. Reverse engineering projects paid off,
> and so did working hand-in-hand with industry actors in multiple ways
> (both openly and behind the scenes). One could then legitimately ask
> why we're still scared.

I don't think I am.  We're mostly fully capable of expounding at length
on the business rationale for being open if the thing they're hiding
isn't much of a differentiator anyway (or they're simply hiding it to
try to retain some illusion of control), so we shouldn't have any fear
of being able to make our case in language business people understand.

I also think this fear is partly a mindset problem on our part.  We
came out of the real fight for openness and we do embrace things like a
licence that forces open code (GPL) and symbols that discourage
proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
FSF coolaid that if we don't stand over manufacturers every second and
force them they'll slide back to their old proprietary ways.  However,
if you look at the entirely permissive ecosystem that grew up after we
did (openstack, docker, kubernetes, etc.) they don't have any such fear
and yet they still have large amounts of uncompelled openness and give
back.

> I can't fully answer that question, but there are two points that I
> think are relevant. Note that due to my background and experience,
> this will be heavily biased towards consumer and embedded hardware,
> not data centre-grade devices. Some technologies from the latter
> however have a tendency to migrate to the former over time, so the
> distinction isn't necessarily as relevant as one may consider.
> 
> The first point is that hardware gets more complicated over time, and
> in some markets there's also an increase in the number of vendors and
> devices. There's a perceived (whether true or not) danger that we
> won't be able to keep up with just reverse engineering and a
> development model relying on hobyists. Getting vendors involved is
> important if we want to scale.

Yes, but there are lots of not very useful complex devices being
produced every day that fail to capture market share.  Not having
reverse engineered drivers for them is no real loss.  If a device does
gain market share, it gains a huge pool of users some of whom become
interested in reverse engineering, so I think market forces actually
work in our favour: we get reverse engineering mostly where the devices
are actually interesting and capture market share.  It's self scaling.

> Second, I think there's a fear of regression. For some categories of
> devices, we have made slow but real progress to try and convince the
> industry to be more open. This sometimes took a decade of work,
> patiently building bridges and creating ecosystems brick by brick.
> Some of those ecosystems are sturdy, some not so. Giving pass-through
> a blank check will likely have very different effects in different
> areas. I don't personally believe it will shatter everything, but I'm
> convinced it carries risk in areas where cooperation with vendors is
> in its infancy or is fragile for any other reason.

I also think we're on the rise in this space.  Since most cloud
workloads are on Linux, there's huge market pressure on most "found in
the cloud" devices (like accelerators and GPUs) to have an easy to
consume Linux story.  Nvidia is a case in point.  When it only cared
about fast games on some other OS, we get shafted with a proprietary
graphics drivers.  Now it's under pressure to be the number one AI
accelerator provider for the cloud it's suddenly wondering about open
source drivers to make adoption easier.

> Finally, let's not forget that pass-through APIs are not an all or
> nothing option. To cite that example only, DRM requires GPU drivers
> to have an open-source userspace implementation to merge the kernel
> driver, and the same subsystems strongly pushes for API
> standardization for display controllers. We can set different rules
> for different cases.

I certainly think we can afford to experiment here, yes.

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Steven Rostedt]

On Wed, 24 Jul 2024 16:37:21 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > The first point is that hardware gets more complicated over time, and
> > in some markets there's also an increase in the number of vendors and
> > devices. There's a perceived (whether true or not) danger that we
> > won't be able to keep up with just reverse engineering and a
> > development model relying on hobyists. Getting vendors involved is
> > important if we want to scale.  
> 
> Yes, but there are lots of not very useful complex devices being
> produced every day that fail to capture market share.  Not having
> reverse engineered drivers for them is no real loss.  If a device does
> gain market share, it gains a huge pool of users some of whom become
> interested in reverse engineering, so I think market forces actually
> work in our favour: we get reverse engineering mostly where the devices
> are actually interesting and capture market share.  It's self scaling.

I agree with this. If a small vendor with low market share has a
proprietary device where they could easily port to Linux via a pass
through, they may do that. But if they don't have that, and require
engineering resources to port to Linux, they will not bother. As they would
only care about the Windows market. So the device becomes useless for the
Linux system. There's not enough Linux users to make a small vendor care
about losing us. That just shrinks the number of devices that are available
to Linux.

My guess is that vendors want to write one piece of code. If it they only
need to modify a small portion to get it to another OS, they would do that.
But if it takes more effort than that, there's not enough cost incentive to
bother.

For devices with a larger market share, it would make it more worth their
while to open source their work, otherwise there's more incentive for us to
reverse engineer it anyway.

-- Steve

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:

>
> While "userspace drivers" often cause allergic reactions, I think I
> won't cause a controversy if I say that we are all used to them in
> certain areas. My heart rate will increase if someone proposes replacing
> a USB webcam driver with a libusb-based solution, but I don't lose sleep
> over the fact that my GPU is mostly controlled by code in Mesa.

I think the key point here is that USB webcams follow a standard, and
GPUs don't.


>
> What I get from the discussions I've followed or partcipated in over the
> years is that the main worry of free software communities is being
> forced to use closed-source userspace components, whether that would be
> to make the device usable at all, or to achieve decent level of
> performance or full feature set. We've been through years of mostly
> closed-source GPU support, of printer "windrivers", and quite a few
> other horrors. The good news is that we've so far overcome lots (most)
> of those challenges. Reverse engineering projects paid off, and so did
> working hand-in-hand with industry actors in multiple ways (both openly
> and behind the scenes). One could then legitimately ask why we're still
> scared.


It would be great to define what are the free software communities
here. Distros and final users are also "free software communities" and
they do not care about niche use cases covered by proprietary
software.
They only care (and should care) about normal workflows.


>
> I can't fully answer that question, but there are two points that I
> think are relevant. Note that due to my background and experience, this
> will be heavily biased towards consumer and embedded hardware, not data
> centre-grade devices. Some technologies from the latter however have a
> tendency to migrate to the former over time, so the distinction isn't
> necessarily as relevant as one may consider.
>
> The first point is that hardware gets more complicated over time, and in
> some markets there's also an increase in the number of vendors and
> devices. There's a perceived (whether true or not) danger that we won't
> be able to keep up with just reverse engineering and a development model
> relying on hobyists. Getting vendors involved is important if we want to
> scale.

If we want vendors involved, we need to build an ecosystem where they
feel invited.

We should not take as hostages our users and impose rules on how they
should build or even sell their product.

>
> Second, I think there's a fear of regression. For some categories of
> devices, we have made slow but real progress to try and convince the
> industry to be more open. This sometimes took a decade of work,
> patiently building bridges and creating ecosystems brick by brick. Some
> of those ecosystems are sturdy, some not so. Giving pass-through a blank
> check will likely have very different effects in different areas. I
> don't personally believe it will shatter everything, but I'm convinced
> it carries risk in areas where cooperation with vendors is in its
> infancy or is fragile for any other reason.

We control what is accepted and what is not. We just need clear rules,
to avoid regressions like:
- For areas where there is a standard (NVME, UVC,...) most of the
drivers must be in-kernel, and use generic system calls.
- For areas with no standard, custom system calls are allowed, and
some part of the driver can be in userspace.
- To land a driver, there must be a full open source stack capable of
using it for standard use cases.
- If there is an established open source stack (mesa, openVINO,
libcamera...), the open source stack must be based on it.
- Vendor passthrough mechanisms are allowed for niche use cases or
development/experimentation.

I believe those rules are already in place in some subsystems. We just
have to agree what rules should apply to all the kernel by policy.

We can agree that this kind of discussion is done better face to face.

Regards!



>
> Finally, let's not forget that pass-through APIs are not an all or
> nothing option. To cite that example only, DRM requires GPU drivers to
> have an open-source userspace implementation to merge the kernel driver,
> and the same subsystems strongly pushes for API standardization for
> display controllers. We can set different rules for different cases.
>
> --
> Regards,
>
> Laurent Pinchart
>


--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

Hi Ricardo,

> It would be great to define what are the free software communities
> here. Distros and final users are also "free software communities" and
> they do not care about niche use cases covered by proprietary
> software.
> They only care (and should care) about normal workflows.

Don't you think there are distros and final users who do care about
proprietary influences, in general. For sure, though, I think they
should not be told what they should care about?

> If we want vendors involved, we need to build an ecosystem where they
> feel invited.

Definitely. Invited as in "you are welcome to work with us on this",
though. Not as in "come here and do what you want". This is why we have
a coding-style etc...

> We should not take as hostages our users and impose rules on how they
> should build or even sell their product.

Right, it should be a plain buisness decision. I totally agree with your
"clear rules" request below. And yeah, face to face is probably best
suited.

> - Vendor passthrough mechanisms are allowed for niche use cases or
> development/experimentation.

Problem with "niche" is that it can grow really big. See Linux :)

All the best,

   Wolfram


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:

<...>

> 
> It would be great to define what are the free software communities
> here. Distros and final users are also "free software communities" and
> they do not care about niche use cases covered by proprietary
> software.

Are you certain about that?

> They only care (and should care) about normal workflows.

What is a normal workflow?
Does it mean that if user bought something very expensive he
should not be able to use it with free software, because his
usage is different from yours?

Thanks

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com> wrote:
>
> <...>
>
> >
> > It would be great to define what are the free software communities
> > here. Distros and final users are also "free software communities" and
> > they do not care about niche use cases covered by proprietary
> > software.
>
> Are you certain about that?

As a user, and as an open source Distro developer I have a small hint.
But you could also ask users what they think about not being able to
use their notebook's cameras. The last time that I could not use some
basic hardware from a notebook with Linux was 20 years ago.

>
> > They only care (and should care) about normal workflows.
>
> What is a normal workflow?
> Does it mean that if user bought something very expensive he
> should not be able to use it with free software, because his
> usage is different from yours?
>
> Thanks

It means that we should not block the standard usage for 99% of the
population just because 1% of the users cannot do something fancy with
their device.

Let me give you an example. When I buy a camera I want to be able to
do Video Conferencing and take some static photos of documents. I do
not care about: automatic makeup, AI generated background, unicorn
filters, eyes recentering... But we need to give a way to vendors to
implement those things closely, without the marketing differentiators,
vendors have zero incentive to invest in Linux, and that affects all
the population.

This challenge seems to be solved for GPUs. I am using my AMD GPU
freely and my nephew can install the amdgpu-pro proprietary user space
driver to play duke nukem (or whatever kids play now) at 2000 fps.

There are other other subsystems that allow vendor passthrough and
their ecosystem has not collapsed.

Can we have some general guidance of what is acceptable? Can we define
together the "normal workflow" and focus on a *full* open source
implementation of that?

-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart
> > > <laurent.pinchart@ideasonboard.com> wrote:
> >
> > <...>
> >
> > >
> > > It would be great to define what are the free software communities
> > > here. Distros and final users are also "free software communities" and
> > > they do not care about niche use cases covered by proprietary
> > > software.
> >
> > Are you certain about that?
> 
> As a user, and as an open source Distro developer I have a small hint.
> But you could also ask users what they think about not being able to
> use their notebook's cameras. The last time that I could not use some
> basic hardware from a notebook with Linux was 20 years ago.

Lucky you, I still have consumer hardware (speaker) that doesn't work
with Linux, and even now, there is basic hardware in my current
laptop (HP docking station) that doesn't work reliably in Linux.

> 
> >
> > > They only care (and should care) about normal workflows.
> >
> > What is a normal workflow?
> > Does it mean that if user bought something very expensive he
> > should not be able to use it with free software, because his
> > usage is different from yours?
> >
> > Thanks
> 
> It means that we should not block the standard usage for 99% of the
> population just because 1% of the users cannot do something fancy with
> their device.

Right, the problem is that in some areas the statistics slightly different.
99% population is blocked because 1% of the users don't need it and
don't think that it is "normal" flow.

> 
> Let me give you an example. When I buy a camera I want to be able to
> do Video Conferencing and take some static photos of documents. I do
> not care about: automatic makeup, AI generated background, unicorn
> filters, eyes recentering... But we need to give a way to vendors to
> implement those things closely, without the marketing differentiators,
> vendors have zero incentive to invest in Linux, and that affects all
> the population.
> 
> This challenge seems to be solved for GPUs. I am using my AMD GPU
> freely and my nephew can install the amdgpu-pro proprietary user space
> driver to play duke nukem (or whatever kids play now) at 2000 fps.
> 
> There are other other subsystems that allow vendor passthrough and
> their ecosystem has not collapsed.

Yes, I completely agree with you on that.

> 
> Can we have some general guidance of what is acceptable? Can we define
> together the "normal workflow" and focus on a *full* open source
> implementation of that?

I don't think that is possible to define "normal workflow". Requirement
to have open-source counterpart to everything exposed through UAPI is a
valid one. I'm all for that.

Thanks

> 
> -- 
> Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:

> > As a user, and as an open source Distro developer I have a small hint.
> > But you could also ask users what they think about not being able to
> > use their notebook's cameras. The last time that I could not use some
> > basic hardware from a notebook with Linux was 20 years ago.

> Lucky you, I still have consumer hardware (speaker) that doesn't work
> with Linux, and even now, there is basic hardware in my current
> laptop (HP docking station) that doesn't work reliably in Linux.

FWIW for most audio issues (especially built in stuff) with laptops if
you report it upstream it'll generally be relatively easy to quirk.
Unfortunately it's idiomatic for ACPI systems to quirk off DMI
information for almost everything which means a constant stream of per
system quirks for subsystems like audio.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Steven Rostedt]

On Thu, 25 Jul 2024 15:02:13 +0200
Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com> wrote:

> As a user, and as an open source Distro developer I have a small hint.
> But you could also ask users what they think about not being able to
> use their notebook's cameras. The last time that I could not use some
> basic hardware from a notebook with Linux was 20 years ago.

FYI, I love my Dell XPS 13. But it's old. The battery died once and I had
to replace it. I would love to buy a new one, but the new one's webcam is
not supported by Linux. I'm just waiting for when this will be resolved. :-p

-- Steve

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:
> On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> 
> > > As a user, and as an open source Distro developer I have a small hint.
> > > But you could also ask users what they think about not being able to
> > > use their notebook's cameras. The last time that I could not use some
> > > basic hardware from a notebook with Linux was 20 years ago.
> 
> > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > with Linux, and even now, there is basic hardware in my current
> > laptop (HP docking station) that doesn't work reliably in Linux.
> 
> FWIW for most audio issues (especially built in stuff) with laptops if
> you report it upstream it'll generally be relatively easy to quirk.
> Unfortunately it's idiomatic for ACPI systems to quirk off DMI
> information for almost everything which means a constant stream of per
> system quirks for subsystems like audio.

It is Jabra USB speaker. One day, I will have time to debug it :).

Thanks

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Thu, 2024-07-25 at 17:18 +0300, Leon Romanovsky wrote:
> On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:
> > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado
> > > wrote:
> > 
> > > > As a user, and as an open source Distro developer I have a
> > > > small hint.  But you could also ask users what they think about
> > > > not being able to use their notebook's cameras. The last time
> > > > that I could not use some basic hardware from a notebook with
> > > > Linux was 20 years ago.
> > 
> > > Lucky you, I still have consumer hardware (speaker) that doesn't
> > > work with Linux, and even now, there is basic hardware in my
> > > current laptop (HP docking station) that doesn't work reliably in
> > > Linux.
> > 
> > FWIW for most audio issues (especially built in stuff) with laptops
> > if you report it upstream it'll generally be relatively easy to
> > quirk. Unfortunately it's idiomatic for ACPI systems to quirk off
> > DMI information for almost everything which means a constant stream
> > of per system quirks for subsystems like audio.
> 
> It is Jabra USB speaker. One day, I will have time to debug it :).

Those actually do work with Linux.  It's the speakers Plumbers has
provided to BoF/Hack sessions that needed remote attendees.  There was
a pulseaudio issue a while ago that any Mic/Speaker combination that
exported HFP instead of HSP didn't work, but that's fixed upstream (but
you might need to upgrade your pulseaudio).

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Leon Romanovsky]

On Thu, Jul 25, 2024 at 10:22:48AM -0400, James Bottomley wrote:
> On Thu, 2024-07-25 at 17:18 +0300, Leon Romanovsky wrote:
> > On Thu, Jul 25, 2024 at 02:29:36PM +0100, Mark Brown wrote:
> > > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado
> > > > wrote:
> > > 
> > > > > As a user, and as an open source Distro developer I have a
> > > > > small hint.  But you could also ask users what they think about
> > > > > not being able to use their notebook's cameras. The last time
> > > > > that I could not use some basic hardware from a notebook with
> > > > > Linux was 20 years ago.
> > > 
> > > > Lucky you, I still have consumer hardware (speaker) that doesn't
> > > > work with Linux, and even now, there is basic hardware in my
> > > > current laptop (HP docking station) that doesn't work reliably in
> > > > Linux.
> > > 
> > > FWIW for most audio issues (especially built in stuff) with laptops
> > > if you report it upstream it'll generally be relatively easy to
> > > quirk. Unfortunately it's idiomatic for ACPI systems to quirk off
> > > DMI information for almost everything which means a constant stream
> > > of per system quirks for subsystems like audio.
> > 
> > It is Jabra USB speaker. One day, I will have time to debug it :).
> 
> Those actually do work with Linux.  It's the speakers Plumbers has
> provided to BoF/Hack sessions that needed remote attendees.  There was
> a pulseaudio issue a while ago that any Mic/Speaker combination that
> exported HFP instead of HSP didn't work, but that's fixed upstream (but
> you might need to upgrade your pulseaudio).

I don't want to hijack this thread, but I tried now my two Jabra
speakers (one is used and another is brand new which I got from IT)
and both don't work on my FC40 laptop.

Thanks

> 
> James
> 
> 

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> [...]
> > What I get from the discussions I've followed or partcipated in over
> > the years is that the main worry of free software communities is
> > being forced to use closed-source userspace components, whether that
> > would be to make the device usable at all, or to achieve decent level
> > of performance or full feature set. We've been through years of
> > mostly closed-source GPU support, of printer "windrivers", and quite
> > a few other horrors. The good news is that we've so far overcome lots
> > (most) of those challenges. Reverse engineering projects paid off,
> > and so did working hand-in-hand with industry actors in multiple ways
> > (both openly and behind the scenes). One could then legitimately ask
> > why we're still scared.
> 
> I don't think I am.  We're mostly fully capable of expounding at length
> on the business rationale for being open if the thing they're hiding
> isn't much of a differentiator anyway (or they're simply hiding it to
> try to retain some illusion of control), so we shouldn't have any fear
> of being able to make our case in language business people understand.
> 
> I also think this fear is partly a mindset problem on our part.  We
> came out of the real fight for openness and we do embrace things like a
> licence that forces open code (GPL) and symbols that discourage
> proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> FSF coolaid that if we don't stand over manufacturers every second and
> force them they'll slide back to their old proprietary ways.  However,
> if you look at the entirely permissive ecosystem that grew up after we
> did (openstack, docker, kubernetes, etc.) they don't have any such fear
> and yet they still have large amounts of uncompelled openness and give
> back.

I don't think those are necessarily relevant examples, as far as device
pass-through goes. Vendors have many times reverted to proprietary ways,
and they still do, at least in the areas of the kernel I'm most active
in. I've seen first hand a large SoC vendor very close to opening a
significant part of their camera stack and changing their mind at the
last minute when they heard they could possibly merge their code through
a different subsystem with a pass-through blank cheque.

I'm willing to believe it can be different in other areas, which may
partly explain why different subsystems and different developers have
different biases and have trouble understand each other's point of view.

> > I can't fully answer that question, but there are two points that I
> > think are relevant. Note that due to my background and experience,
> > this will be heavily biased towards consumer and embedded hardware,
> > not data centre-grade devices. Some technologies from the latter
> > however have a tendency to migrate to the former over time, so the
> > distinction isn't necessarily as relevant as one may consider.
> > 
> > The first point is that hardware gets more complicated over time, and
> > in some markets there's also an increase in the number of vendors and
> > devices. There's a perceived (whether true or not) danger that we
> > won't be able to keep up with just reverse engineering and a
> > development model relying on hobyists. Getting vendors involved is
> > important if we want to scale.
> 
> Yes, but there are lots of not very useful complex devices being
> produced every day that fail to capture market share.  Not having
> reverse engineered drivers for them is no real loss.  If a device does
> gain market share, it gains a huge pool of users some of whom become
> interested in reverse engineering, so I think market forces actually
> work in our favour: we get reverse engineering mostly where the devices
> are actually interesting and capture market share.  It's self scaling.

I can't agree with that, sorry. Not only is the difficulty to
reverse-engineer some classes of devices increasing, but saying that
only devices that make it to the top of the market share chart are worth
considering will leave many users on the side of the road.

> > Second, I think there's a fear of regression. For some categories of
> > devices, we have made slow but real progress to try and convince the
> > industry to be more open. This sometimes took a decade of work,
> > patiently building bridges and creating ecosystems brick by brick.
> > Some of those ecosystems are sturdy, some not so. Giving pass-through
> > a blank check will likely have very different effects in different
> > areas. I don't personally believe it will shatter everything, but I'm
> > convinced it carries risk in areas where cooperation with vendors is
> > in its infancy or is fragile for any other reason.
> 
> I also think we're on the rise in this space.  Since most cloud
> workloads are on Linux, there's huge market pressure on most "found in
> the cloud" devices (like accelerators and GPUs) to have an easy to
> consume Linux story.  Nvidia is a case in point.  When it only cared
> about fast games on some other OS, we get shafted with a proprietary
> graphics drivers.  Now it's under pressure to be the number one AI
> accelerator provider for the cloud it's suddenly wondering about open
> source drivers to make adoption easier.

I can't comment on Nvidia and their inference engines in particular. The
server market may be in a better position that the consumer and embedded
market, and if that's the case, I'm happy for the servers. That doesn't
solve the issues in other markets though.

> > Finally, let's not forget that pass-through APIs are not an all or
> > nothing option. To cite that example only, DRM requires GPU drivers
> > to have an open-source userspace implementation to merge the kernel
> > driver, and the same subsystems strongly pushes for API
> > standardization for display controllers. We can set different rules
> > for different cases.
> 
> I certainly think we can afford to experiment here, yes.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > >
> > > <...>
> > >
> > > > It would be great to define what are the free software communities
> > > > here. Distros and final users are also "free software communities" and
> > > > they do not care about niche use cases covered by proprietary
> > > > software.
> > >
> > > Are you certain about that?
> > 
> > As a user, and as an open source Distro developer I have a small hint.
> > But you could also ask users what they think about not being able to
> > use their notebook's cameras. The last time that I could not use some
> > basic hardware from a notebook with Linux was 20 years ago.
> 
> Lucky you, I still have consumer hardware (speaker) that doesn't work
> with Linux, and even now, there is basic hardware in my current
> laptop (HP docking station) that doesn't work reliably in Linux.
> 
> > > > They only care (and should care) about normal workflows.
> > >
> > > What is a normal workflow?
> > > Does it mean that if user bought something very expensive he
> > > should not be able to use it with free software, because his
> > > usage is different from yours?
> > >
> > > Thanks
> > 
> > It means that we should not block the standard usage for 99% of the
> > population just because 1% of the users cannot do something fancy with
> > their device.
> 
> Right, the problem is that in some areas the statistics slightly different.
> 99% population is blocked because 1% of the users don't need it and
> don't think that it is "normal" flow.
> 
> > Let me give you an example. When I buy a camera I want to be able to
> > do Video Conferencing and take some static photos of documents. I do
> > not care about: automatic makeup, AI generated background, unicorn
> > filters, eyes recentering... But we need to give a way to vendors to
> > implement those things closely, without the marketing differentiators,
> > vendors have zero incentive to invest in Linux, and that affects all
> > the population.

I've seen these kind of examples being repeatedly given in discussions
related to camera ISP support in Linux. They are very misleading. These
are not the kind of features that are relevant for the device
pass-through discussion these day. Those are high-level use cases
implemented in userspace, and vendors can ship any closed-source
binaries they want there. What I care about is the features exposed by
the kernel to userspace API.

> > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > freely and my nephew can install the amdgpu-pro proprietary user space
> > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > 
> > There are other other subsystems that allow vendor passthrough and
> > their ecosystem has not collapsed.
> 
> Yes, I completely agree with you on that.
> 
> > Can we have some general guidance of what is acceptable? Can we define
> > together the "normal workflow" and focus on a *full* open source
> > implementation of that?
> 
> I don't think that is possible to define "normal workflow". Requirement
> to have open-source counterpart to everything exposed through UAPI is a
> valid one. I'm all for that.

That's my current opinion as well, as least when it comes to the kernel
areas I mostly work with.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:

> I don't think those are necessarily relevant examples, as far as device
> pass-through goes. Vendors have many times reverted to proprietary ways,
> and they still do, at least in the areas of the kernel I'm most active
> in. I've seen first hand a large SoC vendor very close to opening a
> significant part of their camera stack and changing their mind at the
> last minute when they heard they could possibly merge their code through
> a different subsystem with a pass-through blank cheque.

If someone came with a fully open source framework for (say) some
camera, with a passthrough kernel driver design, would you reject it
soley because it is passthrough based and you are scared that
something else will use it to do something not open source?

I wouldn't agree with that position, I think denying users useful open
source solutions out of fear is not what Linux should be doing.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Ricardo,

On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > James Bottomley wrote:
> > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > >
> > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > positions?
> > > > >
> > > > > This thread has that discussion:
> > > > >
> > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > >
> > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > consideration of fwctl for CXL.
> > > > >
> > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > wrapper.
> > > > >
> > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > get device-specific commands past another subsystem's objection?
> > > >
> > > > My default answer would be to trust the maintainers of the relevant
> > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > should they know the technical implications best, they should also have
> > > > a good view of the whole vertical stack, and the implications of
> > > > pass-through for their ecosystem. This may result in a single NAK
> > > > overriding ACKs, but we could also try to find technical solutions when
> > > > we'll face such issues, to enforce different sets of rules for the
> > > > different functions of a device.
> > > >
> > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > firmware interface and ship userspace binary blobs with no way for free
> > > > software to use all the device's features. That's something we have been
> > > > fighting against for years, trying to convince vendors that they can
> > > > provide better and more open camera support without the world
> > > > collapsing, with increasing success recently. Saying amen to
> > > > pass-through in this case would be a huge step back that would hurt
> > > > users and the whole ecosystem in the short and long term.
> > >
> > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> >
> > I know we disagree on this topic :-) I'm sure we'll continue the
> > conversation, but I think the technical discussion likely belongs to a
> > different mail thread.
> >
> > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > graphics cards.
> > >
> > > - Lack of Standardization: There's no universal standard for ISPs;
> > > each vendor implements unique features and usage patterns. DRM
> > > addresses this through vendor-specific IOCTLs
> > >
> > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > implementation, not comprehensive documentation.
> >
> > This point isn't technical and is more on-topic for this mail thread.
> >
> > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > text form. An open-source userspace implementation that covers the
> > feature set exposed by the driver is acceptable in place of
> > documentation (provided, of course, that the userspace code wouldn't be
> > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > DRM drivers.
> 
> In DRM vendors typically define a custom IOCTL per driver to pass
> command buffers.
> Only the command buffer structure, and a mesa implementation using
> that command buffer to support the standard features is required.
> 
> In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> be passed from userspace, they are typically formed in the driver from
> a strictly checked struct.

V4L2 has a mechanism to pass buffers between userspace and kernelspace,
and that mechanism is used in mainline drivers to pass camera ISP
parameters. They're not called "command buffers" but that's just a
difference in terminology. The technical means to pass command buffers
to the driver is thus there, I see no meaningful difference with DRM.
Where things can differ is in the contents of those buffers, and the
requirements for documentation or open userspace implementations, but
that's not a technical question.

> > > Our current approach of pushing back against vendors, instead of
> > > seeking compromise, has resulted in the vast majority of the market
> > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > with no options for utilizing their cameras outside of Android.
> > >
> > > DRM allows a hybrid model, where:
> > > - Open Source Foundation: Standard use cases are covered by a fully
> > > open-source stack.
> > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > proprietary features (e.g., automatic makeup) as closed source.
> >
> > V4L2 does as well, you can implement all kind of closed-source ISP
> > control algorithms in userspace, as long as there's an open-source
> > implementation that exercises the same hardware features. A good analogy
> 
> Is it really mandatory to have an open-source 3A algorithm? I thought
> defining the input and output from the algorithm was good enough.

What really matters is documenting the ISP parameters with enough
details to allow for the implementation of open-source userspace code.
Once you have that, 3A is quite simple. You can refine it (especially
AWB) to great length, for instance using NPUs to compute parameters, and
there's absolutely no issue with such userspace implementations being
closed.

In practice, some vendors prefer documenting the parameters by writing
an open-source userspace implementation, partly maybe because developers
are more familiar writing code than formal documentation. I would be
fine either way, as long as there's enough information to make use of
the ISP.

> AFAIK for some time there was no ipu3 open source algorithm, and the
> driver has been upstream.

It sneaked in before we realized we had to enforce rules :-) That's
actually a good example, when we wrote open-source userspace support, we
realized that the level of documentation included in the IPU3 kernel
header was nowhere close to what was needed to make use of the device.

> > for people less familiar with ISPs is shader compilers, GPU vendors are
> > free to ship closed-source implementations that include more
> > optimizations, as long as the open-source, less optimized implementation
> > covers the same GPU ISA, so that open-source developers can also work on
> > optimizing it.
> 
> I believe a more accurate description is that in v4l2 is that we
> expect that all the registers, device architecture and behaviour to be
> documented and accessed with standard IOCTLs. Anything not documented
> cannot be accessed by userspace.
> 
> In DRM their concern is that there is a fully open source
> implementation that the user can use. Vendors have custom IOCTLs and
> they can offer proprietary software for some use cases.

Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
vs. standard ioctls isn't very relevant to this discussion. I really
don't see how this would be about ioctls, it's about making the featured
exposed by the drivers, through whatever means a particular subsystem
allows, usable by open userspace.

> > Thinking that DRM would offer a free pass-through path compared to V4L2
> > doesn't seem realistic to me. Both subsystems will have similar rules.
> 
> DRM does indeed allow vendors to pass random command buffers and they
> will be sent to the hardware. We cannot do that in v4l2.

You can pass a command buffer to a V4L2 device and have the driver send
it to the device firmware (ISPs using real command buffers usually run a
firmware). If you want the driver to be merged upstream, you have to
document the command buffer in enough details.

> I might be wrong, but GPU drivers do not deeply inspect the command
> buffers to make sure that they do not use any feature not covered by
> mesa.

That's correct, but I don't think that's relevant. The GPU market has
GLSL and Vulkan. An open-source compliant implementation will end up
exercising a very very large part of the device ISA, command submission
mechanism and synchronization primitives, if not all of it. There's
little a vendor would keep under the hood and use in closed-source
userspace only. For cameras, there's no standard userspace API that
covers by design a very large part of what is the ISP equivalent of a
GPU ISA. Even "command buffers" are not a proper description of the
parameters the vast majority of ISPs consume.

> > > This approach would allow billions of users to access their hardware
> > > more securely and with in-tree driver support. Our current stubborn
> > > pursuit of an idealistic goal has already negatively impacted both
> > > users and the ecosystem.
> > >
> > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > Linux will remain a niche market for ISPs.
> > >
> > > If such a hybrid model goes against Linux goals, this is something
> > > that should be agreed upon by the whole community, so we have the same
> > > criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Jason,

On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:
> On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:
> 
> > I don't think those are necessarily relevant examples, as far as device
> > pass-through goes. Vendors have many times reverted to proprietary ways,
> > and they still do, at least in the areas of the kernel I'm most active
> > in. I've seen first hand a large SoC vendor very close to opening a
> > significant part of their camera stack and changing their mind at the
> > last minute when they heard they could possibly merge their code through
> > a different subsystem with a pass-through blank cheque.
> 
> If someone came with a fully open source framework for (say) some
> camera,

We have such a framework, it's called libcamera :-) Multiple vendors are
already collaborating.

> with a passthrough kernel driver design, would you reject it
> soley because it is passthrough based and you are scared that
> something else will use it to do something not open source?

It depends what "passthrough kernel driver design" means. If it means
accessing the PCI registers directly from userspace, yes. That's what X
used to do before KMS, and I'm glad it's now a distant past.

If it means a kernel driver that takes the majority of its runtime
parameters from a buffer blob assembled by userspace, while controlling
clocks, power domains and performing basic validation in kernelspace,
then I've already acked multiple drivers with such a design, exactly
because they have open-source userspace that doesn't try to keep many
device features proprietary and usable by closed-source userspace only.

> I wouldn't agree with that position, I think denying users useful open
> source solutions out of fear is not what Linux should be doing.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Thu, Jul 25, 2024 at 11:07:03PM +0300, Laurent Pinchart wrote:

> If it means a kernel driver that takes the majority of its runtime
> parameters from a buffer blob assembled by userspace, while controlling
> clocks, power domains and performing basic validation in kernelspace,
> then I've already acked multiple drivers with such a design, exactly
> because they have open-source userspace that doesn't try to keep many
> device features proprietary and usable by closed-source userspace only.

This also pretty much matches where we are with RDMA as well. Lots and
lots of stuff in userspace, but lots and lots is open. We've been able
to keep open kernel and userspace drivers quite well.

Thanks,
Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

Hi Laurent

On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > >
> > > > <...>
> > > >
> > > > > It would be great to define what are the free software communities
> > > > > here. Distros and final users are also "free software communities" and
> > > > > they do not care about niche use cases covered by proprietary
> > > > > software.
> > > >
> > > > Are you certain about that?
> > >
> > > As a user, and as an open source Distro developer I have a small hint.
> > > But you could also ask users what they think about not being able to
> > > use their notebook's cameras. The last time that I could not use some
> > > basic hardware from a notebook with Linux was 20 years ago.
> >
> > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > with Linux, and even now, there is basic hardware in my current
> > laptop (HP docking station) that doesn't work reliably in Linux.
> >
> > > > > They only care (and should care) about normal workflows.
> > > >
> > > > What is a normal workflow?
> > > > Does it mean that if user bought something very expensive he
> > > > should not be able to use it with free software, because his
> > > > usage is different from yours?
> > > >
> > > > Thanks
> > >
> > > It means that we should not block the standard usage for 99% of the
> > > population just because 1% of the users cannot do something fancy with
> > > their device.
> >
> > Right, the problem is that in some areas the statistics slightly different.
> > 99% population is blocked because 1% of the users don't need it and
> > don't think that it is "normal" flow.
> >
> > > Let me give you an example. When I buy a camera I want to be able to
> > > do Video Conferencing and take some static photos of documents. I do
> > > not care about: automatic makeup, AI generated background, unicorn
> > > filters, eyes recentering... But we need to give a way to vendors to
> > > implement those things closely, without the marketing differentiators,
> > > vendors have zero incentive to invest in Linux, and that affects all
> > > the population.
>
> I've seen these kind of examples being repeatedly given in discussions
> related to camera ISP support in Linux. They are very misleading. These
> are not the kind of features that are relevant for the device
> pass-through discussion these day. Those are high-level use cases
> implemented in userspace, and vendors can ship any closed-source
> binaries they want there. What I care about is the features exposed by
> the kernel to userspace API.

The ISPs are gradually becoming programmable devices and they indeed
help during all of those examples.

Userspace needs to send/receive information from the ISP, and that is
exactly what vendors want to keep in the close.

Describing how they implement those algorithms is a patent minefield
and their differentiating factor.

>
> > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > >
> > > There are other other subsystems that allow vendor passthrough and
> > > their ecosystem has not collapsed.
> >
> > Yes, I completely agree with you on that.
> >
> > > Can we have some general guidance of what is acceptable? Can we define
> > > together the "normal workflow" and focus on a *full* open source
> > > implementation of that?
> >
> > I don't think that is possible to define "normal workflow". Requirement
> > to have open-source counterpart to everything exposed through UAPI is a
> > valid one. I'm all for that.
>
> That's my current opinion as well, as least when it comes to the kernel
> areas I mostly work with.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

Hi Laurent

On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > [...]
> > > What I get from the discussions I've followed or partcipated in over
> > > the years is that the main worry of free software communities is
> > > being forced to use closed-source userspace components, whether that
> > > would be to make the device usable at all, or to achieve decent level
> > > of performance or full feature set. We've been through years of
> > > mostly closed-source GPU support, of printer "windrivers", and quite
> > > a few other horrors. The good news is that we've so far overcome lots
> > > (most) of those challenges. Reverse engineering projects paid off,
> > > and so did working hand-in-hand with industry actors in multiple ways
> > > (both openly and behind the scenes). One could then legitimately ask
> > > why we're still scared.
> >
> > I don't think I am.  We're mostly fully capable of expounding at length
> > on the business rationale for being open if the thing they're hiding
> > isn't much of a differentiator anyway (or they're simply hiding it to
> > try to retain some illusion of control), so we shouldn't have any fear
> > of being able to make our case in language business people understand.
> >
> > I also think this fear is partly a mindset problem on our part.  We
> > came out of the real fight for openness and we do embrace things like a
> > licence that forces open code (GPL) and symbols that discourage
> > proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> > FSF coolaid that if we don't stand over manufacturers every second and
> > force them they'll slide back to their old proprietary ways.  However,
> > if you look at the entirely permissive ecosystem that grew up after we
> > did (openstack, docker, kubernetes, etc.) they don't have any such fear
> > and yet they still have large amounts of uncompelled openness and give
> > back.
>
> I don't think those are necessarily relevant examples, as far as device
> pass-through goes. Vendors have many times reverted to proprietary ways,
> and they still do, at least in the areas of the kernel I'm most active
> in. I've seen first hand a large SoC vendor very close to opening a
> significant part of their camera stack and changing their mind at the
> last minute when they heard they could possibly merge their code through
> a different subsystem with a pass-through blank cheque.

Without knowing who that large SoC vendor is, and what they will be
willing to open, it is difficult to know what opportunity has been
lost. I would argue that if they have cancelled their open plans based
on an hypothesis, their willingness to open was not that high.

It would be more healthy for the ecosystem, if those discussions were
done more openly, at least all the core maintainers should be
involved.

>
> I'm willing to believe it can be different in other areas, which may
> partly explain why different subsystems and different developers have
> different biases and have trouble understand each other's point of view.

It is not different in other areas, it is the same area. At the end of
the day it is the same chip manufacturers, with the same legal teams.
It is our attitude that is different.


>
> > > I can't fully answer that question, but there are two points that I
> > > think are relevant. Note that due to my background and experience,
> > > this will be heavily biased towards consumer and embedded hardware,
> > > not data centre-grade devices. Some technologies from the latter
> > > however have a tendency to migrate to the former over time, so the
> > > distinction isn't necessarily as relevant as one may consider.
> > >
> > > The first point is that hardware gets more complicated over time, and
> > > in some markets there's also an increase in the number of vendors and
> > > devices. There's a perceived (whether true or not) danger that we
> > > won't be able to keep up with just reverse engineering and a
> > > development model relying on hobyists. Getting vendors involved is
> > > important if we want to scale.
> >
> > Yes, but there are lots of not very useful complex devices being
> > produced every day that fail to capture market share.  Not having
> > reverse engineered drivers for them is no real loss.  If a device does
> > gain market share, it gains a huge pool of users some of whom become
> > interested in reverse engineering, so I think market forces actually
> > work in our favour: we get reverse engineering mostly where the devices
> > are actually interesting and capture market share.  It's self scaling.
>
> I can't agree with that, sorry. Not only is the difficulty to
> reverse-engineer some classes of devices increasing, but saying that
> only devices that make it to the top of the market share chart are worth
> considering will leave many users on the side of the road.

Today we have left BILLIONS of users at the other side of the road.


>
> > > Second, I think there's a fear of regression. For some categories of
> > > devices, we have made slow but real progress to try and convince the
> > > industry to be more open. This sometimes took a decade of work,
> > > patiently building bridges and creating ecosystems brick by brick.
> > > Some of those ecosystems are sturdy, some not so. Giving pass-through
> > > a blank check will likely have very different effects in different
> > > areas. I don't personally believe it will shatter everything, but I'm
> > > convinced it carries risk in areas where cooperation with vendors is
> > > in its infancy or is fragile for any other reason.
> >
> > I also think we're on the rise in this space.  Since most cloud
> > workloads are on Linux, there's huge market pressure on most "found in
> > the cloud" devices (like accelerators and GPUs) to have an easy to
> > consume Linux story.  Nvidia is a case in point.  When it only cared
> > about fast games on some other OS, we get shafted with a proprietary
> > graphics drivers.  Now it's under pressure to be the number one AI
> > accelerator provider for the cloud it's suddenly wondering about open
> > source drivers to make adoption easier.
>
> I can't comment on Nvidia and their inference engines in particular. The
> server market may be in a better position that the consumer and embedded
> market, and if that's the case, I'm happy for the servers. That doesn't
> solve the issues in other markets though.
>
> > > Finally, let's not forget that pass-through APIs are not an all or
> > > nothing option. To cite that example only, DRM requires GPU drivers
> > > to have an open-source userspace implementation to merge the kernel
> > > driver, and the same subsystems strongly pushes for API
> > > standardization for display controllers. We can set different rules
> > > for different cases.
> >
> > I certainly think we can afford to experiment here, yes.
>
> --
> Regards,
>
> Laurent Pinchart
>


--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Ricardo,
>
> On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > James Bottomley wrote:
> > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > >
> > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > positions?
> > > > > >
> > > > > > This thread has that discussion:
> > > > > >
> > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > >
> > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > consideration of fwctl for CXL.
> > > > > >
> > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > wrapper.
> > > > > >
> > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > get device-specific commands past another subsystem's objection?
> > > > >
> > > > > My default answer would be to trust the maintainers of the relevant
> > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > should they know the technical implications best, they should also have
> > > > > a good view of the whole vertical stack, and the implications of
> > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > different functions of a device.
> > > > >
> > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > software to use all the device's features. That's something we have been
> > > > > fighting against for years, trying to convince vendors that they can
> > > > > provide better and more open camera support without the world
> > > > > collapsing, with increasing success recently. Saying amen to
> > > > > pass-through in this case would be a huge step back that would hurt
> > > > > users and the whole ecosystem in the short and long term.
> > > >
> > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > >
> > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > conversation, but I think the technical discussion likely belongs to a
> > > different mail thread.
> > >
> > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > graphics cards.
> > > >
> > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > each vendor implements unique features and usage patterns. DRM
> > > > addresses this through vendor-specific IOCTLs
> > > >
> > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > implementation, not comprehensive documentation.
> > >
> > > This point isn't technical and is more on-topic for this mail thread.
> > >
> > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > text form. An open-source userspace implementation that covers the
> > > feature set exposed by the driver is acceptable in place of
> > > documentation (provided, of course, that the userspace code wouldn't be
> > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > DRM drivers.
> >
> > In DRM vendors typically define a custom IOCTL per driver to pass
> > command buffers.
> > Only the command buffer structure, and a mesa implementation using
> > that command buffer to support the standard features is required.
> >
> > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > be passed from userspace, they are typically formed in the driver from
> > a strictly checked struct.
>
> V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> and that mechanism is used in mainline drivers to pass camera ISP
> parameters. They're not called "command buffers" but that's just a
> difference in terminology. The technical means to pass command buffers
> to the driver is thus there, I see no meaningful difference with DRM.
> Where things can differ is in the contents of those buffers, and the
> requirements for documentation or open userspace implementations, but
> that's not a technical question.

There are two things here:

- The political/strategic/philosophical/religious aspect: The industry
definitely prefers the strategic requirements imposed by DRM. In fact
some vendors had some huge legal troubles when they had tried to
follow v4l2 requirements.
- The technical aspect: DRM is more mature when it comes to
sending/receiving buffers to the hardware, and an ISP looks *much*
more similar to an accel device or a GPU than a UVC camera.


>
> > > > Our current approach of pushing back against vendors, instead of
> > > > seeking compromise, has resulted in the vast majority of the market
> > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > with no options for utilizing their cameras outside of Android.
> > > >
> > > > DRM allows a hybrid model, where:
> > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > open-source stack.
> > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > proprietary features (e.g., automatic makeup) as closed source.
> > >
> > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > control algorithms in userspace, as long as there's an open-source
> > > implementation that exercises the same hardware features. A good analogy
> >
> > Is it really mandatory to have an open-source 3A algorithm? I thought
> > defining the input and output from the algorithm was good enough.
>
> What really matters is documenting the ISP parameters with enough
> details to allow for the implementation of open-source userspace code.
> Once you have that, 3A is quite simple. You can refine it (especially
> AWB) to great length, for instance using NPUs to compute parameters, and
> there's absolutely no issue with such userspace implementations being
> closed.
>
> In practice, some vendors prefer documenting the parameters by writing
> an open-source userspace implementation, partly maybe because developers
> are more familiar writing code than formal documentation. I would be
> fine either way, as long as there's enough information to make use of
> the ISP.
>

Even with vendor passthrough there is still a need to provide a full
open source implementation (probably based on libcamera).

So you will have enough information to use all the common
functionality of a camera.

> > AFAIK for some time there was no ipu3 open source algorithm, and the
> > driver has been upstream.
>
> It sneaked in before we realized we had to enforce rules :-) That's
> actually a good example, when we wrote open-source userspace support, we
> realized that the level of documentation included in the IPU3 kernel
> header was nowhere close to what was needed to make use of the device.
>
> > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > free to ship closed-source implementations that include more
> > > optimizations, as long as the open-source, less optimized implementation
> > > covers the same GPU ISA, so that open-source developers can also work on
> > > optimizing it.
> >
> > I believe a more accurate description is that in v4l2 is that we
> > expect that all the registers, device architecture and behaviour to be
> > documented and accessed with standard IOCTLs. Anything not documented
> > cannot be accessed by userspace.
> >
> > In DRM their concern is that there is a fully open source
> > implementation that the user can use. Vendors have custom IOCTLs and
> > they can offer proprietary software for some use cases.
>
> Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> vs. standard ioctls isn't very relevant to this discussion. I really
> don't see how this would be about ioctls, it's about making the featured
> exposed by the drivers, through whatever means a particular subsystem
> allows, usable by open userspace.
>
> > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > doesn't seem realistic to me. Both subsystems will have similar rules.
> >
> > DRM does indeed allow vendors to pass random command buffers and they
> > will be sent to the hardware. We cannot do that in v4l2.
>
> You can pass a command buffer to a V4L2 device and have the driver send
> it to the device firmware (ISPs using real command buffers usually run a
> firmware). If you want the driver to be merged upstream, you have to
> document the command buffer in enough details.

Documenting with "enough details" is not enough. In V4L2, we have to
deeply inspect every single buffer to make sure that it is not sending
an unknown combination of command+arguments, or in other situations we
construct the command buffer in the driver.

>
> > I might be wrong, but GPU drivers do not deeply inspect the command
> > buffers to make sure that they do not use any feature not covered by
> > mesa.
>
> That's correct, but I don't think that's relevant. The GPU market has
> GLSL and Vulkan. An open-source compliant implementation will end up
> exercising a very very large part of the device ISA, command submission
> mechanism and synchronization primitives, if not all of it. There's
> little a vendor would keep under the hood and use in closed-source
> userspace only. For cameras, there's no standard userspace API that
> covers by design a very large part of what is the ISP equivalent of a

Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...

If we are afraid of vendors providing "toy" implemententations to pass
the openness requirements, we can add more features/tests to
libcamera.

And at the end of the day, there will be humans deciding if what a
vendor has provided is good enough or not.

> GPU ISA. Even "command buffers" are not a proper description of the
> parameters the vast majority of ISPs consume.

Modern ISPs are definitely going in the direction of "command buffers"


>
> > > > This approach would allow billions of users to access their hardware
> > > > more securely and with in-tree driver support. Our current stubborn
> > > > pursuit of an idealistic goal has already negatively impacted both
> > > > users and the ecosystem.
> > > >
> > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > Linux will remain a niche market for ISPs.
> > > >
> > > > If such a hybrid model goes against Linux goals, this is something
> > > > that should be agreed upon by the whole community, so we have the same
> > > > criteria for all subsystems.
>
> --
> Regards,
>
> Laurent Pinchart



--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

Hi Laurent

On Thu, Jul 25, 2024 at 10:07 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Jason,
>
> On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:
> > On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:
> >
> > > I don't think those are necessarily relevant examples, as far as device
> > > pass-through goes. Vendors have many times reverted to proprietary ways,
> > > and they still do, at least in the areas of the kernel I'm most active
> > > in. I've seen first hand a large SoC vendor very close to opening a
> > > significant part of their camera stack and changing their mind at the
> > > last minute when they heard they could possibly merge their code through
> > > a different subsystem with a pass-through blank cheque.
> >
> > If someone came with a fully open source framework for (say) some
> > camera,
>
> We have such a framework, it's called libcamera :-) Multiple vendors are
> already collaborating.
>
> > with a passthrough kernel driver design, would you reject it
> > soley because it is passthrough based and you are scared that
> > something else will use it to do something not open source?
>
> It depends what "passthrough kernel driver design" means. If it means
> accessing the PCI registers directly from userspace, yes. That's what X
> used to do before KMS, and I'm glad it's now a distant past.


Nobody has suggested giving PCI register access to userspace.


>
> If it means a kernel driver that takes the majority of its runtime
> parameters from a buffer blob assembled by userspace, while controlling
> clocks, power domains and performing basic validation in kernelspace,
> then I've already acked multiple drivers with such a design, exactly
> because they have open-source userspace that doesn't try to keep many
> device features proprietary and usable by closed-source userspace only.

If that was an option we would not be having this discussion.

Vendors cannot have vendor access in v4l2.
"""
It is not an option to upstream a driver that has support for
undocumented closed features. Basically maintainers can't put their
name on something that contains unverifiable (for them) and unusable
(by all except the vendor) features.
"""
https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil


>
> > I wouldn't agree with that position, I think denying users useful open
> > source solutions out of fear is not what Linux should be doing.
>
> --
> Regards,
>
> Laurent Pinchart
>


--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > James Bottomley wrote:
> > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > >
> > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > positions?
> > > > > > >
> > > > > > > This thread has that discussion:
> > > > > > >
> > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > >
> > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > consideration of fwctl for CXL.
> > > > > > >
> > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > wrapper.
> > > > > > >
> > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > get device-specific commands past another subsystem's objection?
> > > > > >
> > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > should they know the technical implications best, they should also have
> > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > different functions of a device.
> > > > > >
> > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > software to use all the device's features. That's something we have been
> > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > provide better and more open camera support without the world
> > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > users and the whole ecosystem in the short and long term.
> > > > >
> > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > >
> > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > conversation, but I think the technical discussion likely belongs to a
> > > > different mail thread.
> > > >
> > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > graphics cards.
> > > > >
> > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > each vendor implements unique features and usage patterns. DRM
> > > > > addresses this through vendor-specific IOCTLs
> > > > >
> > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > implementation, not comprehensive documentation.
> > > >
> > > > This point isn't technical and is more on-topic for this mail thread.
> > > >
> > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > text form. An open-source userspace implementation that covers the
> > > > feature set exposed by the driver is acceptable in place of
> > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > DRM drivers.
> > >
> > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > command buffers.
> > > Only the command buffer structure, and a mesa implementation using
> > > that command buffer to support the standard features is required.
> > >
> > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > be passed from userspace, they are typically formed in the driver from
> > > a strictly checked struct.
> >
> > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > and that mechanism is used in mainline drivers to pass camera ISP
> > parameters. They're not called "command buffers" but that's just a
> > difference in terminology. The technical means to pass command buffers
> > to the driver is thus there, I see no meaningful difference with DRM.
> > Where things can differ is in the contents of those buffers, and the
> > requirements for documentation or open userspace implementations, but
> > that's not a technical question.
> 
> There are two things here:
> 
> - The political/strategic/philosophical/religious aspect: The industry
> definitely prefers the strategic requirements imposed by DRM. In fact
> some vendors had some huge legal troubles when they had tried to
> follow v4l2 requirements.

That's I'm willing to debate.

> - The technical aspect: DRM is more mature when it comes to
> sending/receiving buffers to the hardware, and an ISP looks *much*
> more similar to an accel device or a GPU than a UVC camera.

But this I don't agree with. I think we should forgo the technical
discussion and stop pretending that DRM is better for this use case from
a technical point of view, and focus on the other aspect of the
discussion. (We can of course reopen the technical discussion if new
concrete arguments emerge.)

> > > > > Our current approach of pushing back against vendors, instead of
> > > > > seeking compromise, has resulted in the vast majority of the market
> > > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > > with no options for utilizing their cameras outside of Android.
> > > > >
> > > > > DRM allows a hybrid model, where:
> > > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > > open-source stack.
> > > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > > proprietary features (e.g., automatic makeup) as closed source.
> > > >
> > > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > > control algorithms in userspace, as long as there's an open-source
> > > > implementation that exercises the same hardware features. A good analogy
> > >
> > > Is it really mandatory to have an open-source 3A algorithm? I thought
> > > defining the input and output from the algorithm was good enough.
> >
> > What really matters is documenting the ISP parameters with enough
> > details to allow for the implementation of open-source userspace code.
> > Once you have that, 3A is quite simple. You can refine it (especially
> > AWB) to great length, for instance using NPUs to compute parameters, and
> > there's absolutely no issue with such userspace implementations being
> > closed.
> >
> > In practice, some vendors prefer documenting the parameters by writing
> > an open-source userspace implementation, partly maybe because developers
> > are more familiar writing code than formal documentation. I would be
> > fine either way, as long as there's enough information to make use of
> > the ISP.
> 
> Even with vendor passthrough there is still a need to provide a full
> open source implementation (probably based on libcamera).

We have a different interpretation of "full" :-) I want to aim for
"full" to cover all the features exposed by the driver UAPI. There could
be some exceptions that we can discuss if there are compeling arguments,
but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.

> So you will have enough information to use all the common
> functionality of a camera.
> 
> > > AFAIK for some time there was no ipu3 open source algorithm, and the
> > > driver has been upstream.
> >
> > It sneaked in before we realized we had to enforce rules :-) That's
> > actually a good example, when we wrote open-source userspace support, we
> > realized that the level of documentation included in the IPU3 kernel
> > header was nowhere close to what was needed to make use of the device.
> >
> > > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > > free to ship closed-source implementations that include more
> > > > optimizations, as long as the open-source, less optimized implementation
> > > > covers the same GPU ISA, so that open-source developers can also work on
> > > > optimizing it.
> > >
> > > I believe a more accurate description is that in v4l2 is that we
> > > expect that all the registers, device architecture and behaviour to be
> > > documented and accessed with standard IOCTLs. Anything not documented
> > > cannot be accessed by userspace.
> > >
> > > In DRM their concern is that there is a fully open source
> > > implementation that the user can use. Vendors have custom IOCTLs and
> > > they can offer proprietary software for some use cases.
> >
> > Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> > vs. standard ioctls isn't very relevant to this discussion. I really
> > don't see how this would be about ioctls, it's about making the featured
> > exposed by the drivers, through whatever means a particular subsystem
> > allows, usable by open userspace.
> >
> > > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > > doesn't seem realistic to me. Both subsystems will have similar rules.
> > >
> > > DRM does indeed allow vendors to pass random command buffers and they
> > > will be sent to the hardware. We cannot do that in v4l2.
> >
> > You can pass a command buffer to a V4L2 device and have the driver send
> > it to the device firmware (ISPs using real command buffers usually run a
> > firmware). If you want the driver to be merged upstream, you have to
> > document the command buffer in enough details.
> 
> Documenting with "enough details" is not enough. In V4L2, we have to
> deeply inspect every single buffer to make sure that it is not sending
> an unknown combination of command+arguments, or in other situations we
> construct the command buffer in the driver.
> 
> > > I might be wrong, but GPU drivers do not deeply inspect the command
> > > buffers to make sure that they do not use any feature not covered by
> > > mesa.
> >
> > That's correct, but I don't think that's relevant. The GPU market has
> > GLSL and Vulkan. An open-source compliant implementation will end up
> > exercising a very very large part of the device ISA, command submission
> > mechanism and synchronization primitives, if not all of it. There's
> > little a vendor would keep under the hood and use in closed-source
> > userspace only. For cameras, there's no standard userspace API that
> > covers by design a very large part of what is the ISP equivalent of a
> 
> Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...

Those are not comparable to GLSL or Vulkan, they are much higher level.

> If we are afraid of vendors providing "toy" implemententations to pass
> the openness requirements, we can add more features/tests to
> libcamera.
> 
> And at the end of the day, there will be humans deciding if what a
> vendor has provided is good enough or not.
> 
> > GPU ISA. Even "command buffers" are not a proper description of the
> > parameters the vast majority of ISPs consume.
> 
> Modern ISPs are definitely going in the direction of "command buffers"

Examples please, with technical details. I've seen this argument being
used for at least a year, with very little evidence. There are ISPs that
are more programmable than others, but in practice firmwares mostly
expose fixed functions, not ISA-level programmability.

> > > > > This approach would allow billions of users to access their hardware
> > > > > more securely and with in-tree driver support. Our current stubborn
> > > > > pursuit of an idealistic goal has already negatively impacted both
> > > > > users and the ecosystem.
> > > > >
> > > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > > Linux will remain a niche market for ISPs.
> > > > >
> > > > > If such a hybrid model goes against Linux goals, this is something
> > > > > that should be agreed upon by the whole community, so we have the same
> > > > > criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 10:04:48AM +0200, Ricardo Ribalda Delgado wrote:
> On Thu, Jul 25, 2024 at 10:07 PM Laurent Pinchart wrote:
> > On Thu, Jul 25, 2024 at 04:43:14PM -0300, Jason Gunthorpe wrote:
> > > On Thu, Jul 25, 2024 at 10:31:25PM +0300, Laurent Pinchart wrote:
> > >
> > > > I don't think those are necessarily relevant examples, as far as device
> > > > pass-through goes. Vendors have many times reverted to proprietary ways,
> > > > and they still do, at least in the areas of the kernel I'm most active
> > > > in. I've seen first hand a large SoC vendor very close to opening a
> > > > significant part of their camera stack and changing their mind at the
> > > > last minute when they heard they could possibly merge their code through
> > > > a different subsystem with a pass-through blank cheque.
> > >
> > > If someone came with a fully open source framework for (say) some
> > > camera,
> >
> > We have such a framework, it's called libcamera :-) Multiple vendors are
> > already collaborating.
> >
> > > with a passthrough kernel driver design, would you reject it
> > > soley because it is passthrough based and you are scared that
> > > something else will use it to do something not open source?
> >
> > It depends what "passthrough kernel driver design" means. If it means
> > accessing the PCI registers directly from userspace, yes. That's what X
> > used to do before KMS, and I'm glad it's now a distant past.
> 
> Nobody has suggested giving PCI register access to userspace.

I know you didn't, but as I didn't expect Jason to be familiar with the
camera ISP discussions, I didn't want to make any specific assumption
regarding what he meant by passthrough kernel driver design.

> > If it means a kernel driver that takes the majority of its runtime
> > parameters from a buffer blob assembled by userspace, while controlling
> > clocks, power domains and performing basic validation in kernelspace,
> > then I've already acked multiple drivers with such a design, exactly
> > because they have open-source userspace that doesn't try to keep many
> > device features proprietary and usable by closed-source userspace only.
> 
> If that was an option we would not be having this discussion.
> 
> Vendors cannot have vendor access in v4l2.
> """
> It is not an option to upstream a driver that has support for
> undocumented closed features. Basically maintainers can't put their
> name on something that contains unverifiable (for them) and unusable
> (by all except the vendor) features.
> """
> https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

What is not an option exactly in my description above ? We have multiple
V4L2 drivers for ISPs. They receive ISP parameters from userspace
through a data buffer. It's not allowed to be opaque, but it doesn't
prevent vendor closed-source userspace implementations with additional
*camera* features, as long as the *hardware* features are available to
everybody.

> > > I wouldn't agree with that position, I think denying users useful open
> > > source solutions out of fear is not what Linux should be doing.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:

> What is not an option exactly in my description above ? We have multiple
> V4L2 drivers for ISPs. They receive ISP parameters from userspace
> through a data buffer. It's not allowed to be opaque, but it doesn't
> prevent vendor closed-source userspace implementations with additional
> *camera* features, as long as the *hardware* features are available to
> everybody.

How far do you take opaque?

In mlx5 we pass command buffers from user to kernel to HW and the
kernel does only a little checking.

There is a 12kloc file describing the layout of alot of commands:
include/linux/mlx5/mlx5_ifc.h

There is an open PDF describing in detail some subset of this:
https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf

There are in-kernel implementations driving most of those commands.

Other commands are only issued by userspace, and we have open source
DPDK, rdma-core and UCX implementations driving them.

ie, this is really quite good as far as a device providing open source
solutions goes.

However, no doubt there is more FW capability and commands than even
this vast amount documents - so lets guess that propritary code is
using this interface with unknown commands too.

From a camera perspective would you be OK with that? Let's guess that
90% of use cases are covered with fully open code. Do we also need to
forcefully close the door to an imagined 10% of proprietary cases just
to be sure open always wins?

Does closing the door have to come at the cost of a technically clean
solution? Doing validation in the kernel to enforce an ideological
position would severely degrade mlx5, and would probably never really
be robust.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:
> > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > >
> > > > > <...>
> > > > >
> > > > > > It would be great to define what are the free software communities
> > > > > > here. Distros and final users are also "free software communities" and
> > > > > > they do not care about niche use cases covered by proprietary
> > > > > > software.
> > > > >
> > > > > Are you certain about that?
> > > >
> > > > As a user, and as an open source Distro developer I have a small hint.
> > > > But you could also ask users what they think about not being able to
> > > > use their notebook's cameras. The last time that I could not use some
> > > > basic hardware from a notebook with Linux was 20 years ago.
> > >
> > > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > > with Linux, and even now, there is basic hardware in my current
> > > laptop (HP docking station) that doesn't work reliably in Linux.
> > >
> > > > > > They only care (and should care) about normal workflows.
> > > > >
> > > > > What is a normal workflow?
> > > > > Does it mean that if user bought something very expensive he
> > > > > should not be able to use it with free software, because his
> > > > > usage is different from yours?
> > > > >
> > > > > Thanks
> > > >
> > > > It means that we should not block the standard usage for 99% of the
> > > > population just because 1% of the users cannot do something fancy with
> > > > their device.
> > >
> > > Right, the problem is that in some areas the statistics slightly different.
> > > 99% population is blocked because 1% of the users don't need it and
> > > don't think that it is "normal" flow.
> > >
> > > > Let me give you an example. When I buy a camera I want to be able to
> > > > do Video Conferencing and take some static photos of documents. I do
> > > > not care about: automatic makeup, AI generated background, unicorn
> > > > filters, eyes recentering... But we need to give a way to vendors to
> > > > implement those things closely, without the marketing differentiators,
> > > > vendors have zero incentive to invest in Linux, and that affects all
> > > > the population.
> >
> > I've seen these kind of examples being repeatedly given in discussions
> > related to camera ISP support in Linux. They are very misleading. These
> > are not the kind of features that are relevant for the device
> > pass-through discussion these day. Those are high-level use cases
> > implemented in userspace, and vendors can ship any closed-source
> > binaries they want there. What I care about is the features exposed by
> > the kernel to userspace API.
> 
> The ISPs are gradually becoming programmable devices and they indeed
> help during all of those examples.

I'd like to see more technical information to substantiate this claim.
So far what I've sometimes seen is ISPs that include programmable
elements, but hiding those behind a firmware that exposes a fixed
(configurable) pipeline. I've also heard of attempts to expose some of
that programmability to the operating system, which were abandoned in
the end due to lack usefulness.

> Userspace needs to send/receive information from the ISP, and that is
> exactly what vendors want to keep in the close.

But that's exactly what we need to implement an open userspace ecosystem
:-)

> Describing how they implement those algorithms is a patent minefield
> and their differentiating factor.

Those are also arguments I've heard many times before. The
differentiating factor for cameras today is mostly in userspace ISP
control algorithms, and nobody is telling vendors they need to open all
that.

When it comes to patents, we all know how software patents is a
minefield, and hardware is also affected. I can't have much sympathy for
this argument though, those patents mostly benefit the largest players
in the market, and those are the ones who currently claim they can't
open anything due to patents.

> > > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > > >
> > > > There are other other subsystems that allow vendor passthrough and
> > > > their ecosystem has not collapsed.
> > >
> > > Yes, I completely agree with you on that.
> > >
> > > > Can we have some general guidance of what is acceptable? Can we define
> > > > together the "normal workflow" and focus on a *full* open source
> > > > implementation of that?
> > >
> > > I don't think that is possible to define "normal workflow". Requirement
> > > to have open-source counterpart to everything exposed through UAPI is a
> > > valid one. I'm all for that.
> >
> > That's my current opinion as well, as least when it comes to the kernel
> > areas I mostly work with.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 10:03:36AM +0200, Ricardo Ribalda Delgado wrote:
> On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart wrote:
> > On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > > [...]
> > > > What I get from the discussions I've followed or partcipated in over
> > > > the years is that the main worry of free software communities is
> > > > being forced to use closed-source userspace components, whether that
> > > > would be to make the device usable at all, or to achieve decent level
> > > > of performance or full feature set. We've been through years of
> > > > mostly closed-source GPU support, of printer "windrivers", and quite
> > > > a few other horrors. The good news is that we've so far overcome lots
> > > > (most) of those challenges. Reverse engineering projects paid off,
> > > > and so did working hand-in-hand with industry actors in multiple ways
> > > > (both openly and behind the scenes). One could then legitimately ask
> > > > why we're still scared.
> > >
> > > I don't think I am.  We're mostly fully capable of expounding at length
> > > on the business rationale for being open if the thing they're hiding
> > > isn't much of a differentiator anyway (or they're simply hiding it to
> > > try to retain some illusion of control), so we shouldn't have any fear
> > > of being able to make our case in language business people understand.
> > >
> > > I also think this fear is partly a mindset problem on our part.  We
> > > came out of the real fight for openness and we do embrace things like a
> > > licence that forces open code (GPL) and symbols that discourage
> > > proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> > > FSF coolaid that if we don't stand over manufacturers every second and
> > > force them they'll slide back to their old proprietary ways.  However,
> > > if you look at the entirely permissive ecosystem that grew up after we
> > > did (openstack, docker, kubernetes, etc.) they don't have any such fear
> > > and yet they still have large amounts of uncompelled openness and give
> > > back.
> >
> > I don't think those are necessarily relevant examples, as far as device
> > pass-through goes. Vendors have many times reverted to proprietary ways,
> > and they still do, at least in the areas of the kernel I'm most active
> > in. I've seen first hand a large SoC vendor very close to opening a
> > significant part of their camera stack and changing their mind at the
> > last minute when they heard they could possibly merge their code through
> > a different subsystem with a pass-through blank cheque.
> 
> Without knowing who that large SoC vendor is, and what they will be
> willing to open, it is difficult to know what opportunity has been
> lost.

I would also like to discuss this in more details, but I can't do so
publicly :-( That's why I don't want to focus too much on this argument.

> I would argue that if they have cancelled their open plans based
> on an hypothesis, their willingness to open was not that high.

That's exactly what I previously described in this mail thread as areas
where cooperation with vendors is in its infancy. Convincing large
industry players is long term work, and it's very demotivating when you
think there's finally light at the end of the tunel, and plans gets
cancelled at the last minute. We're not giving up though.

> It would be more healthy for the ecosystem, if those discussions were
> done more openly, at least all the core maintainers should be
> involved.

Can you get all the camera vendors you work with to talk about all this
openly ? :-) We all know it's a difficult and long process.

> > I'm willing to believe it can be different in other areas, which may
> > partly explain why different subsystems and different developers have
> > different biases and have trouble understand each other's point of view.
> 
> It is not different in other areas, it is the same area. At the end of
> the day it is the same chip manufacturers, with the same legal teams.
> It is our attitude that is different.

I don't agree with that. Not only are there different manufacturers,
there are also different teams, with different approaches. The world is
not uniform.

> > > > I can't fully answer that question, but there are two points that I
> > > > think are relevant. Note that due to my background and experience,
> > > > this will be heavily biased towards consumer and embedded hardware,
> > > > not data centre-grade devices. Some technologies from the latter
> > > > however have a tendency to migrate to the former over time, so the
> > > > distinction isn't necessarily as relevant as one may consider.
> > > >
> > > > The first point is that hardware gets more complicated over time, and
> > > > in some markets there's also an increase in the number of vendors and
> > > > devices. There's a perceived (whether true or not) danger that we
> > > > won't be able to keep up with just reverse engineering and a
> > > > development model relying on hobyists. Getting vendors involved is
> > > > important if we want to scale.
> > >
> > > Yes, but there are lots of not very useful complex devices being
> > > produced every day that fail to capture market share.  Not having
> > > reverse engineered drivers for them is no real loss.  If a device does
> > > gain market share, it gains a huge pool of users some of whom become
> > > interested in reverse engineering, so I think market forces actually
> > > work in our favour: we get reverse engineering mostly where the devices
> > > are actually interesting and capture market share.  It's self scaling.
> >
> > I can't agree with that, sorry. Not only is the difficulty to
> > reverse-engineer some classes of devices increasing, but saying that
> > only devices that make it to the top of the market share chart are worth
> > considering will leave many users on the side of the road.
> 
> Today we have left BILLIONS of users at the other side of the road.

Because we're the only ones to be blamed, or more accuretely, the blame
is solely on me. Vendors are all saints who want the best for the world.
This argument doesn't hold. Please stop with this, I don't want to get
to name calling.

> > > > Second, I think there's a fear of regression. For some categories of
> > > > devices, we have made slow but real progress to try and convince the
> > > > industry to be more open. This sometimes took a decade of work,
> > > > patiently building bridges and creating ecosystems brick by brick.
> > > > Some of those ecosystems are sturdy, some not so. Giving pass-through
> > > > a blank check will likely have very different effects in different
> > > > areas. I don't personally believe it will shatter everything, but I'm
> > > > convinced it carries risk in areas where cooperation with vendors is
> > > > in its infancy or is fragile for any other reason.
> > >
> > > I also think we're on the rise in this space.  Since most cloud
> > > workloads are on Linux, there's huge market pressure on most "found in
> > > the cloud" devices (like accelerators and GPUs) to have an easy to
> > > consume Linux story.  Nvidia is a case in point.  When it only cared
> > > about fast games on some other OS, we get shafted with a proprietary
> > > graphics drivers.  Now it's under pressure to be the number one AI
> > > accelerator provider for the cloud it's suddenly wondering about open
> > > source drivers to make adoption easier.
> >
> > I can't comment on Nvidia and their inference engines in particular. The
> > server market may be in a better position that the consumer and embedded
> > market, and if that's the case, I'm happy for the servers. That doesn't
> > solve the issues in other markets though.
> >
> > > > Finally, let's not forget that pass-through APIs are not an all or
> > > > nothing option. To cite that example only, DRM requires GPU drivers
> > > > to have an open-source userspace implementation to merge the kernel
> > > > driver, and the same subsystems strongly pushes for API
> > > > standardization for display controllers. We can set different rules
> > > > for different cases.
> > >
> > > I certainly think we can afford to experiment here, yes.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Thu, 2024-07-25 at 20:37 +0300, Leon Romanovsky wrote:
> I don't want to hijack this thread, but I tried now my two Jabra
> speakers (one is used and another is brand new which I got from IT)
> and both don't work on my FC40 laptop.

Bring it to Plumbers and we'll try it on a variety of systems ... just
in case it's a fluke that the Jabras we use work (would be embarrassing
for us to buy the wrong ones next time).

Regards,

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:
> On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:
> 
> > What is not an option exactly in my description above ? We have multiple
> > V4L2 drivers for ISPs. They receive ISP parameters from userspace
> > through a data buffer. It's not allowed to be opaque, but it doesn't
> > prevent vendor closed-source userspace implementations with additional
> > *camera* features, as long as the *hardware* features are available to
> > everybody.
> 
> How far do you take opaque?
> 
> In mlx5 we pass command buffers from user to kernel to HW and the
> kernel does only a little checking.

I won't comment on the mlx5 case in particular, I don't have enough
knowledge to do so.

When it comes to validation of commands by the kernel, at the very least
the kernel driver needs to be able to guarantee safety. For instance,
that means that any command that can result in DMA operations need to be
validated (e.g. verifying buffer addresses and sizes) and/or executed
partly by the driver (e.g. mapping a buffer through an IOMMU), depending
on hardware constraints.

> There is a 12kloc file describing the layout of alot of commands:
> include/linux/mlx5/mlx5_ifc.h
> 
> There is an open PDF describing in detail some subset of this:
> https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
> 
> There are in-kernel implementations driving most of those commands.

For camera ISPs, most of the "commands" wouldn't be driven by the
kernel. I don't have expectations when it comes to what commands the
kernel exercises directly, I think that highly depends on the device
type.

> Other commands are only issued by userspace, and we have open source
> DPDK, rdma-core and UCX implementations driving them.
> 
> ie, this is really quite good as far as a device providing open source
> solutions goes.
> 
> However, no doubt there is more FW capability and commands than even
> this vast amount documents - so lets guess that propritary code is
> using this interface with unknown commands too.
> 
> From a camera perspective would you be OK with that? Let's guess that
> 90% of use cases are covered with fully open code. Do we also need to
> forcefully close the door to an imagined 10% of proprietary cases just
> to be sure open always wins?

For command buffers interpreted by a firmware, it would be extremely
difficult, if not impossible, to ensure that nothing undocumented is
possible to access from userspace. I think we have two issues here, one
of them is to define what we required to be openly accessible, and the
other to avoid vendors cheating by claiming to be more open than they
actually are.

I believe the latter can't be solved technically. At the end of the day
it's a matter of trust, and the only option I can see is to build that
trust with the vendors, and to make it clear that breaches of trust will
have consequences. A vendor that deliberately lies would end up on my
personal backlist for as long as they don't implement structural
solutions to be a better player in the ecosystem. What is acceptable as
consequences is for the communities to decide. We have previously banned
people or organizations from contributing to the kernel for certain
periods of time (the University of Minnesota case comes to my mind for
instance), and other options can be considered too.

As for the first issue, I think it's a difficult one as it is very
difficult to quantify the features covered by open implementations, as
well as their importance. You could have a 99% open command set where
the 1% would impact open implementations extremely negatively, the same
way you could have a 50% open command set where the other 50% isn't of
any use to anyone but the vendor (for instance used to debug the
firmware implementation).

For cameras, the feature set can I believe be expressed in terms of ISP
processing blocks that are usable by open implementations, as well as in
terms of the type of usage of those features. For instance, is it
acceptable that a vendor documents how to use 2D noise reduction but
makes 3D (temporal) noise reduction available only to close-source
userspace ? My personal answer is no. I want to aim for very close to
100% of the features, and certainly 100% of the "large" features. I can
close my eyes on features that are very very niche, but what is niche
today may not be tomorrow. For instance, if a camera ISP implements a
feature used only for very specific camera sensors targetted at
autonomous driving in a new generation of research prototypes that only
one company has access to, I'll be inclined to ignore that. That
technology may however become much more mainstream 5 years later.

The other aspect is the type of usage of the features. For camera ISPs
again, some parameters will be computed through a tuning process, and
will be fixed for the lifetime of the camera. I want those parameters to
be documented, to enable users to tune cameras for their own use cases.
This is less important in some market segments, such as laptops for
instance, but is crucial for embedded devices. This is an area where
I've previously been called unreasonable, and I don't think that's fair.
The tuned-and-immutable parameters are not plentiful, as most of the
tuning results in data that needs to be combined with runtime
information to compute ISP parameters, so the "this is for tuning only"
argument doesn't hold as much as one may think. For real immutable
parameters, a large number of them are related to image processing steps
that are very common and found in most ISPs, such as lens shading
correction or geometric distorsion correction. I don't see why we should
let vendors keep those closed.

I'm sorry that this discussion is turning into something very
camera-centric, but that's the relevant area I know best. I hope that
discussing problems related to device pass-through in different areas in
the open will help build a wider shared understanding of the problems,
and hopefully help designing solutions by better taking into account the
various aspects of the issues.

> Does closing the door have to come at the cost of a technically clean
> solution? Doing validation in the kernel to enforce an ideological
> position would severely degrade mlx5, and would probably never really
> be robust.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

I know this is a topic proposed for the maintainers summit, but given
the number of people who seem to have an opinion and be interested in
dicussing it, would a session at LPC be a better candidate ? I don't
expect the maintainer summit to invite all relevant experts from all
subsystems, that would likely overflow the room.

The downside of an LPC session is that it could easily turn into a
heated stage fight, and there are probably also quite a few arguments
that can't really be made in the open :-S

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Steven Rostedt]

On Fri, 26 Jul 2024 17:27:31 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> I know this is a topic proposed for the maintainers summit, but given
> the number of people who seem to have an opinion and be interested in
> dicussing it, would a session at LPC be a better candidate ? I don't
> expect the maintainer summit to invite all relevant experts from all
> subsystems, that would likely overflow the room.
> 
> The downside of an LPC session is that it could easily turn into a
> heated stage fight, and there are probably also quite a few arguments
> that can't really be made in the open :-S
> 

You can always submit a BoF. We have also had last minute BoFs made if
there's rooms available.

-- Steve

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> > On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > > James Bottomley wrote:
> > > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > >
> > > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > > positions?
> > > > > > > >
> > > > > > > > This thread has that discussion:
> > > > > > > >
> > > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > >
> > > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > > consideration of fwctl for CXL.
> > > > > > > >
> > > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > > wrapper.
> > > > > > > >
> > > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > >
> > > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > > should they know the technical implications best, they should also have
> > > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > > different functions of a device.
> > > > > > >
> > > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > > software to use all the device's features. That's something we have been
> > > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > > provide better and more open camera support without the world
> > > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > > users and the whole ecosystem in the short and long term.
> > > > > >
> > > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > > >
> > > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > > conversation, but I think the technical discussion likely belongs to a
> > > > > different mail thread.
> > > > >
> > > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > > graphics cards.
> > > > > >
> > > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > > each vendor implements unique features and usage patterns. DRM
> > > > > > addresses this through vendor-specific IOCTLs
> > > > > >
> > > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > > implementation, not comprehensive documentation.
> > > > >
> > > > > This point isn't technical and is more on-topic for this mail thread.
> > > > >
> > > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > > text form. An open-source userspace implementation that covers the
> > > > > feature set exposed by the driver is acceptable in place of
> > > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > > DRM drivers.
> > > >
> > > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > > command buffers.
> > > > Only the command buffer structure, and a mesa implementation using
> > > > that command buffer to support the standard features is required.
> > > >
> > > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > > be passed from userspace, they are typically formed in the driver from
> > > > a strictly checked struct.
> > >
> > > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > > and that mechanism is used in mainline drivers to pass camera ISP
> > > parameters. They're not called "command buffers" but that's just a
> > > difference in terminology. The technical means to pass command buffers
> > > to the driver is thus there, I see no meaningful difference with DRM.
> > > Where things can differ is in the contents of those buffers, and the
> > > requirements for documentation or open userspace implementations, but
> > > that's not a technical question.
> >
> > There are two things here:
> >
> > - The political/strategic/philosophical/religious aspect: The industry
> > definitely prefers the strategic requirements imposed by DRM. In fact
> > some vendors had some huge legal troubles when they had tried to
> > follow v4l2 requirements.
>
> That's I'm willing to debate.
>
> > - The technical aspect: DRM is more mature when it comes to
> > sending/receiving buffers to the hardware, and an ISP looks *much*
> > more similar to an accel device or a GPU than a UVC camera.
>
> But this I don't agree with. I think we should forgo the technical
> discussion and stop pretending that DRM is better for this use case from
> a technical point of view, and focus on the other aspect of the
> discussion. (We can of course reopen the technical discussion if new
> concrete arguments emerge.)

I disagree. ISP devices are EXACTLY the same as accel devices. I
suspect that you want to avoid them handled in DRM because then you
prefer the v4l2 openness requirements. There are no technical benefit
from v4l2 for using ISPs:

```
Everyone agrees that the current V4L2 API is not very suitable for the
current generation of ISPs: it is too cumbersome.
```
https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

>
> > > > > > Our current approach of pushing back against vendors, instead of
> > > > > > seeking compromise, has resulted in the vast majority of the market
> > > > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > > > with no options for utilizing their cameras outside of Android.
> > > > > >
> > > > > > DRM allows a hybrid model, where:
> > > > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > > > open-source stack.
> > > > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > > > proprietary features (e.g., automatic makeup) as closed source.
> > > > >
> > > > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > > > control algorithms in userspace, as long as there's an open-source
> > > > > implementation that exercises the same hardware features. A good analogy
> > > >
> > > > Is it really mandatory to have an open-source 3A algorithm? I thought
> > > > defining the input and output from the algorithm was good enough.
> > >
> > > What really matters is documenting the ISP parameters with enough
> > > details to allow for the implementation of open-source userspace code.
> > > Once you have that, 3A is quite simple. You can refine it (especially
> > > AWB) to great length, for instance using NPUs to compute parameters, and
> > > there's absolutely no issue with such userspace implementations being
> > > closed.
> > >
> > > In practice, some vendors prefer documenting the parameters by writing
> > > an open-source userspace implementation, partly maybe because developers
> > > are more familiar writing code than formal documentation. I would be
> > > fine either way, as long as there's enough information to make use of
> > > the ISP.
> >
> > Even with vendor passthrough there is still a need to provide a full
> > open source implementation (probably based on libcamera).
>
> We have a different interpretation of "full" :-) I want to aim for
> "full" to cover all the features exposed by the driver UAPI. There could
> be some exceptions that we can discuss if there are compeling arguments,
> but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.

Full means that I can use my camera using debian main and nothing more
than that.  It is a pretty accepted description of what fully open
means.


>
> > So you will have enough information to use all the common
> > functionality of a camera.
> >
> > > > AFAIK for some time there was no ipu3 open source algorithm, and the
> > > > driver has been upstream.
> > >
> > > It sneaked in before we realized we had to enforce rules :-) That's
> > > actually a good example, when we wrote open-source userspace support, we
> > > realized that the level of documentation included in the IPU3 kernel
> > > header was nowhere close to what was needed to make use of the device.
> > >
> > > > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > > > free to ship closed-source implementations that include more
> > > > > optimizations, as long as the open-source, less optimized implementation
> > > > > covers the same GPU ISA, so that open-source developers can also work on
> > > > > optimizing it.
> > > >
> > > > I believe a more accurate description is that in v4l2 is that we
> > > > expect that all the registers, device architecture and behaviour to be
> > > > documented and accessed with standard IOCTLs. Anything not documented
> > > > cannot be accessed by userspace.
> > > >
> > > > In DRM their concern is that there is a fully open source
> > > > implementation that the user can use. Vendors have custom IOCTLs and
> > > > they can offer proprietary software for some use cases.
> > >
> > > Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> > > vs. standard ioctls isn't very relevant to this discussion. I really
> > > don't see how this would be about ioctls, it's about making the featured
> > > exposed by the drivers, through whatever means a particular subsystem
> > > allows, usable by open userspace.
> > >
> > > > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > > > doesn't seem realistic to me. Both subsystems will have similar rules.
> > > >
> > > > DRM does indeed allow vendors to pass random command buffers and they
> > > > will be sent to the hardware. We cannot do that in v4l2.
> > >
> > > You can pass a command buffer to a V4L2 device and have the driver send
> > > it to the device firmware (ISPs using real command buffers usually run a
> > > firmware). If you want the driver to be merged upstream, you have to
> > > document the command buffer in enough details.
> >
> > Documenting with "enough details" is not enough. In V4L2, we have to
> > deeply inspect every single buffer to make sure that it is not sending
> > an unknown combination of command+arguments, or in other situations we
> > construct the command buffer in the driver.
> >
> > > > I might be wrong, but GPU drivers do not deeply inspect the command
> > > > buffers to make sure that they do not use any feature not covered by
> > > > mesa.
> > >
> > > That's correct, but I don't think that's relevant. The GPU market has
> > > GLSL and Vulkan. An open-source compliant implementation will end up
> > > exercising a very very large part of the device ISA, command submission
> > > mechanism and synchronization primitives, if not all of it. There's
> > > little a vendor would keep under the hood and use in closed-source
> > > userspace only. For cameras, there's no standard userspace API that
> > > covers by design a very large part of what is the ISP equivalent of a
> >
> > Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...
>
> Those are not comparable to GLSL or Vulkan, they are much higher level.
>
> > If we are afraid of vendors providing "toy" implemententations to pass
> > the openness requirements, we can add more features/tests to
> > libcamera.
> >
> > And at the end of the day, there will be humans deciding if what a
> > vendor has provided is good enough or not.
> >
> > > GPU ISA. Even "command buffers" are not a proper description of the
> > > parameters the vast majority of ISPs consume.
> >
> > Modern ISPs are definitely going in the direction of "command buffers"
>
> Examples please, with technical details. I've seen this argument being
> used for at least a year, with very little evidence. There are ISPs that
> are more programmable than others, but in practice firmwares mostly
> expose fixed functions, not ISA-level programmability.

You have not seen many programmable ISPs in v4l2 because they cannot
be supported with v4l2.
Instead vendors have to create huge firmwares that limits the
capability of the device.
I do not want to look into those firmwares.


>
> > > > > > This approach would allow billions of users to access their hardware
> > > > > > more securely and with in-tree driver support. Our current stubborn
> > > > > > pursuit of an idealistic goal has already negatively impacted both
> > > > > > users and the ecosystem.
> > > > > >
> > > > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > > > Linux will remain a niche market for ISPs.
> > > > > >
> > > > > > If such a hybrid model goes against Linux goals, this is something
> > > > > > that should be agreed upon by the whole community, so we have the same
> > > > > > criteria for all subsystems.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Fri, Jul 26, 2024 at 3:11 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> > On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:
> > > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > >
> > > > > > <...>
> > > > > >
> > > > > > > It would be great to define what are the free software communities
> > > > > > > here. Distros and final users are also "free software communities" and
> > > > > > > they do not care about niche use cases covered by proprietary
> > > > > > > software.
> > > > > >
> > > > > > Are you certain about that?
> > > > >
> > > > > As a user, and as an open source Distro developer I have a small hint.
> > > > > But you could also ask users what they think about not being able to
> > > > > use their notebook's cameras. The last time that I could not use some
> > > > > basic hardware from a notebook with Linux was 20 years ago.
> > > >
> > > > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > > > with Linux, and even now, there is basic hardware in my current
> > > > laptop (HP docking station) that doesn't work reliably in Linux.
> > > >
> > > > > > > They only care (and should care) about normal workflows.
> > > > > >
> > > > > > What is a normal workflow?
> > > > > > Does it mean that if user bought something very expensive he
> > > > > > should not be able to use it with free software, because his
> > > > > > usage is different from yours?
> > > > > >
> > > > > > Thanks
> > > > >
> > > > > It means that we should not block the standard usage for 99% of the
> > > > > population just because 1% of the users cannot do something fancy with
> > > > > their device.
> > > >
> > > > Right, the problem is that in some areas the statistics slightly different.
> > > > 99% population is blocked because 1% of the users don't need it and
> > > > don't think that it is "normal" flow.
> > > >
> > > > > Let me give you an example. When I buy a camera I want to be able to
> > > > > do Video Conferencing and take some static photos of documents. I do
> > > > > not care about: automatic makeup, AI generated background, unicorn
> > > > > filters, eyes recentering... But we need to give a way to vendors to
> > > > > implement those things closely, without the marketing differentiators,
> > > > > vendors have zero incentive to invest in Linux, and that affects all
> > > > > the population.
> > >
> > > I've seen these kind of examples being repeatedly given in discussions
> > > related to camera ISP support in Linux. They are very misleading. These
> > > are not the kind of features that are relevant for the device
> > > pass-through discussion these day. Those are high-level use cases
> > > implemented in userspace, and vendors can ship any closed-source
> > > binaries they want there. What I care about is the features exposed by
> > > the kernel to userspace API.
> >
> > The ISPs are gradually becoming programmable devices and they indeed
> > help during all of those examples.
>
> I'd like to see more technical information to substantiate this claim.
> So far what I've sometimes seen is ISPs that include programmable
> elements, but hiding those behind a firmware that exposes a fixed
> (configurable) pipeline. I've also heard of attempts to expose some of
> that programmability to the operating system, which were abandoned in
> the end due to lack usefulness.
>
> > Userspace needs to send/receive information from the ISP, and that is
> > exactly what vendors want to keep in the close.
>
> But that's exactly what we need to implement an open userspace ecosystem
> :-)
>
> > Describing how they implement those algorithms is a patent minefield
> > and their differentiating factor.
>
> Those are also arguments I've heard many times before. The
> differentiating factor for cameras today is mostly in userspace ISP
> control algorithms, and nobody is telling vendors they need to open all
> that.

I disagree. The differentiating factor is what the ISP is capable of
doing and how they do it. Otherwise we would not see new ISPs in the
market.

If you define the arguments passed to an ISP you are defining the
algorithm, and that is a trade secret and/or a patent violation.

>
> When it comes to patents, we all know how software patents is a
> minefield, and hardware is also affected. I can't have much sympathy for
> this argument though, those patents mostly benefit the largest players
> in the market, and those are the ones who currently claim they can't
> open anything due to patents.

Big players do not usually sue each other. The big problem is patent
trolls that "shoot at everything that moves".

I dislike patents, but it is the world we have to live in. No vendor
is going to take our approach if they risk a multi million dollar
lawsuit.


>
> > > > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > > > >
> > > > > There are other other subsystems that allow vendor passthrough and
> > > > > their ecosystem has not collapsed.
> > > >
> > > > Yes, I completely agree with you on that.
> > > >
> > > > > Can we have some general guidance of what is acceptable? Can we define
> > > > > together the "normal workflow" and focus on a *full* open source
> > > > > implementation of that?
> > > >
> > > > I don't think that is possible to define "normal workflow". Requirement
> > > > to have open-source counterpart to everything exposed through UAPI is a
> > > > valid one. I'm all for that.
> > >
> > > That's my current opinion as well, as least when it comes to the kernel
> > > areas I mostly work with.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Fri, Jul 26, 2024 at 4:22 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:
> > On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:
> >
> > > What is not an option exactly in my description above ? We have multiple
> > > V4L2 drivers for ISPs. They receive ISP parameters from userspace
> > > through a data buffer. It's not allowed to be opaque, but it doesn't
> > > prevent vendor closed-source userspace implementations with additional
> > > *camera* features, as long as the *hardware* features are available to
> > > everybody.
> >
> > How far do you take opaque?
> >
> > In mlx5 we pass command buffers from user to kernel to HW and the
> > kernel does only a little checking.
>
> I won't comment on the mlx5 case in particular, I don't have enough
> knowledge to do so.
>
> When it comes to validation of commands by the kernel, at the very least
> the kernel driver needs to be able to guarantee safety. For instance,
> that means that any command that can result in DMA operations need to be
> validated (e.g. verifying buffer addresses and sizes) and/or executed
> partly by the driver (e.g. mapping a buffer through an IOMMU), depending
> on hardware constraints.
>
> > There is a 12kloc file describing the layout of alot of commands:
> > include/linux/mlx5/mlx5_ifc.h
> >
> > There is an open PDF describing in detail some subset of this:
> > https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
> >
> > There are in-kernel implementations driving most of those commands.
>
> For camera ISPs, most of the "commands" wouldn't be driven by the
> kernel. I don't have expectations when it comes to what commands the
> kernel exercises directly, I think that highly depends on the device
> type.
>
> > Other commands are only issued by userspace, and we have open source
> > DPDK, rdma-core and UCX implementations driving them.
> >
> > ie, this is really quite good as far as a device providing open source
> > solutions goes.
> >
> > However, no doubt there is more FW capability and commands than even
> > this vast amount documents - so lets guess that propritary code is
> > using this interface with unknown commands too.
> >
> > From a camera perspective would you be OK with that? Let's guess that
> > 90% of use cases are covered with fully open code. Do we also need to
> > forcefully close the door to an imagined 10% of proprietary cases just
> > to be sure open always wins?
>
> For command buffers interpreted by a firmware, it would be extremely
> difficult, if not impossible, to ensure that nothing undocumented is
> possible to access from userspace. I think we have two issues here, one
> of them is to define what we required to be openly accessible, and the
> other to avoid vendors cheating by claiming to be more open than they
> actually are.
>
> I believe the latter can't be solved technically. At the end of the day
> it's a matter of trust, and the only option I can see is to build that
> trust with the vendors, and to make it clear that breaches of trust will
> have consequences. A vendor that deliberately lies would end up on my
> personal backlist for as long as they don't implement structural
> solutions to be a better player in the ecosystem. What is acceptable as
> consequences is for the communities to decide. We have previously banned
> people or organizations from contributing to the kernel for certain
> periods of time (the University of Minnesota case comes to my mind for
> instance), and other options can be considered too.
>
> As for the first issue, I think it's a difficult one as it is very
> difficult to quantify the features covered by open implementations, as
> well as their importance. You could have a 99% open command set where
> the 1% would impact open implementations extremely negatively, the same
> way you could have a 50% open command set where the other 50% isn't of
> any use to anyone but the vendor (for instance used to debug the
> firmware implementation).

It is not that difficult. You just have to define what an acceptable
implementation is. Eg

- Sharpness at specific light.
- Time of convergence for the Auto algorithms
- Noise ratio

We could even use the SoftISP implementation as reference. Is this ISP
working better or worse than SoftISP?


>
> For cameras, the feature set can I believe be expressed in terms of ISP
> processing blocks that are usable by open implementations, as well as in
> terms of the type of usage of those features. For instance, is it
> acceptable that a vendor documents how to use 2D noise reduction but
> makes 3D (temporal) noise reduction available only to close-source
> userspace ? My personal answer is no. I want to aim for very close to
> 100% of the features, and certainly 100% of the "large" features. I can
> close my eyes on features that are very very niche, but what is niche
> today may not be tomorrow. For instance, if a camera ISP implements a
> feature used only for very specific camera sensors targetted at
> autonomous driving in a new generation of research prototypes that only
> one company has access to, I'll be inclined to ignore that. That
> technology may however become much more mainstream 5 years later.

We can update our requirements in 5 years. Nothing is written in stone.

Also it is much easier to reverse engineer an open driver, with an
open userspace and a closed userspace than a close driver with a
closed userspace.


>
> The other aspect is the type of usage of the features. For camera ISPs
> again, some parameters will be computed through a tuning process, and
> will be fixed for the lifetime of the camera. I want those parameters to
> be documented, to enable users to tune cameras for their own use cases.
> This is less important in some market segments, such as laptops for
> instance, but is crucial for embedded devices. This is an area where
> I've previously been called unreasonable, and I don't think that's fair.
> The tuned-and-immutable parameters are not plentiful, as most of the
> tuning results in data that needs to be combined with runtime
> information to compute ISP parameters, so the "this is for tuning only"
> argument doesn't hold as much as one may think. For real immutable
> parameters, a large number of them are related to image processing steps
> that are very common and found in most ISPs, such as lens shading
> correction or geometric distorsion correction. I don't see why we should
> let vendors keep those closed.


We don't have enough leverage for that kind of requirement.

To be fair, we do not ask touchscreen manufacturers to document their
calibration files. Nor any other calibration file in the kernel.

The calibration file for me is like a firmware blob.



>
> I'm sorry that this discussion is turning into something very
> camera-centric, but that's the relevant area I know best. I hope that
> discussing problems related to device pass-through in different areas in
> the open will help build a wider shared understanding of the problems,
> and hopefully help designing solutions by better taking into account the
> various aspects of the issues.
>
> > Does closing the door have to come at the cost of a technically clean
> > solution? Doing validation in the kernel to enforce an ideological
> > position would severely degrade mlx5, and would probably never really
> > be robust.
>
> --
> Regards,
>
> Laurent Pinchart



--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Fri, Jul 26, 2024 at 3:22 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 10:03:36AM +0200, Ricardo Ribalda Delgado wrote:
> > On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart wrote:
> > > On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > > > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > > > [...]
> > > > > What I get from the discussions I've followed or partcipated in over
> > > > > the years is that the main worry of free software communities is
> > > > > being forced to use closed-source userspace components, whether that
> > > > > would be to make the device usable at all, or to achieve decent level
> > > > > of performance or full feature set. We've been through years of
> > > > > mostly closed-source GPU support, of printer "windrivers", and quite
> > > > > a few other horrors. The good news is that we've so far overcome lots
> > > > > (most) of those challenges. Reverse engineering projects paid off,
> > > > > and so did working hand-in-hand with industry actors in multiple ways
> > > > > (both openly and behind the scenes). One could then legitimately ask
> > > > > why we're still scared.
> > > >
> > > > I don't think I am.  We're mostly fully capable of expounding at length
> > > > on the business rationale for being open if the thing they're hiding
> > > > isn't much of a differentiator anyway (or they're simply hiding it to
> > > > try to retain some illusion of control), so we shouldn't have any fear
> > > > of being able to make our case in language business people understand.
> > > >
> > > > I also think this fear is partly a mindset problem on our part.  We
> > > > came out of the real fight for openness and we do embrace things like a
> > > > licence that forces open code (GPL) and symbols that discourage
> > > > proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> > > > FSF coolaid that if we don't stand over manufacturers every second and
> > > > force them they'll slide back to their old proprietary ways.  However,
> > > > if you look at the entirely permissive ecosystem that grew up after we
> > > > did (openstack, docker, kubernetes, etc.) they don't have any such fear
> > > > and yet they still have large amounts of uncompelled openness and give
> > > > back.
> > >
> > > I don't think those are necessarily relevant examples, as far as device
> > > pass-through goes. Vendors have many times reverted to proprietary ways,
> > > and they still do, at least in the areas of the kernel I'm most active
> > > in. I've seen first hand a large SoC vendor very close to opening a
> > > significant part of their camera stack and changing their mind at the
> > > last minute when they heard they could possibly merge their code through
> > > a different subsystem with a pass-through blank cheque.
> >
> > Without knowing who that large SoC vendor is, and what they will be
> > willing to open, it is difficult to know what opportunity has been
> > lost.
>
> I would also like to discuss this in more details, but I can't do so
> publicly :-( That's why I don't want to focus too much on this argument.
>
> > I would argue that if they have cancelled their open plans based
> > on an hypothesis, their willingness to open was not that high.
>
> That's exactly what I previously described in this mail thread as areas
> where cooperation with vendors is in its infancy. Convincing large
> industry players is long term work, and it's very demotivating when you
> think there's finally light at the end of the tunel, and plans gets
> cancelled at the last minute. We're not giving up though.
>
> > It would be more healthy for the ecosystem, if those discussions were
> > done more openly, at least all the core maintainers should be
> > involved.
>
> Can you get all the camera vendors you work with to talk about all this
> openly ? :-) We all know it's a difficult and long process.

We have pushed lots of vendors to contribute directly upstream. You
can see a lot of code that has landed thanks to this effort.
And I have tried very hard to get them involved in all the discussions upstream.


>
> > > I'm willing to believe it can be different in other areas, which may
> > > partly explain why different subsystems and different developers have
> > > different biases and have trouble understand each other's point of view.
> >
> > It is not different in other areas, it is the same area. At the end of
> > the day it is the same chip manufacturers, with the same legal teams.
> > It is our attitude that is different.
>
> I don't agree with that. Not only are there different manufacturers,
> there are also different teams, with different approaches. The world is
> not uniform.
>
> > > > > I can't fully answer that question, but there are two points that I
> > > > > think are relevant. Note that due to my background and experience,
> > > > > this will be heavily biased towards consumer and embedded hardware,
> > > > > not data centre-grade devices. Some technologies from the latter
> > > > > however have a tendency to migrate to the former over time, so the
> > > > > distinction isn't necessarily as relevant as one may consider.
> > > > >
> > > > > The first point is that hardware gets more complicated over time, and
> > > > > in some markets there's also an increase in the number of vendors and
> > > > > devices. There's a perceived (whether true or not) danger that we
> > > > > won't be able to keep up with just reverse engineering and a
> > > > > development model relying on hobyists. Getting vendors involved is
> > > > > important if we want to scale.
> > > >
> > > > Yes, but there are lots of not very useful complex devices being
> > > > produced every day that fail to capture market share.  Not having
> > > > reverse engineered drivers for them is no real loss.  If a device does
> > > > gain market share, it gains a huge pool of users some of whom become
> > > > interested in reverse engineering, so I think market forces actually
> > > > work in our favour: we get reverse engineering mostly where the devices
> > > > are actually interesting and capture market share.  It's self scaling.
> > >
> > > I can't agree with that, sorry. Not only is the difficulty to
> > > reverse-engineer some classes of devices increasing, but saying that
> > > only devices that make it to the top of the market share chart are worth
> > > considering will leave many users on the side of the road.
> >
> > Today we have left BILLIONS of users at the other side of the road.
>
> Because we're the only ones to be blamed, or more accuretely, the blame
> is solely on me. Vendors are all saints who want the best for the world.
> This argument doesn't hold. Please stop with this, I don't want to get
> to name calling.

Nobody said that it was your fault. You and all the media community
have invested thousands of hours of their personal time in Linux.

It is healthy to make a retrospective of where we are now and if we
are helping our users or not.

The reality is that we are reaching a very small percentage of users,
and vendors prefer to work out of the tree than with us. Even vendors
that traditionally have been very good Linux players have issues
landing their code.

Can we try to figure out why it is happening? Can we try something
new? Maybe inspired in other subsystems?


>
> > > > > Second, I think there's a fear of regression. For some categories of
> > > > > devices, we have made slow but real progress to try and convince the
> > > > > industry to be more open. This sometimes took a decade of work,
> > > > > patiently building bridges and creating ecosystems brick by brick.
> > > > > Some of those ecosystems are sturdy, some not so. Giving pass-through
> > > > > a blank check will likely have very different effects in different
> > > > > areas. I don't personally believe it will shatter everything, but I'm
> > > > > convinced it carries risk in areas where cooperation with vendors is
> > > > > in its infancy or is fragile for any other reason.
> > > >
> > > > I also think we're on the rise in this space.  Since most cloud
> > > > workloads are on Linux, there's huge market pressure on most "found in
> > > > the cloud" devices (like accelerators and GPUs) to have an easy to
> > > > consume Linux story.  Nvidia is a case in point.  When it only cared
> > > > about fast games on some other OS, we get shafted with a proprietary
> > > > graphics drivers.  Now it's under pressure to be the number one AI
> > > > accelerator provider for the cloud it's suddenly wondering about open
> > > > source drivers to make adoption easier.
> > >
> > > I can't comment on Nvidia and their inference engines in particular. The
> > > server market may be in a better position that the consumer and embedded
> > > market, and if that's the case, I'm happy for the servers. That doesn't
> > > solve the issues in other markets though.
> > >
> > > > > Finally, let's not forget that pass-through APIs are not an all or
> > > > > nothing option. To cite that example only, DRM requires GPU drivers
> > > > > to have an open-source userspace implementation to merge the kernel
> > > > > driver, and the same subsystems strongly pushes for API
> > > > > standardization for display controllers. We can set different rules
> > > > > for different cases.
> > > >
> > > > I certainly think we can afford to experiment here, yes.
>
> --
> Regards,
>
> Laurent Pinchart



--
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Fri, 2024-07-26 at 16:11 +0300, Laurent Pinchart wrote:
> On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado
> wrote:
[...]
> > Describing how they implement those algorithms is a patent
> > minefield and their differentiating factor.

Just on this argument: The Patent Shield around Linux provided by OIN
is pretty strong and allows us (and entities that contribute to Linux)
to ignore most patent problems because someone else is looking out for
them.  OIN is actually free to join if any company would like to
benefit directly from the Linux Patent Shield:

https://openinventionnetwork.com/about-us/member-benefits/


> Those are also arguments I've heard many times before. The
> differentiating factor for cameras today is mostly in userspace ISP
> control algorithms, and nobody is telling vendors they need to open
> all that.
> 
> When it comes to patents, we all know how software patents is a
> minefield, and hardware is also affected. I can't have much sympathy
> for this argument though, those patents mostly benefit the largest
> players in the market, and those are the ones who currently claim
> they can't open anything due to patents.

In order to get a patent, the claimed invention has to be made public,
so if they hold the patent there should be no problem.  If there is a
problem opening something because it infringes on someone else's patent
and they might see it, then OIN, above, is usually a good answer if the
patent is owned by another ecosystem contributor or an OIN signatory. 
For patent contortia (like MPEG) and patent trolls, it's more
problematic, but, again, OIN can provide help.

Regards,

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Thu, 2024-07-25 at 22:31 +0300, Laurent Pinchart wrote:
> On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > [...]
> > > What I get from the discussions I've followed or partcipated in
> > > over the years is that the main worry of free software
> > > communities is being forced to use closed-source userspace
> > > components, whether that would be to make the device usable at
> > > all, or to achieve decent level of performance or full feature
> > > set. We've been through years of mostly closed-source GPU
> > > support, of printer "windrivers", and quite a few other horrors.
> > > The good news is that we've so far overcome lots (most) of those
> > > challenges. Reverse engineering projects paid off, and so did
> > > working hand-in-hand with industry actors in multiple ways
> > > (both openly and behind the scenes). One could then legitimately
> > > ask why we're still scared.
> > 
> > I don't think I am.  We're mostly fully capable of expounding at
> > length on the business rationale for being open if the thing
> > they're hiding isn't much of a differentiator anyway (or they're
> > simply hiding it to try to retain some illusion of control), so we
> > shouldn't have any fear of being able to make our case in language
> > business people understand.
> > 
> > I also think this fear is partly a mindset problem on our part.  We
> > came out of the real fight for openness and we do embrace things
> > like a licence that forces open code (GPL) and symbols that
> > discourage proprietary drivers (EXPORT_SYMBOL_GPL), so we've
> > somewhat drunk the FSF coolaid that if we don't stand over
> > manufacturers every second and force them they'll slide back to
> > their old proprietary ways.  However, if you look at the entirely
> > permissive ecosystem that grew up after we did (openstack, docker,
> > kubernetes, etc.) they don't have any such fear and yet they still
> > have large amounts of uncompelled openness and give back.
> 
> I don't think those are necessarily relevant examples,

Well they do show it is possible to get an open ecosystem based on the
pull of business need instead of using a forcing function like the
licence.

>  as far as device pass-through goes. Vendors have many times reverted
> to proprietary ways, and they still do, at least in the areas of the
> kernel I'm most active in.

I'm not going to argue that companies don't make bad business
decisions: they definitely do.  And also there are huge reservoirs of
proprietary mindset or fear of losing control in most of them which
tend to drive these bad decisions.  I'm just saying we don't have to be
all stick and no carrot.  If we can use the carrot to help them make
better business decisions, and overcome the proprietary mindset that
way, so much the better.

There are also valid reasons to do pass-through, like you don't want to
invest in standardising the interface until you see that you have an
actual market, which is how mixed standardised/pass through systems
like DRM work (they allow innovation and experimentation, not all of
which works).

>  I've seen first hand a large SoC vendor very close to opening a
> significant part of their camera stack and changing their mind at the
> last minute when they heard they could possibly merge their code
> through a different subsystem with a pass-through blank cheque.

So is the learning here is that perhaps we should co-ordinate better?

> I'm willing to believe it can be different in other areas, which may
> partly explain why different subsystems and different developers have
> different biases and have trouble understand each other's point of
> view.

I think that's true, due to different companies having different levels
of proprietary mindset and other fears of not controlling their own
destiny.  However, the basic fact remains that it will be in their best
business interests to be more open if we can encourage it.

> > > I can't fully answer that question, but there are two points that
> > > I think are relevant. Note that due to my background and
> > > experience, this will be heavily biased towards consumer and
> > > embedded hardware, not data centre-grade devices. Some
> > > technologies from the latter however have a tendency to migrate
> > > to the former over time, so the distinction isn't necessarily as
> > > relevant as one may consider.
> > > 
> > > The first point is that hardware gets more complicated over time,
> > > and in some markets there's also an increase in the number of
> > > vendors and devices. There's a perceived (whether true or not)
> > > danger that we won't be able to keep up with just reverse
> > > engineering and a development model relying on hobyists. Getting
> > > vendors involved is important if we want to scale.
> > 
> > Yes, but there are lots of not very useful complex devices being
> > produced every day that fail to capture market share.  Not having
> > reverse engineered drivers for them is no real loss.  If a device
> > does gain market share, it gains a huge pool of users some of whom
> > become interested in reverse engineering, so I think market forces
> > actually work in our favour: we get reverse engineering mostly
> > where the devices are actually interesting and capture market
> > share.  It's self scaling.
> 
> I can't agree with that, sorry. Not only is the difficulty to
> reverse-engineer some classes of devices increasing,

So it's a higher hill to climb: I didn't say it wasn't.  However for
hugely popular devices there's still more likelihood of finding someone
willing to climb it.  It's not a guarantee, just a probability, though.

> but saying that only devices that make it to the top of the market
> share chart are worth considering will leave many users on the side
> of the road.

I'm afraid this is simple economics.  Suppose we had the budget to pay
for reverse engineering, it's not going to cover everything, so we'd
have to make choices and that would necessarily mean investing in
devices that have utility to the greatest number of people.  So low
market share devices would still be left out.

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Daniel Vetter]

On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> [...]
> > What I get from the discussions I've followed or partcipated in over
> > the years is that the main worry of free software communities is
> > being forced to use closed-source userspace components, whether that
> > would be to make the device usable at all, or to achieve decent level
> > of performance or full feature set. We've been through years of
> > mostly closed-source GPU support, of printer "windrivers", and quite
> > a few other horrors. The good news is that we've so far overcome lots
> > (most) of those challenges. Reverse engineering projects paid off,
> > and so did working hand-in-hand with industry actors in multiple ways
> > (both openly and behind the scenes). One could then legitimately ask
> > why we're still scared.
> 
> I don't think I am.  We're mostly fully capable of expounding at length
> on the business rationale for being open if the thing they're hiding
> isn't much of a differentiator anyway (or they're simply hiding it to
> try to retain some illusion of control), so we shouldn't have any fear
> of being able to make our case in language business people understand.
> 
> I also think this fear is partly a mindset problem on our part.  We
> came out of the real fight for openness and we do embrace things like a
> licence that forces open code (GPL) and symbols that discourage
> proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> FSF coolaid that if we don't stand over manufacturers every second and
> force them they'll slide back to their old proprietary ways.  However,
> if you look at the entirely permissive ecosystem that grew up after we
> did (openstack, docker, kubernetes, etc.) they don't have any such fear
> and yet they still have large amounts of uncompelled openness and give
> back.

This matches my experience. Legal and technical roadblocks help a bit in
the margins, but they don't matter. The open gpu stack is 90% MIT, and
nvidia and all the others have demonstrated for years how easy it is to
ignore the GPL'ed part.

What gets vendors involved is a successful project that drives revenue,
where they have a clear need for a seat at the table to make sure the good
times for them continue. Clear rules what it takes to get that seat is in
my experience really the driving force for private discussions with
vendors, and from that pov the most important thing I've ever done for the
open gpu stack is this little documentation section:

https://dri.freedesktop.org/docs/drm/gpu/drm-uapi.html#open-source-userspace-requirements

Unlike laws you can legalese around and code you can trick with hacks for
an endless game of whack-a-mole the above doc was the real stick. And it
absolutely needed the carrot of the successful project that financially
matters to work.

Cheers, Sima
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 12:01:00PM -0400, James Bottomley wrote:
> On Fri, 2024-07-26 at 16:11 +0300, Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> [...]
> > > Describing how they implement those algorithms is a patent
> > > minefield and their differentiating factor.
> 
> Just on this argument: The Patent Shield around Linux provided by OIN
> is pretty strong and allows us (and entities that contribute to Linux)
> to ignore most patent problems because someone else is looking out for
> them.  OIN is actually free to join if any company would like to
> benefit directly from the Linux Patent Shield:
> 
> https://openinventionnetwork.com/about-us/member-benefits/
> 
> > Those are also arguments I've heard many times before. The
> > differentiating factor for cameras today is mostly in userspace ISP
> > control algorithms, and nobody is telling vendors they need to open
> > all that.
> > 
> > When it comes to patents, we all know how software patents is a
> > minefield, and hardware is also affected. I can't have much sympathy
> > for this argument though, those patents mostly benefit the largest
> > players in the market, and those are the ones who currently claim
> > they can't open anything due to patents.
> 
> In order to get a patent, the claimed invention has to be made public,
> so if they hold the patent there should be no problem.  If there is a
> problem opening something because it infringes on someone else's patent
> and they might see it, then OIN, above, is usually a good answer if the
> patent is owned by another ecosystem contributor or an OIN signatory. 
> For patent contortia (like MPEG) and patent trolls, it's more
> problematic, but, again, OIN can provide help.

While I agree with the above in theory, in practice the patent argument
is raised by large SoC vendors. There's one particular company well
known for having engineering managed by lawyers...

As I said, I don't sympathize with their arguments. The patent mess has
been mostly created by those large actors who benefit the most from it.
They can't at the same time claim that they can't disclose anything due
to fear of patent infringement.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Laurent Pinchart wrote:
> I know this is a topic proposed for the maintainers summit, but given
> the number of people who seem to have an opinion and be interested in
> dicussing it, would a session at LPC be a better candidate ? I don't
> expect the maintainer summit to invite all relevant experts from all
> subsystems, that would likely overflow the room.
> 
> The downside of an LPC session is that it could easily turn into a
> heated stage fight, and there are probably also quite a few arguments
> that can't really be made in the open :-S

A separate LPC session for a subsystem or set of subsystems to explore
local passthrough policy makes sense, but that is not the primary
motivation for also requesting a Maintainer Summit topic slot. The
primary motivation is discussing the provenance and navigation of
cross-subsystem NAKs especially in an environment where the lines
between net, mem, and storage are increasingly blurry at the device
level.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Dan,

On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> Laurent Pinchart wrote:
> > I know this is a topic proposed for the maintainers summit, but given
> > the number of people who seem to have an opinion and be interested in
> > dicussing it, would a session at LPC be a better candidate ? I don't
> > expect the maintainer summit to invite all relevant experts from all
> > subsystems, that would likely overflow the room.
> > 
> > The downside of an LPC session is that it could easily turn into a
> > heated stage fight, and there are probably also quite a few arguments
> > that can't really be made in the open :-S
> 
> A separate LPC session for a subsystem or set of subsystems to explore
> local passthrough policy makes sense, but that is not the primary
> motivation for also requesting a Maintainer Summit topic slot. The
> primary motivation is discussing the provenance and navigation of
> cross-subsystem NAKs especially in an environment where the lines
> between net, mem, and storage are increasingly blurry at the device
> level.

Would there be enough space at the maintainers' summit for all the
relevant people to join the discussion ?

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 05:40:50PM +0200, Ricardo Ribalda Delgado wrote:
> On Fri, Jul 26, 2024 at 3:11 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> >
> > On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> > > On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:
> > > > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > > >
> > > > > > > <...>
> > > > > > >
> > > > > > > > It would be great to define what are the free software communities
> > > > > > > > here. Distros and final users are also "free software communities" and
> > > > > > > > they do not care about niche use cases covered by proprietary
> > > > > > > > software.
> > > > > > >
> > > > > > > Are you certain about that?
> > > > > >
> > > > > > As a user, and as an open source Distro developer I have a small hint.
> > > > > > But you could also ask users what they think about not being able to
> > > > > > use their notebook's cameras. The last time that I could not use some
> > > > > > basic hardware from a notebook with Linux was 20 years ago.
> > > > >
> > > > > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > > > > with Linux, and even now, there is basic hardware in my current
> > > > > laptop (HP docking station) that doesn't work reliably in Linux.
> > > > >
> > > > > > > > They only care (and should care) about normal workflows.
> > > > > > >
> > > > > > > What is a normal workflow?
> > > > > > > Does it mean that if user bought something very expensive he
> > > > > > > should not be able to use it with free software, because his
> > > > > > > usage is different from yours?
> > > > > > >
> > > > > > > Thanks
> > > > > >
> > > > > > It means that we should not block the standard usage for 99% of the
> > > > > > population just because 1% of the users cannot do something fancy with
> > > > > > their device.
> > > > >
> > > > > Right, the problem is that in some areas the statistics slightly different.
> > > > > 99% population is blocked because 1% of the users don't need it and
> > > > > don't think that it is "normal" flow.
> > > > >
> > > > > > Let me give you an example. When I buy a camera I want to be able to
> > > > > > do Video Conferencing and take some static photos of documents. I do
> > > > > > not care about: automatic makeup, AI generated background, unicorn
> > > > > > filters, eyes recentering... But we need to give a way to vendors to
> > > > > > implement those things closely, without the marketing differentiators,
> > > > > > vendors have zero incentive to invest in Linux, and that affects all
> > > > > > the population.
> > > >
> > > > I've seen these kind of examples being repeatedly given in discussions
> > > > related to camera ISP support in Linux. They are very misleading. These
> > > > are not the kind of features that are relevant for the device
> > > > pass-through discussion these day. Those are high-level use cases
> > > > implemented in userspace, and vendors can ship any closed-source
> > > > binaries they want there. What I care about is the features exposed by
> > > > the kernel to userspace API.
> > >
> > > The ISPs are gradually becoming programmable devices and they indeed
> > > help during all of those examples.
> >
> > I'd like to see more technical information to substantiate this claim.
> > So far what I've sometimes seen is ISPs that include programmable
> > elements, but hiding those behind a firmware that exposes a fixed
> > (configurable) pipeline. I've also heard of attempts to expose some of
> > that programmability to the operating system, which were abandoned in
> > the end due to lack usefulness.
> >
> > > Userspace needs to send/receive information from the ISP, and that is
> > > exactly what vendors want to keep in the close.
> >
> > But that's exactly what we need to implement an open userspace ecosystem
> > :-)
> >
> > > Describing how they implement those algorithms is a patent minefield
> > > and their differentiating factor.
> >
> > Those are also arguments I've heard many times before. The
> > differentiating factor for cameras today is mostly in userspace ISP
> > control algorithms, and nobody is telling vendors they need to open all
> > that.
> 
> I disagree. The differentiating factor is what the ISP is capable of
> doing and how they do it. Otherwise we would not see new ISPs in the
> market.

Hardware certainly evolves, but it's far from being the main
differentiating factor in the markets and use cases you're usually
referring to.

> If you define the arguments passed to an ISP you are defining the
> algorithm, and that is a trade secret and/or a patent violation.

Are you confusing ISP processing blocks, sometimes referred to as
algorithms, and ISP control algorithms ? There is absolutely no way to
do anything with an ISP, not even the bare minimum, if you don't know
what parameters to pass to it.

> > When it comes to patents, we all know how software patents is a
> > minefield, and hardware is also affected. I can't have much sympathy for
> > this argument though, those patents mostly benefit the largest players
> > in the market, and those are the ones who currently claim they can't
> > open anything due to patents.
> 
> Big players do not usually sue each other. The big problem is patent
> trolls that "shoot at everything that moves".
> 
> I dislike patents, but it is the world we have to live in. No vendor
> is going to take our approach if they risk a multi million dollar
> lawsuit.

When was the last time anyone heard of big players pushing to reform the
patent system ? At best there are initiatives such as OIN, which some
large companies have supporting. It's still a workaround though.

> > > > > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > > > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > > > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > > > > >
> > > > > > There are other other subsystems that allow vendor passthrough and
> > > > > > their ecosystem has not collapsed.
> > > > >
> > > > > Yes, I completely agree with you on that.
> > > > >
> > > > > > Can we have some general guidance of what is acceptable? Can we define
> > > > > > together the "normal workflow" and focus on a *full* open source
> > > > > > implementation of that?
> > > > >
> > > > > I don't think that is possible to define "normal workflow". Requirement
> > > > > to have open-source counterpart to everything exposed through UAPI is a
> > > > > valid one. I'm all for that.
> > > >
> > > > That's my current opinion as well, as least when it comes to the kernel
> > > > areas I mostly work with.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Greg KH]

On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> Hi Dan,
> 
> On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > Laurent Pinchart wrote:
> > > I know this is a topic proposed for the maintainers summit, but given
> > > the number of people who seem to have an opinion and be interested in
> > > dicussing it, would a session at LPC be a better candidate ? I don't
> > > expect the maintainer summit to invite all relevant experts from all
> > > subsystems, that would likely overflow the room.
> > > 
> > > The downside of an LPC session is that it could easily turn into a
> > > heated stage fight, and there are probably also quite a few arguments
> > > that can't really be made in the open :-S
> > 
> > A separate LPC session for a subsystem or set of subsystems to explore
> > local passthrough policy makes sense, but that is not the primary
> > motivation for also requesting a Maintainer Summit topic slot. The
> > primary motivation is discussing the provenance and navigation of
> > cross-subsystem NAKs especially in an environment where the lines
> > between net, mem, and storage are increasingly blurry at the device
> > level.
> 
> Would there be enough space at the maintainers' summit for all the
> relevant people to join the discussion ?

Who exactly would you consider the "relevant people" here?  It's been a
wide-ranging conversation/thread :)

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 05:43:21PM +0200, Ricardo Ribalda Delgado wrote:
> On Fri, Jul 26, 2024 at 4:22 PM Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:
> > > On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:
> > >
> > > > What is not an option exactly in my description above ? We have multiple
> > > > V4L2 drivers for ISPs. They receive ISP parameters from userspace
> > > > through a data buffer. It's not allowed to be opaque, but it doesn't
> > > > prevent vendor closed-source userspace implementations with additional
> > > > *camera* features, as long as the *hardware* features are available to
> > > > everybody.
> > >
> > > How far do you take opaque?
> > >
> > > In mlx5 we pass command buffers from user to kernel to HW and the
> > > kernel does only a little checking.
> >
> > I won't comment on the mlx5 case in particular, I don't have enough
> > knowledge to do so.
> >
> > When it comes to validation of commands by the kernel, at the very least
> > the kernel driver needs to be able to guarantee safety. For instance,
> > that means that any command that can result in DMA operations need to be
> > validated (e.g. verifying buffer addresses and sizes) and/or executed
> > partly by the driver (e.g. mapping a buffer through an IOMMU), depending
> > on hardware constraints.
> >
> > > There is a 12kloc file describing the layout of alot of commands:
> > > include/linux/mlx5/mlx5_ifc.h
> > >
> > > There is an open PDF describing in detail some subset of this:
> > > https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
> > >
> > > There are in-kernel implementations driving most of those commands.
> >
> > For camera ISPs, most of the "commands" wouldn't be driven by the
> > kernel. I don't have expectations when it comes to what commands the
> > kernel exercises directly, I think that highly depends on the device
> > type.
> >
> > > Other commands are only issued by userspace, and we have open source
> > > DPDK, rdma-core and UCX implementations driving them.
> > >
> > > ie, this is really quite good as far as a device providing open source
> > > solutions goes.
> > >
> > > However, no doubt there is more FW capability and commands than even
> > > this vast amount documents - so lets guess that propritary code is
> > > using this interface with unknown commands too.
> > >
> > > From a camera perspective would you be OK with that? Let's guess that
> > > 90% of use cases are covered with fully open code. Do we also need to
> > > forcefully close the door to an imagined 10% of proprietary cases just
> > > to be sure open always wins?
> >
> > For command buffers interpreted by a firmware, it would be extremely
> > difficult, if not impossible, to ensure that nothing undocumented is
> > possible to access from userspace. I think we have two issues here, one
> > of them is to define what we required to be openly accessible, and the
> > other to avoid vendors cheating by claiming to be more open than they
> > actually are.
> >
> > I believe the latter can't be solved technically. At the end of the day
> > it's a matter of trust, and the only option I can see is to build that
> > trust with the vendors, and to make it clear that breaches of trust will
> > have consequences. A vendor that deliberately lies would end up on my
> > personal backlist for as long as they don't implement structural
> > solutions to be a better player in the ecosystem. What is acceptable as
> > consequences is for the communities to decide. We have previously banned
> > people or organizations from contributing to the kernel for certain
> > periods of time (the University of Minnesota case comes to my mind for
> > instance), and other options can be considered too.
> >
> > As for the first issue, I think it's a difficult one as it is very
> > difficult to quantify the features covered by open implementations, as
> > well as their importance. You could have a 99% open command set where
> > the 1% would impact open implementations extremely negatively, the same
> > way you could have a 50% open command set where the other 50% isn't of
> > any use to anyone but the vendor (for instance used to debug the
> > firmware implementation).
> 
> It is not that difficult. You just have to define what an acceptable
> implementation is. Eg
> 
> - Sharpness at specific light.
> - Time of convergence for the Auto algorithms
> - Noise ratio

Those may just be bad examples, but I think they showcase how little
these discussions are based on technical expertise and facts. Just
singling out one example, the convergence time is driven by the
userspace implementation of the ISP control algorithms. That is *not*
something we want to force vendors to disclose. It isn't related to the
ISP parameters and the UAPI the driver exposes to userspace.

> We could even use the SoftISP implementation as reference. Is this ISP
> working better or worse than SoftISP?
> 
> > For cameras, the feature set can I believe be expressed in terms of ISP
> > processing blocks that are usable by open implementations, as well as in
> > terms of the type of usage of those features. For instance, is it
> > acceptable that a vendor documents how to use 2D noise reduction but
> > makes 3D (temporal) noise reduction available only to close-source
> > userspace ? My personal answer is no. I want to aim for very close to
> > 100% of the features, and certainly 100% of the "large" features. I can
> > close my eyes on features that are very very niche, but what is niche
> > today may not be tomorrow. For instance, if a camera ISP implements a
> > feature used only for very specific camera sensors targetted at
> > autonomous driving in a new generation of research prototypes that only
> > one company has access to, I'll be inclined to ignore that. That
> > technology may however become much more mainstream 5 years later.
> 
> We can update our requirements in 5 years. Nothing is written in stone.

That at least we agree on :-)

> Also it is much easier to reverse engineer an open driver, with an
> open userspace and a closed userspace than a close driver with a
> closed userspace.

Closed kernel drivers are a GPL violation. Do you have any example, or
do you mean out-of-tree drivers ? I don't think out-of-tree vs. mainline
drivers would make much difference when it comes to reverse engineering,
if the mainline driver is just a pass-through driver.

> > The other aspect is the type of usage of the features. For camera ISPs
> > again, some parameters will be computed through a tuning process, and
> > will be fixed for the lifetime of the camera. I want those parameters to
> > be documented, to enable users to tune cameras for their own use cases.
> > This is less important in some market segments, such as laptops for
> > instance, but is crucial for embedded devices. This is an area where
> > I've previously been called unreasonable, and I don't think that's fair.
> > The tuned-and-immutable parameters are not plentiful, as most of the
> > tuning results in data that needs to be combined with runtime
> > information to compute ISP parameters, so the "this is for tuning only"
> > argument doesn't hold as much as one may think. For real immutable
> > parameters, a large number of them are related to image processing steps
> > that are very common and found in most ISPs, such as lens shading
> > correction or geometric distorsion correction. I don't see why we should
> > let vendors keep those closed.
> 
> We don't have enough leverage for that kind of requirement.

I very much disagree with that.

> To be fair, we do not ask touchscreen manufacturers to document their
> calibration files. Nor any other calibration file in the kernel.
> 
> The calibration file for me is like a firmware blob.

It may be for you, but I don't think it is an accurate description for
the rest of the industry.

> > I'm sorry that this discussion is turning into something very
> > camera-centric, but that's the relevant area I know best. I hope that
> > discussing problems related to device pass-through in different areas in
> > the open will help build a wider shared understanding of the problems,
> > and hopefully help designing solutions by better taking into account the
> > various aspects of the issues.
> >
> > > Does closing the door have to come at the cost of a technically clean
> > > solution? Doing validation in the kernel to enforce an ideological
> > > position would severely degrade mlx5, and would probably never really
> > > be robust.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Sun, Jul 28, 2024 at 05:16:28PM +0200, Greg KH wrote:
> On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > > Laurent Pinchart wrote:
> > > > I know this is a topic proposed for the maintainers summit, but given
> > > > the number of people who seem to have an opinion and be interested in
> > > > dicussing it, would a session at LPC be a better candidate ? I don't
> > > > expect the maintainer summit to invite all relevant experts from all
> > > > subsystems, that would likely overflow the room.
> > > > 
> > > > The downside of an LPC session is that it could easily turn into a
> > > > heated stage fight, and there are probably also quite a few arguments
> > > > that can't really be made in the open :-S
> > > 
> > > A separate LPC session for a subsystem or set of subsystems to explore
> > > local passthrough policy makes sense, but that is not the primary
> > > motivation for also requesting a Maintainer Summit topic slot. The
> > > primary motivation is discussing the provenance and navigation of
> > > cross-subsystem NAKs especially in an environment where the lines
> > > between net, mem, and storage are increasingly blurry at the device
> > > level.
> > 
> > Would there be enough space at the maintainers' summit for all the
> > relevant people to join the discussion ?
> 
> Who exactly would you consider the "relevant people" here?  It's been a
> wide-ranging conversation/thread :)

I'd say the maintainers of the related subsystems/drivers, or the
relevant open-source stacks, as well as the people who are pushing for a
change, and overall the main stakeholders who took part in the
discussions so far. I don't think it would be very nice to discuss this
topic and make a decision behind closed doors while excluding some of
those people from the process.

I have opinions on the overall topic, but mostly related to cameras, so
I wouldn't consider myself as relevant when it comes to net/mem/storage,
mlx5 or fwctl in particular.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Sun, 2024-07-28 at 17:16 +0200, Greg KH wrote:
> On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> > Hi Dan,
> > 
> > On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > > Laurent Pinchart wrote:
> > > > I know this is a topic proposed for the maintainers summit, but
> > > > given the number of people who seem to have an opinion and be
> > > > interested in dicussing it, would a session at LPC be a better
> > > > candidate ? I don't expect the maintainer summit to invite all
> > > > relevant experts from all subsystems, that would likely
> > > > overflow the room.
> > > > 
> > > > The downside of an LPC session is that it could easily turn
> > > > into a heated stage fight, and there are probably also quite a
> > > > few arguments that can't really be made in the open :-S
> > > 
> > > A separate LPC session for a subsystem or set of subsystems to
> > > explore local passthrough policy makes sense, but that is not the
> > > primary motivation for also requesting a Maintainer Summit topic
> > > slot. The primary motivation is discussing the provenance and
> > > navigation of cross-subsystem NAKs especially in an environment
> > > where the lines between net, mem, and storage are increasingly
> > > blurry at the device level.
> > 
> > Would there be enough space at the maintainers' summit for all the
> > relevant people to join the discussion ?
> 
> Who exactly would you consider the "relevant people" here?  It's been
> a wide-ranging conversation/thread :)

This is a bit of a trick question, since there seem to be three
separate but intertwined things here

   1. What to do about cross subsystem NAKs (as in how far does one
      subsystem have the ability to NAK something another does because
      they fear it will impact them ... passthrough being only one
      example).
   2. Industry education to help manufacturers making bad decisions
      about openness and APIs make better ones that actually benefit
      their business in the long run.
   3. Standards for open drivers (i.e. is passthrough always evil).

1. is definitely Maintainer Summit material. 2. was something the LF
used to help us with but seems to have foundered of late (I think on
the general assumption that CNCF gets it right, so we can stop pushing)
and 3. is definitely where Plumbers could host a wide ranging debate.

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 11:34:34AM -0400, Steven Rostedt wrote:
> On Fri, 26 Jul 2024 17:27:31 +0300 Laurent Pinchart wrote:
> 
> > I know this is a topic proposed for the maintainers summit, but given
> > the number of people who seem to have an opinion and be interested in
> > dicussing it, would a session at LPC be a better candidate ? I don't
> > expect the maintainer summit to invite all relevant experts from all
> > subsystems, that would likely overflow the room.
> > 
> > The downside of an LPC session is that it could easily turn into a
> > heated stage fight, and there are probably also quite a few arguments
> > that can't really be made in the open :-S
> 
> You can always submit a BoF. We have also had last minute BoFs made if
> there's rooms available.

We could also possibly expand the V4L2 vs. DRM discussion in the Complex
Camera MC to cover this, as there's an important overlap.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 12:49:20PM -0400, James Bottomley wrote:
> On Thu, 2024-07-25 at 22:31 +0300, Laurent Pinchart wrote:
> > On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > > [...]
> > > > What I get from the discussions I've followed or partcipated in
> > > > over the years is that the main worry of free software
> > > > communities is being forced to use closed-source userspace
> > > > components, whether that would be to make the device usable at
> > > > all, or to achieve decent level of performance or full feature
> > > > set. We've been through years of mostly closed-source GPU
> > > > support, of printer "windrivers", and quite a few other horrors.
> > > > The good news is that we've so far overcome lots (most) of those
> > > > challenges. Reverse engineering projects paid off, and so did
> > > > working hand-in-hand with industry actors in multiple ways
> > > > (both openly and behind the scenes). One could then legitimately
> > > > ask why we're still scared.
> > > 
> > > I don't think I am.  We're mostly fully capable of expounding at
> > > length on the business rationale for being open if the thing
> > > they're hiding isn't much of a differentiator anyway (or they're
> > > simply hiding it to try to retain some illusion of control), so we
> > > shouldn't have any fear of being able to make our case in language
> > > business people understand.
> > > 
> > > I also think this fear is partly a mindset problem on our part.  We
> > > came out of the real fight for openness and we do embrace things
> > > like a licence that forces open code (GPL) and symbols that
> > > discourage proprietary drivers (EXPORT_SYMBOL_GPL), so we've
> > > somewhat drunk the FSF coolaid that if we don't stand over
> > > manufacturers every second and force them they'll slide back to
> > > their old proprietary ways.  However, if you look at the entirely
> > > permissive ecosystem that grew up after we did (openstack, docker,
> > > kubernetes, etc.) they don't have any such fear and yet they still
> > > have large amounts of uncompelled openness and give back.
> > 
> > I don't think those are necessarily relevant examples,
> 
> Well they do show it is possible to get an open ecosystem based on the
> pull of business need instead of using a forcing function like the
> licence.

Yes, that's true. I still think it depends on the area, the market
players and the maturity of the technology, but licenses should
certainly not be regarded as being the only tool we can use.

> >  as far as device pass-through goes. Vendors have many times reverted
> > to proprietary ways, and they still do, at least in the areas of the
> > kernel I'm most active in.
> 
> I'm not going to argue that companies don't make bad business
> decisions: they definitely do.  And also there are huge reservoirs of
> proprietary mindset or fear of losing control in most of them which
> tend to drive these bad decisions.  I'm just saying we don't have to be
> all stick and no carrot.  If we can use the carrot to help them make
> better business decisions, and overcome the proprietary mindset that
> way, so much the better.
> 
> There are also valid reasons to do pass-through, like you don't want to
> invest in standardising the interface until you see that you have an
> actual market, which is how mixed standardised/pass through systems
> like DRM work (they allow innovation and experimentation, not all of
> which works).

This also matches what we routinely do in the kernel, where in-kernel
abstractions are usually developed when we have a handful of drivers
that need them. Standardizing interfaces with a single user is more
often than not a recipe for failure, not even mentioning the incurred
cost.

If the main motivation for pass-through is avoidance of the large costs
related to standardization when the market is in its infancy, I have no
big objection in principle. There could be other issues to take into
account though, that would make pass-through problematic for particular
cases. In any case, this wouldn't preclude documenting the whole device
API. When vendors want to keep part of the API secret and usable by
closed-source userspace, the discussion becomes quite different.

> >  I've seen first hand a large SoC vendor very close to opening a
> > significant part of their camera stack and changing their mind at the
> > last minute when they heard they could possibly merge their code
> > through a different subsystem with a pass-through blank cheque.
> 
> So is the learning here is that perhaps we should co-ordinate better?

I don't see how that could hurt :-) My experience with the pass-through
discussions so far is that many people in the kernel community are aware
of the overall issue, but most lack deep understanding of the
case-specific constraints. That's not really surprising as we can't be
expert in everything, but it easily leads to the wrong decisions being
made.

> > I'm willing to believe it can be different in other areas, which may
> > partly explain why different subsystems and different developers have
> > different biases and have trouble understand each other's point of
> > view.
> 
> I think that's true, due to different companies having different levels
> of proprietary mindset and other fears of not controlling their own
> destiny.  However, the basic fact remains that it will be in their best
> business interests to be more open if we can encourage it.

That's what we've been telling ISP vendors for 5 years, and it's
starting to show results. It's a long walk, but I agree it's the best
long term strategy. We however sometimes need to combine it with other
carrots or sticks along the way, in the shorter term.

> > > > I can't fully answer that question, but there are two points that
> > > > I think are relevant. Note that due to my background and
> > > > experience, this will be heavily biased towards consumer and
> > > > embedded hardware, not data centre-grade devices. Some
> > > > technologies from the latter however have a tendency to migrate
> > > > to the former over time, so the distinction isn't necessarily as
> > > > relevant as one may consider.
> > > > 
> > > > The first point is that hardware gets more complicated over time,
> > > > and in some markets there's also an increase in the number of
> > > > vendors and devices. There's a perceived (whether true or not)
> > > > danger that we won't be able to keep up with just reverse
> > > > engineering and a development model relying on hobyists. Getting
> > > > vendors involved is important if we want to scale.
> > > 
> > > Yes, but there are lots of not very useful complex devices being
> > > produced every day that fail to capture market share.  Not having
> > > reverse engineered drivers for them is no real loss.  If a device
> > > does gain market share, it gains a huge pool of users some of whom
> > > become interested in reverse engineering, so I think market forces
> > > actually work in our favour: we get reverse engineering mostly
> > > where the devices are actually interesting and capture market
> > > share.  It's self scaling.
> > 
> > I can't agree with that, sorry. Not only is the difficulty to
> > reverse-engineer some classes of devices increasing,
> 
> So it's a higher hill to climb: I didn't say it wasn't.  However for
> hugely popular devices there's still more likelihood of finding someone
> willing to climb it.  It's not a guarantee, just a probability, though.

Agreed.

> > but saying that only devices that make it to the top of the market
> > share chart are worth considering will leave many users on the side
> > of the road.
> 
> I'm afraid this is simple economics.  Suppose we had the budget to pay
> for reverse engineering, it's not going to cover everything, so we'd
> have to make choices and that would necessarily mean investing in
> devices that have utility to the greatest number of people.  So low
> market share devices would still be left out.

Sure, but my point was that we need to get vendors on board and not just
rely on reverse engineering if we want to scale.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 05:44:09PM +0200, Ricardo Ribalda Delgado wrote:
> On Fri, Jul 26, 2024 at 3:22 PM Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 10:03:36AM +0200, Ricardo Ribalda Delgado wrote:
> > > On Thu, Jul 25, 2024 at 9:32 PM Laurent Pinchart wrote:
> > > > On Wed, Jul 24, 2024 at 04:37:21PM -0400, James Bottomley wrote:
> > > > > On Wed, 2024-07-24 at 23:00 +0300, Laurent Pinchart wrote:
> > > > > [...]
> > > > > > What I get from the discussions I've followed or partcipated in over
> > > > > > the years is that the main worry of free software communities is
> > > > > > being forced to use closed-source userspace components, whether that
> > > > > > would be to make the device usable at all, or to achieve decent level
> > > > > > of performance or full feature set. We've been through years of
> > > > > > mostly closed-source GPU support, of printer "windrivers", and quite
> > > > > > a few other horrors. The good news is that we've so far overcome lots
> > > > > > (most) of those challenges. Reverse engineering projects paid off,
> > > > > > and so did working hand-in-hand with industry actors in multiple ways
> > > > > > (both openly and behind the scenes). One could then legitimately ask
> > > > > > why we're still scared.
> > > > >
> > > > > I don't think I am.  We're mostly fully capable of expounding at length
> > > > > on the business rationale for being open if the thing they're hiding
> > > > > isn't much of a differentiator anyway (or they're simply hiding it to
> > > > > try to retain some illusion of control), so we shouldn't have any fear
> > > > > of being able to make our case in language business people understand.
> > > > >
> > > > > I also think this fear is partly a mindset problem on our part.  We
> > > > > came out of the real fight for openness and we do embrace things like a
> > > > > licence that forces open code (GPL) and symbols that discourage
> > > > > proprietary drivers (EXPORT_SYMBOL_GPL), so we've somewhat drunk the
> > > > > FSF coolaid that if we don't stand over manufacturers every second and
> > > > > force them they'll slide back to their old proprietary ways.  However,
> > > > > if you look at the entirely permissive ecosystem that grew up after we
> > > > > did (openstack, docker, kubernetes, etc.) they don't have any such fear
> > > > > and yet they still have large amounts of uncompelled openness and give
> > > > > back.
> > > >
> > > > I don't think those are necessarily relevant examples, as far as device
> > > > pass-through goes. Vendors have many times reverted to proprietary ways,
> > > > and they still do, at least in the areas of the kernel I'm most active
> > > > in. I've seen first hand a large SoC vendor very close to opening a
> > > > significant part of their camera stack and changing their mind at the
> > > > last minute when they heard they could possibly merge their code through
> > > > a different subsystem with a pass-through blank cheque.
> > >
> > > Without knowing who that large SoC vendor is, and what they will be
> > > willing to open, it is difficult to know what opportunity has been
> > > lost.
> >
> > I would also like to discuss this in more details, but I can't do so
> > publicly :-( That's why I don't want to focus too much on this argument.
> >
> > > I would argue that if they have cancelled their open plans based
> > > on an hypothesis, their willingness to open was not that high.
> >
> > That's exactly what I previously described in this mail thread as areas
> > where cooperation with vendors is in its infancy. Convincing large
> > industry players is long term work, and it's very demotivating when you
> > think there's finally light at the end of the tunel, and plans gets
> > cancelled at the last minute. We're not giving up though.
> >
> > > It would be more healthy for the ecosystem, if those discussions were
> > > done more openly, at least all the core maintainers should be
> > > involved.
> >
> > Can you get all the camera vendors you work with to talk about all this
> > openly ? :-) We all know it's a difficult and long process.
> 
> We have pushed lots of vendors to contribute directly upstream. You
> can see a lot of code that has landed thanks to this effort.
> And I have tried very hard to get them involved in all the discussions upstream.

I'm not blaming your lack of effort, and I agree with your statement
that it would be more healthy for the ecosystem if those discussions
were done more openly. Few, if any, of the big vendors have taken part
in public discussions though, so it's not something we can easily
organize and make scale at the moment. We of course must keep trying,
and I think we're getting closer to a tipping point. The future will
tell us if I've been over optimistic for a change :-)

> > > > I'm willing to believe it can be different in other areas, which may
> > > > partly explain why different subsystems and different developers have
> > > > different biases and have trouble understand each other's point of view.
> > >
> > > It is not different in other areas, it is the same area. At the end of
> > > the day it is the same chip manufacturers, with the same legal teams.
> > > It is our attitude that is different.
> >
> > I don't agree with that. Not only are there different manufacturers,
> > there are also different teams, with different approaches. The world is
> > not uniform.
> >
> > > > > > I can't fully answer that question, but there are two points that I
> > > > > > think are relevant. Note that due to my background and experience,
> > > > > > this will be heavily biased towards consumer and embedded hardware,
> > > > > > not data centre-grade devices. Some technologies from the latter
> > > > > > however have a tendency to migrate to the former over time, so the
> > > > > > distinction isn't necessarily as relevant as one may consider.
> > > > > >
> > > > > > The first point is that hardware gets more complicated over time, and
> > > > > > in some markets there's also an increase in the number of vendors and
> > > > > > devices. There's a perceived (whether true or not) danger that we
> > > > > > won't be able to keep up with just reverse engineering and a
> > > > > > development model relying on hobyists. Getting vendors involved is
> > > > > > important if we want to scale.
> > > > >
> > > > > Yes, but there are lots of not very useful complex devices being
> > > > > produced every day that fail to capture market share.  Not having
> > > > > reverse engineered drivers for them is no real loss.  If a device does
> > > > > gain market share, it gains a huge pool of users some of whom become
> > > > > interested in reverse engineering, so I think market forces actually
> > > > > work in our favour: we get reverse engineering mostly where the devices
> > > > > are actually interesting and capture market share.  It's self scaling.
> > > >
> > > > I can't agree with that, sorry. Not only is the difficulty to
> > > > reverse-engineer some classes of devices increasing, but saying that
> > > > only devices that make it to the top of the market share chart are worth
> > > > considering will leave many users on the side of the road.
> > >
> > > Today we have left BILLIONS of users at the other side of the road.
> >
> > Because we're the only ones to be blamed, or more accuretely, the blame
> > is solely on me. Vendors are all saints who want the best for the world.
> > This argument doesn't hold. Please stop with this, I don't want to get
> > to name calling.
> 
> Nobody said that it was your fault. You and all the media community
> have invested thousands of hours of their personal time in Linux.

Let's conclude I've misinterpreted what you were trying to say then.

> It is healthy to make a retrospective of where we are now and if we
> are helping our users or not.
> 
> The reality is that we are reaching a very small percentage of users,
> and vendors prefer to work out of the tree than with us. Even vendors
> that traditionally have been very good Linux players have issues
> landing their code.
> 
> Can we try to figure out why it is happening? Can we try something
> new? Maybe inspired in other subsystems?

Analyzing the reasons certainly seems a good idea to me. I'm not sure if
we will agree on the conclusions, but I think we should still try. That
may be a good topic for the media summit in September, although without
vendors being present, we may have a hard time doing anything but
speculating.

> > > > > > Second, I think there's a fear of regression. For some categories of
> > > > > > devices, we have made slow but real progress to try and convince the
> > > > > > industry to be more open. This sometimes took a decade of work,
> > > > > > patiently building bridges and creating ecosystems brick by brick.
> > > > > > Some of those ecosystems are sturdy, some not so. Giving pass-through
> > > > > > a blank check will likely have very different effects in different
> > > > > > areas. I don't personally believe it will shatter everything, but I'm
> > > > > > convinced it carries risk in areas where cooperation with vendors is
> > > > > > in its infancy or is fragile for any other reason.
> > > > >
> > > > > I also think we're on the rise in this space.  Since most cloud
> > > > > workloads are on Linux, there's huge market pressure on most "found in
> > > > > the cloud" devices (like accelerators and GPUs) to have an easy to
> > > > > consume Linux story.  Nvidia is a case in point.  When it only cared
> > > > > about fast games on some other OS, we get shafted with a proprietary
> > > > > graphics drivers.  Now it's under pressure to be the number one AI
> > > > > accelerator provider for the cloud it's suddenly wondering about open
> > > > > source drivers to make adoption easier.
> > > >
> > > > I can't comment on Nvidia and their inference engines in particular. The
> > > > server market may be in a better position that the consumer and embedded
> > > > market, and if that's the case, I'm happy for the servers. That doesn't
> > > > solve the issues in other markets though.
> > > >
> > > > > > Finally, let's not forget that pass-through APIs are not an all or
> > > > > > nothing option. To cite that example only, DRM requires GPU drivers
> > > > > > to have an open-source userspace implementation to merge the kernel
> > > > > > driver, and the same subsystems strongly pushes for API
> > > > > > standardization for display controllers. We can set different rules
> > > > > > for different cases.
> > > > >
> > > > > I certainly think we can afford to experiment here, yes.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Fri, Jul 26, 2024 at 05:40:16PM +0200, Ricardo Ribalda Delgado wrote:
> On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> > > On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > > > James Bottomley wrote:
> > > > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > > >
> > > > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > > > positions?
> > > > > > > > >
> > > > > > > > > This thread has that discussion:
> > > > > > > > >
> > > > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > > >
> > > > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > > > consideration of fwctl for CXL.
> > > > > > > > >
> > > > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > > > wrapper.
> > > > > > > > >
> > > > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > > >
> > > > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > > > should they know the technical implications best, they should also have
> > > > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > > > different functions of a device.
> > > > > > > >
> > > > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > > > software to use all the device's features. That's something we have been
> > > > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > > > provide better and more open camera support without the world
> > > > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > > > users and the whole ecosystem in the short and long term.
> > > > > > >
> > > > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > > > >
> > > > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > > > conversation, but I think the technical discussion likely belongs to a
> > > > > > different mail thread.
> > > > > >
> > > > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > > > graphics cards.
> > > > > > >
> > > > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > > > each vendor implements unique features and usage patterns. DRM
> > > > > > > addresses this through vendor-specific IOCTLs
> > > > > > >
> > > > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > > > implementation, not comprehensive documentation.
> > > > > >
> > > > > > This point isn't technical and is more on-topic for this mail thread.
> > > > > >
> > > > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > > > text form. An open-source userspace implementation that covers the
> > > > > > feature set exposed by the driver is acceptable in place of
> > > > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > > > DRM drivers.
> > > > >
> > > > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > > > command buffers.
> > > > > Only the command buffer structure, and a mesa implementation using
> > > > > that command buffer to support the standard features is required.
> > > > >
> > > > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > > > be passed from userspace, they are typically formed in the driver from
> > > > > a strictly checked struct.
> > > >
> > > > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > > > and that mechanism is used in mainline drivers to pass camera ISP
> > > > parameters. They're not called "command buffers" but that's just a
> > > > difference in terminology. The technical means to pass command buffers
> > > > to the driver is thus there, I see no meaningful difference with DRM.
> > > > Where things can differ is in the contents of those buffers, and the
> > > > requirements for documentation or open userspace implementations, but
> > > > that's not a technical question.
> > >
> > > There are two things here:
> > >
> > > - The political/strategic/philosophical/religious aspect: The industry
> > > definitely prefers the strategic requirements imposed by DRM. In fact
> > > some vendors had some huge legal troubles when they had tried to
> > > follow v4l2 requirements.
> >
> > That's I'm willing to debate.
> >
> > > - The technical aspect: DRM is more mature when it comes to
> > > sending/receiving buffers to the hardware, and an ISP looks *much*
> > > more similar to an accel device or a GPU than a UVC camera.
> >
> > But this I don't agree with. I think we should forgo the technical
> > discussion and stop pretending that DRM is better for this use case from
> > a technical point of view, and focus on the other aspect of the
> > discussion. (We can of course reopen the technical discussion if new
> > concrete arguments emerge.)
> 
> I disagree. ISP devices are EXACTLY the same as accel devices.

I'm sorry, but this kind of argument makes me wonder if you know how
ISPs work :-(

> I suspect that you want to avoid them handled in DRM because then you
> prefer the v4l2 openness requirements.

I certainly prefer that, but I also think we would end up having the
exact same openness requirement with DRM. That's at least what I would
aim for.

I do also suspect that you want to handle ISPs in DRM only because you
believe it would give vendors a free pass-through card, and that the
technical arguments you brought up in the discussions are just a
distraction.

> There are no technical benefit
> from v4l2 for using ISPs:
> 
> ```
> Everyone agrees that the current V4L2 API is not very suitable for the
> current generation of ISPs: it is too cumbersome.
> ```
> https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil

And we've improved the situation since then, and landed/landing more ISP
drivers. There's further ongoing improvements pending.

> > > > > > > Our current approach of pushing back against vendors, instead of
> > > > > > > seeking compromise, has resulted in the vast majority of the market
> > > > > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > > > > with no options for utilizing their cameras outside of Android.
> > > > > > >
> > > > > > > DRM allows a hybrid model, where:
> > > > > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > > > > open-source stack.
> > > > > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > > > > proprietary features (e.g., automatic makeup) as closed source.
> > > > > >
> > > > > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > > > > control algorithms in userspace, as long as there's an open-source
> > > > > > implementation that exercises the same hardware features. A good analogy
> > > > >
> > > > > Is it really mandatory to have an open-source 3A algorithm? I thought
> > > > > defining the input and output from the algorithm was good enough.
> > > >
> > > > What really matters is documenting the ISP parameters with enough
> > > > details to allow for the implementation of open-source userspace code.
> > > > Once you have that, 3A is quite simple. You can refine it (especially
> > > > AWB) to great length, for instance using NPUs to compute parameters, and
> > > > there's absolutely no issue with such userspace implementations being
> > > > closed.
> > > >
> > > > In practice, some vendors prefer documenting the parameters by writing
> > > > an open-source userspace implementation, partly maybe because developers
> > > > are more familiar writing code than formal documentation. I would be
> > > > fine either way, as long as there's enough information to make use of
> > > > the ISP.
> > >
> > > Even with vendor passthrough there is still a need to provide a full
> > > open source implementation (probably based on libcamera).
> >
> > We have a different interpretation of "full" :-) I want to aim for
> > "full" to cover all the features exposed by the driver UAPI. There could
> > be some exceptions that we can discuss if there are compeling arguments,
> > but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.
> 
> Full means that I can use my camera using debian main and nothing more
> than that.  It is a pretty accepted description of what fully open
> means.

Thanks to the soft ISP support we've just landed in libcamera, and the
IPU6 ISYS driver merged in mainline, you will be able to use your camera
with Debian main as soon as it integrates recent enough versions of the
kernel and libcamera packages. I hope you don't consider that as full
support.

> > > So you will have enough information to use all the common
> > > functionality of a camera.
> > >
> > > > > AFAIK for some time there was no ipu3 open source algorithm, and the
> > > > > driver has been upstream.
> > > >
> > > > It sneaked in before we realized we had to enforce rules :-) That's
> > > > actually a good example, when we wrote open-source userspace support, we
> > > > realized that the level of documentation included in the IPU3 kernel
> > > > header was nowhere close to what was needed to make use of the device.
> > > >
> > > > > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > > > > free to ship closed-source implementations that include more
> > > > > > optimizations, as long as the open-source, less optimized implementation
> > > > > > covers the same GPU ISA, so that open-source developers can also work on
> > > > > > optimizing it.
> > > > >
> > > > > I believe a more accurate description is that in v4l2 is that we
> > > > > expect that all the registers, device architecture and behaviour to be
> > > > > documented and accessed with standard IOCTLs. Anything not documented
> > > > > cannot be accessed by userspace.
> > > > >
> > > > > In DRM their concern is that there is a fully open source
> > > > > implementation that the user can use. Vendors have custom IOCTLs and
> > > > > they can offer proprietary software for some use cases.
> > > >
> > > > Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> > > > vs. standard ioctls isn't very relevant to this discussion. I really
> > > > don't see how this would be about ioctls, it's about making the featured
> > > > exposed by the drivers, through whatever means a particular subsystem
> > > > allows, usable by open userspace.
> > > >
> > > > > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > > > > doesn't seem realistic to me. Both subsystems will have similar rules.
> > > > >
> > > > > DRM does indeed allow vendors to pass random command buffers and they
> > > > > will be sent to the hardware. We cannot do that in v4l2.
> > > >
> > > > You can pass a command buffer to a V4L2 device and have the driver send
> > > > it to the device firmware (ISPs using real command buffers usually run a
> > > > firmware). If you want the driver to be merged upstream, you have to
> > > > document the command buffer in enough details.
> > >
> > > Documenting with "enough details" is not enough. In V4L2, we have to
> > > deeply inspect every single buffer to make sure that it is not sending
> > > an unknown combination of command+arguments, or in other situations we
> > > construct the command buffer in the driver.
> > >
> > > > > I might be wrong, but GPU drivers do not deeply inspect the command
> > > > > buffers to make sure that they do not use any feature not covered by
> > > > > mesa.
> > > >
> > > > That's correct, but I don't think that's relevant. The GPU market has
> > > > GLSL and Vulkan. An open-source compliant implementation will end up
> > > > exercising a very very large part of the device ISA, command submission
> > > > mechanism and synchronization primitives, if not all of it. There's
> > > > little a vendor would keep under the hood and use in closed-source
> > > > userspace only. For cameras, there's no standard userspace API that
> > > > covers by design a very large part of what is the ISP equivalent of a
> > >
> > > Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...
> >
> > Those are not comparable to GLSL or Vulkan, they are much higher level.
> >
> > > If we are afraid of vendors providing "toy" implemententations to pass
> > > the openness requirements, we can add more features/tests to
> > > libcamera.
> > >
> > > And at the end of the day, there will be humans deciding if what a
> > > vendor has provided is good enough or not.
> > >
> > > > GPU ISA. Even "command buffers" are not a proper description of the
> > > > parameters the vast majority of ISPs consume.
> > >
> > > Modern ISPs are definitely going in the direction of "command buffers"
> >
> > Examples please, with technical details. I've seen this argument being
> > used for at least a year, with very little evidence. There are ISPs that
> > are more programmable than others, but in practice firmwares mostly
> > expose fixed functions, not ISA-level programmability.
> 
> You have not seen many programmable ISPs in v4l2 because they cannot
> be supported with v4l2.

I have not seen many programmable ISPs, period. I would like to see
concrete examples of those ISPs, with technical details of the hardware
and firmware, disregarding which kernel API we would use.

> Instead vendors have to create huge firmwares that limits the
> capability of the device.
> I do not want to look into those firmwares.
> 
> > > > > > > This approach would allow billions of users to access their hardware
> > > > > > > more securely and with in-tree driver support. Our current stubborn
> > > > > > > pursuit of an idealistic goal has already negatively impacted both
> > > > > > > users and the ecosystem.
> > > > > > >
> > > > > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > > > > Linux will remain a niche market for ISPs.
> > > > > > >
> > > > > > > If such a hybrid model goes against Linux goals, this is something
> > > > > > > that should be agreed upon by the whole community, so we have the same
> > > > > > > criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Greg KH]

On Sun, Jul 28, 2024 at 11:49:44AM -0400, James Bottomley wrote:
> On Sun, 2024-07-28 at 17:16 +0200, Greg KH wrote:
> > On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> > > Hi Dan,
> > > 
> > > On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > > > Laurent Pinchart wrote:
> > > > > I know this is a topic proposed for the maintainers summit, but
> > > > > given the number of people who seem to have an opinion and be
> > > > > interested in dicussing it, would a session at LPC be a better
> > > > > candidate ? I don't expect the maintainer summit to invite all
> > > > > relevant experts from all subsystems, that would likely
> > > > > overflow the room.
> > > > > 
> > > > > The downside of an LPC session is that it could easily turn
> > > > > into a heated stage fight, and there are probably also quite a
> > > > > few arguments that can't really be made in the open :-S
> > > > 
> > > > A separate LPC session for a subsystem or set of subsystems to
> > > > explore local passthrough policy makes sense, but that is not the
> > > > primary motivation for also requesting a Maintainer Summit topic
> > > > slot. The primary motivation is discussing the provenance and
> > > > navigation of cross-subsystem NAKs especially in an environment
> > > > where the lines between net, mem, and storage are increasingly
> > > > blurry at the device level.
> > > 
> > > Would there be enough space at the maintainers' summit for all the
> > > relevant people to join the discussion ?
> > 
> > Who exactly would you consider the "relevant people" here?  It's been
> > a wide-ranging conversation/thread :)
> 
> This is a bit of a trick question, since there seem to be three
> separate but intertwined things here
> 
>    1. What to do about cross subsystem NAKs (as in how far does one
>       subsystem have the ability to NAK something another does because
>       they fear it will impact them ... passthrough being only one
>       example).
>    2. Industry education to help manufacturers making bad decisions
>       about openness and APIs make better ones that actually benefit
>       their business in the long run.
>    3. Standards for open drivers (i.e. is passthrough always evil).
> 
> 1. is definitely Maintainer Summit material.

And to ask again, who do you should participate in this?

> 2. was something the LF used to help us with but seems to have
> foundered of late (I think on the general assumption that CNCF gets it
> right, so we can stop pushing)

Based on the number of meetings and trips I keep having with different
companies over the past years, 2. is not something that the LF is no
longer doing, as they fund me doing this all the time.  Including
visiting and educating your current employer about these very issues
recently :)

So don't fall into the "because it's not done in a loud way it must not
be happening" trap here please.

thanks,

greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Sun, Jul 28, 2024 at 1:24 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 05:40:50PM +0200, Ricardo Ribalda Delgado wrote:
> > On Fri, Jul 26, 2024 at 3:11 PM Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com> wrote:
> > >
> > > On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:
> > > > > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > > > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > > > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > > > >
> > > > > > > > <...>
> > > > > > > >
> > > > > > > > > It would be great to define what are the free software communities
> > > > > > > > > here. Distros and final users are also "free software communities" and
> > > > > > > > > they do not care about niche use cases covered by proprietary
> > > > > > > > > software.
> > > > > > > >
> > > > > > > > Are you certain about that?
> > > > > > >
> > > > > > > As a user, and as an open source Distro developer I have a small hint.
> > > > > > > But you could also ask users what they think about not being able to
> > > > > > > use their notebook's cameras. The last time that I could not use some
> > > > > > > basic hardware from a notebook with Linux was 20 years ago.
> > > > > >
> > > > > > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > > > > > with Linux, and even now, there is basic hardware in my current
> > > > > > laptop (HP docking station) that doesn't work reliably in Linux.
> > > > > >
> > > > > > > > > They only care (and should care) about normal workflows.
> > > > > > > >
> > > > > > > > What is a normal workflow?
> > > > > > > > Does it mean that if user bought something very expensive he
> > > > > > > > should not be able to use it with free software, because his
> > > > > > > > usage is different from yours?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > >
> > > > > > > It means that we should not block the standard usage for 99% of the
> > > > > > > population just because 1% of the users cannot do something fancy with
> > > > > > > their device.
> > > > > >
> > > > > > Right, the problem is that in some areas the statistics slightly different.
> > > > > > 99% population is blocked because 1% of the users don't need it and
> > > > > > don't think that it is "normal" flow.
> > > > > >
> > > > > > > Let me give you an example. When I buy a camera I want to be able to
> > > > > > > do Video Conferencing and take some static photos of documents. I do
> > > > > > > not care about: automatic makeup, AI generated background, unicorn
> > > > > > > filters, eyes recentering... But we need to give a way to vendors to
> > > > > > > implement those things closely, without the marketing differentiators,
> > > > > > > vendors have zero incentive to invest in Linux, and that affects all
> > > > > > > the population.
> > > > >
> > > > > I've seen these kind of examples being repeatedly given in discussions
> > > > > related to camera ISP support in Linux. They are very misleading. These
> > > > > are not the kind of features that are relevant for the device
> > > > > pass-through discussion these day. Those are high-level use cases
> > > > > implemented in userspace, and vendors can ship any closed-source
> > > > > binaries they want there. What I care about is the features exposed by
> > > > > the kernel to userspace API.
> > > >
> > > > The ISPs are gradually becoming programmable devices and they indeed
> > > > help during all of those examples.
> > >
> > > I'd like to see more technical information to substantiate this claim.
> > > So far what I've sometimes seen is ISPs that include programmable
> > > elements, but hiding those behind a firmware that exposes a fixed
> > > (configurable) pipeline. I've also heard of attempts to expose some of
> > > that programmability to the operating system, which were abandoned in
> > > the end due to lack usefulness.
> > >
> > > > Userspace needs to send/receive information from the ISP, and that is
> > > > exactly what vendors want to keep in the close.
> > >
> > > But that's exactly what we need to implement an open userspace ecosystem
> > > :-)
> > >
> > > > Describing how they implement those algorithms is a patent minefield
> > > > and their differentiating factor.
> > >
> > > Those are also arguments I've heard many times before. The
> > > differentiating factor for cameras today is mostly in userspace ISP
> > > control algorithms, and nobody is telling vendors they need to open all
> > > that.
> >
> > I disagree. The differentiating factor is what the ISP is capable of
> > doing and how they do it. Otherwise we would not see new ISPs in the
> > market.
>
> Hardware certainly evolves, but it's far from being the main
> differentiating factor in the markets and use cases you're usually
> referring to.
>
> > If you define the arguments passed to an ISP you are defining the
> > algorithm, and that is a trade secret and/or a patent violation.
>
> Are you confusing ISP processing blocks, sometimes referred to as
> algorithms, and ISP control algorithms ? There is absolutely no way to
> do anything with an ISP, not even the bare minimum, if you don't know
> what parameters to pass to it.

Any ISP released in the last few years has *hundreds of thousands* of
parameters.

We only modify hundreds of parameters during runtime. Those are the
ones we need to be documented.

If we enforce a "usable open camera stack", we will have the
documentation and the code needed to use the ISP.

Asking vendors to document *ALL* the parameters means describing how
they have implemented the internals of the ISP camera algorithms.


>
> > > When it comes to patents, we all know how software patents is a
> > > minefield, and hardware is also affected. I can't have much sympathy for
> > > this argument though, those patents mostly benefit the largest players
> > > in the market, and those are the ones who currently claim they can't
> > > open anything due to patents.
> >
> > Big players do not usually sue each other. The big problem is patent
> > trolls that "shoot at everything that moves".
> >
> > I dislike patents, but it is the world we have to live in. No vendor
> > is going to take our approach if they risk a multi million dollar
> > lawsuit.
>
> When was the last time anyone heard of big players pushing to reform the
> patent system ? At best there are initiatives such as OIN, which some
> large companies have supporting. It's still a workaround though.
>
> > > > > > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > > > > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > > > > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > > > > > >
> > > > > > > There are other other subsystems that allow vendor passthrough and
> > > > > > > their ecosystem has not collapsed.
> > > > > >
> > > > > > Yes, I completely agree with you on that.
> > > > > >
> > > > > > > Can we have some general guidance of what is acceptable? Can we define
> > > > > > > together the "normal workflow" and focus on a *full* open source
> > > > > > > implementation of that?
> > > > > >
> > > > > > I don't think that is possible to define "normal workflow". Requirement
> > > > > > to have open-source counterpart to everything exposed through UAPI is a
> > > > > > valid one. I'm all for that.
> > > > >
> > > > > That's my current opinion as well, as least when it comes to the kernel
> > > > > areas I mostly work with.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Sun, Jul 28, 2024 at 5:25 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 05:43:21PM +0200, Ricardo Ribalda Delgado wrote:
> > On Fri, Jul 26, 2024 at 4:22 PM Laurent Pinchart wrote:
> > > On Fri, Jul 26, 2024 at 10:11:06AM -0300, Jason Gunthorpe wrote:
> > > > On Fri, Jul 26, 2024 at 03:49:49PM +0300, Laurent Pinchart wrote:
> > > >
> > > > > What is not an option exactly in my description above ? We have multiple
> > > > > V4L2 drivers for ISPs. They receive ISP parameters from userspace
> > > > > through a data buffer. It's not allowed to be opaque, but it doesn't
> > > > > prevent vendor closed-source userspace implementations with additional
> > > > > *camera* features, as long as the *hardware* features are available to
> > > > > everybody.
> > > >
> > > > How far do you take opaque?
> > > >
> > > > In mlx5 we pass command buffers from user to kernel to HW and the
> > > > kernel does only a little checking.
> > >
> > > I won't comment on the mlx5 case in particular, I don't have enough
> > > knowledge to do so.
> > >
> > > When it comes to validation of commands by the kernel, at the very least
> > > the kernel driver needs to be able to guarantee safety. For instance,
> > > that means that any command that can result in DMA operations need to be
> > > validated (e.g. verifying buffer addresses and sizes) and/or executed
> > > partly by the driver (e.g. mapping a buffer through an IOMMU), depending
> > > on hardware constraints.
> > >
> > > > There is a 12kloc file describing the layout of alot of commands:
> > > > include/linux/mlx5/mlx5_ifc.h
> > > >
> > > > There is an open PDF describing in detail some subset of this:
> > > > https://network.nvidia.com/files/doc-2020/ethernet-adapters-programming-manual.pdf
> > > >
> > > > There are in-kernel implementations driving most of those commands.
> > >
> > > For camera ISPs, most of the "commands" wouldn't be driven by the
> > > kernel. I don't have expectations when it comes to what commands the
> > > kernel exercises directly, I think that highly depends on the device
> > > type.
> > >
> > > > Other commands are only issued by userspace, and we have open source
> > > > DPDK, rdma-core and UCX implementations driving them.
> > > >
> > > > ie, this is really quite good as far as a device providing open source
> > > > solutions goes.
> > > >
> > > > However, no doubt there is more FW capability and commands than even
> > > > this vast amount documents - so lets guess that propritary code is
> > > > using this interface with unknown commands too.
> > > >
> > > > From a camera perspective would you be OK with that? Let's guess that
> > > > 90% of use cases are covered with fully open code. Do we also need to
> > > > forcefully close the door to an imagined 10% of proprietary cases just
> > > > to be sure open always wins?
> > >
> > > For command buffers interpreted by a firmware, it would be extremely
> > > difficult, if not impossible, to ensure that nothing undocumented is
> > > possible to access from userspace. I think we have two issues here, one
> > > of them is to define what we required to be openly accessible, and the
> > > other to avoid vendors cheating by claiming to be more open than they
> > > actually are.
> > >
> > > I believe the latter can't be solved technically. At the end of the day
> > > it's a matter of trust, and the only option I can see is to build that
> > > trust with the vendors, and to make it clear that breaches of trust will
> > > have consequences. A vendor that deliberately lies would end up on my
> > > personal backlist for as long as they don't implement structural
> > > solutions to be a better player in the ecosystem. What is acceptable as
> > > consequences is for the communities to decide. We have previously banned
> > > people or organizations from contributing to the kernel for certain
> > > periods of time (the University of Minnesota case comes to my mind for
> > > instance), and other options can be considered too.
> > >
> > > As for the first issue, I think it's a difficult one as it is very
> > > difficult to quantify the features covered by open implementations, as
> > > well as their importance. You could have a 99% open command set where
> > > the 1% would impact open implementations extremely negatively, the same
> > > way you could have a 50% open command set where the other 50% isn't of
> > > any use to anyone but the vendor (for instance used to debug the
> > > firmware implementation).
> >
> > It is not that difficult. You just have to define what an acceptable
> > implementation is. Eg
> >
> > - Sharpness at specific light.
> > - Time of convergence for the Auto algorithms
> > - Noise ratio
>
> Those may just be bad examples, but I think they showcase how little
> these discussions are based on technical expertise and facts. Just

Could you perhaps provide better examples of what are good objective
characteristics of an open implementation. ?

Those characteristics have been directly extracted from actual OS
certification requirements for Camera stacks.

> singling out one example, the convergence time is driven by the
> userspace implementation of the ISP control algorithms. That is *not*
> something we want to force vendors to disclose. It isn't related to the
> ISP parameters and the UAPI the driver exposes to userspace.

I am not talking about the driver UAPI. I am saying that the Open
implementation provided by vendors to land their code must not be a
"toy" implementation, but have good characteristics.

You said yourself that with the "new rules" we also demand a reference
open ISP control algorithm.

Would you be OK with an open algorithm that cannot converge in a
reasonable time? I hope not.


>
> > We could even use the SoftISP implementation as reference. Is this ISP
> > working better or worse than SoftISP?
> >
> > > For cameras, the feature set can I believe be expressed in terms of ISP
> > > processing blocks that are usable by open implementations, as well as in
> > > terms of the type of usage of those features. For instance, is it
> > > acceptable that a vendor documents how to use 2D noise reduction but
> > > makes 3D (temporal) noise reduction available only to close-source
> > > userspace ? My personal answer is no. I want to aim for very close to
> > > 100% of the features, and certainly 100% of the "large" features. I can
> > > close my eyes on features that are very very niche, but what is niche
> > > today may not be tomorrow. For instance, if a camera ISP implements a
> > > feature used only for very specific camera sensors targetted at
> > > autonomous driving in a new generation of research prototypes that only
> > > one company has access to, I'll be inclined to ignore that. That
> > > technology may however become much more mainstream 5 years later.
> >
> > We can update our requirements in 5 years. Nothing is written in stone.
>
> That at least we agree on :-)
>
> > Also it is much easier to reverse engineer an open driver, with an
> > open userspace and a closed userspace than a close driver with a
> > closed userspace.
>
> Closed kernel drivers are a GPL violation. Do you have any example, or
> do you mean out-of-tree drivers ? I don't think out-of-tree vs. mainline

I am not a lawyer, so I am not capable of discussing violations of any License.

I mean close and out-of-tree. Just two name two well known drivers
with a binary blob:
- fgrlx
- nvidia proprietary.

> drivers would make much difference when it comes to reverse engineering,
> if the mainline driver is just a pass-through driver.

If you have an usable open source camera stack, reverse engineering
the userspace closed implementation should not be a big challenge.

>
> > > The other aspect is the type of usage of the features. For camera ISPs
> > > again, some parameters will be computed through a tuning process, and
> > > will be fixed for the lifetime of the camera. I want those parameters to
> > > be documented, to enable users to tune cameras for their own use cases.
> > > This is less important in some market segments, such as laptops for
> > > instance, but is crucial for embedded devices. This is an area where
> > > I've previously been called unreasonable, and I don't think that's fair.
> > > The tuned-and-immutable parameters are not plentiful, as most of the
> > > tuning results in data that needs to be combined with runtime
> > > information to compute ISP parameters, so the "this is for tuning only"
> > > argument doesn't hold as much as one may think. For real immutable
> > > parameters, a large number of them are related to image processing steps
> > > that are very common and found in most ISPs, such as lens shading
> > > correction or geometric distorsion correction. I don't see why we should
> > > let vendors keep those closed.
> >
> > We don't have enough leverage for that kind of requirement.
>
> I very much disagree with that.
>
> > To be fair, we do not ask touchscreen manufacturers to document their
> > calibration files. Nor any other calibration file in the kernel.
> >
> > The calibration file for me is like a firmware blob.
>
> It may be for you, but I don't think it is an accurate description for
> the rest of the industry.

Can you define who the industry is here?

In my job I have visibility on who and how they create that
calibration file, and the release cycle is identical to the firmware.

>
> > > I'm sorry that this discussion is turning into something very
> > > camera-centric, but that's the relevant area I know best. I hope that
> > > discussing problems related to device pass-through in different areas in
> > > the open will help build a wider shared understanding of the problems,
> > > and hopefully help designing solutions by better taking into account the
> > > various aspects of the issues.
> > >
> > > > Does closing the door have to come at the cost of a technically clean
> > > > solution? Doing validation in the kernel to enforce an ideological
> > > > position would severely degrade mlx5, and would probably never really
> > > > be robust.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Ricardo Ribalda Delgado]

On Sun, Jul 28, 2024 at 7:18 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> On Fri, Jul 26, 2024 at 05:40:16PM +0200, Ricardo Ribalda Delgado wrote:
> > On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart wrote:
> > > On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> > > > On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > > > > James Bottomley wrote:
> > > > > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > > > >
> > > > > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > > > > positions?
> > > > > > > > > >
> > > > > > > > > > This thread has that discussion:
> > > > > > > > > >
> > > > > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > > > >
> > > > > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > > > > consideration of fwctl for CXL.
> > > > > > > > > >
> > > > > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > > > > wrapper.
> > > > > > > > > >
> > > > > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > > > >
> > > > > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > > > > should they know the technical implications best, they should also have
> > > > > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > > > > different functions of a device.
> > > > > > > > >
> > > > > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > > > > software to use all the device's features. That's something we have been
> > > > > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > > > > provide better and more open camera support without the world
> > > > > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > > > > users and the whole ecosystem in the short and long term.
> > > > > > > >
> > > > > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > > > > >
> > > > > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > > > > conversation, but I think the technical discussion likely belongs to a
> > > > > > > different mail thread.
> > > > > > >
> > > > > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > > > > graphics cards.
> > > > > > > >
> > > > > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > > > > each vendor implements unique features and usage patterns. DRM
> > > > > > > > addresses this through vendor-specific IOCTLs
> > > > > > > >
> > > > > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > > > > implementation, not comprehensive documentation.
> > > > > > >
> > > > > > > This point isn't technical and is more on-topic for this mail thread.
> > > > > > >
> > > > > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > > > > text form. An open-source userspace implementation that covers the
> > > > > > > feature set exposed by the driver is acceptable in place of
> > > > > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > > > > DRM drivers.
> > > > > >
> > > > > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > > > > command buffers.
> > > > > > Only the command buffer structure, and a mesa implementation using
> > > > > > that command buffer to support the standard features is required.
> > > > > >
> > > > > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > > > > be passed from userspace, they are typically formed in the driver from
> > > > > > a strictly checked struct.
> > > > >
> > > > > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > > > > and that mechanism is used in mainline drivers to pass camera ISP
> > > > > parameters. They're not called "command buffers" but that's just a
> > > > > difference in terminology. The technical means to pass command buffers
> > > > > to the driver is thus there, I see no meaningful difference with DRM.
> > > > > Where things can differ is in the contents of those buffers, and the
> > > > > requirements for documentation or open userspace implementations, but
> > > > > that's not a technical question.
> > > >
> > > > There are two things here:
> > > >
> > > > - The political/strategic/philosophical/religious aspect: The industry
> > > > definitely prefers the strategic requirements imposed by DRM. In fact
> > > > some vendors had some huge legal troubles when they had tried to
> > > > follow v4l2 requirements.
> > >
> > > That's I'm willing to debate.
> > >
> > > > - The technical aspect: DRM is more mature when it comes to
> > > > sending/receiving buffers to the hardware, and an ISP looks *much*
> > > > more similar to an accel device or a GPU than a UVC camera.
> > >
> > > But this I don't agree with. I think we should forgo the technical
> > > discussion and stop pretending that DRM is better for this use case from
> > > a technical point of view, and focus on the other aspect of the
> > > discussion. (We can of course reopen the technical discussion if new
> > > concrete arguments emerge.)
> >
> > I disagree. ISP devices are EXACTLY the same as accel devices.
>
> I'm sorry, but this kind of argument makes me wonder if you know how
> ISPs work :-(

Laurent, you know for a while and I am not new in this area. Let's
keep the discussion civil. Can we?

>
> > I suspect that you want to avoid them handled in DRM because then you
> > prefer the v4l2 openness requirements.
>
> I certainly prefer that, but I also think we would end up having the
> exact same openness requirement with DRM. That's at least what I would
> aim for.
>
> I do also suspect that you want to handle ISPs in DRM only because you
> believe it would give vendors a free pass-through card, and that the
> technical arguments you brought up in the discussions are just a
> distraction.

Let's put it this way:

If tomorrow a vendor sends to the ML a camera stack that looks like this:

- V4L2 driver for capture
- DRM driver for ISP
- Libcamera implementation with open algorithms
- Everything released with an Open Licence
- The camera stack is production ready: Meaning that it could be
included without any vendor blob in a distro like ChromeOS.

Would you NACK it because the vendor did not release the calibration software?
Would you NACK it because the vendor could also release another userspace?

That NACK will benefit the users? in what timeframe? Another decade?

Does the rest of the kernel community agree on that? I thought that we
were doing Linux for world domination :)


>
> > There are no technical benefit
> > from v4l2 for using ISPs:
> >
> > ```
> > Everyone agrees that the current V4L2 API is not very suitable for the
> > current generation of ISPs: it is too cumbersome.
> > ```
> > https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil
>
> And we've improved the situation since then, and landed/landing more ISP
> drivers. There's further ongoing improvements pending.

We are indeed improving, but I still agree with the original statement.

>
> > > > > > > > Our current approach of pushing back against vendors, instead of
> > > > > > > > seeking compromise, has resulted in the vast majority of the market
> > > > > > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > > > > > with no options for utilizing their cameras outside of Android.
> > > > > > > >
> > > > > > > > DRM allows a hybrid model, where:
> > > > > > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > > > > > open-source stack.
> > > > > > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > > > > > proprietary features (e.g., automatic makeup) as closed source.
> > > > > > >
> > > > > > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > > > > > control algorithms in userspace, as long as there's an open-source
> > > > > > > implementation that exercises the same hardware features. A good analogy
> > > > > >
> > > > > > Is it really mandatory to have an open-source 3A algorithm? I thought
> > > > > > defining the input and output from the algorithm was good enough.
> > > > >
> > > > > What really matters is documenting the ISP parameters with enough
> > > > > details to allow for the implementation of open-source userspace code.
> > > > > Once you have that, 3A is quite simple. You can refine it (especially
> > > > > AWB) to great length, for instance using NPUs to compute parameters, and
> > > > > there's absolutely no issue with such userspace implementations being
> > > > > closed.
> > > > >
> > > > > In practice, some vendors prefer documenting the parameters by writing
> > > > > an open-source userspace implementation, partly maybe because developers
> > > > > are more familiar writing code than formal documentation. I would be
> > > > > fine either way, as long as there's enough information to make use of
> > > > > the ISP.
> > > >
> > > > Even with vendor passthrough there is still a need to provide a full
> > > > open source implementation (probably based on libcamera).
> > >
> > > We have a different interpretation of "full" :-) I want to aim for
> > > "full" to cover all the features exposed by the driver UAPI. There could
> > > be some exceptions that we can discuss if there are compeling arguments,
> > > but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.
> >
> > Full means that I can use my camera using debian main and nothing more
> > than that.  It is a pretty accepted description of what fully open
> > means.
>
> Thanks to the soft ISP support we've just landed in libcamera, and the
> IPU6 ISYS driver merged in mainline, you will be able to use your camera
> with Debian main as soon as it integrates recent enough versions of the
> kernel and libcamera packages. I hope you don't consider that as full
> support.

It is not "full support" because it is missing a lot of the basic
characteristics..

Why don't you want to describe what are the objective characteristics
that an Open Camera stack should have?

Asking vendors to fulfill a set of characteristics is more realistic
than asking them to document the ISP hardware, the ISP algos,  the
calibration files, the calibration tools...


>
> > > > So you will have enough information to use all the common
> > > > functionality of a camera.
> > > >
> > > > > > AFAIK for some time there was no ipu3 open source algorithm, and the
> > > > > > driver has been upstream.
> > > > >
> > > > > It sneaked in before we realized we had to enforce rules :-) That's
> > > > > actually a good example, when we wrote open-source userspace support, we
> > > > > realized that the level of documentation included in the IPU3 kernel
> > > > > header was nowhere close to what was needed to make use of the device.
> > > > >
> > > > > > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > > > > > free to ship closed-source implementations that include more
> > > > > > > optimizations, as long as the open-source, less optimized implementation
> > > > > > > covers the same GPU ISA, so that open-source developers can also work on
> > > > > > > optimizing it.
> > > > > >
> > > > > > I believe a more accurate description is that in v4l2 is that we
> > > > > > expect that all the registers, device architecture and behaviour to be
> > > > > > documented and accessed with standard IOCTLs. Anything not documented
> > > > > > cannot be accessed by userspace.
> > > > > >
> > > > > > In DRM their concern is that there is a fully open source
> > > > > > implementation that the user can use. Vendors have custom IOCTLs and
> > > > > > they can offer proprietary software for some use cases.
> > > > >
> > > > > Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> > > > > vs. standard ioctls isn't very relevant to this discussion. I really
> > > > > don't see how this would be about ioctls, it's about making the featured
> > > > > exposed by the drivers, through whatever means a particular subsystem
> > > > > allows, usable by open userspace.
> > > > >
> > > > > > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > > > > > doesn't seem realistic to me. Both subsystems will have similar rules.
> > > > > >
> > > > > > DRM does indeed allow vendors to pass random command buffers and they
> > > > > > will be sent to the hardware. We cannot do that in v4l2.
> > > > >
> > > > > You can pass a command buffer to a V4L2 device and have the driver send
> > > > > it to the device firmware (ISPs using real command buffers usually run a
> > > > > firmware). If you want the driver to be merged upstream, you have to
> > > > > document the command buffer in enough details.
> > > >
> > > > Documenting with "enough details" is not enough. In V4L2, we have to
> > > > deeply inspect every single buffer to make sure that it is not sending
> > > > an unknown combination of command+arguments, or in other situations we
> > > > construct the command buffer in the driver.
> > > >
> > > > > > I might be wrong, but GPU drivers do not deeply inspect the command
> > > > > > buffers to make sure that they do not use any feature not covered by
> > > > > > mesa.
> > > > >
> > > > > That's correct, but I don't think that's relevant. The GPU market has
> > > > > GLSL and Vulkan. An open-source compliant implementation will end up
> > > > > exercising a very very large part of the device ISA, command submission
> > > > > mechanism and synchronization primitives, if not all of it. There's
> > > > > little a vendor would keep under the hood and use in closed-source
> > > > > userspace only. For cameras, there's no standard userspace API that
> > > > > covers by design a very large part of what is the ISP equivalent of a
> > > >
> > > > Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...
> > >
> > > Those are not comparable to GLSL or Vulkan, they are much higher level.
> > >
> > > > If we are afraid of vendors providing "toy" implemententations to pass
> > > > the openness requirements, we can add more features/tests to
> > > > libcamera.
> > > >
> > > > And at the end of the day, there will be humans deciding if what a
> > > > vendor has provided is good enough or not.
> > > >
> > > > > GPU ISA. Even "command buffers" are not a proper description of the
> > > > > parameters the vast majority of ISPs consume.
> > > >
> > > > Modern ISPs are definitely going in the direction of "command buffers"
> > >
> > > Examples please, with technical details. I've seen this argument being
> > > used for at least a year, with very little evidence. There are ISPs that
> > > are more programmable than others, but in practice firmwares mostly
> > > expose fixed functions, not ISA-level programmability.
> >
> > You have not seen many programmable ISPs in v4l2 because they cannot
> > be supported with v4l2.
>
> I have not seen many programmable ISPs, period. I would like to see
> concrete examples of those ISPs, with technical details of the hardware
> and firmware, disregarding which kernel API we would use.

Just to name one with some public details, Synopsys has a family of devices:
https://www.synopsys.com/dw/ipdir.php?ds=ev7x-vision-processors
You can do standard ISP + AI with one core, pretty cool.

Most of the time, Camera ISPs are nothing more than DSPs plus a
firmware with the computer vision algos.

If there is no framework to use a programmable ISP, vendors will keep
putting everything on the firmware and exposing parameters.

>
> > Instead vendors have to create huge firmwares that limits the
> > capability of the device.
> > I do not want to look into those firmwares.
> >
> > > > > > > > This approach would allow billions of users to access their hardware
> > > > > > > > more securely and with in-tree driver support. Our current stubborn
> > > > > > > > pursuit of an idealistic goal has already negatively impacted both
> > > > > > > > users and the ecosystem.
> > > > > > > >
> > > > > > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > > > > > Linux will remain a niche market for ISPs.
> > > > > > > >
> > > > > > > > If such a hybrid model goes against Linux goals, this is something
> > > > > > > > that should be agreed upon by the whole community, so we have the same
> > > > > > > > criteria for all subsystems.
>
> --
> Regards,
>
> Laurent Pinchart



-- 
Ricardo Ribalda

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Ricardo,

On Mon, Jul 29, 2024 at 11:58:43AM +0200, Ricardo Ribalda Delgado wrote:
> On Sun, Jul 28, 2024 at 7:18 PM Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 05:40:16PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart wrote:
> > > > On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > > > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > > > > > James Bottomley wrote:
> > > > > > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > > > > > positions?
> > > > > > > > > > >
> > > > > > > > > > > This thread has that discussion:
> > > > > > > > > > >
> > > > > > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > > > > >
> > > > > > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > > > > > consideration of fwctl for CXL.
> > > > > > > > > > >
> > > > > > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > > > > > wrapper.
> > > > > > > > > > >
> > > > > > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > > > > >
> > > > > > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > > > > > should they know the technical implications best, they should also have
> > > > > > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > > > > > different functions of a device.
> > > > > > > > > >
> > > > > > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > > > > > software to use all the device's features. That's something we have been
> > > > > > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > > > > > provide better and more open camera support without the world
> > > > > > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > > > > > users and the whole ecosystem in the short and long term.
> > > > > > > > >
> > > > > > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > > > > > >
> > > > > > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > > > > > conversation, but I think the technical discussion likely belongs to a
> > > > > > > > different mail thread.
> > > > > > > >
> > > > > > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > > > > > graphics cards.
> > > > > > > > >
> > > > > > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > > > > > each vendor implements unique features and usage patterns. DRM
> > > > > > > > > addresses this through vendor-specific IOCTLs
> > > > > > > > >
> > > > > > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > > > > > implementation, not comprehensive documentation.
> > > > > > > >
> > > > > > > > This point isn't technical and is more on-topic for this mail thread.
> > > > > > > >
> > > > > > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > > > > > text form. An open-source userspace implementation that covers the
> > > > > > > > feature set exposed by the driver is acceptable in place of
> > > > > > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > > > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > > > > > DRM drivers.
> > > > > > >
> > > > > > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > > > > > command buffers.
> > > > > > > Only the command buffer structure, and a mesa implementation using
> > > > > > > that command buffer to support the standard features is required.
> > > > > > >
> > > > > > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > > > > > be passed from userspace, they are typically formed in the driver from
> > > > > > > a strictly checked struct.
> > > > > >
> > > > > > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > > > > > and that mechanism is used in mainline drivers to pass camera ISP
> > > > > > parameters. They're not called "command buffers" but that's just a
> > > > > > difference in terminology. The technical means to pass command buffers
> > > > > > to the driver is thus there, I see no meaningful difference with DRM.
> > > > > > Where things can differ is in the contents of those buffers, and the
> > > > > > requirements for documentation or open userspace implementations, but
> > > > > > that's not a technical question.
> > > > >
> > > > > There are two things here:
> > > > >
> > > > > - The political/strategic/philosophical/religious aspect: The industry
> > > > > definitely prefers the strategic requirements imposed by DRM. In fact
> > > > > some vendors had some huge legal troubles when they had tried to
> > > > > follow v4l2 requirements.
> > > >
> > > > That's I'm willing to debate.
> > > >
> > > > > - The technical aspect: DRM is more mature when it comes to
> > > > > sending/receiving buffers to the hardware, and an ISP looks *much*
> > > > > more similar to an accel device or a GPU than a UVC camera.
> > > >
> > > > But this I don't agree with. I think we should forgo the technical
> > > > discussion and stop pretending that DRM is better for this use case from
> > > > a technical point of view, and focus on the other aspect of the
> > > > discussion. (We can of course reopen the technical discussion if new
> > > > concrete arguments emerge.)
> > >
> > > I disagree. ISP devices are EXACTLY the same as accel devices.
> >
> > I'm sorry, but this kind of argument makes me wonder if you know how
> > ISPs work :-(
> 
> Laurent, you know for a while and I am not new in this area. Let's
> keep the discussion civil. Can we?

That's what puzzles me. I've known you for a while and know all the
expertise you've accumulated in different technical areas related to
cameras. Despite that, we seem to speak different languages, our world
views are complete mismatches. I wonder where the disconnection is.

> > > I suspect that you want to avoid them handled in DRM because then you
> > > prefer the v4l2 openness requirements.
> >
> > I certainly prefer that, but I also think we would end up having the
> > exact same openness requirement with DRM. That's at least what I would
> > aim for.
> >
> > I do also suspect that you want to handle ISPs in DRM only because you
> > believe it would give vendors a free pass-through card, and that the
> > technical arguments you brought up in the discussions are just a
> > distraction.
> 
> Let's put it this way:
> 
> If tomorrow a vendor sends to the ML a camera stack that looks like this:
> 
> - V4L2 driver for capture

No issue with that.

> - DRM driver for ISP

No issue if there are compeling *technical* arguments to do so.

> - Libcamera implementation with open algorithms
> - Everything released with an Open Licence
> - The camera stack is production ready: Meaning that it could be
> included without any vendor blob in a distro like ChromeOS.

Obviously no issues with this either.

> Would you NACK it because the vendor did not release the calibration software?

I've never asked for the calibration/tuning software to be made
open-source or even publicly available, so no nack for that.

> Would you NACK it because the vendor could also release another userspace?

We've made it clear since the beginning in libcamera that closed
implementation of ISP control algorithms are totally fine as an
alternate implementation, so no issue either.

> That NACK will benefit the users? in what timeframe? Another decade?
> 
> Does the rest of the kernel community agree on that? I thought that we
> were doing Linux for world domination :)
> 
> > > There are no technical benefit
> > > from v4l2 for using ISPs:
> > >
> > > ```
> > > Everyone agrees that the current V4L2 API is not very suitable for the
> > > current generation of ISPs: it is too cumbersome.
> > > ```
> > > https://linuxtv.org/news.php?entry=2022-11-14-1.hverkuil
> >
> > And we've improved the situation since then, and landed/landing more ISP
> > drivers. There's further ongoing improvements pending.
> 
> We are indeed improving, but I still agree with the original statement.
> 
> > > > > > > > > Our current approach of pushing back against vendors, instead of
> > > > > > > > > seeking compromise, has resulted in the vast majority of the market
> > > > > > > > > (99% if not more) relying on out-of-tree drivers. This leaves users
> > > > > > > > > with no options for utilizing their cameras outside of Android.
> > > > > > > > >
> > > > > > > > > DRM allows a hybrid model, where:
> > > > > > > > > - Open Source Foundation: Standard use cases are covered by a fully
> > > > > > > > > open-source stack.
> > > > > > > > > - Vendor Differentiation: Vendors retain the freedom to implement
> > > > > > > > > proprietary features (e.g., automatic makeup) as closed source.
> > > > > > > >
> > > > > > > > V4L2 does as well, you can implement all kind of closed-source ISP
> > > > > > > > control algorithms in userspace, as long as there's an open-source
> > > > > > > > implementation that exercises the same hardware features. A good analogy
> > > > > > >
> > > > > > > Is it really mandatory to have an open-source 3A algorithm? I thought
> > > > > > > defining the input and output from the algorithm was good enough.
> > > > > >
> > > > > > What really matters is documenting the ISP parameters with enough
> > > > > > details to allow for the implementation of open-source userspace code.
> > > > > > Once you have that, 3A is quite simple. You can refine it (especially
> > > > > > AWB) to great length, for instance using NPUs to compute parameters, and
> > > > > > there's absolutely no issue with such userspace implementations being
> > > > > > closed.
> > > > > >
> > > > > > In practice, some vendors prefer documenting the parameters by writing
> > > > > > an open-source userspace implementation, partly maybe because developers
> > > > > > are more familiar writing code than formal documentation. I would be
> > > > > > fine either way, as long as there's enough information to make use of
> > > > > > the ISP.
> > > > >
> > > > > Even with vendor passthrough there is still a need to provide a full
> > > > > open source implementation (probably based on libcamera).
> > > >
> > > > We have a different interpretation of "full" :-) I want to aim for
> > > > "full" to cover all the features exposed by the driver UAPI. There could
> > > > be some exceptions that we can discuss if there are compeling arguments,
> > > > but that would result in a 99% coverage, not a 20%, 50% or 80% coverage.
> > >
> > > Full means that I can use my camera using debian main and nothing more
> > > than that.  It is a pretty accepted description of what fully open
> > > means.
> >
> > Thanks to the soft ISP support we've just landed in libcamera, and the
> > IPU6 ISYS driver merged in mainline, you will be able to use your camera
> > with Debian main as soon as it integrates recent enough versions of the
> > kernel and libcamera packages. I hope you don't consider that as full
> > support.
> 
> It is not "full support" because it is missing a lot of the basic
> characteristics..
> 
> Why don't you want to describe what are the objective characteristics
> that an Open Camera stack should have?

Because that is difficult to do, and may not lead to a very useful
outcome. I'm willing to brainstorm it though, but I don't have a
ready-made answer at the moment.

> Asking vendors to fulfill a set of characteristics is more realistic
> than asking them to document the ISP hardware, the ISP algos,  the
> calibration files, the calibration tools...
> 
> > > > > So you will have enough information to use all the common
> > > > > functionality of a camera.
> > > > >
> > > > > > > AFAIK for some time there was no ipu3 open source algorithm, and the
> > > > > > > driver has been upstream.
> > > > > >
> > > > > > It sneaked in before we realized we had to enforce rules :-) That's
> > > > > > actually a good example, when we wrote open-source userspace support, we
> > > > > > realized that the level of documentation included in the IPU3 kernel
> > > > > > header was nowhere close to what was needed to make use of the device.
> > > > > >
> > > > > > > > for people less familiar with ISPs is shader compilers, GPU vendors are
> > > > > > > > free to ship closed-source implementations that include more
> > > > > > > > optimizations, as long as the open-source, less optimized implementation
> > > > > > > > covers the same GPU ISA, so that open-source developers can also work on
> > > > > > > > optimizing it.
> > > > > > >
> > > > > > > I believe a more accurate description is that in v4l2 is that we
> > > > > > > expect that all the registers, device architecture and behaviour to be
> > > > > > > documented and accessed with standard IOCTLs. Anything not documented
> > > > > > > cannot be accessed by userspace.
> > > > > > >
> > > > > > > In DRM their concern is that there is a fully open source
> > > > > > > implementation that the user can use. Vendors have custom IOCTLs and
> > > > > > > they can offer proprietary software for some use cases.
> > > > > >
> > > > > > Custom ioctls are not closed secrets in DRM, so comparing custom ioctls
> > > > > > vs. standard ioctls isn't very relevant to this discussion. I really
> > > > > > don't see how this would be about ioctls, it's about making the featured
> > > > > > exposed by the drivers, through whatever means a particular subsystem
> > > > > > allows, usable by open userspace.
> > > > > >
> > > > > > > > Thinking that DRM would offer a free pass-through path compared to V4L2
> > > > > > > > doesn't seem realistic to me. Both subsystems will have similar rules.
> > > > > > >
> > > > > > > DRM does indeed allow vendors to pass random command buffers and they
> > > > > > > will be sent to the hardware. We cannot do that in v4l2.
> > > > > >
> > > > > > You can pass a command buffer to a V4L2 device and have the driver send
> > > > > > it to the device firmware (ISPs using real command buffers usually run a
> > > > > > firmware). If you want the driver to be merged upstream, you have to
> > > > > > document the command buffer in enough details.
> > > > >
> > > > > Documenting with "enough details" is not enough. In V4L2, we have to
> > > > > deeply inspect every single buffer to make sure that it is not sending
> > > > > an unknown combination of command+arguments, or in other situations we
> > > > > construct the command buffer in the driver.
> > > > >
> > > > > > > I might be wrong, but GPU drivers do not deeply inspect the command
> > > > > > > buffers to make sure that they do not use any feature not covered by
> > > > > > > mesa.
> > > > > >
> > > > > > That's correct, but I don't think that's relevant. The GPU market has
> > > > > > GLSL and Vulkan. An open-source compliant implementation will end up
> > > > > > exercising a very very large part of the device ISA, command submission
> > > > > > mechanism and synchronization primitives, if not all of it. There's
> > > > > > little a vendor would keep under the hood and use in closed-source
> > > > > > userspace only. For cameras, there's no standard userspace API that
> > > > > > covers by design a very large part of what is the ISP equivalent of a
> > > > >
> > > > > Libcamera supports Camera HAL3, gstreamer, v4l2, pipewire...
> > > >
> > > > Those are not comparable to GLSL or Vulkan, they are much higher level.
> > > >
> > > > > If we are afraid of vendors providing "toy" implemententations to pass
> > > > > the openness requirements, we can add more features/tests to
> > > > > libcamera.
> > > > >
> > > > > And at the end of the day, there will be humans deciding if what a
> > > > > vendor has provided is good enough or not.
> > > > >
> > > > > > GPU ISA. Even "command buffers" are not a proper description of the
> > > > > > parameters the vast majority of ISPs consume.
> > > > >
> > > > > Modern ISPs are definitely going in the direction of "command buffers"
> > > >
> > > > Examples please, with technical details. I've seen this argument being
> > > > used for at least a year, with very little evidence. There are ISPs that
> > > > are more programmable than others, but in practice firmwares mostly
> > > > expose fixed functions, not ISA-level programmability.
> > >
> > > You have not seen many programmable ISPs in v4l2 because they cannot
> > > be supported with v4l2.
> >
> > I have not seen many programmable ISPs, period. I would like to see
> > concrete examples of those ISPs, with technical details of the hardware
> > and firmware, disregarding which kernel API we would use.
> 
> Just to name one with some public details, Synopsys has a family of devices:
> https://www.synopsys.com/dw/ipdir.php?ds=ev7x-vision-processors
> You can do standard ISP + AI with one core, pretty cool.

It's a vision processor, not a camera ISP. Those are different beasts.

> Most of the time, Camera ISPs are nothing more than DSPs plus a
> firmware with the computer vision algos.

Not for camera ISPs, no.

> If there is no framework to use a programmable ISP, vendors will keep
> putting everything on the firmware and exposing parameters.

For vision processors that's not even the case, vendors use a full
pass-through implementation, with fully closed compilers. It's a
completely different story.

> > > Instead vendors have to create huge firmwares that limits the
> > > capability of the device.
> > > I do not want to look into those firmwares.
> > >
> > > > > > > > > This approach would allow billions of users to access their hardware
> > > > > > > > > more securely and with in-tree driver support. Our current stubborn
> > > > > > > > > pursuit of an idealistic goal has already negatively impacted both
> > > > > > > > > users and the ecosystem.
> > > > > > > > >
> > > > > > > > > The late wins, in my opinion, cannot scale to the consumer market, and
> > > > > > > > > Linux will remain a niche market for ISPs.
> > > > > > > > >
> > > > > > > > > If such a hybrid model goes against Linux goals, this is something
> > > > > > > > > that should be agreed upon by the whole community, so we have the same
> > > > > > > > > criteria for all subsystems.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Mon, Jul 29, 2024 at 11:56:45AM +0200, Ricardo Ribalda Delgado wrote:
> On Sun, Jul 28, 2024 at 1:24 PM Laurent Pinchart wrote:
> > On Fri, Jul 26, 2024 at 05:40:50PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Fri, Jul 26, 2024 at 3:11 PM Laurent Pinchart wrote:
> > > > On Fri, Jul 26, 2024 at 10:02:27AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Thu, Jul 25, 2024 at 9:44 PM Laurent Pinchart wrote:
> > > > > > On Thu, Jul 25, 2024 at 04:20:35PM +0300, Leon Romanovsky wrote:
> > > > > > > On Thu, Jul 25, 2024 at 03:02:13PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > On Thu, Jul 25, 2024 at 2:23 PM Leon Romanovsky wrote:
> > > > > > > > > On Thu, Jul 25, 2024 at 11:26:38AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > > > On Wed, Jul 24, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > > > > >
> > > > > > > > > <...>
> > > > > > > > >
> > > > > > > > > > It would be great to define what are the free software communities
> > > > > > > > > > here. Distros and final users are also "free software communities" and
> > > > > > > > > > they do not care about niche use cases covered by proprietary
> > > > > > > > > > software.
> > > > > > > > >
> > > > > > > > > Are you certain about that?
> > > > > > > >
> > > > > > > > As a user, and as an open source Distro developer I have a small hint.
> > > > > > > > But you could also ask users what they think about not being able to
> > > > > > > > use their notebook's cameras. The last time that I could not use some
> > > > > > > > basic hardware from a notebook with Linux was 20 years ago.
> > > > > > >
> > > > > > > Lucky you, I still have consumer hardware (speaker) that doesn't work
> > > > > > > with Linux, and even now, there is basic hardware in my current
> > > > > > > laptop (HP docking station) that doesn't work reliably in Linux.
> > > > > > >
> > > > > > > > > > They only care (and should care) about normal workflows.
> > > > > > > > >
> > > > > > > > > What is a normal workflow?
> > > > > > > > > Does it mean that if user bought something very expensive he
> > > > > > > > > should not be able to use it with free software, because his
> > > > > > > > > usage is different from yours?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > >
> > > > > > > > It means that we should not block the standard usage for 99% of the
> > > > > > > > population just because 1% of the users cannot do something fancy with
> > > > > > > > their device.
> > > > > > >
> > > > > > > Right, the problem is that in some areas the statistics slightly different.
> > > > > > > 99% population is blocked because 1% of the users don't need it and
> > > > > > > don't think that it is "normal" flow.
> > > > > > >
> > > > > > > > Let me give you an example. When I buy a camera I want to be able to
> > > > > > > > do Video Conferencing and take some static photos of documents. I do
> > > > > > > > not care about: automatic makeup, AI generated background, unicorn
> > > > > > > > filters, eyes recentering... But we need to give a way to vendors to
> > > > > > > > implement those things closely, without the marketing differentiators,
> > > > > > > > vendors have zero incentive to invest in Linux, and that affects all
> > > > > > > > the population.
> > > > > >
> > > > > > I've seen these kind of examples being repeatedly given in discussions
> > > > > > related to camera ISP support in Linux. They are very misleading. These
> > > > > > are not the kind of features that are relevant for the device
> > > > > > pass-through discussion these day. Those are high-level use cases
> > > > > > implemented in userspace, and vendors can ship any closed-source
> > > > > > binaries they want there. What I care about is the features exposed by
> > > > > > the kernel to userspace API.
> > > > >
> > > > > The ISPs are gradually becoming programmable devices and they indeed
> > > > > help during all of those examples.
> > > >
> > > > I'd like to see more technical information to substantiate this claim.
> > > > So far what I've sometimes seen is ISPs that include programmable
> > > > elements, but hiding those behind a firmware that exposes a fixed
> > > > (configurable) pipeline. I've also heard of attempts to expose some of
> > > > that programmability to the operating system, which were abandoned in
> > > > the end due to lack usefulness.
> > > >
> > > > > Userspace needs to send/receive information from the ISP, and that is
> > > > > exactly what vendors want to keep in the close.
> > > >
> > > > But that's exactly what we need to implement an open userspace ecosystem
> > > > :-)
> > > >
> > > > > Describing how they implement those algorithms is a patent minefield
> > > > > and their differentiating factor.
> > > >
> > > > Those are also arguments I've heard many times before. The
> > > > differentiating factor for cameras today is mostly in userspace ISP
> > > > control algorithms, and nobody is telling vendors they need to open all
> > > > that.
> > >
> > > I disagree. The differentiating factor is what the ISP is capable of
> > > doing and how they do it. Otherwise we would not see new ISPs in the
> > > market.
> >
> > Hardware certainly evolves, but it's far from being the main
> > differentiating factor in the markets and use cases you're usually
> > referring to.
> >
> > > If you define the arguments passed to an ISP you are defining the
> > > algorithm, and that is a trade secret and/or a patent violation.
> >
> > Are you confusing ISP processing blocks, sometimes referred to as
> > algorithms, and ISP control algorithms ? There is absolutely no way to
> > do anything with an ISP, not even the bare minimum, if you don't know
> > what parameters to pass to it.
> 
> Any ISP released in the last few years has *hundreds of thousands* of
> parameters.

Could you substantiate that claim ? That doesn't match what I've seen
(unless perhaps you count each entry in LSC tables or large tone mapping
LUTs as separate parameters).

> We only modify hundreds of parameters during runtime. Those are the
> ones we need to be documented.
> 
> If we enforce a "usable open camera stack", we will have the
> documentation and the code needed to use the ISP.
> 
> Asking vendors to document *ALL* the parameters means describing how
> they have implemented the internals of the ISP camera algorithms.
> 
> > > > When it comes to patents, we all know how software patents is a
> > > > minefield, and hardware is also affected. I can't have much sympathy for
> > > > this argument though, those patents mostly benefit the largest players
> > > > in the market, and those are the ones who currently claim they can't
> > > > open anything due to patents.
> > >
> > > Big players do not usually sue each other. The big problem is patent
> > > trolls that "shoot at everything that moves".
> > >
> > > I dislike patents, but it is the world we have to live in. No vendor
> > > is going to take our approach if they risk a multi million dollar
> > > lawsuit.
> >
> > When was the last time anyone heard of big players pushing to reform the
> > patent system ? At best there are initiatives such as OIN, which some
> > large companies have supporting. It's still a workaround though.
> >
> > > > > > > > This challenge seems to be solved for GPUs. I am using my AMD GPU
> > > > > > > > freely and my nephew can install the amdgpu-pro proprietary user space
> > > > > > > > driver to play duke nukem (or whatever kids play now) at 2000 fps.
> > > > > > > >
> > > > > > > > There are other other subsystems that allow vendor passthrough and
> > > > > > > > their ecosystem has not collapsed.
> > > > > > >
> > > > > > > Yes, I completely agree with you on that.
> > > > > > >
> > > > > > > > Can we have some general guidance of what is acceptable? Can we define
> > > > > > > > together the "normal workflow" and focus on a *full* open source
> > > > > > > > implementation of that?
> > > > > > >
> > > > > > > I don't think that is possible to define "normal workflow". Requirement
> > > > > > > to have open-source counterpart to everything exposed through UAPI is a
> > > > > > > valid one. I'm all for that.
> > > > > >
> > > > > > That's my current opinion as well, as least when it comes to the kernel
> > > > > > areas I mostly work with.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

> Enter the fwctl proposal [1]. From the CXL subsystem perspective it
> looks like a long-term solution to the problem of managing expectations
> between hardware vendors and mainline subsystems. It disclaims support
> for the fast-path (data-plane) and is targeted at the long tail of
> slow-path (config/debug plane) device-specific operations that are often
> uninteresting to mainline. It sets expectations that the device must
> advertise the effect of all commands so that the kernel can deploy
> reasonable Kernel Lockdown policy, or otherwise require CAP_SYS_RAWIO
> for commands that may affect user-data. It sets common expectations for
> device designers, distribution maintainers, and kernel developers. It is
> complimentary to the Linux-command path for operations that need deeper
> kernel coordination.

I'm reasonably on board with the basic justification for the
fwctl proposal (and it may solve some long term challenges for us), but
I'm concerned by ABI guarantees.  In particularly what happens when
we get decision wrong and expose something via fwctl that we later
have more general kernel support for.

This was triggered by Dave Jiang's (perhaps unintended) use of Patrol Scrub
as a test case for the CXL FWCTL RFC.
https://lore.kernel.org/linux-cxl/20240729130528.0000139b@Huawei.com/T/#t
I'm using this specific example here, but I think it is a more general
question.

By exposing the Get / Set feature controls, a lot of effective user space
ABI is added (some of it setting a taint).

Scrub control is an interesting example, because there is an active
proposal to extend EDAC to cover this and similar RAS related control
features, something we are going to be discussing at LPC.
https://lore.kernel.org/linux-cxl/20240726160556.2079-1-shiju.jose@huawei.com/
One of the key bits of feedback we've had on that series is that it
should be integrated with EDAC.  Part of the reason being need to get
appropriate RAS expert review. Something fwctl won't naturally get.

If we expose that particular Feature via Set Feature we may run into
future problems.  It is probably possible to make the driver stateless
so any interference from a userspace program using fwctl is not fatal
- in this case userspace code should probably be safe to state changes
anyway. We know about this clash today, so could easily block fwctl
from exposing this feature, but it is illustrative of a wider problem.

We will get some decisions about what should be exposed via fwctl wrong
in the long term, even if they are correct at time of initial decision.
So how do we cope with that?

1) Make no guarantees on ABI for taint causing operations.
   So we can block this FWCTL in a kernel if EDAC / ras control is in place
   for the same feature.  I'm fine with this but it's not obviously
   a correct thing to do!
2) Allow the footgun. Keep the fwctl interface and harden the other kernel
   support against state changes that result. If userspace code breaks,
   then tough luck.  (Another form of ABI break, perhaps comprehended by
   existing proposed FWCTL rules).
3) We are stuck for ever with not supporting anything via other interfaces
   that would break if fwctl was in use.  Ouch.
Note that I think this only matters for the Set path as Get side shouldn't
have side effects and is fine to expose without synchronization with
a clear statement that values read are a snapshot only.

So before I'd be happy with fwctl in CXL, I'd want a very clear policy
decision on this that isn't going leave us in a mess long term.

We could say it can only be used for features we have 'opted' in +
vendor defined features, but I'm not sure that helps.  If a vendor
defines a feature for generation A, and does what we want them to by
proposing a spec addition they use in generation B, we would want a
path to single upstream interface for both generations.  So I don't
think restricting this to particular classes of command helps us.

Jonathan

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Borislav Petkov]

On Mon, Jul 29, 2024 at 01:45:12PM +0100, Jonathan Cameron wrote:
> One of the key bits of feedback we've had on that series is that it
> should be integrated with EDAC.  Part of the reason being need to get
> appropriate RAS expert review.

If you mean me with that, my only question back then was: if you're going to
integrate it somewhere and instead of defining something completely new - you
can simply reuse what's there. That's why I suggested EDAC.

IOW, the question becomes, why should it be a completely new thing and not
part of EDAC?

That's all.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Fri, Jul 26, 2024 at 07:33:34PM +0200, Daniel Vetter wrote:

> What gets vendors involved is a successful project that drives revenue,
> where they have a clear need for a seat at the table to make sure the good
> times for them continue. Clear rules what it takes to get that seat is in
> my experience really the driving force for private discussions with
> vendors, and from that pov the most important thing I've ever done for the
> open gpu stack is this little documentation section:

Yeah, +1 on this, we can't forget the important role of the end
customer. In my view a successful project drving revenue is one that
has customer pull attached to it. If the market isn't also pushing for
open in the same direction the chances of success drop dramatically.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Fri, Jul 26, 2024 at 05:22:17PM +0300, Laurent Pinchart wrote:

> I believe the latter can't be solved technically. At the end of the day
> it's a matter of trust, and the only option I can see is to build that
> trust with the vendors, and to make it clear that breaches of trust will
> have consequences. A vendor that deliberately lies would end up on my
> personal backlist for as long as they don't implement structural
> solutions to be a better player in the ecosystem.

FWIW this is largely my thinking with fwctl as well.

> As for the first issue, I think it's a difficult one as it is very
> difficult to quantify the features covered by open implementations, as
> well as their importance. You could have a 99% open command set where
> the 1% would impact open implementations extremely negatively, the same
> way you could have a 50% open command set where the other 50% isn't of
> any use to anyone but the vendor (for instance used to debug the
> firmware implementation).

I think it is likely cameras, there are lots and lots of use cases and
scenarios. If you focus on a specific use case you need everything for
that case. Another view would be how many scenarios and users does the
open stack cover.

> I'm sorry that this discussion is turning into something very
> camera-centric, but that's the relevant area I know best. I hope that
> discussing problems related to device pass-through in different areas in
> the open will help build a wider shared understanding of the problems,
> and hopefully help designing solutions by better taking into account the
> various aspects of the issues.

To be clear I wouldn't imagine fwctl as relating much to cameras,
maybe you'd use fwctl to get diagnostics out of the camera fw or
something, but it shouldn't touch the image path topics here.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jonathan Cameron]

On Mon, 29 Jul 2024 15:38:39 +0200
Borislav Petkov <bp@alien8.de> wrote:

> On Mon, Jul 29, 2024 at 01:45:12PM +0100, Jonathan Cameron wrote:
> > One of the key bits of feedback we've had on that series is that it
> > should be integrated with EDAC.  Part of the reason being need to get
> > appropriate RAS expert review.  
> 
> If you mean me with that, my only question back then was: if you're going to
> integrate it somewhere and instead of defining something completely new - you
> can simply reuse what's there. That's why I suggested EDAC.

Ah fair enough. I'd taken stronger meaning from what you said than
you intended. Thanks for the clarification.

> 
> IOW, the question becomes, why should it be a completely new thing and not
> part of EDAC?

So that particular feedback perhaps doesn't apply here.

I still have a concern with things ending up in fwctl that
are later generalized and how that process can happen.

Jonathan

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jakub Kicinski]

On Sun, 28 Jul 2024 11:49:44 -0400 James Bottomley wrote:
> cross subsystem NAKs

Could y'all please stop saying "cross subsystem NAKs"..
It makes it sound like networking is nacking an addition to RDMA 
or storage. The problem is that nVidia insists on making their
proprietary gateway a "misc driver" usable in all subsystems.

If they want to add something at the top level, all affected
subsystems should have a say.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Mon, Jul 29, 2024 at 03:29:43PM +0100, Jonathan Cameron wrote:
> On Mon, 29 Jul 2024 15:38:39 +0200
> Borislav Petkov <bp@alien8.de> wrote:
> 
> > On Mon, Jul 29, 2024 at 01:45:12PM +0100, Jonathan Cameron wrote:
> > > One of the key bits of feedback we've had on that series is that it
> > > should be integrated with EDAC.  Part of the reason being need to get
> > > appropriate RAS expert review.  
> > 
> > If you mean me with that, my only question back then was: if you're going to
> > integrate it somewhere and instead of defining something completely new - you
> > can simply reuse what's there. That's why I suggested EDAC.
> 
> Ah fair enough. I'd taken stronger meaning from what you said than
> you intended. Thanks for the clarification.
> 
> > 
> > IOW, the question becomes, why should it be a completely new thing and not
> > part of EDAC?
> 
> So that particular feedback perhaps doesn't apply here.
> 
> I still have a concern with things ending up in fwctl that
> are later generalized and how that process can happen.

My intention with fwctl is that it should never technically inhibit
generalization. Someone should be able to come and implement a
concurrent kernel subsystem to operate the generalized thing. The
documentation attempts to explain this position.

I don't know anything about CXL here, but broadly you should very
thoughtfully put things into fwctl that are single-instance and become
"captured" by it, because this would impede a kernel mediated resource
sharing in future.

Like continuous memory scrubbing and EDAC is not really fwctl since it
is part of the main mission of a memory device. However evaluating the
memory to measure current ECC error rate for data collection and
debugging would be appropriate for fwctl.

If the HW can't share the units that are doing this ECC work then
ensuring fwctl is optional and secondary would be the best
option. Turn off in-kernel use of the scrubber unit then you can use
that unit for debugging. (as an example)

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Greg KH]

On Mon, Jul 29, 2024 at 07:56:07AM -0700, Jakub Kicinski wrote:
> On Sun, 28 Jul 2024 11:49:44 -0400 James Bottomley wrote:
> > cross subsystem NAKs
> 
> Could y'all please stop saying "cross subsystem NAKs"..
> It makes it sound like networking is nacking an addition to RDMA 
> or storage. The problem is that nVidia insists on making their
> proprietary gateway a "misc driver" usable in all subsystems.
> 
> If they want to add something at the top level, all affected
> subsystems should have a say.
> 

Also, the misc driver maintainer (i.e. me), explicitly asked for the
review by the networking maintainer here.  It wasn't a drive-by-nak at
all.

greg k-h

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Mon, Jul 29, 2024 at 07:56:07AM -0700, Jakub Kicinski wrote:
> On Sun, 28 Jul 2024 11:49:44 -0400 James Bottomley wrote:
> > cross subsystem NAKs
> 
> Could y'all please stop saying "cross subsystem NAKs"..
> It makes it sound like networking is nacking an addition to RDMA 
> or storage. 

Well, that is exactly what is happening. You are NAKing stuff we need
to operate RDMA, DPDK, and other parts of the device.

> The problem is that nVidia insists on making their proprietary
> gateway a "misc driver" usable in all subsystems.

Sigh, open code is not "proprietary", we've been over this.

fwctl manages the whole device, it is not tied to a subsystem and it
is not "usable" by any specific subsystem. This reflects the design of
HW/FW that is obviously not split up into per-Linux-subsystem
blocks. It is technically appropriate that is global and not somehow
weirdly split by subsystem.

I'm not going to backdoor generic full device fw access through RDMA
into the kernel.

> If they want to add something at the top level, all affected
> subsystems should have a say.

A say is fine, but it is not like netdev is suffering any real
technical harm to justify a full blown NAK fight like this.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Mon, Jul 29, 2024 at 01:45:12PM +0100, Jonathan Cameron wrote:

> If we expose that particular Feature via Set Feature we may run into
> future problems.  It is probably possible to make the driver stateless
> so any interference from a userspace program using fwctl is not fatal
> - in this case userspace code should probably be safe to state changes
> anyway. We know about this clash today, so could easily block fwctl
> from exposing this feature, but it is illustrative of a wider problem.

> We will get some decisions about what should be exposed via fwctl wrong
> in the long term, even if they are correct at time of initial decision.
> So how do we cope with that?
> 
> 1) Make no guarantees on ABI for taint causing operations.
>    So we can block this FWCTL in a kernel if EDAC / ras control is in place
>    for the same feature.  I'm fine with this but it's not obviously
>    a correct thing to do!

Maybe, I think that is a bit suboptimal.

> 2) Allow the footgun. Keep the fwctl interface and harden the other kernel
>    support against state changes that result. If userspace code breaks,
>    then tough luck.  (Another form of ABI break, perhaps comprehended by
>    existing proposed FWCTL rules).

This one is certainly closer to being in line with the fwctl doc as
written, but I'd say fwctl should be unable to hijack a scrubber block
from the kernel in the first place.

> 3) We are stuck for ever with not supporting anything via other interfaces
>    that would break if fwctl was in use.  Ouch.

Definately not this option.

> Note that I think this only matters for the Set path as Get side shouldn't
> have side effects and is fine to expose without synchronization with
> a clear statement that values read are a snapshot only.

Yes, that makes sense.

> We could say it can only be used for features we have 'opted' in +
> vendor defined features, but I'm not sure that helps.  If a vendor
> defines a feature for generation A, and does what we want them to by
> proposing a spec addition they use in generation B, we would want a
> path to single upstream interface for both generations.  So I don't
> think restricting this to particular classes of command helps us.

My expectation for fwctl was that it would own things that are
reasonably sharable by the kernel and userspace.

As an example, instead of a turning on a feature dynamically at run
time, you'd want to instead tell the FW that on next reboot that
feature will be forced on.

Another take would be things that are clearly contained to fwctl
multi-instance features where fwctl gets its own private thing that
cannot disturb the kernel.

I'm really not familiar with cxl to give any comment here - but
dynamically control the single global scrubber unit seems like a poor
fit to me.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Jason Gunthorpe wrote:
[..]
> > We could say it can only be used for features we have 'opted' in +
> > vendor defined features, but I'm not sure that helps.  If a vendor
> > defines a feature for generation A, and does what we want them to by
> > proposing a spec addition they use in generation B, we would want a
> > path to single upstream interface for both generations.  So I don't
> > think restricting this to particular classes of command helps us.
> 
> My expectation for fwctl was that it would own things that are
> reasonably sharable by the kernel and userspace.
> 
> As an example, instead of a turning on a feature dynamically at run
> time, you'd want to instead tell the FW that on next reboot that
> feature will be forced on.
> 
> Another take would be things that are clearly contained to fwctl
> multi-instance features where fwctl gets its own private thing that
> cannot disturb the kernel.
> 
> I'm really not familiar with cxl to give any comment here - but
> dynamically control the single global scrubber unit seems like a poor
> fit to me.

Right, one of the mistakes from NVDIMM that was corrected for CXL was to
explicitly remove the passthrough capability for global state machine
controls like scrubbing.

Many of the "Immediate Configuration Change" CXL commands fall into this
bucket of things that may want to have a kernel-managed global view
rather than let userspace and the kernel get into fights about the
configuration. So, I think it is reasonable to say that scrub has a
kernel interface that goes through EDAC and not fwctl.

For the "anonymous" "Features" that advertise an "Immediate
Configuration Change" effect those need CAP_SYS_RAWIO at a minimum,
possibly a kernel taint, and/or compile time option to block them. Maybe
that encourages more "Configuration Change after Reset" Set Feature
capabilities which carry less risk of confusing a running kernel.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Daniel Vetter]

On Mon, Jul 29, 2024 at 03:37:10PM -0700, Dan Williams wrote:
> Jason Gunthorpe wrote:
> [..]
> > > We could say it can only be used for features we have 'opted' in +
> > > vendor defined features, but I'm not sure that helps.  If a vendor
> > > defines a feature for generation A, and does what we want them to by
> > > proposing a spec addition they use in generation B, we would want a
> > > path to single upstream interface for both generations.  So I don't
> > > think restricting this to particular classes of command helps us.
> > 
> > My expectation for fwctl was that it would own things that are
> > reasonably sharable by the kernel and userspace.
> > 
> > As an example, instead of a turning on a feature dynamically at run
> > time, you'd want to instead tell the FW that on next reboot that
> > feature will be forced on.
> > 
> > Another take would be things that are clearly contained to fwctl
> > multi-instance features where fwctl gets its own private thing that
> > cannot disturb the kernel.
> > 
> > I'm really not familiar with cxl to give any comment here - but
> > dynamically control the single global scrubber unit seems like a poor
> > fit to me.
> 
> Right, one of the mistakes from NVDIMM that was corrected for CXL was to
> explicitly remove the passthrough capability for global state machine
> controls like scrubbing.
> 
> Many of the "Immediate Configuration Change" CXL commands fall into this
> bucket of things that may want to have a kernel-managed global view
> rather than let userspace and the kernel get into fights about the
> configuration. So, I think it is reasonable to say that scrub has a
> kernel interface that goes through EDAC and not fwctl.
> 
> For the "anonymous" "Features" that advertise an "Immediate
> Configuration Change" effect those need CAP_SYS_RAWIO at a minimum,
> possibly a kernel taint, and/or compile time option to block them. Maybe
> that encourages more "Configuration Change after Reset" Set Feature
> capabilities which carry less risk of confusing a running kernel.

I think a solid consensus on the topics above would be really useful for
gpu/accel too. We're still busy with more pressing community/ecosystem
building needs, but gpu fw has become rather complex and it's not
stopping. And there's random other devices attached too nowadays, so fwctl
makes a ton of sense.

But for me the more important stuff would be some clear guidelines like
what should be in other more across-devices subsystems like edac (or other
ras features), what should be in functional subsystems like netdev, rdma,
gpu/accel, ... whatever else, and what should be exposed through some
special purpose subsystems like hwmon.

And then also what the access control guidelines should be around tainting
and and premission checks.

We've got plenty of experience in enforcing such a community contract with
vendors, but the hard part is creating a clear and ideally concise
documentation page I can just point vendors at as the ground truth.

I'm also not super worried about the uapi breakage scenarios. We have
plenty of experience in drm with sometimes horrendous hacks to keep
existing userspace and real-world use-cases going, while still being able
to move the subsystem forward and standardize more stuff as it starts to
make sense. But the "here's the goal" documentation, maybe with the
occasional update when a new subsystem like edac shows up, is imo the hard
part and would be really, really useful.

Cheers, Sima
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Borislav Petkov]

On Mon, Jul 29, 2024 at 11:58:50AM -0300, Jason Gunthorpe wrote:
> Like continuous memory scrubbing and EDAC is not really fwctl since it
> is part of the main mission of a memory device. However evaluating the
> memory to measure current ECC error rate for data collection and
> debugging would be appropriate for fwctl.

fwctl?

What's that? Something to do with "firmware"? If so, how does this thing have
anything to do with RAS stuff?

We have rasdaemon for that where all the RAS controlling and evaluation goes
into.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Sakari Ailus]

Hi Ricardo, Laurent,

On Mon, Jul 29, 2024 at 11:58:43AM +0200, Ricardo Ribalda Delgado wrote:
> On Sun, Jul 28, 2024 at 7:18 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> >
> > On Fri, Jul 26, 2024 at 05:40:16PM +0200, Ricardo Ribalda Delgado wrote:
> > > On Fri, Jul 26, 2024 at 12:59 PM Laurent Pinchart wrote:
> > > > On Fri, Jul 26, 2024 at 10:04:33AM +0200, Ricardo Ribalda Delgado wrote:
> > > > > On Thu, Jul 25, 2024 at 10:02 PM Laurent Pinchart wrote:
> > > > > > On Mon, Jul 22, 2024 at 01:56:11PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > On Mon, Jul 22, 2024 at 1:18 PM Laurent Pinchart wrote:
> > > > > > > > On Mon, Jul 22, 2024 at 12:42:52PM +0200, Ricardo Ribalda Delgado wrote:
> > > > > > > > > On Sun, Jul 21, 2024 at 9:25 PM Laurent Pinchart wrote:
> > > > > > > > > > On Tue, Jul 09, 2024 at 03:15:13PM -0700, Dan Williams wrote:
> > > > > > > > > > > James Bottomley wrote:
> > > > > > > > > > > > > The upstream discussion has yielded the full spectrum of positions on
> > > > > > > > > > > > > device specific functionality, and it is a topic that needs cross-
> > > > > > > > > > > > > kernel consensus as hardware increasingly spans cross-subsystem
> > > > > > > > > > > > > concerns. Please consider it for a Maintainers Summit discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm with Greg on this ... can you point to some of the contrary
> > > > > > > > > > > > positions?
> > > > > > > > > > >
> > > > > > > > > > > This thread has that discussion:
> > > > > > > > > > >
> > > > > > > > > > > http://lore.kernel.org/0-v1-9912f1a11620+2a-fwctl_jgg@nvidia.com
> > > > > > > > > > >
> > > > > > > > > > > I do not want to speak for others on the saliency of their points, all I
> > > > > > > > > > > can say is that the contrary positions have so far not moved me to drop
> > > > > > > > > > > consideration of fwctl for CXL.
> > > > > > > > > > >
> > > > > > > > > > > Where CXL has a Command Effects Log that is a reasonable protocol for
> > > > > > > > > > > making decisions about opaque command codes, and that CXL already has a
> > > > > > > > > > > few years of experience with the commands that *do* need a Linux-command
> > > > > > > > > > > wrapper.
> > > > > > > > > > >
> > > > > > > > > > > Some open questions from that thread are: what does it mean for the fate
> > > > > > > > > > > of a proposal if one subsystem Acks the ABI and another Naks it for a
> > > > > > > > > > > device that crosses subsystem functionality? Would a cynical hardware
> > > > > > > > > > > response just lead to plumbing an NVME admin queue, or CXL mailbox to
> > > > > > > > > > > get device-specific commands past another subsystem's objection?
> > > > > > > > > >
> > > > > > > > > > My default answer would be to trust the maintainers of the relevant
> > > > > > > > > > subsystems (or try to convince them when you disagree :-)). Not only
> > > > > > > > > > should they know the technical implications best, they should also have
> > > > > > > > > > a good view of the whole vertical stack, and the implications of
> > > > > > > > > > pass-through for their ecosystem. This may result in a single NAK
> > > > > > > > > > overriding ACKs, but we could also try to find technical solutions when
> > > > > > > > > > we'll face such issues, to enforce different sets of rules for the
> > > > > > > > > > different functions of a device.
> > > > > > > > > >
> > > > > > > > > > Subsystem hopping is something we're recently noticed for camera ISPs,
> > > > > > > > > > where a vendor wanted to move from V4L2 to DRM. Technical reasons for
> > > > > > > > > > doing so were given, and they were (in my opinion) rather excuses. The
> > > > > > > > > > unspoken real (again in my opinion) reason was to avoid documenting the
> > > > > > > > > > firmware interface and ship userspace binary blobs with no way for free
> > > > > > > > > > software to use all the device's features. That's something we have been
> > > > > > > > > > fighting against for years, trying to convince vendors that they can
> > > > > > > > > > provide better and more open camera support without the world
> > > > > > > > > > collapsing, with increasing success recently. Saying amen to
> > > > > > > > > > pass-through in this case would be a huge step back that would hurt
> > > > > > > > > > users and the whole ecosystem in the short and long term.
> > > > > > > > >
> > > > > > > > > In my view, DRM is a more suitable model for complex ISPs than V4L2:
> > > > > > > >
> > > > > > > > I know we disagree on this topic :-) I'm sure we'll continue the
> > > > > > > > conversation, but I think the technical discussion likely belongs to a
> > > > > > > > different mail thread.
> > > > > > > >
> > > > > > > > > - Userspace Complexity: ISPs demand a highly complex and evolving API,
> > > > > > > > > similar to Vulkan or OpenGL. Applications typically need a framework
> > > > > > > > > like libcamera to utilize ISPs effectively, much like Mesa for
> > > > > > > > > graphics cards.
> > > > > > > > >
> > > > > > > > > - Lack of Standardization: There's no universal standard for ISPs;
> > > > > > > > > each vendor implements unique features and usage patterns. DRM
> > > > > > > > > addresses this through vendor-specific IOCTLs
> > > > > > > > >
> > > > > > > > > - Proprietary Architectures: Vendors often don't fully disclose their
> > > > > > > > > hardware architectures. DRM cleverly only necessitates a Mesa
> > > > > > > > > implementation, not comprehensive documentation.
> > > > > > > >
> > > > > > > > This point isn't technical and is more on-topic for this mail thread.
> > > > > > > >
> > > > > > > > V4L2 doesn't require hundreds of pages of comprehensive documentation in
> > > > > > > > text form. An open-source userspace implementation that covers the
> > > > > > > > feature set exposed by the driver is acceptable in place of
> > > > > > > > documentation (provided, of course, that the userspace code wouldn't be
> > > > > > > > deliberately obfuscated). This is similar in spirit to the rule for GPU
> > > > > > > > DRM drivers.
> > > > > > >
> > > > > > > In DRM vendors typically define a custom IOCTL per driver to pass
> > > > > > > command buffers.
> > > > > > > Only the command buffer structure, and a mesa implementation using
> > > > > > > that command buffer to support the standard features is required.
> > > > > > >
> > > > > > > In V4l2 custom IOCTLs are discouraged. Random command buffers cannot
> > > > > > > be passed from userspace, they are typically formed in the driver from
> > > > > > > a strictly checked struct.
> > > > > >
> > > > > > V4L2 has a mechanism to pass buffers between userspace and kernelspace,
> > > > > > and that mechanism is used in mainline drivers to pass camera ISP
> > > > > > parameters. They're not called "command buffers" but that's just a
> > > > > > difference in terminology. The technical means to pass command buffers
> > > > > > to the driver is thus there, I see no meaningful difference with DRM.
> > > > > > Where things can differ is in the contents of those buffers, and the
> > > > > > requirements for documentation or open userspace implementations, but
> > > > > > that's not a technical question.
> > > > >
> > > > > There are two things here:
> > > > >
> > > > > - The political/strategic/philosophical/religious aspect: The industry
> > > > > definitely prefers the strategic requirements imposed by DRM. In fact
> > > > > some vendors had some huge legal troubles when they had tried to
> > > > > follow v4l2 requirements.
> > > >
> > > > That's I'm willing to debate.
> > > >
> > > > > - The technical aspect: DRM is more mature when it comes to
> > > > > sending/receiving buffers to the hardware, and an ISP looks *much*
> > > > > more similar to an accel device or a GPU than a UVC camera.
> > > >
> > > > But this I don't agree with. I think we should forgo the technical
> > > > discussion and stop pretending that DRM is better for this use case from
> > > > a technical point of view, and focus on the other aspect of the
> > > > discussion. (We can of course reopen the technical discussion if new
> > > > concrete arguments emerge.)
> > >
> > > I disagree. ISP devices are EXACTLY the same as accel devices.

...

> Most of the time, Camera ISPs are nothing more than DSPs plus a
> firmware with the computer vision algos.
> 
> If there is no framework to use a programmable ISP, vendors will keep
> putting everything on the firmware and exposing parameters.

ISPs, as the name suggests, tend to be purpose-built devices, even if some
support some degree of programmability. Even in those cases, the
programmable hardware often assumes pixels are being processed so they
can't be meaningfully used for general-purpose computation. The majority of
devices still has a hardware pipeline with no programmability.

This is also very different from GPUs or accel devices that are built to be
user-programmable. If I'd compare ISPs to different devices, then the
closest match would probably be video codecs -- which also use V4L2.

Also many ISPs use sensor input directly. That introduces timing
constraints that have implications on the API, too, as well as the use of
V4L2/MC APIs by those devices today. I'd rather not see device using an
entirely different API such as DRM in the same pipeline.

I believe we agree V4L2 isn't a great API for camera ISPs but then again I
don't think just switching to DRM is a solution that reasonably covers even
much of the problem area. Some ISPs could be fine using it as such from
purely technical point of view but there are be those for which DRM isn't
an option at all (mainly those having a sensor input).

So in my opinion we either need to improve V4L2/MC to better support those
devices or have an entirely new UAPI for them.

-- 
Kind regards,

Sakari Ailus

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On Mon, 2024-07-29 at 08:10 +0200, Greg KH wrote:
> On Sun, Jul 28, 2024 at 11:49:44AM -0400, James Bottomley wrote:
> > On Sun, 2024-07-28 at 17:16 +0200, Greg KH wrote:
> > > On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> > > > Hi Dan,
> > > > 
> > > > On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > > > > Laurent Pinchart wrote:
> > > > > > I know this is a topic proposed for the maintainers summit,
> > > > > > but given the number of people who seem to have an opinion
> > > > > > and be interested in dicussing it, would a session at LPC
> > > > > > be a better candidate ? I don't expect the maintainer
> > > > > > summit to invite all relevant experts from all subsystems,
> > > > > > that would likely overflow the room.
> > > > > > 
> > > > > > The downside of an LPC session is that it could easily turn
> > > > > > into a heated stage fight, and there are probably also
> > > > > > quite a few arguments that can't really be made in the open
> > > > > > :-S
> > > > > 
> > > > > A separate LPC session for a subsystem or set of subsystems
> > > > > to explore local passthrough policy makes sense, but that is
> > > > > not the primary motivation for also requesting a Maintainer
> > > > > Summit topic slot. The primary motivation is discussing the
> > > > > provenance and navigation of cross-subsystem NAKs especially
> > > > > in an environment where the lines between net, mem, and
> > > > > storage are increasingly blurry at the device level.
> > > > 
> > > > Would there be enough space at the maintainers' summit for all
> > > > the relevant people to join the discussion ?
> > > 
> > > Who exactly would you consider the "relevant people" here?  It's
> > > been a wide-ranging conversation/thread :)
> > 
> > This is a bit of a trick question, since there seem to be three
> > separate but intertwined things here
> > 
> >    1. What to do about cross subsystem NAKs (as in how far does one
> >       subsystem have the ability to NAK something another does
> > because
> >       they fear it will impact them ... passthrough being only one
> >       example).
> >    2. Industry education to help manufacturers making bad decisions
> >       about openness and APIs make better ones that actually
> > benefit
> >       their business in the long run.
> >    3. Standards for open drivers (i.e. is passthrough always evil).
> > 
> > 1. is definitely Maintainer Summit material.
> 
> And to ask again, who do you should participate in this?

Well it's a generic process issue, so the usual suspects at the
Maintainer Summit will do, the question being how do we resolve
disagreements between subsystems that think code in one impacts
another. The answer could be what we already have: resolve it on a case
by case basis, in which case see below, but it would be interesting to
see if something better can come out.  If nothing does then no harm
done and if something comes out no-one likes then the Maintainers won't
follow it anyway.

For the specific issue of discussing fwctl, the Plumbers session would
be better because it can likely gather all interested parties.

> > 2. was something the LF used to help us with but seems to have
> > foundered of late (I think on the general assumption that CNCF gets
> > it right, so we can stop pushing)
> 
> Based on the number of meetings and trips I keep having with
> different companies over the past years, 2. is not something that the
> LF is no longer doing, as they fund me doing this all the time. 
> Including visiting and educating your current employer about these
> very issues recently :)

I didn't say it was something you were no longer doing, I said it was
something the LF is no longer helping us do.  You doing this is great,
but you're just one person.  To scale this we need go to resources for
helping the person dealing with it on the mailing list get up the
management chain when these problems arise.  Plus probably a list of
go-to resources who're used to explaining the business end of open
source to corporate executives, just in case the person dealing with
the patches doesn't have the time or doesn't want to.

Regards,

James

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Wed, Jul 31, 2024 at 08:33:36AM -0400, James Bottomley wrote:
> On Mon, 2024-07-29 at 08:10 +0200, Greg KH wrote:
> > On Sun, Jul 28, 2024 at 11:49:44AM -0400, James Bottomley wrote:
> > > On Sun, 2024-07-28 at 17:16 +0200, Greg KH wrote:
> > > > On Sun, Jul 28, 2024 at 02:18:26PM +0300, Laurent Pinchart wrote:
> > > > > On Fri, Jul 26, 2024 at 05:16:08PM -0700, Dan Williams wrote:
> > > > > > Laurent Pinchart wrote:
> > > > > > > I know this is a topic proposed for the maintainers summit,
> > > > > > > but given the number of people who seem to have an opinion
> > > > > > > and be interested in dicussing it, would a session at LPC
> > > > > > > be a better candidate ? I don't expect the maintainer
> > > > > > > summit to invite all relevant experts from all subsystems,
> > > > > > > that would likely overflow the room.
> > > > > > > 
> > > > > > > The downside of an LPC session is that it could easily turn
> > > > > > > into a heated stage fight, and there are probably also
> > > > > > > quite a few arguments that can't really be made in the open
> > > > > > > :-S
> > > > > > 
> > > > > > A separate LPC session for a subsystem or set of subsystems
> > > > > > to explore local passthrough policy makes sense, but that is
> > > > > > not the primary motivation for also requesting a Maintainer
> > > > > > Summit topic slot. The primary motivation is discussing the
> > > > > > provenance and navigation of cross-subsystem NAKs especially
> > > > > > in an environment where the lines between net, mem, and
> > > > > > storage are increasingly blurry at the device level.
> > > > > 
> > > > > Would there be enough space at the maintainers' summit for all
> > > > > the relevant people to join the discussion ?
> > > > 
> > > > Who exactly would you consider the "relevant people" here?  It's
> > > > been a wide-ranging conversation/thread :)
> > > 
> > > This is a bit of a trick question, since there seem to be three
> > > separate but intertwined things here
> > > 
> > >    1. What to do about cross subsystem NAKs (as in how far does one
> > >       subsystem have the ability to NAK something another does because
> > >       they fear it will impact them ... passthrough being only one
> > >       example).
> > >    2. Industry education to help manufacturers making bad decisions
> > >       about openness and APIs make better ones that actually benefit
> > >       their business in the long run.
> > >    3. Standards for open drivers (i.e. is passthrough always evil).
> > > 
> > > 1. is definitely Maintainer Summit material.
> > 
> > And to ask again, who do you should participate in this?
> 
> Well it's a generic process issue, so the usual suspects at the
> Maintainer Summit will do, the question being how do we resolve
> disagreements between subsystems that think code in one impacts
> another. The answer could be what we already have: resolve it on a case
> by case basis, in which case see below, but it would be interesting to
> see if something better can come out.  If nothing does then no harm
> done and if something comes out no-one likes then the Maintainers won't
> follow it anyway.
> 
> For the specific issue of discussing fwctl, the Plumbers session would
> be better because it can likely gather all interested parties.
> 
> > > 2. was something the LF used to help us with but seems to have
> > > foundered of late (I think on the general assumption that CNCF gets
> > > it right, so we can stop pushing)
> > 
> > Based on the number of meetings and trips I keep having with
> > different companies over the past years, 2. is not something that the
> > LF is no longer doing, as they fund me doing this all the time. 
> > Including visiting and educating your current employer about these
> > very issues recently :)
> 
> I didn't say it was something you were no longer doing, I said it was
> something the LF is no longer helping us do.  You doing this is great,
> but you're just one person.  To scale this we need go to resources for
> helping the person dealing with it on the mailing list get up the
> management chain when these problems arise.  Plus probably a list of
> go-to resources who're used to explaining the business end of open
> source to corporate executives, just in case the person dealing with
> the patches doesn't have the time or doesn't want to.

Having spent the last 5 years having these kind of discussions with
camera vendors, I can tell it's a time consuming job. It's wearisome,
rewarding at times when your message gets through, and depressing when
you experience set backs. Not only could we do with more people and
resources to help with this work, but coordinating the efforts to ensure
vendors won't go back and forth due to contradicting messages would also
help.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Daniel Vetter]

On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> This is also very different from GPUs or accel devices that are built to be
> user-programmable. If I'd compare ISPs to different devices, then the
> closest match would probably be video codecs -- which also use V4L2.

Really just aside, but I figured I should correct this. DRM supports
plenty of video codecs. They're all tied to gpus, but the real reason
really is that the hw has decent command submission support so that
running the entire codec in userspace except the basic memory and
batch execution and synchronization handling in the kernel is a
feasible design. And actually good, because your kernel wont ever blow
up trying to parse complex media formats because it just doesn't.
-Sima
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Jul 30, 2024 at 09:13:00AM +0200, Daniel Vetter wrote:
> I think a solid consensus on the topics above would be really useful for
> gpu/accel too. We're still busy with more pressing community/ecosystem
> building needs, but gpu fw has become rather complex and it's not
> stopping. And there's random other devices attached too nowadays, so fwctl
> makes a ton of sense.

Yeah, I'm pretty sure GPU is going to need fwctl too, the GPU's are
going to have the same issues as NIC does. I see people are already
struggling with topics like how to get debug traces out of the GPU FW.

> But for me the more important stuff would be some clear guidelines like
> what should be in other more across-devices subsystems like edac (or other
> ras features), what should be in functional subsystems like netdev, rdma,
> gpu/accel, ... whatever else, and what should be exposed through some
> special purpose subsystems like hwmon.

In my mind the most important part is that fwctl is not exclusive, the
FW interface and things being manipulated must be sharable or blocked
from fwctl. We should never get in a situation where a fwctl
implementation becomes a reason we cannot have a functional subsystem
interface.

> We've got plenty of experience in enforcing such a community contract with
> vendors, but the hard part is creating a clear and ideally concise
> documentation page I can just point vendors at as the ground truth.

Well, I tried with the documentation in the fwctl patch series..

https://lore.kernel.org/linux-rdma/6-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com/

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Jul 30, 2024 at 03:19:30PM +0200, Borislav Petkov wrote:
> On Mon, Jul 29, 2024 at 11:58:50AM -0300, Jason Gunthorpe wrote:
> > Like continuous memory scrubbing and EDAC is not really fwctl since it
> > is part of the main mission of a memory device. However evaluating the
> > memory to measure current ECC error rate for data collection and
> > debugging would be appropriate for fwctl.
> 
> fwctl?
> 
> What's that? Something to do with "firmware"? If so, how does this thing have
> anything to do with RAS stuff?

https://lore.kernel.org/linux-rdma/6-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com/
https://lore.kernel.org/r/20240718213446.1750135-1-dave.jiang@intel.com

Jonathan is concerned that Dave exposed some of the scrubber FW RPCs
through fwctl.

> We have rasdaemon for that where all the RAS controlling and evaluation goes
> into.

Sure

Jaso 

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Wed, Jul 31, 2024 at 08:33:36AM -0400, James Bottomley wrote:

> For the specific issue of discussing fwctl, the Plumbers session would
> be better because it can likely gather all interested parties.

Keep in mind fwctl is already at the end of a long journey of
conference discussions and talks spanning 3 years back now. It now
represents the generalized consensus between multiple driver
maintainers for at least one side of the debate.

There was also a fwctl presentation at netdev conf a few weeks ago.

In as far as the cross-subsystem NAK, I don't expect more discussion
to result in any change to people's opinions. RDMA side will continue
to want access to the shared device FW, and netdev side will continue
to want to deny access to the shared device FW.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Wed, Jul 31, 2024 at 03:15:39PM +0200, Daniel Vetter wrote:
> On Wed, 31 Jul 2024 at 13:55, Sakari Ailus wrote:
> > This is also very different from GPUs or accel devices that are built to be
> > user-programmable. If I'd compare ISPs to different devices, then the
> > closest match would probably be video codecs -- which also use V4L2.
> 
> Really just aside, but I figured I should correct this. DRM supports
> plenty of video codecs. They're all tied to gpus, but the real reason
> really is that the hw has decent command submission support so that
> running the entire codec in userspace except the basic memory and
> batch execution and synchronization handling in the kernel is a
> feasible design. And actually good, because your kernel wont ever blow
> up trying to parse complex media formats because it just doesn't.

I don't think V4L2 codecs parse the bitstream in the kernel either, at
least not the recent ones.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Daniel Vetter]

On Thu, Aug 01, 2024 at 11:22:23AM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 30, 2024 at 09:13:00AM +0200, Daniel Vetter wrote:
> > I think a solid consensus on the topics above would be really useful for
> > gpu/accel too. We're still busy with more pressing community/ecosystem
> > building needs, but gpu fw has become rather complex and it's not
> > stopping. And there's random other devices attached too nowadays, so fwctl
> > makes a ton of sense.
> 
> Yeah, I'm pretty sure GPU is going to need fwctl too, the GPU's are
> going to have the same issues as NIC does. I see people are already
> struggling with topics like how to get debug traces out of the GPU FW.
> 
> > But for me the more important stuff would be some clear guidelines like
> > what should be in other more across-devices subsystems like edac (or other
> > ras features), what should be in functional subsystems like netdev, rdma,
> > gpu/accel, ... whatever else, and what should be exposed through some
> > special purpose subsystems like hwmon.
> 
> In my mind the most important part is that fwctl is not exclusive, the
> FW interface and things being manipulated must be sharable or blocked
> from fwctl. We should never get in a situation where a fwctl
> implementation becomes a reason we cannot have a functional subsystem
> interface.

Hm still not clear to me how you want to achive that, but I guess best
I'll jump over to the fwctl thread and ask about those details there.

> > We've got plenty of experience in enforcing such a community contract with
> > vendors, but the hard part is creating a clear and ideally concise
> > documentation page I can just point vendors at as the ground truth.
> 
> Well, I tried with the documentation in the fwctl patch series..
> 
> https://lore.kernel.org/linux-rdma/6-v2-940e479ceba9+3821-fwctl_jgg@nvidia.com/

I'll head over and drop some acks and comments.
-Sima
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe]

On Tue, Aug 06, 2024 at 09:14:20AM +0200, Daniel Vetter wrote:
> On Thu, Aug 01, 2024 at 11:22:23AM -0300, Jason Gunthorpe wrote:
> > On Tue, Jul 30, 2024 at 09:13:00AM +0200, Daniel Vetter wrote:
> > > I think a solid consensus on the topics above would be really useful for
> > > gpu/accel too. We're still busy with more pressing community/ecosystem
> > > building needs, but gpu fw has become rather complex and it's not
> > > stopping. And there's random other devices attached too nowadays, so fwctl
> > > makes a ton of sense.
> > 
> > Yeah, I'm pretty sure GPU is going to need fwctl too, the GPU's are
> > going to have the same issues as NIC does. I see people are already
> > struggling with topics like how to get debug traces out of the GPU FW.
> > 
> > > But for me the more important stuff would be some clear guidelines like
> > > what should be in other more across-devices subsystems like edac (or other
> > > ras features), what should be in functional subsystems like netdev, rdma,
> > > gpu/accel, ... whatever else, and what should be exposed through some
> > > special purpose subsystems like hwmon.
> > 
> > In my mind the most important part is that fwctl is not exclusive, the
> > FW interface and things being manipulated must be sharable or blocked
> > from fwctl. We should never get in a situation where a fwctl
> > implementation becomes a reason we cannot have a functional subsystem
> > interface.
> 
> Hm still not clear to me how you want to achive that, but I guess best
> I'll jump over to the fwctl thread and ask about those details
> there.

I'm looking at it from the perspective of mlx5 which has deep
multi-user support in the FW. There is almost nothing in the interface
that is "global" and would become a problem. Everything else can, and
often already is, reasonably be shared.

I think that would have to be the baseline for what you could expose
here.

Like with the memory scrubbing example. It would be fine if fwctl can
read any related counters concurrently with the EDAC driver reading
the same counters. But fwctl shouldn't clear counters or program a
single global scrubber unit.

This limitation has to be baked into the FW/driver on the fwctl side
to undertsand and block these things.

Jason

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Dan Williams]

Jason Gunthorpe wrote:
> On Wed, Jul 31, 2024 at 08:33:36AM -0400, James Bottomley wrote:
> 
> > For the specific issue of discussing fwctl, the Plumbers session would
> > be better because it can likely gather all interested parties.
> 
> Keep in mind fwctl is already at the end of a long journey of
> conference discussions and talks spanning 3 years back now. It now
> represents the generalized consensus between multiple driver
> maintainers for at least one side of the debate.
> 
> There was also a fwctl presentation at netdev conf a few weeks ago.
> 
> In as far as the cross-subsystem NAK, I don't expect more discussion
> to result in any change to people's opinions. RDMA side will continue
> to want access to the shared device FW, and netdev side will continue
> to want to deny access to the shared device FW.

As I mentioned before, this is what I hoped to mediate. The on-list
discussion has seem to hit a deficit of trust roadblock, not a deficit
of technical merit.

All I can say is the discussion is worth a try. With respect to a
precedent for a stalemate moving forward, I point to the MGLRU example.
That proposal had all of the technical merit on the list, but was not
making any clear progress to being merged. It was interesting to watch
that all thaw in real time at LSF/MM (2022) where in person
collaboration yielded strategy concessions, and mutual understanding
that email was never going to produce.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[James Bottomley]

On August 6, 2024 8:06:09 PM EDT, Dan Williams <dan.j.williams@intel.com> wrote:
>Jason Gunthorpe wrote:
>> On Wed, Jul 31, 2024 at 08:33:36AM -0400, James Bottomley wrote:
>> 
>> > For the specific issue of discussing fwctl, the Plumbers session would
>> > be better because it can likely gather all interested parties.
>> 
>> Keep in mind fwctl is already at the end of a long journey of
>> conference discussions and talks spanning 3 years back now. It now
>> represents the generalized consensus between multiple driver
>> maintainers for at least one side of the debate.
>> 
>> There was also a fwctl presentation at netdev conf a few weeks ago.
>> 
>> In as far as the cross-subsystem NAK, I don't expect more discussion
>> to result in any change to people's opinions. RDMA side will continue
>> to want access to the shared device FW, and netdev side will continue
>> to want to deny access to the shared device FW.
>
>As I mentioned before, this is what I hoped to mediate. The on-list
>discussion has seem to hit a deficit of trust roadblock, not a deficit
>of technical merit.
>
>All I can say is the discussion is worth a try. With respect to a
>precedent for a stalemate moving forward, I point to the MGLRU example.
>That proposal had all of the technical merit on the list, but was not
>making any clear progress to being merged. It was interesting to watch
>that all thaw in real time at LSF/MM (2022) where in person
>collaboration yielded strategy concessions, and mutual understanding
>that email was never going to produce.

Well, plumbers stands ready.  We're out of A/V rooms, but if you can do your own A/V with one of the owl cameras we can do a BoF session that can be open to remote participants as well.  I'll be happy to do the setup.

Regards,

James

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Tomasz Figa]

2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
>
> On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > This is also very different from GPUs or accel devices that are built to be
> > user-programmable. If I'd compare ISPs to different devices, then the
> > closest match would probably be video codecs -- which also use V4L2.
>
> Really just aside, but I figured I should correct this. DRM supports
> plenty of video codecs. They're all tied to gpus, but the real reason
> really is that the hw has decent command submission support so that
> running the entire codec in userspace except the basic memory and
> batch execution and synchronization handling in the kernel is a
> feasible design.

FWIW, V4L2 also has an interface for video decoders that require
bitstream processing in software, it's called the V4L2 Stateless
Decoder interface [1]. It defines low level data structures that map
directly to the particular codec specification, so the kernel
interface is generic and the userspace doesn't need to have
hardware-specific components. Hardware that consumes command buffers
can be supported simply by having the kernel driver fill the command
buffers as needed (as opposed to writing the registers directly).
On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
KMS interface for display controllers, rather than a command buffer
passthrough, even though some display controllers actually are driven
by command buffers.
So arguably it's possible and practical to do both command
buffer-based and fixed interfaces for both display controllers and
video codecs. Do you happen to know some background behind why one or
the other was chosen for each of them in DRM?

For how it applies to ISPs, there are both types of ISPs out in the
wild, some support command buffers, while some are programmed directly
via registers. For the former, I can see some loss of flexibility if
the command buffers are hidden behind a fixed function API, because
the userspace would only be able to do what the kernel driver supports
internally, which could make some use case-specific optimizations very
challenging if not impossible.

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html

> And actually good, because your kernel wont ever blow
> up trying to parse complex media formats because it just doesn't.
> -Sima
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
>

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

Hi Tomasz,

On Tue, Aug 13, 2024 at 07:17:07PM +0900, Tomasz Figa wrote:
> 2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
> >
> > On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > This is also very different from GPUs or accel devices that are built to be
> > > user-programmable. If I'd compare ISPs to different devices, then the
> > > closest match would probably be video codecs -- which also use V4L2.
> >
> > Really just aside, but I figured I should correct this. DRM supports
> > plenty of video codecs. They're all tied to gpus, but the real reason
> > really is that the hw has decent command submission support so that
> > running the entire codec in userspace except the basic memory and
> > batch execution and synchronization handling in the kernel is a
> > feasible design.
> 
> FWIW, V4L2 also has an interface for video decoders that require
> bitstream processing in software, it's called the V4L2 Stateless
> Decoder interface [1]. It defines low level data structures that map
> directly to the particular codec specification, so the kernel
> interface is generic and the userspace doesn't need to have
> hardware-specific components. Hardware that consumes command buffers
> can be supported simply by having the kernel driver fill the command
> buffers as needed (as opposed to writing the registers directly).
> On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
> KMS interface for display controllers, rather than a command buffer
> passthrough, even though some display controllers actually are driven
> by command buffers.
> So arguably it's possible and practical to do both command
> buffer-based and fixed interfaces for both display controllers and
> video codecs. Do you happen to know some background behind why one or
> the other was chosen for each of them in DRM?
> 
> For how it applies to ISPs, there are both types of ISPs out in the
> wild, some support command buffers, while some are programmed directly
> via registers.

Could you provide examples of ISPs that use command buffers ? The
discussion has remained fairly vague so far, which I think hinders
progress.

> For the former, I can see some loss of flexibility if
> the command buffers are hidden behind a fixed function API, because
> the userspace would only be able to do what the kernel driver supports
> internally, which could make some use case-specific optimizations very
> challenging if not impossible.

Let's try to discuss this with specific examples.

> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
> 
> > And actually good, because your kernel wont ever blow
> > up trying to parse complex media formats because it just doesn't.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Tomasz Figa]

2024年8月13日(火) 19:27 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>
> Hi Tomasz,
>
> On Tue, Aug 13, 2024 at 07:17:07PM +0900, Tomasz Figa wrote:
> > 2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
> > >
> > > On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > > This is also very different from GPUs or accel devices that are built to be
> > > > user-programmable. If I'd compare ISPs to different devices, then the
> > > > closest match would probably be video codecs -- which also use V4L2.
> > >
> > > Really just aside, but I figured I should correct this. DRM supports
> > > plenty of video codecs. They're all tied to gpus, but the real reason
> > > really is that the hw has decent command submission support so that
> > > running the entire codec in userspace except the basic memory and
> > > batch execution and synchronization handling in the kernel is a
> > > feasible design.
> >
> > FWIW, V4L2 also has an interface for video decoders that require
> > bitstream processing in software, it's called the V4L2 Stateless
> > Decoder interface [1]. It defines low level data structures that map
> > directly to the particular codec specification, so the kernel
> > interface is generic and the userspace doesn't need to have
> > hardware-specific components. Hardware that consumes command buffers
> > can be supported simply by having the kernel driver fill the command
> > buffers as needed (as opposed to writing the registers directly).
> > On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
> > KMS interface for display controllers, rather than a command buffer
> > passthrough, even though some display controllers actually are driven
> > by command buffers.
> > So arguably it's possible and practical to do both command
> > buffer-based and fixed interfaces for both display controllers and
> > video codecs. Do you happen to know some background behind why one or
> > the other was chosen for each of them in DRM?
> >
> > For how it applies to ISPs, there are both types of ISPs out in the
> > wild, some support command buffers, while some are programmed directly
> > via registers.
>
> Could you provide examples of ISPs that use command buffers ? The
> discussion has remained fairly vague so far, which I think hinders
> progress.
>
> > For the former, I can see some loss of flexibility if
> > the command buffers are hidden behind a fixed function API, because
> > the userspace would only be able to do what the kernel driver supports
> > internally, which could make some use case-specific optimizations very
> > challenging if not impossible.
>
> Let's try to discuss this with specific examples.

AFAIK Intel IPU6 and newer, Qualcomm and MediaTek ISPs use command
buffers natively.

>
> > [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
> >
> > > And actually good, because your kernel wont ever blow
> > > up trying to parse complex media formats because it just doesn't.
>
> --
> Regards,
>
> Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Laurent Pinchart]

On Tue, Aug 13, 2024 at 07:33:59PM +0900, Tomasz Figa wrote:
> 2024年8月13日(火) 19:27 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> > On Tue, Aug 13, 2024 at 07:17:07PM +0900, Tomasz Figa wrote:
> > > 2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
> > > >
> > > > On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > > > This is also very different from GPUs or accel devices that are built to be
> > > > > user-programmable. If I'd compare ISPs to different devices, then the
> > > > > closest match would probably be video codecs -- which also use V4L2.
> > > >
> > > > Really just aside, but I figured I should correct this. DRM supports
> > > > plenty of video codecs. They're all tied to gpus, but the real reason
> > > > really is that the hw has decent command submission support so that
> > > > running the entire codec in userspace except the basic memory and
> > > > batch execution and synchronization handling in the kernel is a
> > > > feasible design.
> > >
> > > FWIW, V4L2 also has an interface for video decoders that require
> > > bitstream processing in software, it's called the V4L2 Stateless
> > > Decoder interface [1]. It defines low level data structures that map
> > > directly to the particular codec specification, so the kernel
> > > interface is generic and the userspace doesn't need to have
> > > hardware-specific components. Hardware that consumes command buffers
> > > can be supported simply by having the kernel driver fill the command
> > > buffers as needed (as opposed to writing the registers directly).
> > > On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
> > > KMS interface for display controllers, rather than a command buffer
> > > passthrough, even though some display controllers actually are driven
> > > by command buffers.
> > > So arguably it's possible and practical to do both command
> > > buffer-based and fixed interfaces for both display controllers and
> > > video codecs. Do you happen to know some background behind why one or
> > > the other was chosen for each of them in DRM?
> > >
> > > For how it applies to ISPs, there are both types of ISPs out in the
> > > wild, some support command buffers, while some are programmed directly
> > > via registers.
> >
> > Could you provide examples of ISPs that use command buffers ? The
> > discussion has remained fairly vague so far, which I think hinders
> > progress.
> >
> > > For the former, I can see some loss of flexibility if
> > > the command buffers are hidden behind a fixed function API, because
> > > the userspace would only be able to do what the kernel driver supports
> > > internally, which could make some use case-specific optimizations very
> > > challenging if not impossible.
> >
> > Let's try to discuss this with specific examples.
> 
> AFAIK Intel IPU6 and newer, Qualcomm and MediaTek ISPs use command
> buffers natively.

At the hardware level, firmware level, or both ? Is there a way we can
get more information about the structure of the command buffer and how
it is handled by the ISP for any of those three platforms ?

> > > [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
> > >
> > > > And actually good, because your kernel wont ever blow
> > > > up trying to parse complex media formats because it just doesn't.

-- 
Regards,

Laurent Pinchart

Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Hannes Reinecke]

On 8/7/24 02:06, Dan Williams wrote:
> Jason Gunthorpe wrote:
>> On Wed, Jul 31, 2024 at 08:33:36AM -0400, James Bottomley wrote:
>>
>>> For the specific issue of discussing fwctl, the Plumbers session would
>>> be better because it can likely gather all interested parties.
>>
>> Keep in mind fwctl is already at the end of a long journey of
>> conference discussions and talks spanning 3 years back now. It now
>> represents the generalized consensus between multiple driver
>> maintainers for at least one side of the debate.
>>
>> There was also a fwctl presentation at netdev conf a few weeks ago.
>>
>> In as far as the cross-subsystem NAK, I don't expect more discussion
>> to result in any change to people's opinions. RDMA side will continue
>> to want access to the shared device FW, and netdev side will continue
>> to want to deny access to the shared device FW.
> 
> As I mentioned before, this is what I hoped to mediate. The on-list
> discussion has seem to hit a deficit of trust roadblock, not a deficit
> of technical merit.
> 
> All I can say is the discussion is worth a try. With respect to a
> precedent for a stalemate moving forward, I point to the MGLRU example.
> That proposal had all of the technical merit on the list, but was not
> making any clear progress to being merged. It was interesting to watch
> that all thaw in real time at LSF/MM (2022) where in person
> collaboration yielded strategy concessions, and mutual understanding
> that email was never going to produce.
> 
Well, my experience does not _quite_ match this, but I fully support
the attempt to resolve it.

FWIW, we (ie 'me' with my SUSE distro hat on) are facing similar issues;
every now and again vendors come along asking us to take this very 
important out-of-tree module to allow them to configure their device.
The SCSI stack is _littered_ with vendor-specific commands allowing them 
to reprogram their devices (had a fun experiment once reflashing a 
megaraid HBA to suddenly show up as mpt2sas ... try to code that in 
command effects ...)

So yes, I'd be in favour having an interface for this kind of stuff.
Less sure if there is a generic interface to be found; what we should
try to avoid is having an too generic one (aka: send command with this 
payload, get this result, and heaven knows what it did to the device).
That would surely be useful, but security and operational aspects of
that are a nightmare.

I'd be happy to participate on the discussion at LPC if and when it happens.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.com                               +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich