Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Tomasz Figa <tomasz.figa@gmail.com>]

2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
>
> On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > This is also very different from GPUs or accel devices that are built to be
> > user-programmable. If I'd compare ISPs to different devices, then the
> > closest match would probably be video codecs -- which also use V4L2.
>
> Really just aside, but I figured I should correct this. DRM supports
> plenty of video codecs. They're all tied to gpus, but the real reason
> really is that the hw has decent command submission support so that
> running the entire codec in userspace except the basic memory and
> batch execution and synchronization handling in the kernel is a
> feasible design.

FWIW, V4L2 also has an interface for video decoders that require
bitstream processing in software, it's called the V4L2 Stateless
Decoder interface [1]. It defines low level data structures that map
directly to the particular codec specification, so the kernel
interface is generic and the userspace doesn't need to have
hardware-specific components. Hardware that consumes command buffers
can be supported simply by having the kernel driver fill the command
buffers as needed (as opposed to writing the registers directly).
On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
KMS interface for display controllers, rather than a command buffer
passthrough, even though some display controllers actually are driven
by command buffers.
So arguably it's possible and practical to do both command
buffer-based and fixed interfaces for both display controllers and
video codecs. Do you happen to know some background behind why one or
the other was chosen for each of them in DRM?

For how it applies to ISPs, there are both types of ISPs out in the
wild, some support command buffers, while some are programmed directly
via registers. For the former, I can see some loss of flexibility if
the command buffers are hidden behind a fixed function API, because
the userspace would only be able to do what the kernel driver supports
internally, which could make some use case-specific optimizations very
challenging if not impossible.

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html

> And actually good, because your kernel wont ever blow
> up trying to parse complex media formats because it just doesn't.
> -Sima
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
>