[Ksummit-discuss] [TECH TOPIC] Media Controller

[Ksummit-discuss] [TECH TOPIC] Media Controller

[Mauro Carvalho Chehab]

Along the years, we've been developing a mechanism at the Kernel to store
and present graphs to userspace, called Media Controller. The original
scope of the Media Controller were to store and represent pipeline connections
between hardware components on complex streaming devices, like the ones
that are used on cell phones, where there are typically two camera sensors,
and lots of processing units to enhance the image and to convert it into
different formats.

Since the beginning, we found that it would be useful to use it some day
on other subsystems. Well, now such day has come ;)

Driven by the need of using the media controller on TV sets, where the
pipelines can be really complex and may cover more than one subsystem,
we're reworking at both the core implementation and at the userspace API
to allow it to be used where needed.

On a TV set, the streaming data pipeline covers camera sensors, analog TV,
digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have network
interfaces that may be provided via DVB (like CATV). If we want/need to 
represent and control the full pipeline, the media controller has to have
support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
needs a way to share a common struct between those subsystems without making
them depend on each other. So, it should use devres.

Also, on the MC summit, it was pointed that other subsystems like IIO
(Linux Industrial I/O) have similar needs and also need to track complex
pipelines.

When we have such complex pipelines that interact with different subsystems,
we need to better describe the interfaces between userspace and the
pipeline. In the past, we only needed to represent device nodes, but now
we'll need to be capable of representing network interfaces and sysfs.

We also need to allow dynamic changes in terms of removing and adding
objects at the graph.

Due to that, we're redesigning the MC core, making it more generic and
more capable of representing complex devices. So, we'd like to discuss
it with the KS audience, in order to check if there are other use cases
for it, and if the solution we're designing would require more changes
to fit such other use cases.

For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
Sakari Ailus, Shuah Khan and Lars-Peter Clausen.

PS.: I know we're a week late, but we've been busy this month with the
discussions, including a Media Controller 3-days summit in Helsinki last
week.

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Daniel Vetter]

On Thu, Aug 6, 2015 at 11:20 AM, Mauro Carvalho Chehab
<mchehab@osg.samsung.com> wrote:
> Along the years, we've been developing a mechanism at the Kernel to store
> and present graphs to userspace, called Media Controller. The original
> scope of the Media Controller were to store and represent pipeline connections
> between hardware components on complex streaming devices, like the ones
> that are used on cell phones, where there are typically two camera sensors,
> and lots of processing units to enhance the image and to convert it into
> different formats.
>
> Since the beginning, we found that it would be useful to use it some day
> on other subsystems. Well, now such day has come ;)
>
> Driven by the need of using the media controller on TV sets, where the
> pipelines can be really complex and may cover more than one subsystem,
> we're reworking at both the core implementation and at the userspace API
> to allow it to be used where needed.
>
> On a TV set, the streaming data pipeline covers camera sensors, analog TV,
> digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have network
> interfaces that may be provided via DVB (like CATV). If we want/need to
> represent and control the full pipeline, the media controller has to have
> support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It also
> needs a way to share a common struct between those subsystems without making
> them depend on each other. So, it should use devres.
>
> Also, on the MC summit, it was pointed that other subsystems like IIO
> (Linux Industrial I/O) have similar needs and also need to track complex
> pipelines.
>
> When we have such complex pipelines that interact with different subsystems,
> we need to better describe the interfaces between userspace and the
> pipeline. In the past, we only needed to represent device nodes, but now
> we'll need to be capable of representing network interfaces and sysfs.
>
> We also need to allow dynamic changes in terms of removing and adding
> objects at the graph.
>
> Due to that, we're redesigning the MC core, making it more generic and
> more capable of representing complex devices. So, we'd like to discuss
> it with the KS audience, in order to check if there are other use cases
> for it, and if the solution we're designing would require more changes
> to fit such other use cases.
>
> For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
> Sakari Ailus, Shuah Khan and Lars-Peter Clausen.

There's ideas for live sources/sinks to connect kms/drm drivers with
v4l drivers. I want to participate too to discuss possible integration
issues with drm. Dave Airlie should probably be there too.
-Daniel

> PS.: I know we're a week late, but we've been busy this month with the
> discussions, including a Media Controller 3-days summit in Helsinki last
> week.
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On Thu, Aug 06, 2015 at 06:20:29AM -0300, Mauro Carvalho Chehab wrote:

> For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
> Sakari Ailus, Shuah Khan and Lars-Peter Clausen.

I'm interested in this as well, as you know it's something that's been
discussed for years with ASoC though there's not been much active work
(but Lars has said he's working on it at the minute so perhaps we'll see
something soon).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Jonathan Cameron]

On 6 August 2015 10:20:29 BST, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>Along the years, we've been developing a mechanism at the Kernel to
>store
>and present graphs to userspace, called Media Controller. The original
>scope of the Media Controller were to store and represent pipeline
>connections
>between hardware components on complex streaming devices, like the ones
>that are used on cell phones, where there are typically two camera
>sensors,
>and lots of processing units to enhance the image and to convert it
>into
>different formats.
>
>Since the beginning, we found that it would be useful to use it some
>day
>on other subsystems. Well, now such day has come ;)
>
>Driven by the need of using the media controller on TV sets, where the
>pipelines can be really complex and may cover more than one subsystem,
>we're reworking at both the core implementation and at the userspace
>API
>to allow it to be used where needed.
>
>On a TV set, the streaming data pipeline covers camera sensors, analog
>TV,
>digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have
>network
>interfaces that may be provided via DVB (like CATV). If we want/need to
>
>represent and control the full pipeline, the media controller has to
>have
>support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It
>also
>needs a way to share a common struct between those subsystems without
>making
>them depend on each other. So, it should use devres.
>
>Also, on the MC summit, it was pointed that other subsystems like IIO
>(Linux Industrial I/O) have similar needs and also need to track
>complex
>pipelines.

Not yet as complex as some of your media cases though I wouldn't be
 surprised if Lars has stuff for this underway!

I need to catch up with your framework as I lost track a while back. Lars-Peter
 Clausen is probably the best person to be present for the IIO elements of this (on top of other roles in this discussion!)

Thankfully most of the sensor hub stuff is probably still where media devices were a while back in that the inputs are hardwired to the DSP elements and
 mostly hidden from the host processor.
Of course there are probably more complex topologies already out there
 somewhere!

Jonathan
>
>When we have such complex pipelines that interact with different
>subsystems,
>we need to better describe the interfaces between userspace and the
>pipeline. In the past, we only needed to represent device nodes, but
>now
>we'll need to be capable of representing network interfaces and sysfs.
>
>We also need to allow dynamic changes in terms of removing and adding
>objects at the graph.
>
>Due to that, we're redesigning the MC core, making it more generic and
>more capable of representing complex devices. So, we'd like to discuss
>it with the KS audience, in order to check if there are other use cases
>for it, and if the solution we're designing would require more changes
>to fit such other use cases.
>
>For this discussion, I nominate myself and Laurent Pinchart, Hans
>Verkuil,
>Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
>
>PS.: I know we're a week late, but we've been busy this month with the
>discussions, including a Media Controller 3-days summit in Helsinki
>last
>week.
>_______________________________________________
>Ksummit-discuss mailing list
>Ksummit-discuss@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Jonathan Cameron]

On 6 August 2015 13:42:17 BST, Jonathan Cameron <jic23@jic23.retrosnub.co.uk> wrote:
>
>
>On 6 August 2015 10:20:29 BST, Mauro Carvalho Chehab
><mchehab@osg.samsung.com> wrote:
>>Along the years, we've been developing a mechanism at the Kernel to
>>store
>>and present graphs to userspace, called Media Controller. The original
>>scope of the Media Controller were to store and represent pipeline
>>connections
>>between hardware components on complex streaming devices, like the
>ones
>>that are used on cell phones, where there are typically two camera
>>sensors,
>>and lots of processing units to enhance the image and to convert it
>>into
>>different formats.
>>
>>Since the beginning, we found that it would be useful to use it some
>>day
>>on other subsystems. Well, now such day has come ;)
>>
>>Driven by the need of using the media controller on TV sets, where the
>>pipelines can be really complex and may cover more than one subsystem,
>>we're reworking at both the core implementation and at the userspace
>>API
>>to allow it to be used where needed.
>>
>>On a TV set, the streaming data pipeline covers camera sensors, analog
>>TV,
>>digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have
>>network
>>interfaces that may be provided via DVB (like CATV). If we want/need
>to
>>
>>represent and control the full pipeline, the media controller has to
>>have
>>support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It
>>also
>>needs a way to share a common struct between those subsystems without
>>making
>>them depend on each other. So, it should use devres.
>>
>>Also, on the MC summit, it was pointed that other subsystems like IIO
>>(Linux Industrial I/O) have similar needs and also need to track
>>complex
>>pipelines.
>
>Not yet as complex as some of your media cases though I wouldn't be
> surprised if Lars has stuff for this underway!
>
>I need to catch up with your framework as I lost track a while back.
>Lars-Peter
>Clausen is probably the best person to be present for the IIO elements
>of this (on top of other roles in this discussion!)
>
>Thankfully most of the sensor hub stuff is probably still where media
>devices were a while back in that the inputs are hardwired to the DSP
>elements and
> mostly hidden from the host processor.
>Of course there are probably more complex topologies already out there
> somewhere!

Having had a bit of a read of the docs, I can see the framework could be used 
a lot more widely in IIO than I originally thought.  We have a few different
 dataflows:

* streamed main data flows (triggered samples from ADC channels) might be
 from an analogous accelerometer, through an ADC and on top a sink that
 outputs to userspace as an input driver.

* polled data flows (more like querying a regulator) not sure how these would be
 handled. Note some devices are very slow 1hz? Hence this path is very
 lightweight in terms of driver support needed. More or less the same as reading
 a hwmon channel from sysfs.

* trigger data flows. Hardware of software 'grab a sample now' signals. 
Kind of a one to many sync signal.
* event data flows (threshold detectors
 etc - not yet handled well in IIO)

Can certainly see the streamed stuff and events fitting well with the media controller
framework.  Perhaps the triggering as well.
Any thoughts on the polled data flow? 

Definitely an interesting topic to pursue!

Jonathan
>
>Jonathan
>>
>>When we have such complex pipelines that interact with different
>>subsystems,
>>we need to better describe the interfaces between userspace and the
>>pipeline. In the past, we only needed to represent device nodes, but
>>now
>>we'll need to be capable of representing network interfaces and sysfs.
>>
>>We also need to allow dynamic changes in terms of removing and adding
>>objects at the graph.
>>
>>Due to that, we're redesigning the MC core, making it more generic and
>>more capable of representing complex devices. So, we'd like to discuss
>>it with the KS audience, in order to check if there are other use
>cases
>>for it, and if the solution we're designing would require more changes
>>to fit such other use cases.
>>
>>For this discussion, I nominate myself and Laurent Pinchart, Hans
>>Verkuil,
>>Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
>>
>>PS.: I know we're a week late, but we've been busy this month with the
>>discussions, including a Media Controller 3-days summit in Helsinki
>>last
>>week.
>>_______________________________________________
>>Ksummit-discuss mailing list
>>Ksummit-discuss@lists.linuxfoundation.org
>>https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Lars-Peter Clausen]

On 08/07/2015 09:21 AM, Jonathan Cameron wrote:
> 
> 
> On 6 August 2015 13:42:17 BST, Jonathan Cameron <jic23@jic23.retrosnub.co.uk> wrote:
>>
>>
>> On 6 August 2015 10:20:29 BST, Mauro Carvalho Chehab
>> <mchehab@osg.samsung.com> wrote:
>>> Along the years, we've been developing a mechanism at the Kernel to
>>> store
>>> and present graphs to userspace, called Media Controller. The original
>>> scope of the Media Controller were to store and represent pipeline
>>> connections
>>> between hardware components on complex streaming devices, like the
>> ones
>>> that are used on cell phones, where there are typically two camera
>>> sensors,
>>> and lots of processing units to enhance the image and to convert it
>>> into
>>> different formats.
>>>
>>> Since the beginning, we found that it would be useful to use it some
>>> day
>>> on other subsystems. Well, now such day has come ;)
>>>
>>> Driven by the need of using the media controller on TV sets, where the
>>> pipelines can be really complex and may cover more than one subsystem,
>>> we're reworking at both the core implementation and at the userspace
>>> API
>>> to allow it to be used where needed.
>>>
>>> On a TV set, the streaming data pipeline covers camera sensors, analog
>>> TV,
>>> digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have
>>> network
>>> interfaces that may be provided via DVB (like CATV). If we want/need
>> to
>>>
>>> represent and control the full pipeline, the media controller has to
>>> have
>>> support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It
>>> also
>>> needs a way to share a common struct between those subsystems without
>>> making
>>> them depend on each other. So, it should use devres.
>>>
>>> Also, on the MC summit, it was pointed that other subsystems like IIO
>>> (Linux Industrial I/O) have similar needs and also need to track
>>> complex
>>> pipelines.
>>
>> Not yet as complex as some of your media cases though I wouldn't be
>> surprised if Lars has stuff for this underway!

This[1] is the quick presentation I gave the media controller workshop, it
has a few examples of complex pipelines. Most of the driver support is work
in progress, but it shows why we need a method of describing the data flow
topology. So the goal is to have the MC integration ready when the drivers
get merged.

- Lars

[1] http://metafoo.de/iio_media_controller.pdf

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Shuah Khan]

On Fri, Aug 7, 2015 at 1:48 AM, Lars-Peter Clausen <lars@metafoo.de> wrote:
> On 08/07/2015 09:21 AM, Jonathan Cameron wrote:
>>
>>
>> On 6 August 2015 13:42:17 BST, Jonathan Cameron <jic23@jic23.retrosnub.co.uk> wrote:
>>>
>>>
>>> On 6 August 2015 10:20:29 BST, Mauro Carvalho Chehab
>>> <mchehab@osg.samsung.com> wrote:
>>>> Along the years, we've been developing a mechanism at the Kernel to
>>>> store
>>>> and present graphs to userspace, called Media Controller. The original
>>>> scope of the Media Controller were to store and represent pipeline
>>>> connections
>>>> between hardware components on complex streaming devices, like the
>>> ones
>>>> that are used on cell phones, where there are typically two camera
>>>> sensors,
>>>> and lots of processing units to enhance the image and to convert it
>>>> into
>>>> different formats.
>>>>
>>>> Since the beginning, we found that it would be useful to use it some
>>>> day
>>>> on other subsystems. Well, now such day has come ;)
>>>>
>>>> Driven by the need of using the media controller on TV sets, where the
>>>> pipelines can be really complex and may cover more than one subsystem,
>>>> we're reworking at both the core implementation and at the userspace
>>>> API
>>>> to allow it to be used where needed.
>>>>
>>>> On a TV set, the streaming data pipeline covers camera sensors, analog
>>>> TV,
>>>> digital TV, crypto/decrypto modules, audio and GPU. Also, TVs have
>>>> network
>>>> interfaces that may be provided via DVB (like CATV). If we want/need
>>> to
>>>>
>>>> represent and control the full pipeline, the media controller has to
>>>> have
>>>> support on different subsystems: V4L2, DVB, ALSA, DRM and Network. It
>>>> also
>>>> needs a way to share a common struct between those subsystems without
>>>> making
>>>> them depend on each other. So, it should use devres.
>>>>
>>>> Also, on the MC summit, it was pointed that other subsystems like IIO
>>>> (Linux Industrial I/O) have similar needs and also need to track
>>>> complex
>>>> pipelines.
>>>
>>> Not yet as complex as some of your media cases though I wouldn't be
>>> surprised if Lars has stuff for this underway!
>
> This[1] is the quick presentation I gave the media controller workshop, it
> has a few examples of complex pipelines. Most of the driver support is work
> in progress, but it shows why we need a method of describing the data flow
> topology. So the goal is to have the MC integration ready when the drivers
> get merged.
>
Sorry if you are seeing this meesage twice. My previous message didn't make to
the KS because it was too large.

I have been working on updating snd-usb-audio and a media driver (au0828) that
uses snd-usb-audio as its audio driver to use Media Controller API to share
resources. Sharing resources across these varied set of drivers required adding
Managed Media Controller API and now I am building on top. Managed Media
Controller API creates media_device as a device resource at the parent
USB device
which is the root device for this driver hierarchy.

We reviewed Patch series v5 at the Media Summit and I am working on v6. Patch
series v5 is available at

https://git.kernel.org/cgit/linux/kernel/git/shuah/linux.git/log/?h=media_controller

As go along and do this work, I am refining and adding new MC interfaces as
needed and creating handlers to enable  drivers to coordinate initialization as
each driver adds new entities. Here is the link to media graph for

Hauppauge HVR950Q Hybrid USB TV stick that id driven by au0828 (DVB, Video),
and snd-usb-audio drivers.  The media device is s Managed device resource.

https://drive.google.com/file/d/0B0NIL0BQg-AlSFJPelltTm82UGM/view

I look forward to sharing and discussing this work and MC use-cases for other
driver classes at the workshop.

thanks,
-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] Media Controller

[Vinod Koul]

On Thu, Aug 06, 2015 at 06:20:29AM -0300, Mauro Carvalho Chehab wrote:

> Due to that, we're redesigning the MC core, making it more generic and
> more capable of representing complex devices. So, we'd like to discuss
> it with the KS audience, in order to check if there are other use cases
> for it, and if the solution we're designing would require more changes
> to fit such other use cases.
> 
> For this discussion, I nominate myself and Laurent Pinchart, Hans Verkuil,
> Sakari Ailus, Shuah Khan and Lars-Peter Clausen.
> 
> PS.: I know we're a week late, but we've been busy this month with the
> discussions, including a Media Controller 3-days summit in Helsinki last
> week.

Sorry for replying super duper late, I wanted to ensure that I have travel
go ahead before asking :)

So I would like an invitation for this WS, at Intel we have similar issues
with DSP graphs. We are using shiny new ASoC topology framework in our DSP
to get topology information and not hard code the graphs, but have no way to
tell that to usermode.

I did look at media controller in past and would love to discuss more. In
audio, people have been talking about this and I would be spending time on
this later this year

-- 
~Vinod