[Ksummit-discuss] [TECH TOPIC] PM dependencies

[Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

Hello,

On modern systems many PM dependencies don't follow the Linux kernel device 
model based on parent-child relationships from a control bus point of view. 
For instance a GPU will need the IOMMU that services its memory requests to be 
powered on to perform DMA operations.

Ad-hoc solutions with subsystem-specific or even driver-specific APIs have 
been implemented (see omap_iommu_save_ctx and omap_iommu_restore_ctx in 
include/linux/omap-iommu.h for instance, that push the burden of saving and 
restoring the IOMMU registers to bus master drivers), but they make automatic 
handling of hardware resources difficult. In the IOMMU case again, the goal is 
to hide IOMMU handling inside the DMA mapping API and make its usage 
completely transparent to bus master drivers in most of the cases. An IOMMU-
specific API to explicitly control IOMMU power from the bus master driver 
would make that goal impossible to reach.

The problem is not limited to IOMMUs. We have similar dependencies at 
suspend/resume time with camera interfaces for instance, where two completely 
unrelated device in the Linux device hierarchy (a camera interface platform 
device in the SoC and an I2C camera sensor) need to be suspended and resumed 
in a controlled order. I'm sure many more use cases exist.

Whether we can (partly) reuse existing infrastructure for this is not clear. 
I'd like to gather use cases, to first find out exactly how widespread the 
need is, and whether we can come up with a common solution or we are really 
faced with different classes of similarly looking but distinct issues.

Tomasz Figa (CC'ed) has recently expressed interest for this topic, if he's 
still interested I'd like to nominate him.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Shuah Khan]

On Mon, May 12, 2014 at 11:43 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Hello,
>
> On modern systems many PM dependencies don't follow the Linux kernel device
> model based on parent-child relationships from a control bus point of view.
> For instance a GPU will need the IOMMU that services its memory requests to be
> powered on to perform DMA operations.
>
> Ad-hoc solutions with subsystem-specific or even driver-specific APIs have
> been implemented (see omap_iommu_save_ctx and omap_iommu_restore_ctx in
> include/linux/omap-iommu.h for instance, that push the burden of saving and
> restoring the IOMMU registers to bus master drivers), but they make automatic
> handling of hardware resources difficult. In the IOMMU case again, the goal is
> to hide IOMMU handling inside the DMA mapping API and make its usage
> completely transparent to bus master drivers in most of the cases. An IOMMU-
> specific API to explicitly control IOMMU power from the bus master driver
> would make that goal impossible to reach.
>
> The problem is not limited to IOMMUs. We have similar dependencies at
> suspend/resume time with camera interfaces for instance, where two completely
> unrelated device in the Linux device hierarchy (a camera interface platform
> device in the SoC and an I2C camera sensor) need to be suspended and resumed
> in a controlled order. I'm sure many more use cases exist.

We discussed this topic briefly at the Media Mini-summit.  I looking
into similar
use-cases in media devices. Suspend/resume of a tuner for example should only
be done once as opposed multiple times by drivers that think they have exclusive
access to the tuner on a media device.

I am interested in this discussion and would like to participate.
Requesting nomination.

-- Shuah
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Tomasz Figa]

Hi,

On 12.05.2014 19:43, Laurent Pinchart wrote:
> Hello,
> 
> On modern systems many PM dependencies don't follow the Linux kernel device 
> model based on parent-child relationships from a control bus point of view. 
> For instance a GPU will need the IOMMU that services its memory requests to be 
> powered on to perform DMA operations.
> 
> Ad-hoc solutions with subsystem-specific or even driver-specific APIs have 
> been implemented (see omap_iommu_save_ctx and omap_iommu_restore_ctx in 
> include/linux/omap-iommu.h for instance, that push the burden of saving and 
> restoring the IOMMU registers to bus master drivers), but they make automatic 
> handling of hardware resources difficult. In the IOMMU case again, the goal is 
> to hide IOMMU handling inside the DMA mapping API and make its usage 
> completely transparent to bus master drivers in most of the cases. An IOMMU-
> specific API to explicitly control IOMMU power from the bus master driver 
> would make that goal impossible to reach.
> 
> The problem is not limited to IOMMUs. We have similar dependencies at 
> suspend/resume time with camera interfaces for instance, where two completely 
> unrelated device in the Linux device hierarchy (a camera interface platform 
> device in the SoC and an I2C camera sensor) need to be suspended and resumed 
> in a controlled order. I'm sure many more use cases exist.
> 
> Whether we can (partly) reuse existing infrastructure for this is not clear. 
> I'd like to gather use cases, to first find out exactly how widespread the 
> need is, and whether we can come up with a common solution or we are really 
> faced with different classes of similarly looking but distinct issues.
> 
> Tomasz Figa (CC'ed) has recently expressed interest for this topic, if he's 
> still interested I'd like to nominate him.

Yes, I'm still interested.

IOMMUs are the use case I've been wondering about recently, but I'd like
to take some time to analyze other use cases useful for Samsung
platforms. I will post them in another reply to this thread as soon as I
get to some interesting conclusions.

Best regards,
Tomasz

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

On Mon, May 12, 2014 at 07:43:22PM +0200, Laurent Pinchart wrote:

> The problem is not limited to IOMMUs. We have similar dependencies at 
> suspend/resume time with camera interfaces for instance, where two completely 
> unrelated device in the Linux device hierarchy (a camera interface platform 
> device in the SoC and an I2C camera sensor) need to be suspended and resumed 
> in a controlled order. I'm sure many more use cases exist.

> Whether we can (partly) reuse existing infrastructure for this is not clear. 
> I'd like to gather use cases, to first find out exactly how widespread the 
> need is, and whether we can come up with a common solution or we are really 
> faced with different classes of similarly looking but distinct issues.

This is something ASoC has been resolving since forever with the machine
drivers, Russell King recently created a generic version of the code
that does the dependency deferral glue (see linux/component.h).  It's
not a model of elegance but it ends up getting the job done.

Anyway, I'm definitely interested in this - a couple of the subsystems I
maintain are affected and it's something I've thought about a bit.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

Hi Mark,

On Monday 12 May 2014 21:14:38 Mark Brown wrote:
> On Mon, May 12, 2014 at 07:43:22PM +0200, Laurent Pinchart wrote:
> > The problem is not limited to IOMMUs. We have similar dependencies at
> > suspend/resume time with camera interfaces for instance, where two
> > completely unrelated device in the Linux device hierarchy (a camera
> > interface platform device in the SoC and an I2C camera sensor) need to be
> > suspended and resumed in a controlled order. I'm sure many more use cases
> > exist.
> > 
> > Whether we can (partly) reuse existing infrastructure for this is not
> > clear. I'd like to gather use cases, to first find out exactly how
> > widespread the need is, and whether we can come up with a common solution
> > or we are really faced with different classes of similarly looking but
> > distinct issues.
>
> This is something ASoC has been resolving since forever with the machine
> drivers, Russell King recently created a generic version of the code
> that does the dependency deferral glue (see linux/component.h).  It's
> not a model of elegance but it ends up getting the job done.

That solves the probe time dependencies, but unfortunately not the runtime PM 
dependencies.

> Anyway, I'm definitely interested in this - a couple of the subsystems I
> maintain are affected and it's something I've thought about a bit.

-- 
Regards,

Laurent Pinchart

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:
> On Monday 12 May 2014 21:14:38 Mark Brown wrote:

> > This is something ASoC has been resolving since forever with the machine
> > drivers, Russell King recently created a generic version of the code
> > that does the dependency deferral glue (see linux/component.h).  It's
> > not a model of elegance but it ends up getting the job done.

> That solves the probe time dependencies, but unfortunately not the runtime PM 
> dependencies.

It also solves the system suspend dependencies.  Why don't the runtime
PM dependencies just work with reference counting?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Tomasz Figa]

Hi Mark,

On 12.05.2014 22:31, Mark Brown wrote:
> On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:
>> On Monday 12 May 2014 21:14:38 Mark Brown wrote:
> 
>>> This is something ASoC has been resolving since forever with
>>> the machine drivers, Russell King recently created a generic
>>> version of the code that does the dependency deferral glue (see
>>> linux/component.h).  It's not a model of elegance but it ends
>>> up getting the job done.
> 
>> That solves the probe time dependencies, but unfortunately not
>> the runtime PM dependencies.
> 
> It also solves the system suspend dependencies.  Why don't the
> runtime PM dependencies just work with reference counting?
> 

Runtime PM dependencies work with reference counting just fine, but
only for topologies matching Linux driver model, e.g. devices with
exactly one device they depend on, e.g. SPI controller and SPI devices
on the bus driven by it. Add there an IOMMU and other various strange
things that should be transparent to the drivers and it stops working.

I'm still investigating this issue, so more uses cases are yet to be
found, but I also guess this is the purpose of this thread. Anyway,
for some reason .suspend_late() and .resume_early() callbacks exist in
dev_pm_ops struct and I believe that at least some of the cases
"solved" by them might be related to the issue being discussed here.

Best regards,
Tomasz

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> On 12.05.2014 22:31, Mark Brown wrote:

> > It also solves the system suspend dependencies.  Why don't the
> > runtime PM dependencies just work with reference counting?

> Runtime PM dependencies work with reference counting just fine, but
> only for topologies matching Linux driver model, e.g. devices with
> exactly one device they depend on, e.g. SPI controller and SPI devices
> on the bus driven by it. Add there an IOMMU and other various strange
> things that should be transparent to the drivers and it stops working.

There's no reason why runtime PM references have to follow the topology
- you do get a default reference count up to any parent (though we break
that sometimes, as is the case with SPI controllers being suspended even
though the devices below them are active) but there's nothing stopping
references being taken outside the topology.

> I'm still investigating this issue, so more uses cases are yet to be
> found, but I also guess this is the purpose of this thread. Anyway,
> for some reason .suspend_late() and .resume_early() callbacks exist in
> dev_pm_ops struct and I believe that at least some of the cases
> "solved" by them might be related to the issue being discussed here.

Yes, they're partly solving a particular common case for this sort of
interdependency (though I guess they do also do things like allow us to
make sure the hardware came back in a state where it won't be harmful
to the rest of the system if we start enabling things).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Daniel Vetter]

On Tue, May 13, 2014 at 12:07 AM, Mark Brown <broonie@kernel.org> wrote:
> On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>> On 12.05.2014 22:31, Mark Brown wrote:
>> > It also solves the system suspend dependencies.  Why don't the
>> > runtime PM dependencies just work with reference counting?
>
>> Runtime PM dependencies work with reference counting just fine, but
>> only for topologies matching Linux driver model, e.g. devices with
>> exactly one device they depend on, e.g. SPI controller and SPI devices
>> on the bus driven by it. Add there an IOMMU and other various strange
>> things that should be transparent to the drivers and it stops working.
>
> There's no reason why runtime PM references have to follow the topology
> - you do get a default reference count up to any parent (though we break
> that sometimes, as is the case with SPI controllers being suspended even
> though the devices below them are active) but there's nothing stopping
> references being taken outside the topology.

I guess some helpers to grab/drop runtime PM references on all parts
of a componentized device should resolve this? We propably don't want
to bake this into the driver core since often the driver will know
that it e.g. doesn't need a specific encoder block (if it's not
enabled), so doesn't to grab a reference. But for simple drivers, and
to get runtime PM off the ground when enabling new hardware this could
be rather useful.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

On Tuesday 13 May 2014 09:43:05 Daniel Vetter wrote:
> On Tue, May 13, 2014 at 12:07 AM, Mark Brown <broonie@kernel.org> wrote:
> > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> >> On 12.05.2014 22:31, Mark Brown wrote:
> >> > It also solves the system suspend dependencies.  Why don't the
> >> > runtime PM dependencies just work with reference counting?
> >> 
> >> Runtime PM dependencies work with reference counting just fine, but
> >> only for topologies matching Linux driver model, e.g. devices with
> >> exactly one device they depend on, e.g. SPI controller and SPI devices
> >> on the bus driven by it. Add there an IOMMU and other various strange
> >> things that should be transparent to the drivers and it stops working.
> > 
> > There's no reason why runtime PM references have to follow the topology
> > - you do get a default reference count up to any parent (though we break
> > that sometimes, as is the case with SPI controllers being suspended even
> > though the devices below them are active) but there's nothing stopping
> > references being taken outside the topology.

One of the issues is that we need API(s) to take/release those references. In 
some cases the depender doesn't have direct access to the dependee, like in 
the IOMMU case where the DMA mapping API hides the IOMMU. One of the questions 
here is whether this kind of problems should be solved with ad-hoc solutions 
(for instance adding PM support to the DMA mapping API in this specific case), 
or in a more generic fashion. Answering that will require a comprehensive list 
of use cases to detect common patterns.

> I guess some helpers to grab/drop runtime PM references on all parts
> of a componentized device should resolve this?

That could help in some cases, but there might be ordering issues.

> We propably don't want to bake this into the driver core since often the
> driver will know that it e.g. doesn't need a specific encoder block (if it's
> not enabled), so doesn't to grab a reference. But for simple drivers, and
> to get runtime PM off the ground when enabling new hardware this could
> be rather useful.

This won't solve the IOMMU problem, as the component framework isn't used 
there. I'm not advocating for a common solution to all non parent/child PM 
dependencies just for the sake of it, we could find out that several distinct 
solutions to distinct classes of issues is the way to go.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Shuah Khan]

On Tue, May 13, 2014 at 4:31 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Tuesday 13 May 2014 09:43:05 Daniel Vetter wrote:
>> On Tue, May 13, 2014 at 12:07 AM, Mark Brown <broonie@kernel.org> wrote:
>> > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>> >> On 12.05.2014 22:31, Mark Brown wrote:
>> >> > It also solves the system suspend dependencies.  Why don't the
>> >> > runtime PM dependencies just work with reference counting?
>> >>
>> >> Runtime PM dependencies work with reference counting just fine, but
>> >> only for topologies matching Linux driver model, e.g. devices with
>> >> exactly one device they depend on, e.g. SPI controller and SPI devices
>> >> on the bus driven by it. Add there an IOMMU and other various strange
>> >> things that should be transparent to the drivers and it stops working.
>> >
>> > There's no reason why runtime PM references have to follow the topology
>> > - you do get a default reference count up to any parent (though we break
>> > that sometimes, as is the case with SPI controllers being suspended even
>> > though the devices below them are active) but there's nothing stopping
>> > references being taken outside the topology.
>
> One of the issues is that we need API(s) to take/release those references. In
> some cases the depender doesn't have direct access to the dependee, like in
> the IOMMU case where the DMA mapping API hides the IOMMU. One of the questions
> here is whether this kind of problems should be solved with ad-hoc solutions
> (for instance adding PM support to the DMA mapping API in this specific case),
> or in a more generic fashion. Answering that will require a comprehensive list
> of use cases to detect common patterns.
>
>> I guess some helpers to grab/drop runtime PM references on all parts
>> of a componentized device should resolve this?
>
> That could help in some cases, but there might be ordering issues.

Can we use pm domains concept to solve the problem? As it exists
today, it probably
can't support loosely associated devices, however, could it be
extended to support
handling device groups that don't necessarily share power source and/or clock.

I started looking at this as a way to solve media device PM issues,
however I haven't
had a chance to experiment with this idea.

-- Shuah
>
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> > On 12.05.2014 22:31, Mark Brown wrote:
> 
> > > It also solves the system suspend dependencies.  Why don't the
> > > runtime PM dependencies just work with reference counting?
> 
> > Runtime PM dependencies work with reference counting just fine, but
> > only for topologies matching Linux driver model, e.g. devices with
> > exactly one device they depend on, e.g. SPI controller and SPI devices
> > on the bus driven by it. Add there an IOMMU and other various strange
> > things that should be transparent to the drivers and it stops working.
> 
> There's no reason why runtime PM references have to follow the topology
> - you do get a default reference count up to any parent (though we break
> that sometimes, as is the case with SPI controllers being suspended even
> though the devices below them are active) but there's nothing stopping
> references being taken outside the topology.

Precisely.

> > I'm still investigating this issue, so more uses cases are yet to be
> > found, but I also guess this is the purpose of this thread. Anyway,
> > for some reason .suspend_late() and .resume_early() callbacks exist in
> > dev_pm_ops struct and I believe that at least some of the cases
> > "solved" by them might be related to the issue being discussed here.
> 
> Yes, they're partly solving a particular common case for this sort of
> interdependency (though I guess they do also do things like allow us to
> make sure the hardware came back in a state where it won't be harmful
> to the rest of the system if we start enabling things).

The original reason why .suspend_late/.resume_early were introduced was to
allow the same callback routines to be run for runtime and for system-wide PM.
Specifically, the idea was that .suspend_late may point to the same routine
as .runtime_suspend, for example, because of the analogy of the conditions
in which they are called (the device is known to be not in use).

That didn't work out perfectly, but those callbacks turn out to be useful
nonetheless.

And yes, I'm interested in this topic too.

Rafael (who wonders why he wasn't CCed in this thread to start with)

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> > > On 12.05.2014 22:31, Mark Brown wrote:
> > 
> > > > It also solves the system suspend dependencies.  Why don't the
> > > > runtime PM dependencies just work with reference counting?
> > 
> > > Runtime PM dependencies work with reference counting just fine, but
> > > only for topologies matching Linux driver model, e.g. devices with
> > > exactly one device they depend on, e.g. SPI controller and SPI devices
> > > on the bus driven by it. Add there an IOMMU and other various strange
> > > things that should be transparent to the drivers and it stops working.
> > 
> > There's no reason why runtime PM references have to follow the topology
> > - you do get a default reference count up to any parent (though we break
> > that sometimes, as is the case with SPI controllers being suspended even
> > though the devices below them are active) but there's nothing stopping
> > references being taken outside the topology.
> 
> Precisely.

BTW, I guess that the problem is resume and specifically the fact that if
a child device resumes, the parent will also resume automatically, but the
other devices the child may depend on will not (the child's resume may need
to resume them directly).

But I'm not sure why that is a problem, so can anyone please share some details?

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

On Wed, May 14, 2014 at 02:11:25PM +0200, Rafael J. Wysocki wrote:
> On Monday, May 12, 2014 09:31:53 PM Mark Brown wrote:
> > On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:

> > > That solves the probe time dependencies, but unfortunately not the runtime PM 
> > > dependencies.

> > It also solves the system suspend dependencies.

> Well, I'm not sure about that.  How does it do that, exactly?

Since the core device probe is deferred until everything it depends on
is ready it'll get pushed along the device list so that it appears after
everything it needs and hence is suspended first.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Monday, May 12, 2014 09:31:53 PM Mark Brown wrote:
> On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:
> > On Monday 12 May 2014 21:14:38 Mark Brown wrote:
> 
> > > This is something ASoC has been resolving since forever with the machine
> > > drivers, Russell King recently created a generic version of the code
> > > that does the dependency deferral glue (see linux/component.h).  It's
> > > not a model of elegance but it ends up getting the job done.
> 
> > That solves the probe time dependencies, but unfortunately not the runtime PM 
> > dependencies.
> 
> It also solves the system suspend dependencies.

Well, I'm not sure about that.  How does it do that, exactly?

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Wednesday, May 14, 2014 12:57:45 PM Mark Brown wrote:
> On Wed, May 14, 2014 at 02:11:25PM +0200, Rafael J. Wysocki wrote:
> > On Monday, May 12, 2014 09:31:53 PM Mark Brown wrote:
> > > On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:
> 
> > > > That solves the probe time dependencies, but unfortunately not the runtime PM 
> > > > dependencies.
> 
> > > It also solves the system suspend dependencies.
> 
> > Well, I'm not sure about that.  How does it do that, exactly?
> 
> Since the core device probe is deferred until everything it depends on
> is ready it'll get pushed along the device list so that it appears after
> everything it needs and hence is suspended first.

Except for async suspend, that is?

Rafael

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Wednesday, May 14, 2014 12:34:56 AM Rafael J. Wysocki wrote:
> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> > > > On 12.05.2014 22:31, Mark Brown wrote:
> > > 
> > > > > It also solves the system suspend dependencies.  Why don't the
> > > > > runtime PM dependencies just work with reference counting?
> > > 
> > > > Runtime PM dependencies work with reference counting just fine, but
> > > > only for topologies matching Linux driver model, e.g. devices with
> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
> > > > on the bus driven by it. Add there an IOMMU and other various strange
> > > > things that should be transparent to the drivers and it stops working.
> > > 
> > > There's no reason why runtime PM references have to follow the topology
> > > - you do get a default reference count up to any parent (though we break
> > > that sometimes, as is the case with SPI controllers being suspended even
> > > though the devices below them are active) but there's nothing stopping
> > > references being taken outside the topology.
> > 
> > Precisely.
> 
> BTW, I guess that the problem is resume and specifically the fact that if
> a child device resumes, the parent will also resume automatically, but the
> other devices the child may depend on will not (the child's resume may need
> to resume them directly).
> 
> But I'm not sure why that is a problem, so can anyone please share some details?

Thinking more about this, there seem to be two cases:

(a) The driver knows that its device (X) depends on another device (Y).
(b) The driver doesn't know that there is the dependency (there is a framework
    having that information, but drivers only access it indirectly, for
    example).

In principle (a) may be addressed by the driver itself if it is careful enough
to reference count Y in addition to X during runtime PM operations.  The async
(system) suspend case is more problematic, because it also requies the driver
of Y to wait for X to suspend (that is not required if both X and Y are "sync",
but that case is kind of suboptimal).

However, (b) cannot be taken care of by the driver itself and it looks like some
coordination mechanism would be necessary to address it.

I proposed something like that a few years ago (https://lkml.org/lkml/2009/8/26/485),
but people generally didn't like it and it has locking problems.  Admittedly, I'd
do it differently today, but if there's interest, I can refresh that approach.

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

On Wed, May 14, 2014 at 02:32:07PM +0200, Rafael J. Wysocki wrote:
> On Wednesday, May 14, 2014 12:57:45 PM Mark Brown wrote:

> > Since the core device probe is deferred until everything it depends on
> > is ready it'll get pushed along the device list so that it appears after
> > everything it needs and hence is suspended first.

> Except for async suspend, that is?

Gah, I'd not actually seen that before - few of the systems I've worked
on would have changed it anyway.  Yes, that'll break things too.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

On Wednesday 14 May 2014 12:57:45 Mark Brown wrote:
> On Wed, May 14, 2014 at 02:11:25PM +0200, Rafael J. Wysocki wrote:
> > On Monday, May 12, 2014 09:31:53 PM Mark Brown wrote:
> > > On Mon, May 12, 2014 at 10:27:14PM +0200, Laurent Pinchart wrote:
> > > > That solves the probe time dependencies, but unfortunately not the
> > > > runtime PM dependencies.
> > > 
> > > It also solves the system suspend dependencies.
> > 
> > Well, I'm not sure about that.  How does it do that, exactly?
> 
> Since the core device probe is deferred until everything it depends on
> is ready it'll get pushed along the device list so that it appears after
> everything it needs and hence is suspended first.

That might solve (part of) the master vs. components dependencies (and even in 
that case I'm pretty sure that there will be cases where the master needs to 
be suspended first and cases where th slave(s) need to be suspended first), 
but it doesn't handle inter-slaves dependencies.

In the V4L subsystem we've implemented an explicit power management API that 
"slaves" (called subdevices in V4L) must implement, to be called by the 
master. That's of course more of a workaround than a perfect (or even good) 
solution.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

On Wed, May 14, 2014 at 05:26:20PM +0200, Laurent Pinchart wrote:
> On Wednesday 14 May 2014 12:57:45 Mark Brown wrote:

> > Since the core device probe is deferred until everything it depends on
> > is ready it'll get pushed along the device list so that it appears after
> > everything it needs and hence is suspended first.

> That might solve (part of) the master vs. components dependencies (and even in 
> that case I'm pretty sure that there will be cases where the master needs to 
> be suspended first and cases where th slave(s) need to be suspended first), 
> but it doesn't handle inter-slaves dependencies.

> In the V4L subsystem we've implemented an explicit power management API that 
> "slaves" (called subdevices in V4L) must implement, to be called by the 
> master. That's of course more of a workaround than a perfect (or even good) 
> solution.

Right, that's what the function of the master device is in suspend for
this model - it quiesces the compound device as a whole so that any
interdependencies between the devices have been unwound (including
giving callbacks to the slaves where appropriate).  It gives us a place
to hang everything associated with having the devices so closely
interrelated.

Though as Rafael says with async suspend that gets broken.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Kevin Hilman]

Mark Brown <broonie@kernel.org> writes:

> On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>> On 12.05.2014 22:31, Mark Brown wrote:
>
>> > It also solves the system suspend dependencies.  Why don't the
>> > runtime PM dependencies just work with reference counting?
>
>> Runtime PM dependencies work with reference counting just fine, but
>> only for topologies matching Linux driver model, e.g. devices with
>> exactly one device they depend on, e.g. SPI controller and SPI devices
>> on the bus driven by it. Add there an IOMMU and other various strange
>> things that should be transparent to the drivers and it stops working.
>
> There's no reason why runtime PM references have to follow the topology
> - you do get a default reference count up to any parent (though we break
> that sometimes, as is the case with SPI controllers being suspended even
> though the devices below them are active) but there's nothing stopping
> references being taken outside the topology.

I'm very interested in this topic as well.

Though, I'm reluctant to see new APIs invented when I think already have
the infrastructure to handle this (though admittedly, I haven't seen all
the use-cases where this is problematic either.)

In my experience with this, it seems to me the root cause here is simply
that there are still lots of drivers/subsytems that are not runtime PM
adapted.

IOW, even if device X has a non-parent/child relationship with device Y,
if both are runtime PM adapted, the simple fact of X *using* Y (or using
the framework hiding Y) will be enough to ensure Y is powered up when
used.  

Yes, we could hack up the ability for X to directly refcount Y, but why
should we do that instead of adapdting Y to use runtime PM itself so
that when ever it's requested/used, it's powered up by the runtime PM
infrastructure.

What am I missing ?

>> I'm still investigating this issue, so more uses cases are yet to be
>> found, but I also guess this is the purpose of this thread. Anyway,
>> for some reason .suspend_late() and .resume_early() callbacks exist in
>> dev_pm_ops struct and I believe that at least some of the cases
>> "solved" by them might be related to the issue being discussed here.
>
> Yes, they're partly solving a particular common case for this sort of
> interdependency (though I guess they do also do things like allow us to
> make sure the hardware came back in a state where it won't be harmful
> to the rest of the system if we start enabling things).

IMO, the suspend_late/resume_early callbacks are useful for turning
drivers/subsystems into what I call "runtime PM centric".  That means
the driver/subsytem can focus on a runtime PM centric view of the world,
and then implement suspend/resume using the late/early callbacks to call
the same functions as the runtime suspend/resume.

Kevin

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

On Wednesday 14 May 2014 00:27:47 Rafael J. Wysocki wrote:
> On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> > > On 12.05.2014 22:31, Mark Brown wrote:
> > > > It also solves the system suspend dependencies.  Why don't the
> > > > runtime PM dependencies just work with reference counting?
> > > 
> > > Runtime PM dependencies work with reference counting just fine, but
> > > only for topologies matching Linux driver model, e.g. devices with
> > > exactly one device they depend on, e.g. SPI controller and SPI devices
> > > on the bus driven by it. Add there an IOMMU and other various strange
> > > things that should be transparent to the drivers and it stops working.
> > 
> > There's no reason why runtime PM references have to follow the topology
> > - you do get a default reference count up to any parent (though we break
> > that sometimes, as is the case with SPI controllers being suspended even
> > though the devices below them are active) but there's nothing stopping
> > references being taken outside the topology.
> 
> Precisely.
> 
> > > I'm still investigating this issue, so more uses cases are yet to be
> > > found, but I also guess this is the purpose of this thread. Anyway,
> > > for some reason .suspend_late() and .resume_early() callbacks exist in
> > > dev_pm_ops struct and I believe that at least some of the cases
> > > "solved" by them might be related to the issue being discussed here.
> > 
> > Yes, they're partly solving a particular common case for this sort of
> > interdependency (though I guess they do also do things like allow us to
> > make sure the hardware came back in a state where it won't be harmful
> > to the rest of the system if we start enabling things).
> 
> The original reason why .suspend_late/.resume_early were introduced was to
> allow the same callback routines to be run for runtime and for system-wide
> PM. Specifically, the idea was that .suspend_late may point to the same
> routine as .runtime_suspend, for example, because of the analogy of the
> conditions in which they are called (the device is known to be not in use).
> 
> That didn't work out perfectly, but those callbacks turn out to be useful
> nonetheless.
> 
> And yes, I'm interested in this topic too.
> 
> Rafael (who wonders why he wasn't CCed in this thread to start with)

It was a honest mistake, I'm sorry about that.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> > > > On 12.05.2014 22:31, Mark Brown wrote:
> > > > > It also solves the system suspend dependencies.  Why don't the
> > > > > runtime PM dependencies just work with reference counting?
> > > > 
> > > > Runtime PM dependencies work with reference counting just fine, but
> > > > only for topologies matching Linux driver model, e.g. devices with
> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
> > > > on the bus driven by it. Add there an IOMMU and other various strange
> > > > things that should be transparent to the drivers and it stops working.
> > > 
> > > There's no reason why runtime PM references have to follow the topology
> > > - you do get a default reference count up to any parent (though we break
> > > that sometimes, as is the case with SPI controllers being suspended even
> > > though the devices below them are active) but there's nothing stopping
> > > references being taken outside the topology.
> > 
> > Precisely.
> 
> BTW, I guess that the problem is resume and specifically the fact that if
> a child device resumes, the parent will also resume automatically, but the
> other devices the child may depend on will not (the child's resume may need
> to resume them directly).
> 
> But I'm not sure why that is a problem, so can anyone please share some
> details?

Here are two real life examples.

1. IOMMU and bus master

Bus master devices connected to an IOMMU need the IOMMU to be powered on in 
order to access memory. In order to save power the IOMMU should of course be 
powered off as much as possible.

The tricky part here comes from the fact that the IOMMU is hidden behind the 
DMA API. The bus master driver can't manage the IOMMU power state eplicitly by 
taking/releasing references to the IOMMU device.

This could easily be solved in an ad-hoc fashion by extending the DMA mapping 
API, but I'm wondering whether similar issues wouldn't benefit from a common 
solution (I'm not sure yet what all the similar issues are, hence the topic 
proposal to try and gather use cases).

2. Composite devices

I'll take an embedded camera devices as an example as that's the case I know 
better, but the same problem occurs on the display side as well.

Camera devices on embedded systems are usually made of a camera interface 
inside an SoC (possibly split into several IP cores, such as a CSI - Camera 
Serial Interface - receiver and a backend) and one or more external devices, 
such as sensors, flash controller, lens controller or sometimes dedicated 
video processing accelerators.

The external devices are most of the time controlled through I2C (SPI is an 
option as well, the exact interface doesn't matter much). Those that handle 
video data streams (sensor, video processor) use dedicated parallel or high 
speed serial busses.

Most of the external devices require an input clock. The clock can be fixed 
(easy), provided by a dedicated chip or IP core in the SoC (easy as well), 
sometimes by the SoC camera interface or even by one of the other external 
devices (for instance the lens controller could be clocked by the pixel clock 
output by the sensor - that's a bit far-fetched - or, more boring but equally 
annoying, I've seen a camera interface in an SoC that required the pixel clock 
output by the sensor to be running in order to complete its reset sequence).

On OMAP3 systems implementing a camera the image sensor is usually supplied 
with a clock output by the ISP (Image Signal Processor, the camera interface). 
When resuming the system from suspend (assuming the ISP was capturing video 
when the system got suspended), it's important to restart the ISP first and 
only then restart the sensor. To ensure that the order is followed the ISP 
driver resume path first restarts the ISP and then calls the sensor driver to 
start the video stream. The sensor driver will need to send I2C messages to 
the device in order to start it, which requires the I2C controller to be 
resumed. As the ISP is a platform device at the same level as the I2C 
controller, that ordering is not guaranteed.

Furthermore, if the sensor is resumed first, it might try to access the 
device, which requires the clock output by the ISP to be available, and thus 
requires the ISP to be resumed. To solve this problem the ISP driver only 
restarts the clocks in its PM resume callback, and restarts the video stream 
(following the sequence described above) in its PM complete callback.

When adding more external devices to the mix the problem just becomes more 
complex, especially when the devices are chained (for instance sensor -> video 
processor -> ISP). The problem is similar on the display side, possibly with a 
different resume ordering (it should be noted that the external devices vs. 
internal device ordering might vary even inside the same class of devices - 
camera or display).

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

On Tuesday 13 May 2014 08:26:09 Shuah Khan wrote:
> On Tue, May 13, 2014 at 4:31 AM, Laurent Pinchart wrote:
> > On Tuesday 13 May 2014 09:43:05 Daniel Vetter wrote:
> >> On Tue, May 13, 2014 at 12:07 AM, Mark Brown <broonie@kernel.org> wrote:
> >> > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> >> >> On 12.05.2014 22:31, Mark Brown wrote:
> >> >> > It also solves the system suspend dependencies.  Why don't the
> >> >> > runtime PM dependencies just work with reference counting?
> >> >> 
> >> >> Runtime PM dependencies work with reference counting just fine, but
> >> >> only for topologies matching Linux driver model, e.g. devices with
> >> >> exactly one device they depend on, e.g. SPI controller and SPI devices
> >> >> on the bus driven by it. Add there an IOMMU and other various strange
> >> >> things that should be transparent to the drivers and it stops working.
> >> > 
> >> > There's no reason why runtime PM references have to follow the topology
> >> > - you do get a default reference count up to any parent (though we
> >> > break that sometimes, as is the case with SPI controllers being
> >> > suspended even though the devices below them are active) but there's
> >> > nothing stopping references being taken outside the topology.
> > 
> > One of the issues is that we need API(s) to take/release those references.
> > In some cases the depender doesn't have direct access to the dependee,
> > like in the IOMMU case where the DMA mapping API hides the IOMMU. One of
> > the questions here is whether this kind of problems should be solved with
> > ad-hoc solutions (for instance adding PM support to the DMA mapping API
> > in this specific case), or in a more generic fashion. Answering that will
> > require a comprehensive list of use cases to detect common patterns.
> > 
> >> I guess some helpers to grab/drop runtime PM references on all parts
> >> of a componentized device should resolve this?
> > 
> > That could help in some cases, but there might be ordering issues.
> 
> Can we use pm domains concept to solve the problem? As it exists today, it
> probably can't support loosely associated devices, however, could it be
> extended to support handling device groups that don't necessarily share
> power source and/or clock.

If I understand them properly, power domains model groups of devices that 
share common power handling. I'm not sure how they could be used to model PM 
dependencies between devices. Feel free to prove me wrong though :-)

> I started looking at this as a way to solve media device PM issues,
> however I haven't had a chance to experiment with this idea.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mauro Carvalho Chehab]

Em Mon, 12 May 2014 11:51:54 -0600
Shuah Khan <shuahkhan@gmail.com> escreveu:

> On Mon, May 12, 2014 at 11:43 AM, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> > Hello,
> >
> > On modern systems many PM dependencies don't follow the Linux kernel device
> > model based on parent-child relationships from a control bus point of view.
> > For instance a GPU will need the IOMMU that services its memory requests to be
> > powered on to perform DMA operations.
> >
> > Ad-hoc solutions with subsystem-specific or even driver-specific APIs have
> > been implemented (see omap_iommu_save_ctx and omap_iommu_restore_ctx in
> > include/linux/omap-iommu.h for instance, that push the burden of saving and
> > restoring the IOMMU registers to bus master drivers), but they make automatic
> > handling of hardware resources difficult. In the IOMMU case again, the goal is
> > to hide IOMMU handling inside the DMA mapping API and make its usage
> > completely transparent to bus master drivers in most of the cases. An IOMMU-
> > specific API to explicitly control IOMMU power from the bus master driver
> > would make that goal impossible to reach.
> >
> > The problem is not limited to IOMMUs. We have similar dependencies at
> > suspend/resume time with camera interfaces for instance, where two completely
> > unrelated device in the Linux device hierarchy (a camera interface platform
> > device in the SoC and an I2C camera sensor) need to be suspended and resumed
> > in a controlled order. I'm sure many more use cases exist.
> 
> We discussed this topic briefly at the Media Mini-summit.  I looking
> into similar
> use-cases in media devices. Suspend/resume of a tuner for example should only
> be done once as opposed multiple times by drivers that think they have exclusive
> access to the tuner on a media device.
> 
> I am interested in this discussion and would like to participate.
> Requesting nomination.

PM on media devices is very complex, and this is broken since... ever.

From time to time, someone comes with some patch that fixes it for some
specific drivers, but, after some Kernel cycles, things get broken
again, because the approach taken by those patches were to add a logic
inside the specific PCI (or USB) bridge driver that simply unbinds
all other drivers at suspend, and re-initialize all of them at resume.

Since the end of last year, both me and Shuah are working together in
order to find some strategies to fix it.

One of the problems with media devices is that they are typically multi
function devices that share some resources on very complex ways. So,
both suspend and resume operations should be serialized in a way that
will warrant that the devices internal clocks, GPIOs, I2C buses, I2C,
switches, IOMMUs etc are initialized before talking to each component 
of the hardware that depends on it.

On several such hardware, there are components that are optional, like
IR receivers, IR transmitters, etc, and are only known to be available
at probe time.

Also, some devices like hybrid TV ones disables part of the hardware
when in usage, as they can't typically stream analog TV (V4L2 API and
v4l subsystem) while an ATSC stream is displayed (DVB API and dvb
subsystem).

In other words, the driver would need to build a PM dependency tree
in probe time, adjusting it in runtime, depending on what sub-devices
are in use, in order to assure that the stream will continue being
played after resume.

I'd like to nominate myself and Shuah for this topic.

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Shuah Khan]

On Thu, May 15, 2014 at 5:43 PM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Tuesday 13 May 2014 08:26:09 Shuah Khan wrote:
>>
>> Can we use pm domains concept to solve the problem? As it exists today, it
>> probably can't support loosely associated devices, however, could it be
>> extended to support handling device groups that don't necessarily share
>> power source and/or clock.
>
> If I understand them properly, power domains model groups of devices that
> share common power handling. I'm not sure how they could be used to model PM
> dependencies between devices. Feel free to prove me wrong though :-)
>
>> I started looking at this as a way to solve media device PM issues,
>> however I haven't had a chance to experiment with this idea.

Right now, PM domains currently support grouping of devices that share
power source. What if we extended PM domains to support a concept of
logical grouping of devices that have PM dependencies with a way to
specify dependency ordering?

For example, a media usb device example:
The main driver could create a logical power domain and rest of the
drivers add their devices to the same domain specifying their
dependencies. logical power domain pm handlers can handle ordering. In
some cases, there is a need to save and restore state of tuner i2c and
the existing save_state and restore_state could be used to within the
scope of this logical domain.

I think PM domains can definitely be used for media device case, but I
am not sure about the specific case you are looking at where there is
a clearly identified master driver/device.

These are at the moment ideas and I need to do some work to test these
theories. :)

-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Geert Uytterhoeven]

On Mon, May 19, 2014 at 3:00 AM, Shuah Khan <shuahkhan@gmail.com> wrote:
> On Thu, May 15, 2014 at 5:43 PM, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> On Tuesday 13 May 2014 08:26:09 Shuah Khan wrote:
>>>
>>> Can we use pm domains concept to solve the problem? As it exists today, it
>>> probably can't support loosely associated devices, however, could it be
>>> extended to support handling device groups that don't necessarily share
>>> power source and/or clock.
>>
>> If I understand them properly, power domains model groups of devices that
>> share common power handling. I'm not sure how they could be used to model PM
>> dependencies between devices. Feel free to prove me wrong though :-)
>>
>>> I started looking at this as a way to solve media device PM issues,
>>> however I haven't had a chance to experiment with this idea.
>
> Right now, PM domains currently support grouping of devices that share
> power source. What if we extended PM domains to support a concept of

Physical power domain...

> logical grouping of devices that have PM dependencies with a way to
> specify dependency ordering?
>
> For example, a media usb device example:
> The main driver could create a logical power domain and rest of the
> drivers add their devices to the same domain specifying their
> dependencies. logical power domain pm handlers can handle ordering. In

... and logical power domain...

> some cases, there is a need to save and restore state of tuner i2c and
> the existing save_state and restore_state could be used to within the
> scope of this logical domain.
>
> I think PM domains can definitely be used for media device case, but I
> am not sure about the specific case you are looking at where there is
> a clearly identified master driver/device.

Sure, it can be done.

> These are at the moment ideas and I need to do some work to test these
> theories. :)

.. and we also have clock power domains (cfr. pm_clk_notify()).

Now we do have hardware that would benefit from all three of the above.
AFAIK a device can be a member of only one power domain, through
the struct device.pm_domain pointer. Can/should this be extended,
cfr. the RPM_GET_CALLBACK() stack for Runtime PM (but we do want
to call into more than one of them).

I'm still too pm_domain-illiterate to say much more about this though...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Kevin Hilman]

Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:

> On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
>> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
>> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
>> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>> > > > On 12.05.2014 22:31, Mark Brown wrote:
>> > > > > It also solves the system suspend dependencies.  Why don't the
>> > > > > runtime PM dependencies just work with reference counting?
>> > > > 
>> > > > Runtime PM dependencies work with reference counting just fine, but
>> > > > only for topologies matching Linux driver model, e.g. devices with
>> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
>> > > > on the bus driven by it. Add there an IOMMU and other various strange
>> > > > things that should be transparent to the drivers and it stops working.
>> > > 
>> > > There's no reason why runtime PM references have to follow the topology
>> > > - you do get a default reference count up to any parent (though we break
>> > > that sometimes, as is the case with SPI controllers being suspended even
>> > > though the devices below them are active) but there's nothing stopping
>> > > references being taken outside the topology.
>> > 
>> > Precisely.
>> 
>> BTW, I guess that the problem is resume and specifically the fact that if
>> a child device resumes, the parent will also resume automatically, but the
>> other devices the child may depend on will not (the child's resume may need
>> to resume them directly).
>> 
>> But I'm not sure why that is a problem, so can anyone please share some
>> details?
>
> Here are two real life examples.
>
> 1. IOMMU and bus master
>
> Bus master devices connected to an IOMMU need the IOMMU to be powered on in 
> order to access memory. In order to save power the IOMMU should of course be 
> powered off as much as possible.
>
> The tricky part here comes from the fact that the IOMMU is hidden behind the 
> DMA API. The bus master driver can't manage the IOMMU power state eplicitly by 
> taking/releasing references to the IOMMU device.
>
> This could easily be solved in an ad-hoc fashion by extending the DMA mapping 
> API, but I'm wondering whether similar issues wouldn't benefit from a common 
> solution (I'm not sure yet what all the similar issues are, hence the topic 
> proposal to try and gather use cases).

I'll continue to beat the runtime PM drum...

This would also easily be solved if the bus master device, the IOMMU and
the dmaengine (or the platform-specific dma driver) were all using
runtime PM.  e.g. bus master device is a user of dmaengine, which is a
user of the IOMMU.  If all are using runtime PM, the fact that a device
is "in use", means runtime PM would keep them active when needed.  In
this example, it wouldn't matter that the bus master device doesn't know
about the IOMMU. It suffices that the dmaengine driver knows about the
IOMMU.

> 2. Composite devices
>
> I'll take an embedded camera devices as an example as that's the case I know 
> better, but the same problem occurs on the display side as well.
>
> Camera devices on embedded systems are usually made of a camera interface 
> inside an SoC (possibly split into several IP cores, such as a CSI - Camera 
> Serial Interface - receiver and a backend) and one or more external devices, 
> such as sensors, flash controller, lens controller or sometimes dedicated 
> video processing accelerators.
>
> The external devices are most of the time controlled through I2C (SPI is an 
> option as well, the exact interface doesn't matter much). Those that handle 
> video data streams (sensor, video processor) use dedicated parallel or high 
> speed serial busses.
>
> Most of the external devices require an input clock. The clock can be fixed 
> (easy), provided by a dedicated chip or IP core in the SoC (easy as well), 
> sometimes by the SoC camera interface or even by one of the other external 
> devices (for instance the lens controller could be clocked by the pixel clock 
> output by the sensor - that's a bit far-fetched - or, more boring but equally 
> annoying, I've seen a camera interface in an SoC that required the pixel clock 
> output by the sensor to be running in order to complete its reset sequence).
>
> On OMAP3 systems implementing a camera the image sensor is usually supplied 
> with a clock output by the ISP (Image Signal Processor, the camera interface). 
> When resuming the system from suspend (assuming the ISP was capturing video 
> when the system got suspended), it's important to restart the ISP first and 
> only then restart the sensor. To ensure that the order is followed the ISP 
> driver resume path first restarts the ISP and then calls the sensor driver to 
> start the video stream. The sensor driver will need to send I2C messages to 
> the device in order to start it, which requires the I2C controller to be 
> resumed. As the ISP is a platform device at the same level as the I2C 
> controller, that ordering is not guaranteed.

As you likely already know, at least on OMAP, the I2C ordering would
work just fine because the I2C driver is (what I like to call) runtime
PM centric.  IOW, it doesn't really implement suspend/resume, instead it
only does runtime PM "on demand" on a per-xfer basis.  That means that
the I2C driver can be used anytime during the [runtime] suspend/resume
of any other device without any ordering problems because it's activated
on demand.

Extending this same "runtime PM centric" view of the world to the other
devices, I think that the runtime PM reference counting invovled when
all the components are using runtime PM would solve this problem.

(I realize that this I2C solution doesn't solve the more general "resume
ISP before sensor" problem though, but with better modeling, I think it
can.  More on that below...)

> Furthermore, if the sensor is resumed first, it might try to access the 
> device, which requires the clock output by the ISP to be available, and thus 
> requires the ISP to be resumed. To solve this problem the ISP driver only 
> restarts the clocks in its PM resume callback, and restarts the video stream 
> (following the sequence described above) in its PM complete callback.

For most devices, input clocks are modeled by the clock framework (or
managed by the SoC's runtime PM core), and therefore, a pm_runtime_get()
(or possibly an explict clk_enable()) is used to ensure the input clock
is running.  In this external device example, it sounds to me like the
sensor driver has no knowledge of its input clock so it has to rely on
some other layer to resume things in the right order for correct
functionality.

Maybe I'm wrong here (likely, since I haven't looked at the code, and am
admittedly very ignorant of the camera and display subsystems) but it
sounds to me like what's missing is the sensor driver having knowledge
of it's input clock and/or a way for it to request it's input clock to
be enabled (e.g. clk_get/clk_enable.)

Alternatively, what would proably be even better would be that the
sensor driver has a reference to the actual device that provides its
input clock (possibly via a DT phandle?) so that the sensor driver can
simply do a pm_runtime_get() on the device providing the clock.

> When adding more external devices to the mix the problem just becomes more 
> complex, especially when the devices are chained (for instance sensor -> video 
> processor -> ISP). The problem is similar on the display side, possibly with a 
> different resume ordering (it should be noted that the external devices vs. 
> internal device ordering might vary even inside the same class of devices - 
> camera or display).

IMO, I still think that properly modeling the device dependenies
combined with a "runtime PM centric" view of suspend/resume should allow
the dependencies to be handled correctly for system suspend/resume and
runtime PM.

I think what complicates things here is not the PM specifics but
probably the fact that the device hierarchy (and dependencies) may be
dynamic depending on many factors like which sensors are in use,
post-processing, etc. etc.

Above, I suggested possibly using DT phandles to model these non
parent/child relationships.  That's all fine if the dependencies are not
changing, but if they are dynamic, we'll probably need something
different.

At least for starters though, static dependencies like this should be
(relatively) easy to model in DT and combined with runtime PM
refcounding on the dependencies, should be able to address the problem.

Kevin

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

On Tue, May 20, 2014 at 09:57:14AM -0700, Kevin Hilman wrote:

> For most devices, input clocks are modeled by the clock framework (or
> managed by the SoC's runtime PM core), and therefore, a pm_runtime_get()
> (or possibly an explict clk_enable()) is used to ensure the input clock
> is running.  In this external device example, it sounds to me like the
> sensor driver has no knowledge of its input clock so it has to rely on
> some other layer to resume things in the right order for correct
> functionality.

> Maybe I'm wrong here (likely, since I haven't looked at the code, and am
> admittedly very ignorant of the camera and display subsystems) but it
> sounds to me like what's missing is the sensor driver having knowledge
> of it's input clock and/or a way for it to request it's input clock to
> be enabled (e.g. clk_get/clk_enable.)

> Alternatively, what would proably be even better would be that the
> sensor driver has a reference to the actual device that provides its
> input clock (possibly via a DT phandle?) so that the sensor driver can
> simply do a pm_runtime_get() on the device providing the clock.

We have this problem on ASoC too and for DT converted devices we've been
doing exactly as you suggest and just teaching the device it has a clock
and letting it work out what to do with it.  Ideally we'd be able to
rely on having a clock API (better yet, the common clock API
implementation) and just put the entire clock tree in there.  It's not
been causing us any problems thus far but it's not that widely rolled
out yet either.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Ulf Hansson]

On 20 May 2014 18:57, Kevin Hilman <khilman@linaro.org> wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
>
>> On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
>>> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
>>> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
>>> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>>> > > > On 12.05.2014 22:31, Mark Brown wrote:
>>> > > > > It also solves the system suspend dependencies.  Why don't the
>>> > > > > runtime PM dependencies just work with reference counting?
>>> > > >
>>> > > > Runtime PM dependencies work with reference counting just fine, but
>>> > > > only for topologies matching Linux driver model, e.g. devices with
>>> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
>>> > > > on the bus driven by it. Add there an IOMMU and other various strange
>>> > > > things that should be transparent to the drivers and it stops working.
>>> > >
>>> > > There's no reason why runtime PM references have to follow the topology
>>> > > - you do get a default reference count up to any parent (though we break
>>> > > that sometimes, as is the case with SPI controllers being suspended even
>>> > > though the devices below them are active) but there's nothing stopping
>>> > > references being taken outside the topology.
>>> >
>>> > Precisely.
>>>
>>> BTW, I guess that the problem is resume and specifically the fact that if
>>> a child device resumes, the parent will also resume automatically, but the
>>> other devices the child may depend on will not (the child's resume may need
>>> to resume them directly).
>>>
>>> But I'm not sure why that is a problem, so can anyone please share some
>>> details?
>>
>> Here are two real life examples.
>>
>> 1. IOMMU and bus master
>>
>> Bus master devices connected to an IOMMU need the IOMMU to be powered on in
>> order to access memory. In order to save power the IOMMU should of course be
>> powered off as much as possible.
>>
>> The tricky part here comes from the fact that the IOMMU is hidden behind the
>> DMA API. The bus master driver can't manage the IOMMU power state eplicitly by
>> taking/releasing references to the IOMMU device.
>>
>> This could easily be solved in an ad-hoc fashion by extending the DMA mapping
>> API, but I'm wondering whether similar issues wouldn't benefit from a common
>> solution (I'm not sure yet what all the similar issues are, hence the topic
>> proposal to try and gather use cases).
>
> I'll continue to beat the runtime PM drum...
>
> This would also easily be solved if the bus master device, the IOMMU and
> the dmaengine (or the platform-specific dma driver) were all using
> runtime PM.  e.g. bus master device is a user of dmaengine, which is a
> user of the IOMMU.  If all are using runtime PM, the fact that a device
> is "in use", means runtime PM would keep them active when needed.  In
> this example, it wouldn't matter that the bus master device doesn't know
> about the IOMMU. It suffices that the dmaengine driver knows about the
> IOMMU.
>
>> 2. Composite devices
>>
>> I'll take an embedded camera devices as an example as that's the case I know
>> better, but the same problem occurs on the display side as well.
>>
>> Camera devices on embedded systems are usually made of a camera interface
>> inside an SoC (possibly split into several IP cores, such as a CSI - Camera
>> Serial Interface - receiver and a backend) and one or more external devices,
>> such as sensors, flash controller, lens controller or sometimes dedicated
>> video processing accelerators.
>>
>> The external devices are most of the time controlled through I2C (SPI is an
>> option as well, the exact interface doesn't matter much). Those that handle
>> video data streams (sensor, video processor) use dedicated parallel or high
>> speed serial busses.
>>
>> Most of the external devices require an input clock. The clock can be fixed
>> (easy), provided by a dedicated chip or IP core in the SoC (easy as well),
>> sometimes by the SoC camera interface or even by one of the other external
>> devices (for instance the lens controller could be clocked by the pixel clock
>> output by the sensor - that's a bit far-fetched - or, more boring but equally
>> annoying, I've seen a camera interface in an SoC that required the pixel clock
>> output by the sensor to be running in order to complete its reset sequence).
>>
>> On OMAP3 systems implementing a camera the image sensor is usually supplied
>> with a clock output by the ISP (Image Signal Processor, the camera interface).
>> When resuming the system from suspend (assuming the ISP was capturing video
>> when the system got suspended), it's important to restart the ISP first and
>> only then restart the sensor. To ensure that the order is followed the ISP
>> driver resume path first restarts the ISP and then calls the sensor driver to
>> start the video stream. The sensor driver will need to send I2C messages to
>> the device in order to start it, which requires the I2C controller to be
>> resumed. As the ISP is a platform device at the same level as the I2C
>> controller, that ordering is not guaranteed.
>
> As you likely already know, at least on OMAP, the I2C ordering would
> work just fine because the I2C driver is (what I like to call) runtime
> PM centric.  IOW, it doesn't really implement suspend/resume, instead it
> only does runtime PM "on demand" on a per-xfer basis.  That means that
> the I2C driver can be used anytime during the [runtime] suspend/resume
> of any other device without any ordering problems because it's activated
> on demand.
>
> Extending this same "runtime PM centric" view of the world to the other
> devices, I think that the runtime PM reference counting invovled when
> all the components are using runtime PM would solve this problem.

I like this approach, but we should know that there are some complex
issues you need to address when using it it.

For reasons, the PM core prevent runtime PM suspend during system
suspend, (it invokes pm_runtime_get_noresume() for each device).
Especially, this means you can't guarantee the I2C driver to hold it's
device in in-activated state while the platform is in system suspend
state - unless you explicitly takes care of it from a system PM
callback, somehow and somewhere.

One potential help here, could be to use the new runtime PM helper
functions, pm_runtime_force_suspend|resume().

Kind regards
Ulf Hansson

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Geert Uytterhoeven]

On Tue, May 20, 2014 at 6:57 PM, Kevin Hilman <khilman@linaro.org> wrote:
>> Furthermore, if the sensor is resumed first, it might try to access the
>> device, which requires the clock output by the ISP to be available, and thus
>> requires the ISP to be resumed. To solve this problem the ISP driver only
>> restarts the clocks in its PM resume callback, and restarts the video stream
>> (following the sequence described above) in its PM complete callback.
>
> For most devices, input clocks are modeled by the clock framework (or
> managed by the SoC's runtime PM core), and therefore, a pm_runtime_get()
> (or possibly an explict clk_enable()) is used to ensure the input clock
> is running.  In this external device example, it sounds to me like the
> sensor driver has no knowledge of its input clock so it has to rely on
> some other layer to resume things in the right order for correct
> functionality.
>
> Maybe I'm wrong here (likely, since I haven't looked at the code, and am
> admittedly very ignorant of the camera and display subsystems) but it
> sounds to me like what's missing is the sensor driver having knowledge
> of it's input clock and/or a way for it to request it's input clock to
> be enabled (e.g. clk_get/clk_enable.)

For static configuration, this could indeed be put in DT.

> Alternatively, what would proably be even better would be that the
> sensor driver has a reference to the actual device that provides its
> input clock (possibly via a DT phandle?) so that the sensor driver can
> simply do a pm_runtime_get() on the device providing the clock.

Why not a clocks property instead of a phandle?
Of course the actual device providing the clock needs to advertise
itself as a clock (e.g. in DT).

>> When adding more external devices to the mix the problem just becomes more
>> complex, especially when the devices are chained (for instance sensor -> video
>> processor -> ISP). The problem is similar on the display side, possibly with a
>> different resume ordering (it should be noted that the external devices vs.
>> internal device ordering might vary even inside the same class of devices -
>> camera or display).
>
> IMO, I still think that properly modeling the device dependenies
> combined with a "runtime PM centric" view of suspend/resume should allow
> the dependencies to be handled correctly for system suspend/resume and
> runtime PM.
>
> I think what complicates things here is not the PM specifics but
> probably the fact that the device hierarchy (and dependencies) may be
> dynamic depending on many factors like which sensors are in use,
> post-processing, etc. etc.
>
> Above, I suggested possibly using DT phandles to model these non
> parent/child relationships.  That's all fine if the dependencies are not
> changing, but if they are dynamic, we'll probably need something
> different.
>
> At least for starters though, static dependencies like this should be
> (relatively) easy to model in DT and combined with runtime PM
> refcounding on the dependencies, should be able to address the problem.

I'm still looking for a more generic way to handle such clocks, perhaps in
the driver core. Right now the driver is supposed to handle this, so it needs
to be aware of the existence of the clock, which may be optional.
E.g. if device modules in a new SoC suddenly gets gateable clocks,
do we have to modify all affected drivers? It would be nice if it would
"just work" by adding the clocks to DT.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Tuesday, May 20, 2014 09:57:14 AM Kevin Hilman wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> 
> > On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
> >> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> >> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> >> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> >> > > > On 12.05.2014 22:31, Mark Brown wrote:
> >> > > > > It also solves the system suspend dependencies.  Why don't the
> >> > > > > runtime PM dependencies just work with reference counting?
> >> > > > 
> >> > > > Runtime PM dependencies work with reference counting just fine, but
> >> > > > only for topologies matching Linux driver model, e.g. devices with
> >> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
> >> > > > on the bus driven by it. Add there an IOMMU and other various strange
> >> > > > things that should be transparent to the drivers and it stops working.
> >> > > 
> >> > > There's no reason why runtime PM references have to follow the topology
> >> > > - you do get a default reference count up to any parent (though we break
> >> > > that sometimes, as is the case with SPI controllers being suspended even
> >> > > though the devices below them are active) but there's nothing stopping
> >> > > references being taken outside the topology.
> >> > 
> >> > Precisely.
> >> 
> >> BTW, I guess that the problem is resume and specifically the fact that if
> >> a child device resumes, the parent will also resume automatically, but the
> >> other devices the child may depend on will not (the child's resume may need
> >> to resume them directly).
> >> 
> >> But I'm not sure why that is a problem, so can anyone please share some
> >> details?
> >
> > Here are two real life examples.
> >
> > 1. IOMMU and bus master
> >
> > Bus master devices connected to an IOMMU need the IOMMU to be powered on in 
> > order to access memory. In order to save power the IOMMU should of course be 
> > powered off as much as possible.
> >
> > The tricky part here comes from the fact that the IOMMU is hidden behind the 
> > DMA API. The bus master driver can't manage the IOMMU power state eplicitly by 
> > taking/releasing references to the IOMMU device.
> >
> > This could easily be solved in an ad-hoc fashion by extending the DMA mapping 
> > API, but I'm wondering whether similar issues wouldn't benefit from a common 
> > solution (I'm not sure yet what all the similar issues are, hence the topic 
> > proposal to try and gather use cases).
> 
> I'll continue to beat the runtime PM drum...

What about async system suspend/resume, then?

> This would also easily be solved if the bus master device, the IOMMU and
> the dmaengine (or the platform-specific dma driver) were all using
> runtime PM.  e.g. bus master device is a user of dmaengine, which is a
> user of the IOMMU.  If all are using runtime PM, the fact that a device
> is "in use", means runtime PM would keep them active when needed.  In
> this example, it wouldn't matter that the bus master device doesn't know
> about the IOMMU. It suffices that the dmaengine driver knows about the
> IOMMU.

That requires it to be able to say "now I'm not using you any more" whenever
the device in question suspend and "now I'm going to use you again" when that
device resumes.  Does something like that happen today?

There are more weird cases still.  For example, we have the _DEP object in
ACPI that basically says "this device depends on that one" and there may be
no other relationship between the two whatsoever.  How are we supposed to
implement this within the existing frameworks?

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

On Thu, May 22, 2014 at 02:19:35AM +0200, Rafael J. Wysocki wrote:
> On Tuesday, May 20, 2014 09:57:14 AM Kevin Hilman wrote:

> > This would also easily be solved if the bus master device, the IOMMU and
> > the dmaengine (or the platform-specific dma driver) were all using
> > runtime PM.  e.g. bus master device is a user of dmaengine, which is a
> > user of the IOMMU.  If all are using runtime PM, the fact that a device
> > is "in use", means runtime PM would keep them active when needed.  In
> > this example, it wouldn't matter that the bus master device doesn't know
> > about the IOMMU. It suffices that the dmaengine driver knows about the
> > IOMMU.

> That requires it to be able to say "now I'm not using you any more" whenever
> the device in question suspend and "now I'm going to use you again" when that
> device resumes.  Does something like that happen today?

The easiest thing to do with this sort of thing is usually to put the
referencing into the API that abstracts things (it really ought to know
when things are actually being used anyway after all) - there are some
APIs that do this already.

> There are more weird cases still.  For example, we have the _DEP object in
> ACPI that basically says "this device depends on that one" and there may be
> no other relationship between the two whatsoever.  How are we supposed to
> implement this within the existing frameworks?

That sounds like something that should be baked into however ACPI is
hooked in already?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Kevin Hilman]

"Rafael J. Wysocki" <rjw@rjwysocki.net> writes:

> On Tuesday, May 20, 2014 09:57:14 AM Kevin Hilman wrote:
>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
>> 
>> > On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
>> >> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
>> >> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
>> >> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
>> >> > > > On 12.05.2014 22:31, Mark Brown wrote:
>> >> > > > > It also solves the system suspend dependencies.  Why don't the
>> >> > > > > runtime PM dependencies just work with reference counting?
>> >> > > > 
>> >> > > > Runtime PM dependencies work with reference counting just fine, but
>> >> > > > only for topologies matching Linux driver model, e.g. devices with
>> >> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
>> >> > > > on the bus driven by it. Add there an IOMMU and other various strange
>> >> > > > things that should be transparent to the drivers and it stops working.
>> >> > > 
>> >> > > There's no reason why runtime PM references have to follow the topology
>> >> > > - you do get a default reference count up to any parent (though we break
>> >> > > that sometimes, as is the case with SPI controllers being suspended even
>> >> > > though the devices below them are active) but there's nothing stopping
>> >> > > references being taken outside the topology.
>> >> > 
>> >> > Precisely.
>> >> 
>> >> BTW, I guess that the problem is resume and specifically the fact that if
>> >> a child device resumes, the parent will also resume automatically, but the
>> >> other devices the child may depend on will not (the child's resume may need
>> >> to resume them directly).
>> >> 
>> >> But I'm not sure why that is a problem, so can anyone please share some
>> >> details?
>> >
>> > Here are two real life examples.
>> >
>> > 1. IOMMU and bus master
>> >
>> > Bus master devices connected to an IOMMU need the IOMMU to be powered on in 
>> > order to access memory. In order to save power the IOMMU should of course be 
>> > powered off as much as possible.
>> >
>> > The tricky part here comes from the fact that the IOMMU is hidden behind the 
>> > DMA API. The bus master driver can't manage the IOMMU power state eplicitly by 
>> > taking/releasing references to the IOMMU device.
>> >
>> > This could easily be solved in an ad-hoc fashion by extending the DMA mapping 
>> > API, but I'm wondering whether similar issues wouldn't benefit from a common 
>> > solution (I'm not sure yet what all the similar issues are, hence the topic 
>> > proposal to try and gather use cases).
>> 
>> I'll continue to beat the runtime PM drum...
>
> What about async system suspend/resume, then?

Not sure what you're asking here.  I don't see any (new) conflicts with
async system suspend/resume when combined with runtime PM.

>> This would also easily be solved if the bus master device, the IOMMU and
>> the dmaengine (or the platform-specific dma driver) were all using
>> runtime PM.  e.g. bus master device is a user of dmaengine, which is a
>> user of the IOMMU.  If all are using runtime PM, the fact that a device
>> is "in use", means runtime PM would keep them active when needed.  In
>> this example, it wouldn't matter that the bus master device doesn't know
>> about the IOMMU. It suffices that the dmaengine driver knows about the
>> IOMMU.
>
> That requires it to be able to say "now I'm not using you any more" whenever
> the device in question suspend and "now I'm going to use you again" when that
> device resumes.  Does something like that happen today?

I don't know the specifics of the dmaengine or IOMMU APIs, but they are
certainly request based, so I suspect the infrastructure exist already.
And in the case of DMAs, the core code knows when xfers are done, so
having it request/release the underlying IOMMUs (or other dependencies)
seems rather straight forward.

Even if the APIs dont exist today, instrumenting the frameworks
themsleves seems better to me than trying to invent a generic new
dependency structure, when runtime PM is most of the way there already.

> There are more weird cases still.  For example, we have the _DEP object in
> ACPI that basically says "this device depends on that one" and there may be
> no other relationship between the two whatsoever.  How are we supposed to
> implement this within the existing frameworks?

Sounds like something for the ACPI PM domain to sort out, and it really
depends on what what "this device depends on that one" means.  Does it
mean device Y has to be on whenever device X is on?  Does it mean device
Y has to be on just for device X to suspend/resume?  Does it mean device
Y has to be on for X to have certain functionality?

If it's something like device Y has to be on whenever device X is on,
the PM domain would just need to track the dependencies and ensure that
it does a pm_runtime_get() on device Y whenever device X is enabled.

Kevin

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Laurent Pinchart]

Hi Kevin,

On Tuesday 20 May 2014 09:57:14 Kevin Hilman wrote:
> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> > On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
> >> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> >> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> >> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> >> > > > On 12.05.2014 22:31, Mark Brown wrote:
> >> > > > > It also solves the system suspend dependencies.  Why don't the
> >> > > > > runtime PM dependencies just work with reference counting?
> >> > > > 
> >> > > > Runtime PM dependencies work with reference counting just fine, but
> >> > > > only for topologies matching Linux driver model, e.g. devices with
> >> > > > exactly one device they depend on, e.g. SPI controller and SPI
> >> > > > devices on the bus driven by it. Add there an IOMMU and other
> >> > > > various strange things that should be transparent to the drivers
> >> > > > and it stops working.
> >> > > 
> >> > > There's no reason why runtime PM references have to follow the
> >> > > topology - you do get a default reference count up to any parent
> >> > > (though we break that sometimes, as is the case with SPI controllers
> >> > > being suspended even though the devices below them are active) but
> >> > > there's nothing stopping references being taken outside the topology.
> >> > 
> >> > Precisely.
> >> 
> >> BTW, I guess that the problem is resume and specifically the fact that if
> >> a child device resumes, the parent will also resume automatically, but
> >> the other devices the child may depend on will not (the child's resume
> >> may need to resume them directly).
> >> 
> >> But I'm not sure why that is a problem, so can anyone please share some
> >> details?
> > 
> > Here are two real life examples.
> > 
> > 1. IOMMU and bus master
> > 
> > Bus master devices connected to an IOMMU need the IOMMU to be powered on
> > in order to access memory. In order to save power the IOMMU should of
> > course be powered off as much as possible.
> > 
> > The tricky part here comes from the fact that the IOMMU is hidden behind
> > the DMA API. The bus master driver can't manage the IOMMU power state
> > eplicitly by taking/releasing references to the IOMMU device.
> > 
> > This could easily be solved in an ad-hoc fashion by extending the DMA
> > mapping API, but I'm wondering whether similar issues wouldn't benefit
> > from a common solution (I'm not sure yet what all the similar issues are,
> > hence the topic proposal to try and gather use cases).
> 
> I'll continue to beat the runtime PM drum...

And I'll continue to sing a slightly different song :-)

> This would also easily be solved if the bus master device, the IOMMU and
> the dmaengine (or the platform-specific dma driver) were all using runtime
> PM.  e.g. bus master device is a user of dmaengine, which is a user of the
> IOMMU.  If all are using runtime PM, the fact that a device is "in use",
> means runtime PM would keep them active when needed.  In this example, it
> wouldn't matter that the bus master device doesn't know about the IOMMU. It
> suffices that the dmaengine driver knows about the IOMMU.

You're confusing the DMA engine and DMA mapping APIs here. IOMMU integration 
occurs in the DMA mapping API, which lets the bus master map/unmap memory 
regions for DMA. The DMA mapping API itself doesn't expose any underlying 
device (when no IOMMU is used there's no underlying device to be exposed), so 
the bus master can't take a PM reference on the IOMMU device neither directly 
nor indirectly at the moment.

The DMA mapping API is not request-based, so we can't rely on requests to 
manage the IOMMU power. There's a concept of mappings being created and torn 
down, where no memory access can happen without a mapping. A first step would 
be to keep the IOMMU powered on while mappings exist. However, as creating a 
mapping can be expensive, devices usually keep mappings around even when they 
don't need to access memory.

This could be solved pretty easily (at least in my mind, coding a solution 
would likely reveal issues) by extending the DMA mapping API, but what I'm 
wondering is whether that ad-hoc solution is the right way to go, or if 
something more generic could be useful. I don't think this falls in the same 
PM dependency category as the problem below, but it might still benefit from a 
more generic solution than just a DMA mapping API extension. If nobody can see 
a real added value here I would be fine with just extending the DMA mapping 
API.

> > 2. Composite devices
> > 
> > I'll take an embedded camera devices as an example as that's the case I
> > know better, but the same problem occurs on the display side as well.
> > 
> > Camera devices on embedded systems are usually made of a camera interface
> > inside an SoC (possibly split into several IP cores, such as a CSI -
> > Camera Serial Interface - receiver and a backend) and one or more external
> > devices, such as sensors, flash controller, lens controller or sometimes
> > dedicated video processing accelerators.
> > 
> > The external devices are most of the time controlled through I2C (SPI is
> > an option as well, the exact interface doesn't matter much). Those that
> > handle video data streams (sensor, video processor) use dedicated parallel
> > or high speed serial busses.
> > 
> > Most of the external devices require an input clock. The clock can be
> > fixed (easy), provided by a dedicated chip or IP core in the SoC (easy as
> > well), sometimes by the SoC camera interface or even by one of the other
> > external devices (for instance the lens controller could be clocked by the
> > pixel clock output by the sensor - that's a bit far-fetched - or, more
> > boring but equally annoying, I've seen a camera interface in an SoC that
> > required the pixel clock output by the sensor to be running in order to
> > complete its reset sequence).
> > 
> > On OMAP3 systems implementing a camera the image sensor is usually
> > supplied with a clock output by the ISP (Image Signal Processor, the
> > camera interface). When resuming the system from suspend (assuming the ISP
> > was capturing video when the system got suspended), it's important to
> > restart the ISP first and only then restart the sensor. To ensure that the
> > order is followed the ISP driver resume path first restarts the ISP and
> > then calls the sensor driver to start the video stream. The sensor driver
> > will need to send I2C messages to the device in order to start it, which
> > requires the I2C controller to be resumed. As the ISP is a platform
> > device at the same level as the I2C controller, that ordering is not
> > guaranteed.
> 
> As you likely already know, at least on OMAP, the I2C ordering would
> work just fine because the I2C driver is (what I like to call) runtime
> PM centric.  IOW, it doesn't really implement suspend/resume, instead it
> only does runtime PM "on demand" on a per-xfer basis.  That means that
> the I2C driver can be used anytime during the [runtime] suspend/resume
> of any other device without any ordering problems because it's activated
> on demand.

That's nice. As long as all devices up the chain can work that way (if the bus 
the I2C master sits on is suspended, we would just be moving the problem one 
square up) that should fix at least part of the problem

> Extending this same "runtime PM centric" view of the world to the other
> devices, I think that the runtime PM reference counting invovled when
> all the components are using runtime PM would solve this problem.
> 
> (I realize that this I2C solution doesn't solve the more general "resume
> ISP before sensor" problem though, but with better modeling, I think it
> can.  More on that below...)
> 
> > Furthermore, if the sensor is resumed first, it might try to access the
> > device, which requires the clock output by the ISP to be available, and
> > thus requires the ISP to be resumed. To solve this problem the ISP driver
> > only restarts the clocks in its PM resume callback, and restarts the
> > video stream (following the sequence described above) in its PM complete
> > callback.
>
> For most devices, input clocks are modeled by the clock framework (or
> managed by the SoC's runtime PM core), and therefore, a pm_runtime_get()
> (or possibly an explict clk_enable()) is used to ensure the input clock
> is running.  In this external device example, it sounds to me like the
> sensor driver has no knowledge of its input clock so it has to rely on
> some other layer to resume things in the right order for correct
> functionality.
> 
> Maybe I'm wrong here (likely, since I haven't looked at the code, and am
> admittedly very ignorant of the camera and display subsystems) but it
> sounds to me like what's missing is the sensor driver having knowledge
> of it's input clock and/or a way for it to request it's input clock to
> be enabled (e.g. clk_get/clk_enable.)

I'm glad that you point out likely being wrong yourself, as you are ;-) The 
sensor driver manages the sensor input clock explicitly through CCF (or at 
least it should, not all drivers do their homework properly, but that's just a 
matter of fixing them), so it can enable/disable the clock when needed. If the 
clock provider driver becomes runtime PM centric then part of the problem 
would be fixed.

> Alternatively, what would proably be even better would be that the
> sensor driver has a reference to the actual device that provides its
> input clock (possibly via a DT phandle?) so that the sensor driver can
> simply do a pm_runtime_get() on the device providing the clock.

Isn't it better for the sensor DT node to reference its input clock through 
the clocks property and enable/disable the clock on demand instead of 
explicitly calling pm_runtime_(get|put) on the clock provider device ?

> > When adding more external devices to the mix the problem just becomes more
> > complex, especially when the devices are chained (for instance sensor ->
> > video processor -> ISP). The problem is similar on the display side,
> > possibly with a different resume ordering (it should be noted that the
> > external devices vs. internal device ordering might vary even inside the
> > same class of devices - camera or display).
> 
> IMO, I still think that properly modeling the device dependenies
> combined with a "runtime PM centric" view of suspend/resume should allow
> the dependencies to be handled correctly for system suspend/resume and
> runtime PM.

I definitely need to give this a bit more thought. I agree that it would 
likely solve part of the issue, but I'm not sure whether the rest is solvable 
with the infrastructure we have now.

> I think what complicates things here is not the PM specifics but
> probably the fact that the device hierarchy (and dependencies) may be
> dynamic depending on many factors like which sensors are in use,
> post-processing, etc. etc.
> 
> Above, I suggested possibly using DT phandles to model these non
> parent/child relationships.  That's all fine if the dependencies are not
> changing, but if they are dynamic, we'll probably need something
> different.

They can be dynamic, yes. However, we already model the video data streams 
dependencies using phandles in the V4L2 bindings, so the required information 
is there. It would thus "just" be a matter of orchestrating all the involved 
components. If we go the runtime PM way the problem might be simplified, but 
as Ulf Hansson mentioned interactions between runtime PM and system 
suspend/resume need to be taken care of.

> At least for starters though, static dependencies like this should be
> (relatively) easy to model in DT and combined with runtime PM
> refcounding on the dependencies, should be able to address the problem.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Kevin Hilman]

Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:

> Hi Kevin,
>
> On Tuesday 20 May 2014 09:57:14 Kevin Hilman wrote:
>> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:

[...]

I'll respond to the DMA/IOMMU part separately, as I need some time to
digest it.

>> > Furthermore, if the sensor is resumed first, it might try to access the
>> > device, which requires the clock output by the ISP to be available, and
>> > thus requires the ISP to be resumed. To solve this problem the ISP driver
>> > only restarts the clocks in its PM resume callback, and restarts the
>> > video stream (following the sequence described above) in its PM complete
>> > callback.
>>
>> For most devices, input clocks are modeled by the clock framework (or
>> managed by the SoC's runtime PM core), and therefore, a pm_runtime_get()
>> (or possibly an explict clk_enable()) is used to ensure the input clock
>> is running.  In this external device example, it sounds to me like the
>> sensor driver has no knowledge of its input clock so it has to rely on
>> some other layer to resume things in the right order for correct
>> functionality.
>> 
>> Maybe I'm wrong here (likely, since I haven't looked at the code, and am
>> admittedly very ignorant of the camera and display subsystems) but it
>> sounds to me like what's missing is the sensor driver having knowledge
>> of it's input clock and/or a way for it to request it's input clock to
>> be enabled (e.g. clk_get/clk_enable.)
>
> I'm glad that you point out likely being wrong yourself, as you are ;-) 

Touché.

It's good to have a low standard for oneself. ;)

> The sensor driver manages the sensor input clock explicitly through
> CCF (or at least it should, not all drivers do their homework
> properly, but that's just a matter of fixing them), so it can
> enable/disable the clock when needed. If the clock provider driver
> becomes runtime PM centric then part of the problem would be fixed.

Great, that simplifies things a bit, except...

>> Alternatively, what would proably be even better would be that the
>> sensor driver has a reference to the actual device that provides its
>> input clock (possibly via a DT phandle?) so that the sensor driver can
>> simply do a pm_runtime_get() on the device providing the clock.
>
> Isn't it better for the sensor DT node to reference its input clock through 
> the clocks property and enable/disable the clock on demand instead of 
> explicitly calling pm_runtime_(get|put) on the clock provider device ?

Possibly, but I suspect that's not going to be good enough.

As I mentioned in an earlier response to Geert, I don't think managing
clocks alone is enough.  I can imagine many platforms where a simple
clk_enable() of an input clock is not enough to bring the device
providing that clock out of a low power state.  For this to be generic
enough to handle those cases, I suspect runtime PM get/put is the right
way to go.

Of course, for many "simple" platforms, runtime PM get/put just ends up
doing a clk_enable/disable, but on the platforms where runtime PM is
slightly more... um, "interesting"... just doing a clk enable/disable
won't do what you hope.

>> > When adding more external devices to the mix the problem just becomes more
>> > complex, especially when the devices are chained (for instance sensor ->
>> > video processor -> ISP). The problem is similar on the display side,
>> > possibly with a different resume ordering (it should be noted that the
>> > external devices vs. internal device ordering might vary even inside the
>> > same class of devices - camera or display).
>> 
>> IMO, I still think that properly modeling the device dependenies
>> combined with a "runtime PM centric" view of suspend/resume should allow
>> the dependencies to be handled correctly for system suspend/resume and
>> runtime PM.
>
> I definitely need to give this a bit more thought. I agree that it would 
> likely solve part of the issue, but I'm not sure whether the rest is solvable 
> with the infrastructure we have now.

Not without some enhancments, but IMO ensuring the frameworks are
runtime PM centric, and possibly extending runtime PM slightly where
needed is a better route than creating something new.  Whatever that
"something new" would be would end up duplicating much of the use
counting already done by runtime PM anyways.

>> I think what complicates things here is not the PM specifics but
>> probably the fact that the device hierarchy (and dependencies) may be
>> dynamic depending on many factors like which sensors are in use,
>> post-processing, etc. etc.
>> 
>> Above, I suggested possibly using DT phandles to model these non
>> parent/child relationships.  That's all fine if the dependencies are not
>> changing, but if they are dynamic, we'll probably need something
>> different.
>
> They can be dynamic, yes. However, we already model the video data streams 
> dependencies using phandles in the V4L2 bindings, so the required information 
> is there. It would thus "just" be a matter of orchestrating all the involved 
> components. If we go the runtime PM way the problem might be simplified, but 
> as Ulf Hansson mentioned interactions between runtime PM and system 
> suspend/resume need to be taken care of.

I'm unfortunately all too familiar with the interactions between system
PM and runtime PM, but I believe we have a pretty good grip on that now,
and the infrastructure is in place to solve those problems.

Kevin

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Linus Walleij]

On Fri, May 23, 2014 at 2:18 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Tuesday 20 May 2014 09:57:14 Kevin Hilman wrote:

>> Alternatively, what would proably be even better would be that the
>> sensor driver has a reference to the actual device that provides its
>> input clock (possibly via a DT phandle?) so that the sensor driver can
>> simply do a pm_runtime_get() on the device providing the clock.
>
> Isn't it better for the sensor DT node to reference its input clock through
> the clocks property and enable/disable the clock on demand instead of
> explicitly calling pm_runtime_(get|put) on the clock provider device ?

This is tangent to another discussion we've had off and on whether
a device's resources (clocks, regulators, GPIOs, pins, pwms, D/As...)
should be handled explicitly by the driver or implicitly by e.g. a
PM domain.

It appears that currently we have bias such that for a discrete
component (such as some sensor) soldered on a board with
rails and stuff, these generally do explicit resource management
with clk_prepare_enable() and friends.

On the other hand there are some SoCs (OMAP especially but also
some Reneasa SH stuff IIRC!) that does implicit resource management
of devices on the SoC using PM domains.

This is a typical example of implicit resource management:
drivers/base/power/clock_ops.c

As PM domains are really not just for SoCs the broader question
of whether runtime resource management should be explicit using
handles in the drivers or implicit by referencing PM domain pops up.

The mixture of both that we have right now is a bit confusing,
admittedly.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Linus Walleij]

On Fri, May 23, 2014 at 2:39 AM, Kevin Hilman <khilman@linaro.org> wrote:

> Of course, for many "simple" platforms, runtime PM get/put just ends up
> doing a clk_enable/disable, but on the platforms where runtime PM is
> slightly more... um, "interesting"... just doing a clk enable/disable
> won't do what you hope.

And drivers/base/power/clock_ops.c as invoked from
drivers/sh/pm_runtime.c does exactly that simple thing using
PM domains. Quite elegantly too, I'd say.

But now I get curious - can we define the "interesting PM" more
generally, like get the details out? And I mean like for a few different
platforms, I know OMAP with its hwmod is one such thing, but how
much interesting stuff do we have out there that we know we must
be able to handle?

Cc:ing in Peter De Schrijver who might have some interesting PM
to tell us about :-)

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Ulf Hansson]

On 23 May 2014 10:25, Linus Walleij <linus.walleij@linaro.org> wrote:
> On Fri, May 23, 2014 at 2:18 AM, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> On Tuesday 20 May 2014 09:57:14 Kevin Hilman wrote:
>
>>> Alternatively, what would proably be even better would be that the
>>> sensor driver has a reference to the actual device that provides its
>>> input clock (possibly via a DT phandle?) so that the sensor driver can
>>> simply do a pm_runtime_get() on the device providing the clock.
>>
>> Isn't it better for the sensor DT node to reference its input clock through
>> the clocks property and enable/disable the clock on demand instead of
>> explicitly calling pm_runtime_(get|put) on the clock provider device ?
>
> This is tangent to another discussion we've had off and on whether
> a device's resources (clocks, regulators, GPIOs, pins, pwms, D/As...)
> should be handled explicitly by the driver or implicitly by e.g. a
> PM domain.
>
> It appears that currently we have bias such that for a discrete
> component (such as some sensor) soldered on a board with
> rails and stuff, these generally do explicit resource management
> with clk_prepare_enable() and friends.
>
> On the other hand there are some SoCs (OMAP especially but also
> some Reneasa SH stuff IIRC!) that does implicit resource management
> of devices on the SoC using PM domains.
>
> This is a typical example of implicit resource management:
> drivers/base/power/clock_ops.c
>
> As PM domains are really not just for SoCs the broader question
> of whether runtime resource management should be explicit using
> handles in the drivers or implicit by referencing PM domain pops up.
>
> The mixture of both that we have right now is a bit confusing,
> admittedly.

This discussion, touches several unresolved runtime PM and system PM
issues in general. I think it's good we bring this all together,
because certainly there still some pieces missing in the puzzle.

As we have brought up pm_domains in the discussion, we might as well
add the "generic power domain" here.

Currently I am working on simplifying genpd. I am trying to iron out
what responsibilities actually should reside in it. Like what SOC
specific actions should it control, how should "pm_runtime_irq_safe"
devices be handled, etc.

Kind regards
Ulf Hansson

>
> Yours,
> Linus Walleij
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Kevin Hilman]

Linus Walleij <linus.walleij@linaro.org> writes:

> On Fri, May 23, 2014 at 2:39 AM, Kevin Hilman <khilman@linaro.org> wrote:
>
>> Of course, for many "simple" platforms, runtime PM get/put just ends up
>> doing a clk_enable/disable, but on the platforms where runtime PM is
>> slightly more... um, "interesting"... just doing a clk enable/disable
>> won't do what you hope.
>
> And drivers/base/power/clock_ops.c as invoked from
> drivers/sh/pm_runtime.c does exactly that simple thing using
> PM domains. Quite elegantly too, I'd say.
>
> But now I get curious - can we define the "interesting PM" more
> generally, like get the details out? And I mean like for a few different
> platforms, I know OMAP with its hwmod is one such thing, but how
> much interesting stuff do we have out there that we know we must
> be able to handle?

By "interesting", I was referring to some of the more complicated ones
we have, such as OMAP (though I suspect other SoCs are equally
complicated but just don't have their stuff documented and/or upstream
yet so we don't know.)

That being said, I think any SoC that requires more than a clk_enable()
to bring an IP block out of a low-power state, or clk_disable() to put
it into a low-power state qualifies as "interesting" because it can no
longer rely on the "simple" clock_ops.c you refer to above.

Stated differently, for most SoCs, fine-grained PM is a lot more than
just clock management.  As soon as you have to do more than manage
clocks, the generic clock_ops.c isn't good enough and you need your own
PM domain (or use genpd.)

Kevin

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Thursday, May 22, 2014 11:14:49 AM Mark Brown wrote:
> 
> --msuMhlVPES9Pz2Dp
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> 
> On Thu, May 22, 2014 at 02:19:35AM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, May 20, 2014 09:57:14 AM Kevin Hilman wrote:
> 
> > > This would also easily be solved if the bus master device, the IOMMU and
> > > the dmaengine (or the platform-specific dma driver) were all using
> > > runtime PM.  e.g. bus master device is a user of dmaengine, which is a
> > > user of the IOMMU.  If all are using runtime PM, the fact that a device
> > > is "in use", means runtime PM would keep them active when needed.  In
> > > this example, it wouldn't matter that the bus master device doesn't know
> > > about the IOMMU. It suffices that the dmaengine driver knows about the
> > > IOMMU.
> 
> > That requires it to be able to say "now I'm not using you any more" whenever
> > the device in question suspend and "now I'm going to use you again" when that
> > device resumes.  Does something like that happen today?
> 
> The easiest thing to do with this sort of thing is usually to put the
> referencing into the API that abstracts things (it really ought to know
> when things are actually being used anyway after all) - there are some
> APIs that do this already.
> 
> > There are more weird cases still.  For example, we have the _DEP object in
> > ACPI that basically says "this device depends on that one" and there may be
> > no other relationship between the two whatsoever.  How are we supposed to
> > implement this within the existing frameworks?
> 
> That sounds like something that should be baked into however ACPI is
> hooked in already?

No, it is not.  It is a relatively new addition to ACPI and we don't support
it today.  That's because it hasn't been present in the ACPI tables of any
production systems until recently.

However, today we have systems with it shipping and we need to add support
for it.

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Thursday, May 22, 2014 10:35:46 AM Kevin Hilman wrote:
> "Rafael J. Wysocki" <rjw@rjwysocki.net> writes:
> 
> > On Tuesday, May 20, 2014 09:57:14 AM Kevin Hilman wrote:
> >> Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:
> >> 
> >> > On Wednesday 14 May 2014 00:34:56 Rafael J. Wysocki wrote:
> >> >> On Wednesday, May 14, 2014 12:27:47 AM Rafael J. Wysocki wrote:
> >> >> > On Monday, May 12, 2014 11:07:29 PM Mark Brown wrote:
> >> >> > > On Mon, May 12, 2014 at 11:16:57PM +0200, Tomasz Figa wrote:
> >> >> > > > On 12.05.2014 22:31, Mark Brown wrote:
> >> >> > > > > It also solves the system suspend dependencies.  Why don't the
> >> >> > > > > runtime PM dependencies just work with reference counting?
> >> >> > > > 
> >> >> > > > Runtime PM dependencies work with reference counting just fine, but
> >> >> > > > only for topologies matching Linux driver model, e.g. devices with
> >> >> > > > exactly one device they depend on, e.g. SPI controller and SPI devices
> >> >> > > > on the bus driven by it. Add there an IOMMU and other various strange
> >> >> > > > things that should be transparent to the drivers and it stops working.
> >> >> > > 
> >> >> > > There's no reason why runtime PM references have to follow the topology
> >> >> > > - you do get a default reference count up to any parent (though we break
> >> >> > > that sometimes, as is the case with SPI controllers being suspended even
> >> >> > > though the devices below them are active) but there's nothing stopping
> >> >> > > references being taken outside the topology.
> >> >> > 
> >> >> > Precisely.
> >> >> 
> >> >> BTW, I guess that the problem is resume and specifically the fact that if
> >> >> a child device resumes, the parent will also resume automatically, but the
> >> >> other devices the child may depend on will not (the child's resume may need
> >> >> to resume them directly).
> >> >> 
> >> >> But I'm not sure why that is a problem, so can anyone please share some
> >> >> details?
> >> >
> >> > Here are two real life examples.
> >> >
> >> > 1. IOMMU and bus master
> >> >
> >> > Bus master devices connected to an IOMMU need the IOMMU to be powered on in 
> >> > order to access memory. In order to save power the IOMMU should of course be 
> >> > powered off as much as possible.
> >> >
> >> > The tricky part here comes from the fact that the IOMMU is hidden behind the 
> >> > DMA API. The bus master driver can't manage the IOMMU power state eplicitly by 
> >> > taking/releasing references to the IOMMU device.
> >> >
> >> > This could easily be solved in an ad-hoc fashion by extending the DMA mapping 
> >> > API, but I'm wondering whether similar issues wouldn't benefit from a common 
> >> > solution (I'm not sure yet what all the similar issues are, hence the topic 
> >> > proposal to try and gather use cases).
> >> 
> >> I'll continue to beat the runtime PM drum...
> >
> > What about async system suspend/resume, then?
> 
> Not sure what you're asking here.  I don't see any (new) conflicts with
> async system suspend/resume when combined with runtime PM.

I don't mean conflicts, but how runtime PM can help in that area. :-)

[cut]

> > There are more weird cases still.  For example, we have the _DEP object in
> > ACPI that basically says "this device depends on that one" and there may be
> > no other relationship between the two whatsoever.  How are we supposed to
> > implement this within the existing frameworks?
> 
> Sounds like something for the ACPI PM domain to sort out, and it really
> depends on what what "this device depends on that one" means.  Does it
> mean device Y has to be on whenever device X is on?  Does it mean device
> Y has to be on just for device X to suspend/resume?  Does it mean device
> Y has to be on for X to have certain functionality?
> 
> If it's something like device Y has to be on whenever device X is on,
> the PM domain would just need to track the dependencies and ensure that
> it does a pm_runtime_get() on device Y whenever device X is enabled.

It is kind of hard to explain without going into details about how certain
things in AML work, but roughly an ACPI method for one device may use
resources provided by another device (for example, an ACPI method for
device X uses an I2C connection with controller Y).  In that case the
"resource provider" has to be functional whenever the "consumer" method is
executed.

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Friday, May 23, 2014 10:25:09 AM Linus Walleij wrote:
> On Fri, May 23, 2014 at 2:18 AM, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> > On Tuesday 20 May 2014 09:57:14 Kevin Hilman wrote:
> 
> >> Alternatively, what would proably be even better would be that the
> >> sensor driver has a reference to the actual device that provides its
> >> input clock (possibly via a DT phandle?) so that the sensor driver can
> >> simply do a pm_runtime_get() on the device providing the clock.
> >
> > Isn't it better for the sensor DT node to reference its input clock through
> > the clocks property and enable/disable the clock on demand instead of
> > explicitly calling pm_runtime_(get|put) on the clock provider device ?
> 
> This is tangent to another discussion we've had off and on whether
> a device's resources (clocks, regulators, GPIOs, pins, pwms, D/As...)
> should be handled explicitly by the driver or implicitly by e.g. a
> PM domain.
> 
> It appears that currently we have bias such that for a discrete
> component (such as some sensor) soldered on a board with
> rails and stuff, these generally do explicit resource management
> with clk_prepare_enable() and friends.
> 
> On the other hand there are some SoCs (OMAP especially but also
> some Reneasa SH stuff IIRC!) that does implicit resource management
> of devices on the SoC using PM domains.
> 
> This is a typical example of implicit resource management:
> drivers/base/power/clock_ops.c
> 
> As PM domains are really not just for SoCs the broader question
> of whether runtime resource management should be explicit using
> handles in the drivers or implicit by referencing PM domain pops up.

When I was adding PM domains to the PM core one of the concerns was
that the PM resources the drivers can access may not be independent and
using them would require synchronization between drivers.  Using PM
domains is one of the ways to achieve that.

Also using PM domains helps to address the case when the same driver
needs to be used on different platforms with different PM configurations.

> The mixture of both that we have right now is a bit confusing,
> admittedly.

I suppose that this is for historical reasons mostly (different platforms
chose to do things differently and code duplication between them related
to handling analogous IP blocks in slightly different ways was less of
a concern in the past).

For driver portability, however, there has to be a level of indirection
between the driver and the PM features of the platform and PM domains can be
used for providing something like that.

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Friday, May 23, 2014 10:32:24 AM Linus Walleij wrote:
> On Fri, May 23, 2014 at 2:39 AM, Kevin Hilman <khilman@linaro.org> wrote:
> 
> > Of course, for many "simple" platforms, runtime PM get/put just ends up
> > doing a clk_enable/disable, but on the platforms where runtime PM is
> > slightly more... um, "interesting"... just doing a clk enable/disable
> > won't do what you hope.
> 
> And drivers/base/power/clock_ops.c as invoked from
> drivers/sh/pm_runtime.c does exactly that simple thing using
> PM domains. Quite elegantly too, I'd say.

Well, thanks! :-)

> But now I get curious - can we define the "interesting PM" more
> generally, like get the details out? And I mean like for a few different
> platforms, I know OMAP with its hwmod is one such thing, but how
> much interesting stuff do we have out there that we know we must
> be able to handle?
> 
> Cc:ing in Peter De Schrijver who might have some interesting PM
> to tell us about :-)

I generally like to split PM into things that can be done by the driver
regardless of the underlying platform and things that are platform-dependent.
It generally is better to abstract the latter away from drivers in my opinion.

So if a given clock is guaranteed to be manageable in the same way on every
platform where the driver will ever run, that clock may be managed by the driver
itself.  However, if that clock is correlated with something else on one of
the platforms and the driver would need to take that into account, things
can get quite ugly relatively quickly.  And if one of those platforms is
ACPI, they are ugly right to start with. ;-)

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Friday, May 23, 2014 08:26:12 AM Kevin Hilman wrote:
> Linus Walleij <linus.walleij@linaro.org> writes:
> 
> > On Fri, May 23, 2014 at 2:39 AM, Kevin Hilman <khilman@linaro.org> wrote:
> >
> >> Of course, for many "simple" platforms, runtime PM get/put just ends up
> >> doing a clk_enable/disable, but on the platforms where runtime PM is
> >> slightly more... um, "interesting"... just doing a clk enable/disable
> >> won't do what you hope.
> >
> > And drivers/base/power/clock_ops.c as invoked from
> > drivers/sh/pm_runtime.c does exactly that simple thing using
> > PM domains. Quite elegantly too, I'd say.
> >
> > But now I get curious - can we define the "interesting PM" more
> > generally, like get the details out? And I mean like for a few different
> > platforms, I know OMAP with its hwmod is one such thing, but how
> > much interesting stuff do we have out there that we know we must
> > be able to handle?
> 
> By "interesting", I was referring to some of the more complicated ones
> we have, such as OMAP (though I suspect other SoCs are equally
> complicated but just don't have their stuff documented and/or upstream
> yet so we don't know.)
> 
> That being said, I think any SoC that requires more than a clk_enable()
> to bring an IP block out of a low-power state, or clk_disable() to put
> it into a low-power state qualifies as "interesting" because it can no
> longer rely on the "simple" clock_ops.c you refer to above.
> 
> Stated differently, for most SoCs, fine-grained PM is a lot more than
> just clock management.  As soon as you have to do more than manage
> clocks, the generic clock_ops.c isn't good enough and you need your own
> PM domain (or use genpd.)

Well stated.

Also, as I said in one of the previous messages, if you consider a driver
as a piece of code that should work with different platforms (or different
SoCs if you will, but there's more to that), then it's quite clear that the
driver is probably not the right place to do all PM that's needed.

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

On Sat, May 24, 2014 at 01:15:56AM +0200, Rafael J. Wysocki wrote:
> On Thursday, May 22, 2014 11:14:49 AM Mark Brown wrote:

> > > There are more weird cases still.  For example, we have the _DEP object in
> > > ACPI that basically says "this device depends on that one" and there may be
> > > no other relationship between the two whatsoever.  How are we supposed to
> > > implement this within the existing frameworks?

> > That sounds like something that should be baked into however ACPI is
> > hooked in already?

> No, it is not.  It is a relatively new addition to ACPI and we don't support
> it today.  That's because it hasn't been present in the ACPI tables of any
> production systems until recently.

> However, today we have systems with it shipping and we need to add support
> for it.

Sorry, I wasn't clear - what I meant was that it should be something
that can be handled based on information the ACPI implementation already
has, I had been under the impression it got notification of all the
basic PM transitions already?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Rafael J. Wysocki]

On Saturday, May 24, 2014 11:53:28 AM Mark Brown wrote:
> 
> --6o5AAeie4QQ7Mfd3
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> 
> On Sat, May 24, 2014 at 01:15:56AM +0200, Rafael J. Wysocki wrote:
> > On Thursday, May 22, 2014 11:14:49 AM Mark Brown wrote:
> 
> > > > There are more weird cases still.  For example, we have the _DEP object in
> > > > ACPI that basically says "this device depends on that one" and there may be
> > > > no other relationship between the two whatsoever.  How are we supposed to
> > > > implement this within the existing frameworks?
> 
> > > That sounds like something that should be baked into however ACPI is
> > > hooked in already?
> 
> > No, it is not.  It is a relatively new addition to ACPI and we don't support
> > it today.  That's because it hasn't been present in the ACPI tables of any
> > production systems until recently.
> 
> > However, today we have systems with it shipping and we need to add support
> > for it.
> 
> Sorry, I wasn't clear - what I meant was that it should be something
> that can be handled based on information the ACPI implementation already
> has, I had been under the impression it got notification of all the
> basic PM transitions already?

It is called to do some low-level manipulations of device PM settings during
those transitions, but that may be after the driver's suspend callback has
run (like for PCI, for example).

Rafael

Re: [Ksummit-discuss] [TECH TOPIC] PM dependencies

[Peter De Schrijver]

On Fri, May 23, 2014 at 10:32:24AM +0200, Linus Walleij wrote:
> On Fri, May 23, 2014 at 2:39 AM, Kevin Hilman <khilman@linaro.org> wrote:
> 
> > Of course, for many "simple" platforms, runtime PM get/put just ends up
> > doing a clk_enable/disable, but on the platforms where runtime PM is
> > slightly more... um, "interesting"... just doing a clk enable/disable
> > won't do what you hope.
> 
> And drivers/base/power/clock_ops.c as invoked from
> drivers/sh/pm_runtime.c does exactly that simple thing using
> PM domains. Quite elegantly too, I'd say.
> 
> But now I get curious - can we define the "interesting PM" more
> generally, like get the details out? And I mean like for a few different
> platforms, I know OMAP with its hwmod is one such thing, but how
> much interesting stuff do we have out there that we know we must
> be able to handle?
> 
> Cc:ing in Peter De Schrijver who might have some interesting PM
> to tell us about :-)
> 

On Tegra to powergate a domain you have to:

1) wait until all outstanding memory accesses by DMA clients in the domain have finished and assert the reset of the DMA clients. This is because DMA to/from memory goes via a seperate path compared to control MMIO (register accesses).
2) assert reset of the modules in the domain
3) disable the clocks of the modules in the domain
4) enable clamps
5) turn off power

to powerungate a domain you basically have to do:

1) enable power
2) enable all clocks of the modules in the domain
3) unclamp the domain
4) deassert reset (this is why the clocks have to be on, otherwise the reset signals wouldn't propagate)
6) deassert reset of the DMA clients
5) disable all module clocks again

My guess is that OMAP does most of this sequencing in hw (probably done by the PRCM). Also I guess that OMAP doesn't need the special handling of the DMA clients as both control MMIO and DMA goes via the same interconnect.

Cheers,

Peter.

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------