[Ksummit-discuss] Energy-aware Scheduling Workshop

[Ksummit-discuss] Energy-aware Scheduling Workshop

[Morten Rasmussen]

Hi,

Last year's Energy-Aware Scheduling workshop [1,2] was a good
opportunity for interested parties to discuss some of the open issues in
this area face to face. While work is still ongoing on many of the
topics that were discussed, it might be worth having workshop again this
year to follow up, revise the plans if necessary, and discuss topics
that were not covered last year.

Before submitting a workshop proposal to the Ksummit PC I would like to
probe the interest. IMO, it is important that we have scheduler
maintainers present.

Workshop topic proposals:

Test cases
Use-cases for high-end phones (which some of us care about) consist of
rather complex software stacks which are not suitable for quick patch
testing [3]. While we can't avoid testing using the full software stack
in the end, it would be useful to have configurable micro-benchmarks for
initial testing and to reproduce specific scheduling patterns from the
full use-case for debugging purposes.

Energy Evaluation
A hot topic last year. We need a way to evaluate energy-awareness
patches. Work has started on an idle state analysis tool [4], but we are
not there yet.

Platform Performance/Energy data
Currently the kernel has quite limited knowledge about energy costs of
the platform where it is running. Without this information it is rather
hard to make energy-efficient scheduling decisions. It seems that
various energy-saving techniques don't work equally well on all
platforms and might even depend on the use-case. Should we give the
kernel enough information to construct a simple energy-model to guide
decisions?

CPU utilization and cpu_power
The entity load tracking has given us a much better indication of
individual task loadi. However, priority scaling makes it less suitable
for low load scenarios [5] where we care more about actual cpu
utilization per task when trying to figure out an energy-efficient load
balance. Do we need entity utilization tracking as well? Related to this
topic is the representation of cpu compute capacity. The current
representation, cpu_power, can't deal with heterogeneous systems
correctly. Can we come up with a solution that can handle SMT, SMP, and
heterogeneous systems?


All comments and topic proposals are welcome.

Thanks,
Morten

[1] http://lwn.net/Articles/571414/
[2] http://etherpad.osuosl.org/energy-aware-scheduling-ks-2013
[3] https://lkml.org/lkml/2014/1/7/355
[4] https://lkml.org/lkml/2014/3/24/363
[5] https://lkml.org/lkml/2014/1/7/503 

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Rafael J. Wysocki]

On Monday, May 12, 2014 04:32:34 PM Morten Rasmussen wrote:
> Hi,

Hi,

> Last year's Energy-Aware Scheduling workshop [1,2] was a good
> opportunity for interested parties to discuss some of the open issues in
> this area face to face. While work is still ongoing on many of the
> topics that were discussed, it might be worth having workshop again this
> year to follow up, revise the plans if necessary, and discuss topics
> that were not covered last year.
> 
> Before submitting a workshop proposal to the Ksummit PC I would like to
> probe the interest. IMO, it is important that we have scheduler
> maintainers present.
> 
> Workshop topic proposals:
> 
> Test cases
> Use-cases for high-end phones (which some of us care about) consist of
> rather complex software stacks which are not suitable for quick patch
> testing [3]. While we can't avoid testing using the full software stack
> in the end, it would be useful to have configurable micro-benchmarks for
> initial testing and to reproduce specific scheduling patterns from the
> full use-case for debugging purposes.
> 
> Energy Evaluation
> A hot topic last year. We need a way to evaluate energy-awareness
> patches. Work has started on an idle state analysis tool [4], but we are
> not there yet.
> 
> Platform Performance/Energy data
> Currently the kernel has quite limited knowledge about energy costs of
> the platform where it is running. Without this information it is rather
> hard to make energy-efficient scheduling decisions. It seems that
> various energy-saving techniques don't work equally well on all
> platforms and might even depend on the use-case. Should we give the
> kernel enough information to construct a simple energy-model to guide
> decisions?
> 
> CPU utilization and cpu_power
> The entity load tracking has given us a much better indication of
> individual task loadi. However, priority scaling makes it less suitable
> for low load scenarios [5] where we care more about actual cpu
> utilization per task when trying to figure out an energy-efficient load
> balance. Do we need entity utilization tracking as well? Related to this
> topic is the representation of cpu compute capacity. The current
> representation, cpu_power, can't deal with heterogeneous systems
> correctly. Can we come up with a solution that can handle SMT, SMP, and
> heterogeneous systems?
> 
> 
> All comments and topic proposals are welcome.

I would be interested in participating in that discussion (which also is
related to the energy conservation bias interfaces KS topic proposed by
me).

Thanks!


> [1] http://lwn.net/Articles/571414/
> [2] http://etherpad.osuosl.org/energy-aware-scheduling-ks-2013
> [3] https://lkml.org/lkml/2014/1/7/355
> [4] https://lkml.org/lkml/2014/3/24/363
> [5] https://lkml.org/lkml/2014/1/7/503 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Preeti U Murthy]

Hi Morten,

Thank you for initiating this.

On 05/12/2014 09:02 PM, Morten Rasmussen wrote:
> Hi,
> 
> Last year's Energy-Aware Scheduling workshop [1,2] was a good
> opportunity for interested parties to discuss some of the open issues in
> this area face to face. While work is still ongoing on many of the
> topics that were discussed, it might be worth having workshop again this
> year to follow up, revise the plans if necessary, and discuss topics
> that were not covered last year.
> 
> Before submitting a workshop proposal to the Ksummit PC I would like to
> probe the interest. IMO, it is important that we have scheduler
> maintainers present.
> 
> Workshop topic proposals:
> 
> Test cases
> Use-cases for high-end phones (which some of us care about) consist of
> rather complex software stacks which are not suitable for quick patch
> testing [3]. While we can't avoid testing using the full software stack
> in the end, it would be useful to have configurable micro-benchmarks for
> initial testing and to reproduce specific scheduling patterns from the
> full use-case for debugging purposes.
> 
> Energy Evaluation
> A hot topic last year. We need a way to evaluate energy-awareness
> patches. Work has started on an idle state analysis tool [4], but we are
> not there yet.
> 

An interesting sub topic pertaining to evaluation would be deciding the
target residency of idle states. How do we know if the target residency
set for the different idle states is set optimally? The cpuidle governor
bases its decisions of the idle state to enter into based on numbers
such as these. If it is too stringent we would be hampering power savings.

We could run different benchmarks with different degree of utilization.
And observe how much of the idle time is spent in different idle states.
If it is too less in a particular idle state we can up the target
residency of that idle state to begin with. This is static setting and
we hard code the target residency like now. Will a dynamic setting of
metrics like these help us in any way?

This was just an example to show that there are interesting problems
around "tuning" of power management metrics. It would be very helpful if
the maintainers relevant to this workshop guide us on this.

> Platform Performance/Energy data
> Currently the kernel has quite limited knowledge about energy costs of
> the platform where it is running. Without this information it is rather
> hard to make energy-efficient scheduling decisions. It seems that
> various energy-saving techniques don't work equally well on all
> platforms and might even depend on the use-case. Should we give the
> kernel enough information to construct a simple energy-model to guide
> decisions?
> 
> CPU utilization and cpu_power
> The entity load tracking has given us a much better indication of
> individual task loadi. However, priority scaling makes it less suitable
> for low load scenarios [5] where we care more about actual cpu
> utilization per task when trying to figure out an energy-efficient load
> balance. Do we need entity utilization tracking as well? Related to this
> topic is the representation of cpu compute capacity. The current
> representation, cpu_power, can't deal with heterogeneous systems
> correctly. Can we come up with a solution that can handle SMT, SMP, and
> heterogeneous systems?
> 
> 
> All comments and topic proposals are welcome.

I would be very interested in attending this workshop. We are currently
evaluating cpuidle and cpu-frequency subsystems on PowerPC. I look
forward to share and discuss the findings in this workshop.

Thanks!

Regards
Preeti U Murthy
> 
> Thanks,
> Morten
> 
> [1] http://lwn.net/Articles/571414/
> [2] http://etherpad.osuosl.org/energy-aware-scheduling-ks-2013
> [3] https://lkml.org/lkml/2014/1/7/355
> [4] https://lkml.org/lkml/2014/3/24/363
> [5] https://lkml.org/lkml/2014/1/7/503 
> 

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Amit Kucheria]

On Wed, May 14, 2014 at 3:31 PM, Preeti U Murthy
<preeti@linux.vnet.ibm.com> wrote:
> Hi Morten,
>
> Thank you for initiating this.
>
> On 05/12/2014 09:02 PM, Morten Rasmussen wrote:
>> Hi,
>>
>> Last year's Energy-Aware Scheduling workshop [1,2] was a good
>> opportunity for interested parties to discuss some of the open issues in
>> this area face to face. While work is still ongoing on many of the
>> topics that were discussed, it might be worth having workshop again this
>> year to follow up, revise the plans if necessary, and discuss topics
>> that were not covered last year.
>>
>> Before submitting a workshop proposal to the Ksummit PC I would like to
>> probe the interest. IMO, it is important that we have scheduler
>> maintainers present.
>>
>> Workshop topic proposals:
>>
>> Test cases
>> Use-cases for high-end phones (which some of us care about) consist of
>> rather complex software stacks which are not suitable for quick patch
>> testing [3]. While we can't avoid testing using the full software stack
>> in the end, it would be useful to have configurable micro-benchmarks for
>> initial testing and to reproduce specific scheduling patterns from the
>> full use-case for debugging purposes.
>>
>> Energy Evaluation
>> A hot topic last year. We need a way to evaluate energy-awareness
>> patches. Work has started on an idle state analysis tool [4], but we are
>> not there yet.
>>
>
> An interesting sub topic pertaining to evaluation would be deciding the
> target residency of idle states. How do we know if the target residency
> set for the different idle states is set optimally? The cpuidle governor
> bases its decisions of the idle state to enter into based on numbers
> such as these. If it is too stringent we would be hampering power savings.
>
> We could run different benchmarks with different degree of utilization.
> And observe how much of the idle time is spent in different idle states.
> If it is too less in a particular idle state we can up the target
> residency of that idle state to begin with. This is static setting and
> we hard code the target residency like now. Will a dynamic setting of
> metrics like these help us in any way?

Preeti,

We've created a tool called idlestat[1] that traces the cpuidle and
cpufreq code paths and then creates a report about how often each
state was reached, average residency, etc.

We're currently experimenting with a patch to output how often we
mispredict the idle state and are seeing some good corelation with
actual power measurements. Dynamic residency might be a solution, but
first we need to understand how bad the problem is. That is what this
tool is designed to solve. It also helps provide a qualitative feel
for kernel behaviour without requiring HW hacking to measure various
power rails.

Daniel and I'd be interested in talking about idlestat and share our
findings about using it to study change in kernel behaviour due to
certain patchsets.

Regards,
Amit
[1] https://git.linaro.org/power/idlestat.git

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Lorenzo Pieralisi]

Hi Preeti,

On Wed, May 14, 2014 at 11:01:08AM +0100, Preeti U Murthy wrote:
> Hi Morten,
> 
> Thank you for initiating this.
> 
> On 05/12/2014 09:02 PM, Morten Rasmussen wrote:
> > Hi,
> > 
> > Last year's Energy-Aware Scheduling workshop [1,2] was a good
> > opportunity for interested parties to discuss some of the open issues in
> > this area face to face. While work is still ongoing on many of the
> > topics that were discussed, it might be worth having workshop again this
> > year to follow up, revise the plans if necessary, and discuss topics
> > that were not covered last year.
> > 
> > Before submitting a workshop proposal to the Ksummit PC I would like to
> > probe the interest. IMO, it is important that we have scheduler
> > maintainers present.
> > 
> > Workshop topic proposals:
> > 
> > Test cases
> > Use-cases for high-end phones (which some of us care about) consist of
> > rather complex software stacks which are not suitable for quick patch
> > testing [3]. While we can't avoid testing using the full software stack
> > in the end, it would be useful to have configurable micro-benchmarks for
> > initial testing and to reproduce specific scheduling patterns from the
> > full use-case for debugging purposes.
> > 
> > Energy Evaluation
> > A hot topic last year. We need a way to evaluate energy-awareness
> > patches. Work has started on an idle state analysis tool [4], but we are
> > not there yet.
> > 
> 
> An interesting sub topic pertaining to evaluation would be deciding the
> target residency of idle states. How do we know if the target residency
> set for the different idle states is set optimally? The cpuidle governor
> bases its decisions of the idle state to enter into based on numbers
> such as these. If it is too stringent we would be hampering power savings.
> 
> We could run different benchmarks with different degree of utilization.
> And observe how much of the idle time is spent in different idle states.
> If it is too less in a particular idle state we can up the target
> residency of that idle state to begin with. This is static setting and
> we hard code the target residency like now. Will a dynamic setting of
> metrics like these help us in any way?

It might, as long as you are able to monitor energy in the kernel.
target_residency is expressed in time, but must be evaluated in energy
terms, it is a break-even point. It strictly depends on the dynamic state
of the CPU when it goes idle, hence not really easy to factor in.

On ARM, it depends on a slew of factors, inclusive of the number of
dirty caches lines (ie memory writebacks), running CPU frequency, memory
frequency and the list goes on and on and on.

I am not even taking into account CPU idle states (C-states) whose power
domains span devices such as GPUs.

It is certainly an interesting topic, I gave a presentation on this
at Linaro Connect in 2013, I would be happy to share the results and debate
it if I am able to attend.

> This was just an example to show that there are interesting problems
> around "tuning" of power management metrics. It would be very helpful if
> the maintainers relevant to this workshop guide us on this.

I would be certainly happy to explain to you how idle states work on ARM
systems and the factors contributing to target_residency and other idle
states parameters.

Thanks,
Lorenzo

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Preeti U Murthy]

Hi Amit, Lorenzo,

On 05/14/2014 04:07 PM, Lorenzo Pieralisi wrote:
> Hi Preeti,
> 
> On Wed, May 14, 2014 at 11:01:08AM +0100, Preeti U Murthy wrote:
>> Hi Morten,
>>
>> Thank you for initiating this.
>>
>> On 05/12/2014 09:02 PM, Morten Rasmussen wrote:
>>> Hi,
>>>
>>> Last year's Energy-Aware Scheduling workshop [1,2] was a good
>>> opportunity for interested parties to discuss some of the open issues in
>>> this area face to face. While work is still ongoing on many of the
>>> topics that were discussed, it might be worth having workshop again this
>>> year to follow up, revise the plans if necessary, and discuss topics
>>> that were not covered last year.
>>>
>>> Before submitting a workshop proposal to the Ksummit PC I would like to
>>> probe the interest. IMO, it is important that we have scheduler
>>> maintainers present.
>>>
>>> Workshop topic proposals:
>>>
>>> Test cases
>>> Use-cases for high-end phones (which some of us care about) consist of
>>> rather complex software stacks which are not suitable for quick patch
>>> testing [3]. While we can't avoid testing using the full software stack
>>> in the end, it would be useful to have configurable micro-benchmarks for
>>> initial testing and to reproduce specific scheduling patterns from the
>>> full use-case for debugging purposes.
>>>
>>> Energy Evaluation
>>> A hot topic last year. We need a way to evaluate energy-awareness
>>> patches. Work has started on an idle state analysis tool [4], but we are
>>> not there yet.
>>>
>>
>> An interesting sub topic pertaining to evaluation would be deciding the
>> target residency of idle states. How do we know if the target residency
>> set for the different idle states is set optimally? The cpuidle governor
>> bases its decisions of the idle state to enter into based on numbers
>> such as these. If it is too stringent we would be hampering power savings.
>>
>> We could run different benchmarks with different degree of utilization.
>> And observe how much of the idle time is spent in different idle states.
>> If it is too less in a particular idle state we can up the target
>> residency of that idle state to begin with. This is static setting and
>> we hard code the target residency like now. Will a dynamic setting of
>> metrics like these help us in any way?
> 
> It might, as long as you are able to monitor energy in the kernel.
> target_residency is expressed in time, but must be evaluated in energy
> terms, it is a break-even point. It strictly depends on the dynamic state
> of the CPU when it goes idle, hence not really easy to factor in.
> 
> On ARM, it depends on a slew of factors, inclusive of the number of
> dirty caches lines (ie memory writebacks), running CPU frequency, memory
> frequency and the list goes on and on and on.
> 
> I am not even taking into account CPU idle states (C-states) whose power
> domains span devices such as GPUs.
> 
> It is certainly an interesting topic, I gave a presentation on this
> at Linaro Connect in 2013, I would be happy to share the results and debate
> it if I am able to attend.
> 
>> This was just an example to show that there are interesting problems
>> around "tuning" of power management metrics. It would be very helpful if
>> the maintainers relevant to this workshop guide us on this.
> 
> I would be certainly happy to explain to you how idle states work on ARM
> systems and the factors contributing to target_residency and other idle
> states parameters.

Thanks for sharing your ideas about this.

Lorenzo,
Sure I will be glad if we are able to meet to discuss this.

Amit,
I will take a look at idle stat and verify if I can use it for my purpose.

Regards
Preeti U Murthy
> 
> Thanks,
> Lorenzo
> 

Re: [Ksummit-discuss] Energy-aware Scheduling Workshop

[Du, Yuyang]

Hi Morten,

I am interested in attending this workshop. Especially interested in topics:
CPU utilization, refining load balancing interfaces, and hetero scheduling.
And also tools.

Thanks,
Yuyang