[LSF/MM ATTEND] Expanding OS noise suppression

[LSF/MM ATTEND] Expanding OS noise suppression

[Christoph Lameter]

Recently a lot of work has been done in the kernel to be able to keep OS
threads off low latency cores with the NOHZ work mainly pushed by Frederic
Weisbecker (also also Paul McKenney modifying RCU for that purpose). With
that approach we may now reduce the timer tick to a frequency of 1 per
second. The result of that work is now available in Redhat 7.

I have recently submitted work on the vmstat kworkers that makes the
kworkers run on demand with a shepherd worker checking from a non low
latency processor if there is actual work to be done on a processor in low
latency mode. If not then the kworker requests can be avoided and
therefore activities on that processor are reduced. This approach can be
extended to cover other necessary activities on low latency cores.

There is other work in progress to limit unbound kworker threads to no
NOHZ processors. Also more work is in flight to work on various issues in
the scheduler to enable us to hold off the timer tick for more than one
second.

There are numerous other issues that can impact on a low latency core from
the memory management system. I would like to discuss ways that we can
further ensure that OS activities do not impact latency critical threads
running on special nohz cores.

This may cover:
 - minor and major faults and how to suppress them effectively.
 - Processor cache impacts by sibling threads.
 - IPIs
 - Control over various subsystem specific per cpu threads.
 - Control impacts of scans for defragmentation and THP on these cores.

There was a recent discussion on the subject matter on lkml that mentions
a number of the pending issues in this area:

https://lkml.org/lkml/2014/11/11/679
https://lkml.org/lkml/2014/10/31/364

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM ATTEND] Expanding OS noise suppression

[Christoph Lameter]

Sorry the list address was not correct

On Mon, 24 Nov 2014, Christoph Lameter wrote:

> Recently a lot of work has been done in the kernel to be able to keep OS
> threads off low latency cores with the NOHZ work mainly pushed by Frederic
> Weisbecker (also also Paul McKenney modifying RCU for that purpose). With
> that approach we may now reduce the timer tick to a frequency of 1 per
> second. The result of that work is now available in Redhat 7.
>
> I have recently submitted work on the vmstat kworkers that makes the
> kworkers run on demand with a shepherd worker checking from a non low
> latency processor if there is actual work to be done on a processor in low
> latency mode. If not then the kworker requests can be avoided and
> therefore activities on that processor are reduced. This approach can be
> extended to cover other necessary activities on low latency cores.
>
> There is other work in progress to limit unbound kworker threads to no
> NOHZ processors. Also more work is in flight to work on various issues in
> the scheduler to enable us to hold off the timer tick for more than one
> second.
>
> There are numerous other issues that can impact on a low latency core from
> the memory management system. I would like to discuss ways that we can
> further ensure that OS activities do not impact latency critical threads
> running on special nohz cores.
>
> This may cover:
>  - minor and major faults and how to suppress them effectively.
>  - Processor cache impacts by sibling threads.
>  - IPIs
>  - Control over various subsystem specific per cpu threads.
>  - Control impacts of scans for defragmentation and THP on these cores.
>
> There was a recent discussion on the subject matter on lkml that mentions
> a number of the pending issues in this area:
>
> https://lkml.org/lkml/2014/11/11/679
> https://lkml.org/lkml/2014/10/31/364
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] Expanding OS noise suppression

[Rik van Riel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/01/2014 11:45 AM, Christoph Lameter wrote:
> On Mon, 24 Nov 2014, Christoph Lameter wrote:
> 
>> Recently a lot of work has been done in the kernel to be able to
>> keep OS threads off low latency cores with the NOHZ work mainly
>> pushed by Frederic Weisbecker (also also Paul McKenney modifying
>> RCU for that purpose). With that approach we may now reduce the
>> timer tick to a frequency of 1 per second. The result of that
>> work is now available in Redhat 7.
>> 
>> I have recently submitted work on the vmstat kworkers that makes
>> the kworkers run on demand with a shepherd worker checking from a
>> non low latency processor if there is actual work to be done on a
>> processor in low latency mode. If not then the kworker requests
>> can be avoided and therefore activities on that processor are
>> reduced. This approach can be extended to cover other necessary
>> activities on low latency cores.
>> 
>> There is other work in progress to limit unbound kworker threads
>> to no NOHZ processors. Also more work is in flight to work on
>> various issues in the scheduler to enable us to hold off the
>> timer tick for more than one second.
>> 
>> There are numerous other issues that can impact on a low latency
>> core from the memory management system. I would like to discuss
>> ways that we can further ensure that OS activities do not impact
>> latency critical threads running on special nohz cores.
>> 
>> This may cover: - minor and major faults and how to suppress them
>> effectively. - Processor cache impacts by sibling threads. -
>> IPIs - Control over various subsystem specific per cpu threads. -
>> Control impacts of scans for defragmentation and THP on these
>> cores.

This is a very interesting topic, but I am not sure the right audience
for many of these discussions will be at LSF/MM...

Besides the minor and major faults, and the THP related defragmentation,
which of the problems could actually be addressed by the memory
management subsystem?

Would you have a list of other items in the memory management subsystem
that cause latency issues?

Is the minor & major fault thing an actual problem for people with real
time applications?

Do you have any ideas on how we could solve the defragmentation and THP
issue? Even strawman proposals to start a discussion could be useful...

- -- 
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUfKEqAAoJEM553pKExN6D4n4IAKmEkEHrJkhbaI4xGHHl/Stq
73IWLjd10uLyiTDY2sMn5iKDHpeBLXL63uI8FiPBGO/XBi7YaGdOSGiGUuWLMSzL
oc49dHKI8hMnk9vTG/nRxdrODnRkSgwKudUN6FTe0MwZ9vcDtYXxSTDlN0pMBKKe
hpGHgwUwNzpc/m6gGWdjZ3Dzo5R8WZ84fgdVQlrbjx+9We6XetsbT2CzlxscnKyH
ZgJFae+taqUJt2AN9izeW7cNNRwro3SaS6J9FrO64Y044QCeQ11nyJP2XlAoT7lc
tPv5cOnxXNzBLyulLMHVL1/7jUcx8dfmhdsh8gVpLkL10QypFVS6MhVbyCRZ7dM=
=7AHn
-----END PGP SIGNATURE-----

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] Expanding OS noise suppression

[Christoph Lameter]

On Mon, 1 Dec 2014, Rik van Riel wrote:

> This is a very interesting topic, but I am not sure the right audience
> for many of these discussions will be at LSF/MM...

Well some of it at least is relevant.

> Besides the minor and major faults, and the THP related defragmentation,
> which of the problems could actually be addressed by the memory
> management subsystem?

One of the motivations for the development of SLUB for example was the
long periods of latency generated by SLAB's object expiration. There are
numerous code segment in the mm subsystem that can cause suprisingly long
latencies for the application. Memory allocations through the page
allocator are on of the most severe examples.

The SLUB allocator's per cpu partial pages introduce some new latencies
(not as bad as SLAB but still) and I have seen that RT people compile that
cpu partial page support out because it causes higher variability.

Some way for the application to know and be able to avoid these would be
great.

> Would you have a list of other items in the memory management subsystem
> that cause latency issues?

I mentioned some above. There are numeous issues arising from various
pieces of heavy operations of the mm subsystems which involve page
migration, writeback, general page table walks, statistics keeping etc
etc.

> Is the minor & major fault thing an actual problem for people with real
> time applications?

Yes. The timeframes for electronic trading are lower than the time it
takes for a fault to be processed. A fault occurring at the wrong time
causes an immediate hit on the bottom line.

> Do you have any ideas on how we could solve the defragmentation and THP
> issue? Even strawman proposals to start a discussion could be useful...

Right now we disable automatic defrag and do a run of defrag and THP
before the start of business manually. There are cores that are dedicated
for the OS where the defrag etc can run during business hours and which
could also do these jobs remotely for the low latency cores if one is
careful and does not create too many latency issues on the remote cores.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] Expanding OS noise suppression

[Andy Lutomirski]

On 12/01/2014 10:22 AM, Christoph Lameter wrote:
> On Mon, 1 Dec 2014, Rik van Riel wrote:
> 
>> This is a very interesting topic, but I am not sure the right audience
>> for many of these discussions will be at LSF/MM...
> 
> Well some of it at least is relevant.
> 
>> Besides the minor and major faults, and the THP related defragmentation,
>> which of the problems could actually be addressed by the memory
>> management subsystem?
> 
> One of the motivations for the development of SLUB for example was the
> long periods of latency generated by SLAB's object expiration. There are
> numerous code segment in the mm subsystem that can cause suprisingly long
> latencies for the application. Memory allocations through the page
> allocator are on of the most severe examples.
> 
> The SLUB allocator's per cpu partial pages introduce some new latencies
> (not as bad as SLAB but still) and I have seen that RT people compile that
> cpu partial page support out because it causes higher variability.
> 
> Some way for the application to know and be able to avoid these would be
> great.
> 
>> Would you have a list of other items in the memory management subsystem
>> that cause latency issues?
> 
> I mentioned some above. There are numeous issues arising from various
> pieces of heavy operations of the mm subsystems which involve page
> migration, writeback, general page table walks, statistics keeping etc
> etc.
> 
>> Is the minor & major fault thing an actual problem for people with real
>> time applications?
> 
> Yes. The timeframes for electronic trading are lower than the time it
> takes for a fault to be processed. A fault occurring at the wrong time
> causes an immediate hit on the bottom line.

*snicker* :)

There's also my old complaint that memory mapped files insist on
periodically write-protecting their pages, causing unnecessary minor
faults.  This may or may not affect users, depending on the workload.

FWIW, context tracking for full nohz is *slow*, so it may reduce noise,
but it dramatically increases syscall and fault overhead.  This isn't
really an mm issue, though.

--Andy

> 
>> Do you have any ideas on how we could solve the defragmentation and THP
>> issue? Even strawman proposals to start a discussion could be useful...
> 
> Right now we disable automatic defrag and do a run of defrag and THP
> before the start of business manually. There are cores that are dedicated
> for the OS where the defrag etc can run during business hours and which
> could also do these jobs remotely for the low latency cores if one is
> careful and does not create too many latency issues on the remote cores.
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] Expanding OS noise suppression

[Christoph Lameter]

On Tue, 2 Dec 2014, Andy Lutomirski wrote:

> FWIW, context tracking for full nohz is *slow*, so it may reduce noise,
> but it dramatically increases syscall and fault overhead.  This isn't
> really an mm issue, though.

In general though no syscalls are performed in critical latency sensitive
segments and the I/O is done using kernel bypass. So controlling the OS
causing hiccups becomes an important issue.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>