[LSF/MM TOPIC] Proactive Memory Reclaim

[LSF/MM TOPIC] Proactive Memory Reclaim

[Shakeel Butt]

Though this is quite late, I still want to propose a topic for
discussion during LSFMM'19 which I think will be beneficial for Linux
users in general but particularly the data center users running a
range of different workloads and want to reduce the memory cost.

Topic: Proactive Memory Reclaim

Motivation/Problem: Memory overcommit is most commonly used technique
to reduce the cost of memory by large infrastructure owners. However
memory overcommit can adversely impact the performance of latency
sensitive applications by triggering direct memory reclaim. Direct
reclaim is unpredictable and disastrous for latency sensitive
applications.

Solution: Proactively reclaim memory from the system to drastically
reduce the occurrences of direct reclaim. Target cold memory to keep
the refault rate of the applications acceptable (i.e. no impact on the
performance).

Challenges:
1. Tracking cold memory efficiently.
2. Lack of infrastructure to reclaim specific memory.

Details: Existing "Idle Page Tracking" allows tracking cold memory on
a system but it becomes prohibitively expensive as the machine size
grows. Also there is no way from the user space to reclaim a specific
'cold' page. I want to present our implementation of cold memory
tracking and reclaim. The aim is to make it more generally beneficial
to lot more users and upstream it.

More details:
"Software-driven far-memory in warehouse-scale computers", ASPLOS'19.
https://youtu.be/aKddds6jn1s

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Mel Gorman]

On Tue, Apr 23, 2019 at 08:30:46AM -0700, Shakeel Butt wrote:
> Though this is quite late, I still want to propose a topic for
> discussion during LSFMM'19 which I think will be beneficial for Linux
> users in general but particularly the data center users running a
> range of different workloads and want to reduce the memory cost.
> 
> Topic: Proactive Memory Reclaim
> 
> Motivation/Problem: Memory overcommit is most commonly used technique
> to reduce the cost of memory by large infrastructure owners. However
> memory overcommit can adversely impact the performance of latency
> sensitive applications by triggering direct memory reclaim. Direct
> reclaim is unpredictable and disastrous for latency sensitive
> applications.
> 
> Solution: Proactively reclaim memory from the system to drastically
> reduce the occurrences of direct reclaim. Target cold memory to keep
> the refault rate of the applications acceptable (i.e. no impact on the
> performance).
> 
> Challenges:
> 1. Tracking cold memory efficiently.
> 2. Lack of infrastructure to reclaim specific memory.
> 
> Details: Existing "Idle Page Tracking" allows tracking cold memory on
> a system but it becomes prohibitively expensive as the machine size
> grows. Also there is no way from the user space to reclaim a specific
> 'cold' page. I want to present our implementation of cold memory
> tracking and reclaim. The aim is to make it more generally beneficial
> to lot more users and upstream it.
> 

Why is this not partially addressed by tuning vm.watermark_scale_factor?
As for a specific cold page, why not mmap the page in question,
msync(MS_SYNC) and call madvise(MADV_DONTNEED)? It may not be perfect in
all cases admittedly.

-- 
Mel Gorman
SUSE Labs

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Shakeel Butt]

On Tue, Apr 23, 2019 at 8:58 AM Mel Gorman <mgorman@techsingularity.net> wrote:
>
> On Tue, Apr 23, 2019 at 08:30:46AM -0700, Shakeel Butt wrote:
> > Though this is quite late, I still want to propose a topic for
> > discussion during LSFMM'19 which I think will be beneficial for Linux
> > users in general but particularly the data center users running a
> > range of different workloads and want to reduce the memory cost.
> >
> > Topic: Proactive Memory Reclaim
> >
> > Motivation/Problem: Memory overcommit is most commonly used technique
> > to reduce the cost of memory by large infrastructure owners. However
> > memory overcommit can adversely impact the performance of latency
> > sensitive applications by triggering direct memory reclaim. Direct
> > reclaim is unpredictable and disastrous for latency sensitive
> > applications.
> >
> > Solution: Proactively reclaim memory from the system to drastically
> > reduce the occurrences of direct reclaim. Target cold memory to keep
> > the refault rate of the applications acceptable (i.e. no impact on the
> > performance).
> >
> > Challenges:
> > 1. Tracking cold memory efficiently.
> > 2. Lack of infrastructure to reclaim specific memory.
> >
> > Details: Existing "Idle Page Tracking" allows tracking cold memory on
> > a system but it becomes prohibitively expensive as the machine size
> > grows. Also there is no way from the user space to reclaim a specific
> > 'cold' page. I want to present our implementation of cold memory
> > tracking and reclaim. The aim is to make it more generally beneficial
> > to lot more users and upstream it.
> >
>
> Why is this not partially addressed by tuning vm.watermark_scale_factor?

We want to have more control on exactly which memory pages to reclaim.
The definition of cold memory can be very job specific. With kswapd,
that is not possible.

> As for a specific cold page, why not mmap the page in question,
> msync(MS_SYNC) and call madvise(MADV_DONTNEED)? It may not be perfect in
> all cases admittedly.
>

Wouldn't this throw away the anon memory? We want to swapout that. In
our production we actually only target swapbacked memory due to very
low page fault cost from zswap.

Shakeel

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Yang Shi]

Hi Shakeel,

This sounds interesting. Actually, we have something similar designed
in-house (called "cold" page reclaim). But, we mainly targeted to cold
page cache rather than anonymous page for the time being, and it does
in cgroup scope. We are extending it to anonymous page now.

Look forward to discussing with you.


On Tue, Apr 23, 2019 at 9:34 AM Shakeel Butt <shakeelb@google.com> wrote:
>
> On Tue, Apr 23, 2019 at 8:58 AM Mel Gorman <mgorman@techsingularity.net> wrote:
> >
> > On Tue, Apr 23, 2019 at 08:30:46AM -0700, Shakeel Butt wrote:
> > > Though this is quite late, I still want to propose a topic for
> > > discussion during LSFMM'19 which I think will be beneficial for Linux
> > > users in general but particularly the data center users running a
> > > range of different workloads and want to reduce the memory cost.
> > >
> > > Topic: Proactive Memory Reclaim
> > >
> > > Motivation/Problem: Memory overcommit is most commonly used technique
> > > to reduce the cost of memory by large infrastructure owners. However
> > > memory overcommit can adversely impact the performance of latency
> > > sensitive applications by triggering direct memory reclaim. Direct
> > > reclaim is unpredictable and disastrous for latency sensitive
> > > applications.
> > >
> > > Solution: Proactively reclaim memory from the system to drastically
> > > reduce the occurrences of direct reclaim. Target cold memory to keep
> > > the refault rate of the applications acceptable (i.e. no impact on the
> > > performance).
> > >
> > > Challenges:
> > > 1. Tracking cold memory efficiently.
> > > 2. Lack of infrastructure to reclaim specific memory.
> > >
> > > Details: Existing "Idle Page Tracking" allows tracking cold memory on
> > > a system but it becomes prohibitively expensive as the machine size
> > > grows. Also there is no way from the user space to reclaim a specific
> > > 'cold' page. I want to present our implementation of cold memory
> > > tracking and reclaim. The aim is to make it more generally beneficial
> > > to lot more users and upstream it.
> > >
> >
> > Why is this not partially addressed by tuning vm.watermark_scale_factor?
>
> We want to have more control on exactly which memory pages to reclaim.
> The definition of cold memory can be very job specific. With kswapd,
> that is not possible.
>
> > As for a specific cold page, why not mmap the page in question,
> > msync(MS_SYNC) and call madvise(MADV_DONTNEED)? It may not be perfect in
> > all cases admittedly.
> >
>
> Wouldn't this throw away the anon memory? We want to swapout that. In
> our production we actually only target swapbacked memory due to very
> low page fault cost from zswap.
>
> Shakeel
>

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Shakeel Butt]

On Tue, Apr 23, 2019 at 9:50 AM Yang Shi <shy828301@gmail.com> wrote:
>
> Hi Shakeel,
>
> This sounds interesting. Actually, we have something similar designed
> in-house (called "cold" page reclaim). But, we mainly targeted to cold
> page cache rather than anonymous page for the time being, and it does
> in cgroup scope. We are extending it to anonymous page now.
>
> Look forward to discussing with you.
>

Hi Yang,

Thanks for the info. Is this per-cgroup "cold page reclaim" is
triggered by the job themselves? Are the jobs trying to avoid memcg
limit reclaim by proactively reclaiming their own memory?

Shakeel

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Yang Shi]

On Tue, Apr 23, 2019 at 10:12 AM Shakeel Butt <shakeelb@google.com> wrote:
>
> On Tue, Apr 23, 2019 at 9:50 AM Yang Shi <shy828301@gmail.com> wrote:
> >
> > Hi Shakeel,
> >
> > This sounds interesting. Actually, we have something similar designed
> > in-house (called "cold" page reclaim). But, we mainly targeted to cold
> > page cache rather than anonymous page for the time being, and it does
> > in cgroup scope. We are extending it to anonymous page now.
> >
> > Look forward to discussing with you.
> >
>
> Hi Yang,
>
> Thanks for the info. Is this per-cgroup "cold page reclaim" is
> triggered by the job themselves? Are the jobs trying to avoid memcg

No, it is triggered by admin or cluster management.

> limit reclaim by proactively reclaiming their own memory?

Yes, kind of. And, it also helps to avoid global direct reclaim.

>
> Shakeel

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Tue, 2019-04-23 at 08:30 -0700, Shakeel Butt wrote:

> Topic: Proactive Memory Reclaim
> 
> Motivation/Problem: Memory overcommit is most commonly used technique
> to reduce the cost of memory by large infrastructure owners. However
> memory overcommit can adversely impact the performance of latency
> sensitive applications by triggering direct memory reclaim. Direct
> reclaim is unpredictable and disastrous for latency sensitive
> applications.

This sounds similar to a project Johannes has
been working on, except he is not tracking which
memory is idle at all, but only the pressure on
each cgroup, through the PSI interface:

https://facebookmicrosites.github.io/psi/docs/overview

Discussing the pros and cons, and experiences with
both approaches seems like a useful topic. I'll add
it to the agenda.

-- 
All Rights Reversed.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Shakeel Butt]

On Tue, Apr 23, 2019 at 9:08 AM Rik van Riel <riel@surriel.com> wrote:
>
> On Tue, 2019-04-23 at 08:30 -0700, Shakeel Butt wrote:
>
> > Topic: Proactive Memory Reclaim
> >
> > Motivation/Problem: Memory overcommit is most commonly used technique
> > to reduce the cost of memory by large infrastructure owners. However
> > memory overcommit can adversely impact the performance of latency
> > sensitive applications by triggering direct memory reclaim. Direct
> > reclaim is unpredictable and disastrous for latency sensitive
> > applications.
>
> This sounds similar to a project Johannes has
> been working on, except he is not tracking which
> memory is idle at all, but only the pressure on
> each cgroup, through the PSI interface:
>
> https://facebookmicrosites.github.io/psi/docs/overview
>

I think both techniques are orthogonal and can be used concurrently.
This technique proactively reclaims memory and hopes that we don't go
to direct reclaim but in the worst case if we trigger direct reclaim
then we can use PSI to early detect when to give up on reclaim and
trigger oom-kill.

Another thing I want to point out is our usage model: this proactive
memory reclaim is transparent to the jobs. The admin (infrastructure
owner) is using proactive reclaim to create more schedulable memory
transparently to the job owners.

> Discussing the pros and cons, and experiences with
> both approaches seems like a useful topic. I'll add
> it to the agenda.
>

Thanks a lot.
Shakeel

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Johannes Weiner]

On Tue, Apr 23, 2019 at 10:04:19AM -0700, Shakeel Butt wrote:
> On Tue, Apr 23, 2019 at 9:08 AM Rik van Riel <riel@surriel.com> wrote:
> > On Tue, 2019-04-23 at 08:30 -0700, Shakeel Butt wrote:
> > This sounds similar to a project Johannes has
> > been working on, except he is not tracking which
> > memory is idle at all, but only the pressure on
> > each cgroup, through the PSI interface:
> >
> > https://facebookmicrosites.github.io/psi/docs/overview
> >
> 
> I think both techniques are orthogonal and can be used concurrently.
> This technique proactively reclaims memory and hopes that we don't go
> to direct reclaim but in the worst case if we trigger direct reclaim
> then we can use PSI to early detect when to give up on reclaim and
> trigger oom-kill.
> 
> Another thing I want to point out is our usage model: this proactive
> memory reclaim is transparent to the jobs. The admin (infrastructure
> owner) is using proactive reclaim to create more schedulable memory
> transparently to the job owners.

That's our motivation too.

We want a more accurate sense of actually "required" RAM for each job,
as determined by the job's latency expectations, the access frequency
curve, and IO latency (or compression and CPU latency - whatever is
used for secondary storage). The latter two change dynamically based
on memory and IO access patterns, but psi factors that in.

It's supposed to be transparent to the job owners and not impact their
performance. It's supposed to help them understand their own memory
requirements and the utilization of their resource allotment. Having a
better sense of utilization also helps fleet capacity planning.

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Suren Baghdasaryan]

On Tue, Apr 23, 2019 at 9:08 AM Rik van Riel <riel@surriel.com> wrote:
>
> On Tue, 2019-04-23 at 08:30 -0700, Shakeel Butt wrote:
>
> > Topic: Proactive Memory Reclaim
> >
> > Motivation/Problem: Memory overcommit is most commonly used technique
> > to reduce the cost of memory by large infrastructure owners. However
> > memory overcommit can adversely impact the performance of latency
> > sensitive applications by triggering direct memory reclaim. Direct
> > reclaim is unpredictable and disastrous for latency sensitive
> > applications.
>
> This sounds similar to a project Johannes has
> been working on, except he is not tracking which
> memory is idle at all, but only the pressure on
> each cgroup, through the PSI interface:
>
> https://facebookmicrosites.github.io/psi/docs/overview
>
> Discussing the pros and cons, and experiences with
> both approaches seems like a useful topic. I'll add
> it to the agenda.

This topic sounds interesting and in line with some experiments being
done on Android. Looking forward to this discussion. CC'ing Android
folks that might be interested as well.

> --
> All Rights Reversed.

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Johannes Weiner]

Hi Shakeel,

On Tue, Apr 23, 2019 at 08:30:46AM -0700, Shakeel Butt wrote:
> Though this is quite late, I still want to propose a topic for
> discussion during LSFMM'19 which I think will be beneficial for Linux
> users in general but particularly the data center users running a
> range of different workloads and want to reduce the memory cost.
> 
> Topic: Proactive Memory Reclaim
> 
> Motivation/Problem: Memory overcommit is most commonly used technique
> to reduce the cost of memory by large infrastructure owners. However
> memory overcommit can adversely impact the performance of latency
> sensitive applications by triggering direct memory reclaim. Direct
> reclaim is unpredictable and disastrous for latency sensitive
> applications.
> 
> Solution: Proactively reclaim memory from the system to drastically
> reduce the occurrences of direct reclaim. Target cold memory to keep
> the refault rate of the applications acceptable (i.e. no impact on the
> performance).
> 
> Challenges:
> 1. Tracking cold memory efficiently.
> 2. Lack of infrastructure to reclaim specific memory.
> 
> Details: Existing "Idle Page Tracking" allows tracking cold memory on
> a system but it becomes prohibitively expensive as the machine size
> grows. Also there is no way from the user space to reclaim a specific
> 'cold' page. I want to present our implementation of cold memory
> tracking and reclaim. The aim is to make it more generally beneficial
> to lot more users and upstream it.
> 
> More details:
> "Software-driven far-memory in warehouse-scale computers", ASPLOS'19.
> https://youtu.be/aKddds6jn1s

I would be very interested to hear about this as well.

As Rik mentions, I've been working on a way to determine the "true"
memory workingsets of our workloads. I'm using a pressure feedback
loop of psi and dynamically adjusted cgroup limits, to harness the
kernel's LRU/clock algorithm to sort out what's cold and what isn't.

This does use direct reclaim, but since psi quantifies the exact time
cost of that, it backs off before our SLAs are violated. Of course, if
necessary, this work could easily be punted to a kthread or something.

The additional refault IO also has not been a problem in practice for
us so far, since our pressure parameters are fairly conservative. But
that is a bit harder to manage - by the time you experience those you
might have already oversteered. This is where compression could help
reduce the cost of being aggressive. That said, even with conservative
settings I've managed to shave off 25-30% of the memory footprint of
common interactive jobs without affecting their performance. I suspect
that in many workloads (depending on their exact slope of the access
locality bell curve) shaving off more would require a disproportionate
amount more pressure/CPU/IO, and so might not be worthwile.

Anyway, I'd love to hear your insights on this.

Re: [LSF/MM TOPIC] Proactive Memory Reclaim

[Christopher Lameter]

Could we retitle this "Improve background reclaim"?

This is a basic function of the VM that we seek to improve. It is already
proactive by predicting reclaim based on the LRU. We have tried to improve
that a couple of times over the years. This is bringing new ideas to the
table but its not something entirely new.