hibernation memory usage

hibernation memory usage

[Luigi Semenzato]

[-- Attachment #1: Type: text/plain, Size: 706 bytes --]

Hi,

I am working on a project that uses hibernation, and we've noticed
occasional failures with "echo disk > /sys/power/state" returning ENOMEM.
I added some logging and noticed that the failures seem to correlate with
total anonymous pages being approximately 1/2 of total RAM.  The allocation
strategy isn't explicitly documented and the code is a bit tricky (as
usual), but I am getting the sense that a copy of the entire RAM in use is
made prior to saving it to disk.  Is it the case then that hibernation is
guaranteed to fail if anon memory is more than 50% of RAM?  Since tasks are
frozen, that memory cannot change and the copy seems redundant (except it
probably makes things simpler).

Thanks!

[-- Attachment #2: Type: text/html, Size: 802 bytes --]

Re: hibernation memory usage

[James Morse]

Hi Luigi,

(CC: +linux-pm mailing list)

On 03/10/2019 18:16, Luigi Semenzato wrote:
> I am working on a project that uses hibernation, and we've noticed occasional failures
> with "echo disk > /sys/power/state" returning ENOMEM.  I added some logging and noticed
> that the failures seem to correlate with total anonymous pages being approximately 1/2 of
> total RAM.  The allocation strategy isn't explicitly documented and the code is a bit
> tricky (as usual), but I am getting the sense that a copy of the entire RAM in use is made
> prior to saving it to disk.  Is it the case then that hibernation is guaranteed to fail if
> anon memory is more than 50% of RAM? 

I'm pretty sure it is. If 50% of memory needs saving, you can't create a snapshot of it.


> Since tasks are frozen, that memory cannot change> and the copy seems redundant (except it probably makes things simpler).

Tasks aren't the only thing changing memory. Hibernate save/restores the entire system,
including the kernel data and text. (what happens if a task is waiting for a syscall to
complete?)


Hibernate needs a snapshot of memory, and the disk drivers, block layer etc need to write
to memory (and allocate it) in order to get their work done.

To work with this, hibernate's create_image() stops secondary CPUs and suspends all
devices. Now that only hibernate is running, it calls swsusp_arch_suspend() which then
call swsusp_save(). This creates the snapshot of memory.

Once this is done, devices are thawed, and hibernate() goes on to call swusp_write() to
write the snapshot to disk. Finally, processes are thawed.

(create_image() is called by hibernation_snapshot() from hibernate()).


If you don't need to save/restore the kernel state, you might not need hibernate at all.


Thanks,

James

Re: hibernation memory usage

[Luigi Semenzato]

Thank you for your reply!

I understand the need for saving all state, not just process/task
state.  But for many of the systems that could benefit from
hibernation, the majority of RAM is taken by user processes (I am
thinking laptops).  It should be possible to copy their anonymous
pages to disk more or less directly, without making an extra copy like
it's done for all other pages.  I am not sure what happens with kernel
tasks, but they don't have anonymous pages (that I know).

I am curious to know how/if hibernation is currently used in practice.
It doesn't seem practical to require that user processes take less
than 50% of RAM at all times.  There may be special cases in which the
restriction can be achieved by terminating non-essential processes
before hibernating, but I don't know of any.

I would also like to know how much work it might take to avoid the
extra copy of the anonymous pages of frozen processes.

Thanks!

On Tue, Oct 8, 2019 at 3:33 AM James Morse <james.morse@arm.com> wrote:
>
> Hi Luigi,
>
> (CC: +linux-pm mailing list)
>
> On 03/10/2019 18:16, Luigi Semenzato wrote:
> > I am working on a project that uses hibernation, and we've noticed occasional failures
> > with "echo disk > /sys/power/state" returning ENOMEM.  I added some logging and noticed
> > that the failures seem to correlate with total anonymous pages being approximately 1/2 of
> > total RAM.  The allocation strategy isn't explicitly documented and the code is a bit
> > tricky (as usual), but I am getting the sense that a copy of the entire RAM in use is made
> > prior to saving it to disk.  Is it the case then that hibernation is guaranteed to fail if
> > anon memory is more than 50% of RAM?
>
> I'm pretty sure it is. If 50% of memory needs saving, you can't create a snapshot of it.
>
>
> > Since tasks are frozen, that memory cannot change> and the copy seems redundant (except it probably makes things simpler).
>
> Tasks aren't the only thing changing memory. Hibernate save/restores the entire system,
> including the kernel data and text. (what happens if a task is waiting for a syscall to
> complete?)
>
>
> Hibernate needs a snapshot of memory, and the disk drivers, block layer etc need to write
> to memory (and allocate it) in order to get their work done.
>
> To work with this, hibernate's create_image() stops secondary CPUs and suspends all
> devices. Now that only hibernate is running, it calls swsusp_arch_suspend() which then
> call swsusp_save(). This creates the snapshot of memory.
>
> Once this is done, devices are thawed, and hibernate() goes on to call swusp_write() to
> write the snapshot to disk. Finally, processes are thawed.
>
> (create_image() is called by hibernation_snapshot() from hibernate()).
>
>
> If you don't need to save/restore the kernel state, you might not need hibernate at all.
>
>
> Thanks,
>
> James

Re: hibernation memory usage

[Rafael J. Wysocki]

On Tue, Oct 8, 2019 at 5:26 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> Thank you for your reply!
>
> I understand the need for saving all state, not just process/task
> state.  But for many of the systems that could benefit from
> hibernation, the majority of RAM is taken by user processes (I am
> thinking laptops).  It should be possible to copy their anonymous
> pages to disk more or less directly, without making an extra copy like
> it's done for all other pages.  I am not sure what happens with kernel
> tasks, but they don't have anonymous pages (that I know).
>
> I am curious to know how/if hibernation is currently used in practice.
> It doesn't seem practical to require that user processes take less
> than 50% of RAM at all times.  There may be special cases in which the
> restriction can be achieved by terminating non-essential processes
> before hibernating, but I don't know of any.
>
> I would also like to know how much work it might take to avoid the
> extra copy of the anonymous pages of frozen processes.

Whatever doesn't fit into 50% of RAM needs to be swapped out before
hibernation.  The efficiency of that depends on the swap handling code
and the underlying hardware.  If that is efficient enough overall,
trying to avoid it altogether isn't going to make much of a
difference.

Re: hibernation memory usage

[Luigi Semenzato]

Yes, that makes sense, thank you.  Use separate partitions for swap
and hibernation.

Normally the kernel starts swapping out when there's no reclaimable
memory, so anon usage will be high.  Do you think cranking up
/proc/vm/swappiness would be enough to ensure that file pages stay
over 50%?  Or would you use some tricks, such as running a
high-priority process which allocates >50% of RAM, thus forcing other
anon pages to be swapped out, then killing that process and quickly
hibernating before too many pages are brought back in?  Or changing
the kernel so that in the first part of hibernation we'll just swap
stuff out?

On Tue, Oct 8, 2019 at 8:39 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
>
> On Tue, Oct 8, 2019 at 5:26 PM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > Thank you for your reply!
> >
> > I understand the need for saving all state, not just process/task
> > state.  But for many of the systems that could benefit from
> > hibernation, the majority of RAM is taken by user processes (I am
> > thinking laptops).  It should be possible to copy their anonymous
> > pages to disk more or less directly, without making an extra copy like
> > it's done for all other pages.  I am not sure what happens with kernel
> > tasks, but they don't have anonymous pages (that I know).
> >
> > I am curious to know how/if hibernation is currently used in practice.
> > It doesn't seem practical to require that user processes take less
> > than 50% of RAM at all times.  There may be special cases in which the
> > restriction can be achieved by terminating non-essential processes
> > before hibernating, but I don't know of any.
> >
> > I would also like to know how much work it might take to avoid the
> > extra copy of the anonymous pages of frozen processes.
>
> Whatever doesn't fit into 50% of RAM needs to be swapped out before
> hibernation.  The efficiency of that depends on the swap handling code
> and the underlying hardware.  If that is efficient enough overall,
> trying to avoid it altogether isn't going to make much of a
> difference.

Re: hibernation memory usage

[Luigi Semenzato]

Actually, I think we only need to change the MM watermarks before
hibernation and after resume.  There's a patch that will do just that:

https://lkml.org/lkml/2013/2/17/210

It didn't make it into mainline (which seems kind of unreasonable,
since the watermarks themselves are based on heuristics) but shouldn't
be difficult to apply.  Or are there simpler solutions?

On Tue, Oct 8, 2019 at 9:18 AM Luigi Semenzato <semenzato@google.com> wrote:
>
> Yes, that makes sense, thank you.  Use separate partitions for swap
> and hibernation.
>
> Normally the kernel starts swapping out when there's no reclaimable
> memory, so anon usage will be high.  Do you think cranking up
> /proc/vm/swappiness would be enough to ensure that file pages stay
> over 50%?  Or would you use some tricks, such as running a
> high-priority process which allocates >50% of RAM, thus forcing other
> anon pages to be swapped out, then killing that process and quickly
> hibernating before too many pages are brought back in?  Or changing
> the kernel so that in the first part of hibernation we'll just swap
> stuff out?
>
> On Tue, Oct 8, 2019 at 8:39 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> >
> > On Tue, Oct 8, 2019 at 5:26 PM Luigi Semenzato <semenzato@google.com> wrote:
> > >
> > > Thank you for your reply!
> > >
> > > I understand the need for saving all state, not just process/task
> > > state.  But for many of the systems that could benefit from
> > > hibernation, the majority of RAM is taken by user processes (I am
> > > thinking laptops).  It should be possible to copy their anonymous
> > > pages to disk more or less directly, without making an extra copy like
> > > it's done for all other pages.  I am not sure what happens with kernel
> > > tasks, but they don't have anonymous pages (that I know).
> > >
> > > I am curious to know how/if hibernation is currently used in practice.
> > > It doesn't seem practical to require that user processes take less
> > > than 50% of RAM at all times.  There may be special cases in which the
> > > restriction can be achieved by terminating non-essential processes
> > > before hibernating, but I don't know of any.
> > >
> > > I would also like to know how much work it might take to avoid the
> > > extra copy of the anonymous pages of frozen processes.
> >
> > Whatever doesn't fit into 50% of RAM needs to be swapped out before
> > hibernation.  The efficiency of that depends on the swap handling code
> > and the underlying hardware.  If that is efficient enough overall,
> > trying to avoid it altogether isn't going to make much of a
> > difference.

Re: hibernation memory usage

[Luigi Semenzato]

To make hibernation work, I have been playing with this suggestion:

> Whatever doesn't fit into 50% of RAM needs to be swapped out before
> hibernation.  The efficiency of that depends on the swap handling code
> and the underlying hardware.  If that is efficient enough overall,
> trying to avoid it altogether isn't going to make much of a
> difference.

What's a good way of swapping out 50% of RAM?  I have tried playing
with /proc/sys/vm/min_free_kbytes.  Lowering it below MemFree forces
reclaim and gets swapping started.  Unfortunately the reclaim also
hits file pages, so badly that the system thrashes to a grinding halt.
Swappiness is already set to 100.  Internally it seems that values up
to 200 are valid, and I wish that the entire range was allowed, but I
am not sure it would help.  Right now I am playing with the production
kernel, but similar situations triggered the same behavior in the
Chrome OS kernels, even after internally setting swappiness to values
close to 200.


On Tue, Oct 8, 2019 at 1:10 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> Actually, I think we only need to change the MM watermarks before
> hibernation and after resume.  There's a patch that will do just that:
>
> https://lkml.org/lkml/2013/2/17/210
>
> It didn't make it into mainline (which seems kind of unreasonable,
> since the watermarks themselves are based on heuristics) but shouldn't
> be difficult to apply.  Or are there simpler solutions?
>
> On Tue, Oct 8, 2019 at 9:18 AM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > Yes, that makes sense, thank you.  Use separate partitions for swap
> > and hibernation.
> >
> > Normally the kernel starts swapping out when there's no reclaimable
> > memory, so anon usage will be high.  Do you think cranking up
> > /proc/vm/swappiness would be enough to ensure that file pages stay
> > over 50%?  Or would you use some tricks, such as running a
> > high-priority process which allocates >50% of RAM, thus forcing other
> > anon pages to be swapped out, then killing that process and quickly
> > hibernating before too many pages are brought back in?  Or changing
> > the kernel so that in the first part of hibernation we'll just swap
> > stuff out?
> >
> > On Tue, Oct 8, 2019 at 8:39 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > >
> > > On Tue, Oct 8, 2019 at 5:26 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > >
> > > > Thank you for your reply!
> > > >
> > > > I understand the need for saving all state, not just process/task
> > > > state.  But for many of the systems that could benefit from
> > > > hibernation, the majority of RAM is taken by user processes (I am
> > > > thinking laptops).  It should be possible to copy their anonymous
> > > > pages to disk more or less directly, without making an extra copy like
> > > > it's done for all other pages.  I am not sure what happens with kernel
> > > > tasks, but they don't have anonymous pages (that I know).
> > > >
> > > > I am curious to know how/if hibernation is currently used in practice.
> > > > It doesn't seem practical to require that user processes take less
> > > > than 50% of RAM at all times.  There may be special cases in which the
> > > > restriction can be achieved by terminating non-essential processes
> > > > before hibernating, but I don't know of any.
> > > >
> > > > I would also like to know how much work it might take to avoid the
> > > > extra copy of the anonymous pages of frozen processes.
> > >
> > > Whatever doesn't fit into 50% of RAM needs to be swapped out before
> > > hibernation.  The efficiency of that depends on the swap handling code
> > > and the underlying hardware.  If that is efficient enough overall,
> > > trying to avoid it altogether isn't going to make much of a
> > > difference.

Re: hibernation memory usage

[Luigi Semenzato]

On Thu, Oct 17, 2019 at 3:55 PM Luigi Semenzato <semenzato@google.com> wrote:
>
> To make hibernation work, I have been playing with this suggestion:
>
> > Whatever doesn't fit into 50% of RAM needs to be swapped out before
> > hibernation.  The efficiency of that depends on the swap handling code
> > and the underlying hardware.  If that is efficient enough overall,
> > trying to avoid it altogether isn't going to make much of a
> > difference.
>
> What's a good way of swapping out 50% of RAM?  I have tried playing
> with /proc/sys/vm/min_free_kbytes.  Lowering it below MemFree forces
> reclaim and gets swapping started.  Unfortunately the reclaim also
> hits file pages, so badly that the system thrashes to a grinding halt.
> Swappiness is already set to 100.  Internally it seems that values up
> to 200 are valid, and I wish that the entire range was allowed, but I
> am not sure it would help.  Right now I am playing with the production
> kernel, but similar situations triggered the same behavior in the
> Chrome OS kernels, even after internally setting swappiness to values
> close to 200.

One more data point: playing with min_free_kbytes is flakey: it can
cause OOM-kills even when I increase it slowly (so that I don't
completely destroy the file RSS) and there is plenty of swap space
available.  Not sure why.

Is there perhaps a way of achieving this using the userland suspend
API?  If there is, I am not able to see it.

Thanks!



> On Tue, Oct 8, 2019 at 1:10 PM Luigi Semenzato <semenzato@google.com> wrote:
> >
> > Actually, I think we only need to change the MM watermarks before
> > hibernation and after resume.  There's a patch that will do just that:
> >
> > https://lkml.org/lkml/2013/2/17/210
> >
> > It didn't make it into mainline (which seems kind of unreasonable,
> > since the watermarks themselves are based on heuristics) but shouldn't
> > be difficult to apply.  Or are there simpler solutions?
> >
> > On Tue, Oct 8, 2019 at 9:18 AM Luigi Semenzato <semenzato@google.com> wrote:
> > >
> > > Yes, that makes sense, thank you.  Use separate partitions for swap
> > > and hibernation.
> > >
> > > Normally the kernel starts swapping out when there's no reclaimable
> > > memory, so anon usage will be high.  Do you think cranking up
> > > /proc/vm/swappiness would be enough to ensure that file pages stay
> > > over 50%?  Or would you use some tricks, such as running a
> > > high-priority process which allocates >50% of RAM, thus forcing other
> > > anon pages to be swapped out, then killing that process and quickly
> > > hibernating before too many pages are brought back in?  Or changing
> > > the kernel so that in the first part of hibernation we'll just swap
> > > stuff out?
> > >
> > > On Tue, Oct 8, 2019 at 8:39 AM Rafael J. Wysocki <rafael@kernel.org> wrote:
> > > >
> > > > On Tue, Oct 8, 2019 at 5:26 PM Luigi Semenzato <semenzato@google.com> wrote:
> > > > >
> > > > > Thank you for your reply!
> > > > >
> > > > > I understand the need for saving all state, not just process/task
> > > > > state.  But for many of the systems that could benefit from
> > > > > hibernation, the majority of RAM is taken by user processes (I am
> > > > > thinking laptops).  It should be possible to copy their anonymous
> > > > > pages to disk more or less directly, without making an extra copy like
> > > > > it's done for all other pages.  I am not sure what happens with kernel
> > > > > tasks, but they don't have anonymous pages (that I know).
> > > > >
> > > > > I am curious to know how/if hibernation is currently used in practice.
> > > > > It doesn't seem practical to require that user processes take less
> > > > > than 50% of RAM at all times.  There may be special cases in which the
> > > > > restriction can be achieved by terminating non-essential processes
> > > > > before hibernating, but I don't know of any.
> > > > >
> > > > > I would also like to know how much work it might take to avoid the
> > > > > extra copy of the anonymous pages of frozen processes.
> > > >
> > > > Whatever doesn't fit into 50% of RAM needs to be swapped out before
> > > > hibernation.  The efficiency of that depends on the swap handling code
> > > > and the underlying hardware.  If that is efficient enough overall,
> > > > trying to avoid it altogether isn't going to make much of a
> > > > difference.