8 Gigabytes and constantly swapping

8 Gigabytes and constantly swapping

[Arthur Marsh]

I've been building the Linus git head kernels as the source gets updated 
and the one built about 3 hours ago managed to get stuck with kswapd0 as 
the highest consumer of CPU cycles (but still under 1 percent) of 
processes listed by top for over 15 minutes, after which I hit the power 
switch and rebooted with a Debian 4.11.0 kernel.

The previous kernel built less than 24 hours earlier did not have this 
problem.

CPU is an Athlon64 (Athlon II X4, 4 cores), RAM is 8GiB, swap is 4GiB, 
load was mainly firefox and chromium. Opening a new window in chromium 
seemed to help trigger the problem.

It's not much information to go on, just wondered if anyone else had 
experienced similar issues?

I'm happy to supply more configuration information and run tests 
including with kernels built with test patches applied.

Arthur.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 8 Gigabytes and constantly swapping

[Tetsuo Handa]

On 2017/05/12 17:51, Arthur Marsh wrote:
> I've been building the Linus git head kernels as the source gets updated and
> the one built about 3 hours ago managed to get stuck with kswapd0 as the highest
> consumer of CPU cycles (but still under 1 percent) of processes listed by top for
> over 15 minutes, after which I hit the power switch and rebooted with a Debian
> 4.11.0 kernel.

Did the /bin/top process continue showing up-to-dated statistics rather than
refrain from showing up-to-dated statistics? (I wonder why you had to hit
the power switch before trying SysRq-m/SysRq-f etc.)

If yes, assuming that reading statistics involves memory allocation requests,
there was no load at at all despite firefox and chromium were running?

If no, all allocation requests got stuck waiting for memory reclaim?

> 
> The previous kernel built less than 24 hours earlier did not have this problem.
> 
> CPU is an Athlon64 (Athlon II X4, 4 cores), RAM is 8GiB, swap is 4GiB, load was
> mainly firefox and chromium. Opening a new window in chromium seemed to help
> trigger the problem.
> 
> It's not much information to go on, just wondered if anyone else had experienced
> similar issues?
> 
> I'm happy to supply more configuration information and run tests including with
> kernels built with test patches applied.

I don't know but http://lkml.kernel.org/r/20170316100409.GR802@shells.gnugeneration.com
or http://lkml.kernel.org/r/20170502041235.zqmywvj5tiiom3jk@merlins.org ?
More description of what happened and how you confirmed that
the /bin/top process continued working would be helpful.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 8 Gigabytes and constantly swapping

[Arthur Marsh]

Tetsuo Handa wrote on 12/05/17 20:26:
> On 2017/05/12 17:51, Arthur Marsh wrote:
>> I've been building the Linus git head kernels as the source gets updated and
>> the one built about 3 hours ago managed to get stuck with kswapd0 as the highest
>> consumer of CPU cycles (but still under 1 percent) of processes listed by top for
>> over 15 minutes, after which I hit the power switch and rebooted with a Debian
>> 4.11.0 kernel.
>
> Did the /bin/top process continue showing up-to-dated statistics rather than
> refrain from showing up-to-dated statistics? (I wonder why you had to hit
> the power switch before trying SysRq-m/SysRq-f etc.)

Yes, the /bin/top process continued updating.

The load average reached a high figure (over 14).

kswapd0 was using more CPU cycles than anything else but was still under 
1 percent.

There was still a lot of swap space free (about 3 GiB).

buffer/cache dropped below 1 GiB out of 8 GiB RAM.

I would have used alt-sysrq-f but had only recently enabled it in my 
kernel builds and didn't think to look it up on my other pc.

>
> If yes, assuming that reading statistics involves memory allocation requests,
> there was no load at at all despite firefox and chromium were running?

Wait time was over 98 percent, load average over 14.

>
> If no, all allocation requests got stuck waiting for memory reclaim?
...
> More description of what happened and how you confirmed that
> the /bin/top process continued working would be helpful.
>
>

It was as if kswapd0 was just left waiting to swap pages in and out 
without any other processes getting to complete what they were trying to do.

It does seem to be related to chromium starting up several processes 
when opening extra windows/tabs.

Thanks for your help!

Arthur.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 8 Gigabytes and constantly swapping

[Arthur Marsh]

Arthur Marsh wrote on 12/05/17 23:09:

> It does seem to be related to chromium starting up several processes
> when opening extra windows/tabs.

I tried using ionice -c3 on chromium, which meant that disk I/O by 
chromium should only occur when disk I/O was otherwise idle, and this 
made a major difference.

It appears that chromium's use of multiple processes and threads means 
that it effectively hogs all the disk I/O but ends up not being able to 
do much as different processes/threads are making different demands on 
disk I/O, leading to kswapd0 having the greatest CPU usage of any 
process but still under 1 percent of available CPU time and the system 
spending over 99 percent of the time in wait. All while about 1 GiB of 4 
GiB available swap is being used.

Is there any existing way to limit the chromium process tree to have the 
same amount of access to disk I/O as say firefox running as a single 
process?

Arthur.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 8 Gigabytes and constantly swapping

[Michal Hocko]

On Fri 12-05-17 18:21:27, Arthur Marsh wrote:
> I've been building the Linus git head kernels as the source gets updated and
> the one built about 3 hours ago managed to get stuck with kswapd0 as the
> highest consumer of CPU cycles (but still under 1 percent) of processes
> listed by top for over 15 minutes, after which I hit the power switch and
> rebooted with a Debian 4.11.0 kernel.
> 
> The previous kernel built less than 24 hours earlier did not have this
> problem.
> 
> CPU is an Athlon64 (Athlon II X4, 4 cores), RAM is 8GiB, swap is 4GiB, load
> was mainly firefox and chromium. Opening a new window in chromium seemed to
> help trigger the problem.
> 
> It's not much information to go on, just wondered if anyone else had
> experienced similar issues?
> 
> I'm happy to supply more configuration information and run tests including
> with kernels built with test patches applied.

Is this 32b or 64b kernel? Could you take /proc/vmstat snapshots ever
second while the kswapd is active?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 8 Gigabytes and constantly swapping

[Arthur Marsh]

Michal Hocko wrote on 15/05/17 17:39:

> Is this 32b or 64b kernel? Could you take /proc/vmstat snapshots ever
> second while the kswapd is active?
>

This was with a 64 bit kernel.

Thankfully the problem does not seem to affect 4.12.0-rc1 and later 
kernels, with the amount of swap used and amount of time spent waiting 
on kswapd not being a problem even under heavy load.

Arthur.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>