kswapd consumes 100% CPU when highest zone is small

kswapd consumes 100% CPU when highest zone is small

[Jerry Lee]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Hi,

I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
large
files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
deleted
or the page cache is dropped.  With setting the min_free_kbytes from 16384
to
65536, the symptom is mitigated but I can't totally get rid of the problem.

After some trial and error, I found that highest zone is always unbalanced
with
order-0 page request so that pgdat_blanaced() continuously return false and
kswapd can't sleep.

Here's the watermarks (min_free_kbytes = 65536) in my system:
Node 0, zone      DMA
  pages free     2167
        min      138
        low      172
        high     207
        scanned  0
        spanned  4095
        present  3996
        managed  3974

Node 0, zone    DMA32
  pages free     215375
        min      16226
        low      20282
        high     24339
        scanned  0
        spanned  1044480
        present  490971
        managed  464223

Node 0, zone   Normal
  pages free     7
        min      18
        low      22
        high     27
        scanned  0
        spanned  1536
        present  1536
        managed  523

Besides, when the kswapd crazily spins, the value of the following entries
in vmstat increases quickly even when I stop copying file:

pgalloc_dma 17719
pgalloc_dma32 3262823
slabs_scanned 937728
kswapd_high_wmark_hit_quickly 54333233
pageoutrun 54333235

Is there anything I could do to totally get rid of the problem?

Thanks

[-- Attachment #2: Type: text/html, Size: 2052 bytes --]

Re: kswapd consumes 100% CPU when highest zone is small

[chen feng]

On 2016/3/2 14:20, Jerry Lee wrote:
> Hi,
> 
> I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
> large
> files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
> deleted
> or the page cache is dropped.  With setting the min_free_kbytes from 16384
> to
> 65536, the symptom is mitigated but I can't totally get rid of the problem.
> 
> After some trial and error, I found that highest zone is always unbalanced
> with
> order-0 page request so that pgdat_blanaced() continuously return false and
> kswapd can't sleep.
> 
> Here's the watermarks (min_free_kbytes = 65536) in my system:
> Node 0, zone      DMA
>   pages free     2167
>         min      138
>         low      172
>         high     207
>         scanned  0
>         spanned  4095
>         present  3996
>         managed  3974
> 
> Node 0, zone    DMA32
>   pages free     215375
>         min      16226
>         low      20282
>         high     24339
>         scanned  0
>         spanned  1044480
>         present  490971
>         managed  464223
> 
> Node 0, zone   Normal
>   pages free     7
>         min      18
>         low      22
>         high     27
>         scanned  0
>         spanned  1536
>         present  1536
>         managed  523
> 
> Besides, when the kswapd crazily spins, the value of the following entries
> in vmstat increases quickly even when I stop copying file:
> 
> pgalloc_dma 17719
> pgalloc_dma32 3262823
> slabs_scanned 937728
> kswapd_high_wmark_hit_quickly 54333233
> pageoutrun 54333235
> 
> Is there anything I could do to totally get rid of the problem?
> \
Yes, I have the same issue on arm64 platform.

I think you can increase the normal ZONE size. And I think there will be a memory alloc process
in your system which tigger the kswapd too frequently.

You can set this process to no-kswapd flag will also solve this issue.
> Thanks
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd consumes 100% CPU when highest zone is small

[Jerry Lee]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

Hi,

Thanks for sharing the same experience and workaround with me.
But it's kind of hard for me to set all the possible processes to no-kswapd
flag
in advance so that they would not trigger kswapd in the future.

Cheers,
- Jerry

On 2 March 2016 at 22:21, chen feng <puck.chen@foxmail.com> wrote:

>
>
> On 2016/3/2 14:20, Jerry Lee wrote:
> > Hi,
> >
> > I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
> > large
> > files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
> > deleted
> > or the page cache is dropped.  With setting the min_free_kbytes from
> 16384
> > to
> > 65536, the symptom is mitigated but I can't totally get rid of the
> problem.
> >
> > After some trial and error, I found that highest zone is always
> unbalanced
> > with
> > order-0 page request so that pgdat_blanaced() continuously return false
> and
> > kswapd can't sleep.
> >
> > Here's the watermarks (min_free_kbytes = 65536) in my system:
> > Node 0, zone      DMA
> >   pages free     2167
> >         min      138
> >         low      172
> >         high     207
> >         scanned  0
> >         spanned  4095
> >         present  3996
> >         managed  3974
> >
> > Node 0, zone    DMA32
> >   pages free     215375
> >         min      16226
> >         low      20282
> >         high     24339
> >         scanned  0
> >         spanned  1044480
> >         present  490971
> >         managed  464223
> >
> > Node 0, zone   Normal
> >   pages free     7
> >         min      18
> >         low      22
> >         high     27
> >         scanned  0
> >         spanned  1536
> >         present  1536
> >         managed  523
> >
> > Besides, when the kswapd crazily spins, the value of the following
> entries
> > in vmstat increases quickly even when I stop copying file:
> >
> > pgalloc_dma 17719
> > pgalloc_dma32 3262823
> > slabs_scanned 937728
> > kswapd_high_wmark_hit_quickly 54333233
> > pageoutrun 54333235
> >
> > Is there anything I could do to totally get rid of the problem?
> > \
> Yes, I have the same issue on arm64 platform.
>
> I think you can increase the normal ZONE size. And I think there will be a
> memory alloc process
> in your system which tigger the kswapd too frequently.
>
> You can set this process to no-kswapd flag will also solve this issue.
> > Thanks
> >
>

[-- Attachment #2: Type: text/html, Size: 3368 bytes --]

Re: kswapd consumes 100% CPU when highest zone is small

[Chen Feng]

On 2016/3/3 9:56, Jerry Lee wrote:
> Hi,
> 
> Thanks for sharing the same experience and workaround with me.
> But it's kind of hard for me to set all the possible processes to no-kswapd flag
> in advance so that they would not trigger kswapd in the future.
> 
> Cheers,
> - Jerry
> 
> On 2 March 2016 at 22:21, chen feng <puck.chen@foxmail.com <mailto:puck.chen@foxmail.com>> wrote:
> 
> 
> 
>     On 2016/3/2 14:20, Jerry Lee wrote:
>     > Hi,
>     >
>     > I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
>     > large
>     > files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
>     > deleted
>     > or the page cache is dropped.  With setting the min_free_kbytes from 16384
>     > to
>     > 65536, the symptom is mitigated but I can't totally get rid of the problem.
>     >
>     > After some trial and error, I found that highest zone is always unbalanced
>     > with
>     > order-0 page request so that pgdat_blanaced() continuously return false and
>     > kswapd can't sleep.
>     >
>     > Here's the watermarks (min_free_kbytes = 65536) in my system:
>     > Node 0, zone      DMA
>     >   pages free     2167
>     >         min      138
>     >         low      172
>     >         high     207
>     >         scanned  0
>     >         spanned  4095
>     >         present  3996
>     >         managed  3974
>     >
>     > Node 0, zone    DMA32
>     >   pages free     215375
>     >         min      16226
>     >         low      20282
>     >         high     24339
>     >         scanned  0
>     >         spanned  1044480
>     >         present  490971
>     >         managed  464223
>     >
>     > Node 0, zone   Normal
>     >   pages free     7
>     >         min      18
>     >         low      22
>     >         high     27
>     >         scanned  0
>     >         spanned  1536
>     >         present  1536
>     >         managed  523
>     >
>     > Besides, when the kswapd crazily spins, the value of the following entries
>     > in vmstat increases quickly even when I stop copying file:
>     >
>     > pgalloc_dma 17719
>     > pgalloc_dma32 3262823
>     > slabs_scanned 937728
>     > kswapd_high_wmark_hit_quickly 54333233
>     > pageoutrun 54333235
>     >
>     > Is there anything I could do to totally get rid of the problem?
>     > \
>     Yes, I have the same issue on arm64 platform.
> 
>     I think you can increase the normal ZONE size. And I think there will be a memory alloc process
>     in your system which tigger the kswapd too frequently.
> 
>     You can set this process to no-kswapd flag will also solve this issue.
>     > Thanks
>     >

Just hack the process who tigger it too frequenctly.
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd consumes 100% CPU when highest zone is small

[Michal Hocko]

On Wed 02-03-16 14:20:38, Jerry Lee wrote:
> Hi,
> 
> I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
> large
> files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
> deleted
> or the page cache is dropped.  With setting the min_free_kbytes from 16384
> to
> 65536, the symptom is mitigated but I can't totally get rid of the problem.
> 
> After some trial and error, I found that highest zone is always unbalanced
> with
> order-0 page request so that pgdat_blanaced() continuously return false and
> kswapd can't sleep.
> 
> Here's the watermarks (min_free_kbytes = 65536) in my system:
> Node 0, zone      DMA
>   pages free     2167
>         min      138
>         low      172
>         high     207
>         scanned  0
>         spanned  4095
>         present  3996
>         managed  3974
> 
> Node 0, zone    DMA32
>   pages free     215375
>         min      16226
>         low      20282
>         high     24339
>         scanned  0
>         spanned  1044480
>         present  490971
>         managed  464223
> 
> Node 0, zone   Normal
>   pages free     7
>         min      18
>         low      22
>         high     27
>         scanned  0
>         spanned  1536
>         present  1536
>         managed  523

The zone Normal is just too small and that confuses the reclaim path.

> 
> Besides, when the kswapd crazily spins, the value of the following entries
> in vmstat increases quickly even when I stop copying file:
> 
> pgalloc_dma 17719
> pgalloc_dma32 3262823
> slabs_scanned 937728
> kswapd_high_wmark_hit_quickly 54333233
> pageoutrun 54333235
> 
> Is there anything I could do to totally get rid of the problem?

I would try to sacrifice those few megs and get rid of zone normal
completely. AFAIR mem=4G should limit the max_pfn to 4G so DMA32 should
cover the shole memory.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd consumes 100% CPU when highest zone is small

[Jerry Lee]

[-- Attachment #1: Type: text/plain, Size: 3284 bytes --]

On 3 March 2016 at 01:36, Michal Hocko <mhocko@kernel.org> wrote:

> On Wed 02-03-16 14:20:38, Jerry Lee wrote:
> > Hi,
> >
> > I have a x86_64 system with 2G RAM using linux-3.12.x.  During copying
> > large
> > files (e.g. 100GB), kswapd easily consumes 100% CPU until the file is
> > deleted
> > or the page cache is dropped.  With setting the min_free_kbytes from
> 16384
> > to
> > 65536, the symptom is mitigated but I can't totally get rid of the
> problem.
> >
> > After some trial and error, I found that highest zone is always
> unbalanced
> > with
> > order-0 page request so that pgdat_blanaced() continuously return false
> and
> > kswapd can't sleep.
> >
> > Here's the watermarks (min_free_kbytes = 65536) in my system:
> > Node 0, zone      DMA
> >   pages free     2167
> >         min      138
> >         low      172
> >         high     207
> >         scanned  0
> >         spanned  4095
> >         present  3996
> >         managed  3974
> >
> > Node 0, zone    DMA32
> >   pages free     215375
> >         min      16226
> >         low      20282
> >         high     24339
> >         scanned  0
> >         spanned  1044480
> >         present  490971
> >         managed  464223
> >
> > Node 0, zone   Normal
> >   pages free     7
> >         min      18
> >         low      22
> >         high     27
> >         scanned  0
> >         spanned  1536
> >         present  1536
> >         managed  523
>
> The zone Normal is just too small and that confuses the reclaim path.
>
> >
> > Besides, when the kswapd crazily spins, the value of the following
> entries
> > in vmstat increases quickly even when I stop copying file:
> >
> > pgalloc_dma 17719
> > pgalloc_dma32 3262823
> > slabs_scanned 937728
> > kswapd_high_wmark_hit_quickly 54333233
> > pageoutrun 54333235
> >
> > Is there anything I could do to totally get rid of the problem?
>
> I would try to sacrifice those few megs and get rid of zone normal
> completely. AFAIR mem=4G should limit the max_pfn to 4G so DMA32 should
> cover the shole memory.
>

I came up with a patch that seem to work well on my system.  But, I am
afraid
that it breaks the rule that all zones must be balanced for order-0 request
and
It may cause some other side-effect?  I thought that the patch is just a
workaround
(a bad one) and not a cure-all.

BTW, if I upgrade the RAM from 2G to 4G, the problem is gone because the
Normal zone won't confuse the reclaim path as you said before.

Thanks


--- a/linux-3.12.6/mm/vmscan.c
+++ b/linux-3.12.6/mm/vmscan.c
@@ -2755,6 +2755,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int
order, int classzone_idx)
        unsigned long managed_pages = 0;
        unsigned long balanced_pages = 0;
        int i;
+#define HWMARK_THRESHOLD 128

        /* Check the watermark levels */
        for (i = 0; i <= classzone_idx; i++) {
@@ -2779,7 +2780,8 @@ static bool pgdat_balanced(pg_data_t *pgdat, int
order, int classzone_idx)

                if (zone_balanced(zone, order, 0, i))
                        balanced_pages += zone->managed_pages;
-               else if (!order)
+               else if (!order &&
+                        (high_wmark_pages(zone) > HWMARK_THRESHOLD))
                        return false;
        }



> --
> Michal Hocko
> SUSE Labs
>

[-- Attachment #2: Type: text/html, Size: 4775 bytes --]

Re: kswapd consumes 100% CPU when highest zone is small

[Michal Hocko]

On Thu 03-03-16 10:23:03, Jerry Lee wrote:
> On 3 March 2016 at 01:36, Michal Hocko <mhocko@kernel.org> wrote:
> 
> > On Wed 02-03-16 14:20:38, Jerry Lee wrote:
[...]
> > > Is there anything I could do to totally get rid of the problem?
> >
> > I would try to sacrifice those few megs and get rid of zone normal
> > completely. AFAIR mem=4G should limit the max_pfn to 4G so DMA32 should
> > cover the shole memory.
> >
> 
> I came up with a patch that seem to work well on my system.  But, I
> am afraid that it breaks the rule that all zones must be balanced for
> order-0 request and It may cause some other side-effect?  I thought
> that the patch is just a workaround (a bad one) and not a cure-all.

One thing I haven't noticed previously is that you are running on the 3.12
kernel. I vaguely remember there were some fixes for small zones. Not
sure it would work for such a small zone but it would be worth trying I
guess. Could you retest with 4.4?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd consumes 100% CPU when highest zone is small

[Jerry Lee]

[-- Attachment #1: Type: text/plain, Size: 2058 bytes --]

On 3 March 2016 at 16:22, Michal Hocko <mhocko@kernel.org> wrote:

> On Thu 03-03-16 10:23:03, Jerry Lee wrote:
> > On 3 March 2016 at 01:36, Michal Hocko <mhocko@kernel.org> wrote:
> >
> > > On Wed 02-03-16 14:20:38, Jerry Lee wrote:
> [...]
> > > > Is there anything I could do to totally get rid of the problem?
> > >
> > > I would try to sacrifice those few megs and get rid of zone normal
> > > completely. AFAIR mem=4G should limit the max_pfn to 4G so DMA32 should
> > > cover the shole memory.
> > >
> >
> > I came up with a patch that seem to work well on my system.  But, I
> > am afraid that it breaks the rule that all zones must be balanced for
> > order-0 request and It may cause some other side-effect?  I thought
> > that the patch is just a workaround (a bad one) and not a cure-all.
>
> One thing I haven't noticed previously is that you are running on the 3.12
> kernel. I vaguely remember there were some fixes for small zones. Not
> sure it would work for such a small zone but it would be worth trying I
> guess. Could you retest with 4.4?
>

Hi,

Thanks for the quick feedback!

Before sending a mail to linux-mm, I found that there were discussions and
fixes for the small zone as you remember:
https://lkml.org/lkml/2011/6/24/161 .
However, the fixes is kind of old and should be already included into the
current kernel version.  Speaking of retesting the issue with kernel-4.4,
it's
a bit hard for my right now because there are some customized hardware and
drivers on my system but I could give it a try.

BTW, there are some information I forgot to mention before.  Originally, I
use
kernel-3.4 on my system without the kswapd issue.  After upgrading to
linux-3.12.x, the issue occur.  In addition, I found that there are other
people
encountering the same problem even linux-4.x are used. [1] The idea to
increase the value of min_free_kbytes comes from the post. [1]

[1] https://github.com/GalliumOS/galliumos-distro/issues/52

Anyway, thanks again for your help and suggestion!

- Jerry



> --
> Michal Hocko
> SUSE Labs
>

[-- Attachment #2: Type: text/html, Size: 3190 bytes --]