Kswapd 100% CPU since 3.8 on Sandybridge

linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed

* Kswapd 100% CPU since 3.8 on Sandybridge
       [not found] ` <CABe+QzA-E40bFFXYJBc663Kx0KrE3xy2uZq5xOH2XL6mFPA6+w@mail.gmail.com>
@ 2014-10-04 17:05   ` Sarah A Sharp
  2014-10-06  7:25     ` Daniel Vetter
  2014-10-06  9:37     ` Mel Gorman
  0 siblings, 2 replies; 6+ messages in thread
From: Sarah A Sharp @ 2014-10-04 17:05 UTC (permalink / raw)
  To: linux-mm, mgorman, intel-gfx

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Please excuse the non-wrapped email. My personal system is currently
b0rked, so I'm sending this in frustration from my phone.

My laptop is currently completely hosed. Disk light on full solid
Mouse movement sluggish to the point of moving a couple cms per second.
Firefox window greyed out but not OOM killed yet. When this behavior
occurred in the past, if I ran top, I would see kswapd taking up 100% of
one of my two CPUs.

If I can catch the system in time before mouse movement becomes too
sluggish, closing the browser window will cause kswapd usage to drop, and
the system goes back to a normal state. If I don't catch it in time, I
can't even ssh into the box to kill Firefox because the login times out.
Occasionally Firefox gets OOM killed, but most of the time I have to use
sysreq keys to reboot the system.

This can be reproduced by using either Chrome or Firefox. Chrome fails
faster. I'm not sure whether it's related to loading tabs with a bunch of
images, maybe flash, but it takes around 10-15 tabs being open before it
starts to fail. I can try to characterize it further.

System: Lenovo x220 Intel Sandy Bridge graphics
Ubuntu 14.04 with edgers PPA for Mesa
3.16.3 kernel

Since around the 3.8 kernel time frame, I've been able to reproduce this
behavior. I'm pretty sure it was a kernel change.

I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a
particular mm test. I still don't have time to triage this, but I'm now
frustrated enough to make time.

Mel, what test do you want me to run?

Sarah Sharp

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Kswapd 100% CPU since 3.8 on Sandybridge
  2014-10-04 17:05   ` Kswapd 100% CPU since 3.8 on Sandybridge Sarah A Sharp
@ 2014-10-06  7:25     ` Daniel Vetter
  2014-10-06  9:37     ` Mel Gorman
  1 sibling, 0 replies; 6+ messages in thread
From: Daniel Vetter @ 2014-10-06  7:25 UTC (permalink / raw)
  To: Sarah A Sharp; +Cc: linux-mm, mgorman, intel-gfx

On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote:
> Please excuse the non-wrapped email. My personal system is currently
> b0rked, so I'm sending this in frustration from my phone.
> 
> My laptop is currently completely hosed. Disk light on full solid
> Mouse movement sluggish to the point of moving a couple cms per second.
> Firefox window greyed out but not OOM killed yet. When this behavior
> occurred in the past, if I ran top, I would see kswapd taking up 100% of
> one of my two CPUs.
> 
> If I can catch the system in time before mouse movement becomes too
> sluggish, closing the browser window will cause kswapd usage to drop, and
> the system goes back to a normal state. If I don't catch it in time, I
> can't even ssh into the box to kill Firefox because the login times out.
> Occasionally Firefox gets OOM killed, but most of the time I have to use
> sysreq keys to reboot the system.
> 
> This can be reproduced by using either Chrome or Firefox. Chrome fails
> faster. I'm not sure whether it's related to loading tabs with a bunch of
> images, maybe flash, but it takes around 10-15 tabs being open before it
> starts to fail. I can try to characterize it further.
> 
> System: Lenovo x220 Intel Sandy Bridge graphics
> Ubuntu 14.04 with edgers PPA for Mesa
> 3.16.3 kernel
> 
> Since around the 3.8 kernel time frame, I've been able to reproduce this
> behavior. I'm pretty sure it was a kernel change.

Hm, doesn't ring any bell for i915 bugs, but to make sure can you please
sample debugfs/dri/0/i915_gem_objects while things go south?

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Kswapd 100% CPU since 3.8 on Sandybridge
  2014-10-04 17:05   ` Kswapd 100% CPU since 3.8 on Sandybridge Sarah A Sharp
  2014-10-06  7:25     ` Daniel Vetter
@ 2014-10-06  9:37     ` Mel Gorman
  2014-10-06 13:32       ` Daniel Vetter
  2014-10-26 16:05       ` sarah
  1 sibling, 2 replies; 6+ messages in thread
From: Mel Gorman @ 2014-10-06  9:37 UTC (permalink / raw)
  To: Sarah A Sharp; +Cc: linux-mm, intel-gfx

On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote:
> Please excuse the non-wrapped email. My personal system is currently
> b0rked, so I'm sending this in frustration from my phone.
> 
> My laptop is currently completely hosed. Disk light on full solid
> Mouse movement sluggish to the point of moving a couple cms per second.
> Firefox window greyed out but not OOM killed yet. When this behavior
> occurred in the past, if I ran top, I would see kswapd taking up 100% of
> one of my two CPUs.
> 
> If I can catch the system in time before mouse movement becomes too
> sluggish, closing the browser window will cause kswapd usage to drop, and
> the system goes back to a normal state. If I don't catch it in time, I
> can't even ssh into the box to kill Firefox because the login times out.
> Occasionally Firefox gets OOM killed, but most of the time I have to use
> sysreq keys to reboot the system.
> 
> This can be reproduced by using either Chrome or Firefox. Chrome fails
> faster. I'm not sure whether it's related to loading tabs with a bunch of
> images, maybe flash, but it takes around 10-15 tabs being open before it
> starts to fail. I can try to characterize it further.
> 
> System: Lenovo x220 Intel Sandy Bridge graphics
> Ubuntu 14.04 with edgers PPA for Mesa
> 3.16.3 kernel
> 
> Since around the 3.8 kernel time frame, I've been able to reproduce this
> behavior. I'm pretty sure it was a kernel change.
> 
> I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a
> particular mm test. I still don't have time to triage this, but I'm now
> frustrated enough to make time.
> 
> Mel, what test do you want me to run?
> 

Minimally I wanted you to sample the stack traces for kswapd, narrow down
to the time of its failure and see if it was stuck in a shrinker loop. What
I suspected at the time was that it was hammering on the i915 shrinker and
possibly doing repeated shrinks of the GPU objects in there. At one point
at least, that was an extremely heavy operation if the objections were
not freeable and I wanted to see if that was still the case. I confess I
haven't looked at the code to see what has changed recently.

If that was confirmed then I to modify the mmtests Ftracereclaimcompact
reported to focus exclusively on slab and give a breakdown of which shrinker
it was spending time in. Right now, that reporter only says how much time
is spent in slab which is not enough in this case. I just wanted to first
know if it was worth the effort writing a monitor that gave a per-slab
breakdown. If it can be both identified as shrinker-related and narrowed
down to a specific shrinker then there is more to work with. mmtests can run
in a monitor-only mode so it *should* be possible to turn on this monitor,
wait for the problem to reproduce and focus on the end of the logs.

Unfortunately, none of this explains why the machine completely froze.
If it really was a shrinker problem then I expected the system to be
extremely sluggish but did not predict that it would be so unresponsive
that ssh was not an option. That has left me scratching my head.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Kswapd 100% CPU since 3.8 on Sandybridge
  2014-10-06  9:37     ` Mel Gorman
@ 2014-10-06 13:32       ` Daniel Vetter
  2014-10-26 16:05       ` sarah
  1 sibling, 0 replies; 6+ messages in thread
From: Daniel Vetter @ 2014-10-06 13:32 UTC (permalink / raw)
  To: Mel Gorman; +Cc: Sarah A Sharp, Linux MM, intel-gfx

On Mon, Oct 6, 2014 at 11:37 AM, Mel Gorman <mgorman@suse.de> wrote:
> Minimally I wanted you to sample the stack traces for kswapd, narrow down
> to the time of its failure and see if it was stuck in a shrinker loop. What
> I suspected at the time was that it was hammering on the i915 shrinker and
> possibly doing repeated shrinks of the GPU objects in there. At one point
> at least, that was an extremely heavy operation if the objections were
> not freeable and I wanted to see if that was still the case. I confess I
> haven't looked at the code to see what has changed recently.

We've stopped doing that with

commit 2cfcd32a929b21c3cf77256dd8b858c076803ccc
Author: Chris Wilson <chris@chris-wilson.co.uk>
Date:   Tue May 20 08:28:43 2014 +0100

    drm/i915: Implement an oom-notifier for last resort shrinking


so now we do the handbreak last ditch shrinking really only when the
mm decided that it's time to oom. Until that point we should just do
the proportional shrinking the vm asked us to, but not more.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Kswapd 100% CPU since 3.8 on Sandybridge
  2014-10-06  9:37     ` Mel Gorman
  2014-10-06 13:32       ` Daniel Vetter
@ 2014-10-26 16:05       ` sarah
  2014-10-26 18:20         ` Johannes Weiner
  1 sibling, 1 reply; 6+ messages in thread
From: sarah @ 2014-10-26 16:05 UTC (permalink / raw)
  To: Mel Gorman; +Cc: linux-mm, intel-gfx

On Mon, Oct 06, 2014 at 10:37:40AM +0100, Mel Gorman wrote:
> On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote:
> > Please excuse the non-wrapped email. My personal system is currently
> > b0rked, so I'm sending this in frustration from my phone.
> > 
> > My laptop is currently completely hosed. Disk light on full solid
> > Mouse movement sluggish to the point of moving a couple cms per second.
> > Firefox window greyed out but not OOM killed yet. When this behavior
> > occurred in the past, if I ran top, I would see kswapd taking up 100% of
> > one of my two CPUs.
> > 
> > If I can catch the system in time before mouse movement becomes too
> > sluggish, closing the browser window will cause kswapd usage to drop, and
> > the system goes back to a normal state. If I don't catch it in time, I
> > can't even ssh into the box to kill Firefox because the login times out.
> > Occasionally Firefox gets OOM killed, but most of the time I have to use
> > sysreq keys to reboot the system.
> > 
> > This can be reproduced by using either Chrome or Firefox. Chrome fails
> > faster. I'm not sure whether it's related to loading tabs with a bunch of
> > images, maybe flash, but it takes around 10-15 tabs being open before it
> > starts to fail. I can try to characterize it further.
> > 
> > System: Lenovo x220 Intel Sandy Bridge graphics
> > Ubuntu 14.04 with edgers PPA for Mesa
> > 3.16.3 kernel
> > 
> > Since around the 3.8 kernel time frame, I've been able to reproduce this
> > behavior. I'm pretty sure it was a kernel change.
> > 
> > I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a
> > particular mm test. I still don't have time to triage this, but I'm now
> > frustrated enough to make time.
> > 
> > Mel, what test do you want me to run?
> > 
> 
> Minimally I wanted you to sample the stack traces for kswapd, narrow down
> to the time of its failure and see if it was stuck in a shrinker loop. What
> I suspected at the time was that it was hammering on the i915 shrinker and
> possibly doing repeated shrinks of the GPU objects in there. At one point
> at least, that was an extremely heavy operation if the objections were
> not freeable and I wanted to see if that was still the case. I confess I
> haven't looked at the code to see what has changed recently.

When kswapd is at 100% CPU usage, perf shows top symbols as:

-   46.45%          kswapd0  [i915]                       [k] i915_gem_inactive_count                                                                           a??
   - i915_gem_inactive_count                                                                                                                                    a??
      - 99.97% shrink_slab_node                                                                                                                                 a??
           shrink_slab                                                                                                                                          a??
           balance_pgdat                                                                                                                                        a??
           kswapd                                                                                                                                               a??
           kthread                                                                                                                                              a??
           ret_from_fork                                                                                                                                        a??
-   15.83%          kswapd0  [kernel.kallsyms]            [k] _raw_spin_lock                                                                                    a??
   - _raw_spin_lock                                                                                                                                             a??
      + 47.36% list_lru_count_node                                                                                                                              a??
      + 25.31% grab_super_passive                                                                                                                               a??
      + 21.10% put_super                                                                                                                                        a??
      + 5.19% super_cache_count                                                                                                                                 a??
      + 0.96% mb_cache_shrink_count                                                                                                                             a??
-    5.22%          kswapd0  [kernel.kallsyms]            [k] list_lru_count_node                                                                               a??
   - list_lru_count_node                                                                                                                                        a??
      + 93.65% super_cache_count                                                                                                                                a??
      + 4.71% shrink_slab_node                                                                                                                                  a??
      + 1.64% count_shadow_nodes

The perf trace and kallsyms file is attached.

I'll try updating from 3.15 to 3.17 and see if commit 2cfcd32a929b "drm/i915:
Implement an oom-notifier for last resort shrinking" solves the issue.

> If that was confirmed then I to modify the mmtests Ftracereclaimcompact
> reported to focus exclusively on slab and give a breakdown of which shrinker
> it was spending time in. Right now, that reporter only says how much time
> is spent in slab which is not enough in this case. I just wanted to first
> know if it was worth the effort writing a monitor that gave a per-slab
> breakdown. If it can be both identified as shrinker-related and narrowed
> down to a specific shrinker then there is more to work with. mmtests can run
> in a monitor-only mode so it *should* be possible to turn on this monitor,
> wait for the problem to reproduce and focus on the end of the logs.
> 
> Unfortunately, none of this explains why the machine completely froze.
> If it really was a shrinker problem then I expected the system to be
> extremely sluggish but did not predict that it would be so unresponsive
> that ssh was not an option. That has left me scratching my head.

One hypothesis I had was the system was hanging because kswapd was hammering on
the disk, and it was really low on disk space (< 1 GB).  But I've moved about 20
GB of roadtrip photos to my USB 3.0 drive, and I can still replicate the slow
system behavior.  I was also poking around /etc/fstab, and I realized that I had
actually disabled my swap partition around the 3.12 kernel time frame to see if
that helped the issue.  That lead me to wonder why kswapd was running at all?

Sarah Sharp

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Kswapd 100% CPU since 3.8 on Sandybridge
  2014-10-26 16:05       ` sarah
@ 2014-10-26 18:20         ` Johannes Weiner
  0 siblings, 0 replies; 6+ messages in thread
From: Johannes Weiner @ 2014-10-26 18:20 UTC (permalink / raw)
  To: sarah; +Cc: Mel Gorman, linux-mm, intel-gfx

Hi Sarah,

On Sun, Oct 26, 2014 at 09:05:34AM -0700, sarah wrote:
> One hypothesis I had was the system was hanging because kswapd was hammering on
> the disk, and it was really low on disk space (< 1 GB).  But I've moved about 20
> GB of roadtrip photos to my USB 3.0 drive, and I can still replicate the slow
> system behavior.  I was also poking around /etc/fstab, and I realized that I had
> actually disabled my swap partition around the 3.12 kernel time frame to see if
> that helped the issue.  That lead me to wonder why kswapd was running at all?

Kswapd isn't just there to swap, it reclaims pages in the background
when memory is filling up in order to keep allocation latencies low.
That includes trimming the page cache, shrinking slabs, even writing
back dirty pages.  So "kswapd" is a bit of a misnomer, and it's
expected to run even when you don't have any swap space configured.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2014-10-26 18:20 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CABe+QzA=0YVpQ8rN+3X-cbH6JP1nWTvp2spb93P9PqJhmjBROA@mail.gmail.com>
     [not found] ` <CABe+QzA-E40bFFXYJBc663Kx0KrE3xy2uZq5xOH2XL6mFPA6+w@mail.gmail.com>
2014-10-04 17:05   ` Kswapd 100% CPU since 3.8 on Sandybridge Sarah A Sharp
2014-10-06  7:25     ` Daniel Vetter
2014-10-06  9:37     ` Mel Gorman
2014-10-06 13:32       ` Daniel Vetter
2014-10-26 16:05       ` sarah
2014-10-26 18:20         ` Johannes Weiner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox