Re: ps lockups, cgroup memory reclaim

[Mark Hills <mark@xwax.org>]

On Tue, 17 Sep 2013, Johannes Weiner wrote:

> On Tue, Sep 17, 2013 at 04:50:42PM +0100, Mark Hills wrote:
> > I'm investigating intermitten kernel lockups in an HPC environment, with 
> > the RedHat kernel.
> > 
> > The symptoms are seen as lockups of multiple ps commands, with one 
> > consuming full CPU:
> > 
> >   # ps aux | grep ps
> >   root     19557 68.9  0.0 108100   908 ?        D    Sep16 1045:37 ps --ppid 1 -o args=
> >   root     19871  0.0  0.0 108100   908 ?        D    Sep16   0:00 ps --ppid 1 -o args=
> > 
> > SIGKILL on the busy one causes the other ps processes to run to completion 
> > (TERM has no effect).
> 
> Any chance you can get to the stack of the non-busy blocked tasks?
> 
> It would be /proc/19871/stack in this case.

I had to return the machine above to the cluster, but next time I'll log 
this information.
 
> > In this case I was able to run my own ps to see the process list, but not 
> > always.
> > 
> > perf shows the locality of the spinning, roughly:
> > 
> >   proc_pid_cmdline
> >   get_user_pages
> >   handle_mm_fault
> >   mem_cgroup_try_charge_swapin
> >   mem_cgroup_reclaim
> > 
> > There are two entry points, the codepaths taken are better shown by the 
> > attached profile of CPU time.
> 
> Looks like it's spinning like crazy in shrink_mem_cgroup_zone().
> Maybe an LRU counter underflow, maybe endlessly looping on the
> should_continue_reclaim() compaction condition.  But I don't see an
> obvious connection to why killing the busy task wakes up the blocked
> one.

Maybe it's as simple as the lock taken at quite a high level; perhaps even 
a lock when reading values for /proc.

But no need for me to guess, we'll find out next time from the /proc 
information.
 
> So yeah, it would be helpful to know what that task is waiting for.
> 
> > We've had this behaviour since switching to Scientific Linux 6 (based on 
> > RHEL6, like CentOS) at kernel 2.6.32-279.9.1.el6.x86_64.
> > 
> > The example above is kernel 2.6.32-358.el6.x86_64.
> 
> Can you test with the debug build?  That should trap LRU counter
> underflows at least.

Ah, excellent -- I did not realise there was a kernel-debug package.

I'll see if I can isolate this with more detail from that kernel 
(including the stack trace of each task that is hung)

> If you have the possibility to recompile the distribution kernel I can 
> provide you with debug patches.

Absolutely, I can deploy a patched kernels but not cluster-wide. Let's 
hope I can get enough coverage to catch the bug.

Hopefully I'll have more information soon.

Thanks

-- 
Mark

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>