Re: Softlockup during memory allocation

[Michal Hocko <mhocko@suse.cz>]

On Tue 22-11-16 15:30:56, Michal Hocko wrote:
> On Tue 22-11-16 10:56:51, Nikolay Borisov wrote:
> > 
> > 
> > On 11/21/2016 07:31 AM, Michal Hocko wrote:
> > > Hi,
> > > I am sorry for a late response, but I was offline until this weekend. I
> > > will try to get to this email ASAP but it might take some time.
> > 
> > No worries. I did some further digging up and here is what I got, which
> > I believe is rather strange:
> > 
> > struct scan_control {
> >   nr_to_reclaim = 32,
> >   gfp_mask = 37880010,
> >   order = 0,
> >   nodemask = 0x0,
> >   target_mem_cgroup = 0xffff8823990d1400,
> >   priority = 7,
> >   may_writepage = 1,
> >   may_unmap = 1,
> >   may_swap = 0,
> >   may_thrash = 1,
> >   hibernation_mode = 0,
> >   compaction_ready = 0,
> >   nr_scanned = 0,
> >   nr_reclaimed = 0
> > }
> > 
> > Parsing: 37880010
> > #define ___GFP_HIGHMEM		0x02
> > #define ___GFP_MOVABLE		0x08
> > #define ___GFP_IO		0x40
> > #define ___GFP_FS		0x80
> > #define ___GFP_HARDWALL		0x20000
> > #define ___GFP_DIRECT_RECLAIM	0x400000
> > #define ___GFP_KSWAPD_RECLAIM	0x2000000
> > 
> > And initial_priority is 12 (DEF_PRIORITY). Given that nr_scanned is 0
> > and priority is 7 this means we've gone 5 times through the do {} while
> > in do_try_to_free_pages. Also total_scanned seems to be 0.  Here is the
> > zone which was being reclaimed :
> > 
> > http://sprunge.us/hQBi
> 
> LRUs on that zones seem to be empty from a quick glance. kmem -z in the
> crash can give you per zone counters much more nicely.
> 
> > So what's strange is that the softlockup occurred but then the code
> > proceeded (as evident from the subsequent stack traces), yet inspecting
> > the reclaim progress it seems rather sad (no progress at all)
> 
> Unless I have misread the data above it seems something has either
> isolated all LRU pages for some time or there simply are none while the
> reclaim is desperately trying to make some progress. In any case this
> sounds less than a happy system...

Btw. how do you configure memcgs that the FS workload runs in?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>