From: Michal Hocko <mhocko@kernel.org>
To: David Rientjes <rientjes@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Mel Gorman <mgorman@suse.de>,
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
Hillf Danton <hillf.zj@alibaba-inc.com>,
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/3] OOM detection rework v4
Date: Thu, 4 Feb 2016 13:57:00 +0100 [thread overview]
Message-ID: <20160204125700.GA14425@dhcp22.suse.cz> (raw)
In-Reply-To: <alpine.DEB.2.10.1602031457120.10331@chino.kir.corp.google.com>
On Wed 03-02-16 14:58:06, David Rientjes wrote:
> On Wed, 3 Feb 2016, Michal Hocko wrote:
>
> > Hi,
> > this thread went mostly quite. Are all the main concerns clarified?
> > Are there any new concerns? Are there any objections to targeting
> > this for the next merge window?
>
> Did we ever figure out what was causing the oom killer to be called much
> earlier in Tetsuo's http://marc.info/?l=linux-kernel&m=145096089726481 and
>From the OOM report:
[ 3902.430630] kthreadd invoked oom-killer: order=2, oom_score_adj=0, gfp_mask=0x27000c0(GFP_KERNEL|GFP_NOTRACK|0x100000)
[ 3902.507561] Node 0 DMA32: 3788*4kB (UME) 184*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16624kB
[ 5262.901161] smbd invoked oom-killer: order=2, oom_score_adj=0, gfp_mask=0x27000c0(GFP_KERNEL|GFP_NOTRACK|0x100000)
[ 5262.983496] Node 0 DMA32: 1987*4kB (UME) 14*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8060kB
[ 5269.764580] kthreadd invoked oom-killer: order=2, oom_score_adj=0, gfp_mask=0x27000c0(GFP_KERNEL|GFP_NOTRACK|0x100000)
[ 5269.858330] Node 0 DMA32: 10648*4kB (UME) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42592kB
> http://marc.info/?l=linux-kernel&m=145130454913757 ?
[ 277.884512] Node 0 DMA32: 3438*4kB (UME) 791*8kB (UME) 3*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20128kB
[ 291.349097] Node 0 DMA32: 4221*4kB (UME) 1971*8kB (UME) 436*16kB (UME) 141*32kB (UME) 8*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44652kB
[ 302.916334] Node 0 DMA32: 4304*4kB (UM) 1181*8kB (UME) 59*16kB (UME) 7*32kB (ME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 27832kB
[ 311.034251] Node 0 DMA32: 6*4kB (U) 2401*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19232kB
[ 314.314336] Node 0 DMA32: 1180*4kB (UM) 1449*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16312kB
[ 322.796256] Node 0 DMA32: 86*4kB (UME) 2474*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20136kB
[ 330.826190] Node 0 DMA32: 1637*4kB (UM) 1354*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17380kB
[ 332.846805] Node 0 DMA32: 4108*4kB (UME) 897*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 23608kB
[ 341.073722] Node 0 DMA32: 3309*4kB (UM) 1124*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 22228kB
[ 360.093794] Node 0 DMA32: 2719*4kB (UM) 97*8kB (UM) 14*16kB (UM) 37*32kB (UME) 27*64kB (UME) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15172kB
[ 368.871173] Node 0 DMA32: 5042*4kB (UM) 248*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 22152kB
[ 379.279344] Node 0 DMA32: 2994*4kB (ME) 503*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16000kB
[ 387.385740] Node 0 DMA32: 3638*4kB (UM) 115*8kB (UM) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15488kB
[ 391.228084] Node 0 DMA32: 3374*4kB (UME) 221*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15264kB
[ 395.683137] Node 0 DMA32: 3794*4kB (ME) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15176kB
[ 399.890082] Node 0 DMA32: 4155*4kB (UME) 200*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18220kB
[ 408.465169] Node 0 DMA32: 2804*4kB (ME) 203*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12840kB
[ 416.447247] Node 0 DMA32: 5158*4kB (UME) 68*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 21176kB
[ 418.799643] Node 0 DMA32: 3093*4kB (UME) 1043*8kB (UME) 2*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20748kB
[ 428.109005] Node 0 DMA32: 2943*4kB (UME) 458*8kB (UME) 20*16kB (UME) 11*32kB (UME) 11*64kB (ME) 4*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17324kB
[ 439.032446] Node 0 DMA32: 2761*4kB (UM) 28*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 11268kB
[ 441.731018] Node 0 DMA32: 3130*4kB (UM) 338*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15224kB
[ 442.070867] Node 0 DMA32: 590*4kB (ME) 827*8kB (ME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8976kB
[ 442.245208] Node 0 DMA32: 1902*4kB (UME) 410*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 10888kB
There are cases where order-2 has some pages but I have commented on
that here [1]
> I'd like to take a look at the patch(es) that fixed it.
I am not sure we can fix these pathological loads where we hit the
higher order depletion and there is a chance that one of the thousands
tasks terminates in an unpredictable way which happens to race with the
OOM killer. As I've pointed out in [1] once the watermark check for the
higher order allocation fails for the given order then we cannot rely
on the reclaimable pages ever construct the required order. The current
zone_reclaimable approach just happens to work for this particular load
because the NR_PAGES_SCANNED gets reseted too often with a side effect
of an undeterministic behavior.
[1] http://lkml.kernel.org/r/20160120131355.GE14187@dhcp22.suse.cz
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-02-04 12:57 UTC|newest]
Thread overview: 152+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-15 18:19 Michal Hocko
2015-12-15 18:19 ` [PATCH 1/3] mm, oom: rework oom detection Michal Hocko
2016-01-14 22:58 ` David Rientjes
2016-01-16 1:07 ` Tetsuo Handa
2016-01-19 22:48 ` David Rientjes
2016-01-20 11:13 ` Tetsuo Handa
2016-01-20 13:13 ` Michal Hocko
2016-04-04 8:23 ` Vladimir Davydov
2016-04-04 9:42 ` Michal Hocko
2015-12-15 18:19 ` [PATCH 2/3] mm: throttle on IO only when there are too many dirty and writeback pages Michal Hocko
2016-03-17 11:35 ` Tetsuo Handa
2016-03-17 12:01 ` Michal Hocko
2015-12-15 18:19 ` [PATCH 3/3] mm: use watermak checks for __GFP_REPEAT high order allocations Michal Hocko
2015-12-16 23:35 ` [PATCH 0/3] OOM detection rework v4 Andrew Morton
2015-12-18 12:12 ` Michal Hocko
2015-12-16 23:58 ` Andrew Morton
2015-12-18 13:15 ` Michal Hocko
2015-12-18 16:35 ` Johannes Weiner
2015-12-24 12:41 ` Tetsuo Handa
2015-12-28 12:08 ` Tetsuo Handa
2015-12-28 14:13 ` Tetsuo Handa
2016-01-06 12:44 ` Vlastimil Babka
2016-01-08 12:37 ` Michal Hocko
2015-12-29 16:32 ` Michal Hocko
2015-12-30 15:05 ` Tetsuo Handa
2016-01-02 15:47 ` Tetsuo Handa
2016-01-20 12:24 ` Michal Hocko
2016-01-27 23:18 ` David Rientjes
2016-01-28 21:19 ` Michal Hocko
2015-12-29 16:27 ` Michal Hocko
2016-01-28 20:40 ` [PATCH 4/3] mm, oom: drop the last allocation attempt before out_of_memory Michal Hocko
2016-01-28 21:36 ` Johannes Weiner
2016-01-28 23:19 ` David Rientjes
2016-01-28 23:51 ` Johannes Weiner
2016-01-29 10:39 ` Tetsuo Handa
2016-01-29 15:32 ` Michal Hocko
2016-01-30 12:18 ` Tetsuo Handa
2016-01-29 15:23 ` Michal Hocko
2016-01-29 15:24 ` Michal Hocko
2016-01-28 21:19 ` [PATCH 5/3] mm, vmscan: make zone_reclaimable_pages more precise Michal Hocko
2016-01-28 23:20 ` David Rientjes
2016-01-29 3:41 ` Hillf Danton
2016-01-29 10:35 ` Tetsuo Handa
2016-01-29 15:17 ` Michal Hocko
2016-01-29 21:30 ` Tetsuo Handa
2016-02-03 13:27 ` [PATCH 0/3] OOM detection rework v4 Michal Hocko
2016-02-03 22:58 ` David Rientjes
2016-02-04 12:57 ` Michal Hocko [this message]
2016-02-04 13:10 ` Tetsuo Handa
2016-02-04 13:39 ` Michal Hocko
2016-02-04 14:24 ` Michal Hocko
2016-02-07 4:09 ` Tetsuo Handa
2016-02-15 20:06 ` Michal Hocko
2016-02-16 13:10 ` Tetsuo Handa
2016-02-16 15:19 ` Michal Hocko
2016-02-25 3:47 ` Hugh Dickins
2016-02-25 6:48 ` Sergey Senozhatsky
2016-02-25 9:17 ` Hillf Danton
2016-02-25 9:27 ` Michal Hocko
2016-02-25 9:48 ` Hillf Danton
2016-02-25 11:02 ` Sergey Senozhatsky
2016-02-25 9:23 ` Michal Hocko
2016-02-26 6:32 ` Hugh Dickins
2016-02-26 7:54 ` Hillf Danton
2016-02-26 9:24 ` Michal Hocko
2016-02-26 10:27 ` Hillf Danton
2016-02-26 13:49 ` Michal Hocko
2016-02-26 9:33 ` Michal Hocko
2016-02-29 21:02 ` Michal Hocko
2016-03-02 2:19 ` Joonsoo Kim
2016-03-02 9:50 ` Michal Hocko
2016-03-02 13:32 ` Joonsoo Kim
2016-03-02 14:06 ` Michal Hocko
2016-03-02 14:34 ` Joonsoo Kim
2016-03-03 9:26 ` Michal Hocko
2016-03-03 10:29 ` Tetsuo Handa
2016-03-03 14:10 ` Joonsoo Kim
2016-03-03 15:25 ` Michal Hocko
2016-03-04 5:23 ` Joonsoo Kim
2016-03-04 15:15 ` Michal Hocko
2016-03-04 17:39 ` Michal Hocko
2016-03-07 5:23 ` Joonsoo Kim
2016-03-03 15:50 ` Vlastimil Babka
2016-03-03 16:26 ` Michal Hocko
2016-03-04 7:10 ` Joonsoo Kim
2016-03-02 15:01 ` Minchan Kim
2016-03-07 16:08 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Michal Hocko
2016-03-08 3:51 ` Sergey Senozhatsky
2016-03-08 9:08 ` Michal Hocko
2016-03-08 9:24 ` Sergey Senozhatsky
2016-03-08 9:24 ` [PATCH] mm, oom: protect !costly allocations some more Vlastimil Babka
2016-03-08 9:32 ` Sergey Senozhatsky
2016-03-08 9:46 ` Michal Hocko
2016-03-08 9:52 ` Vlastimil Babka
2016-03-08 10:10 ` Michal Hocko
2016-03-08 11:12 ` Vlastimil Babka
2016-03-08 12:22 ` Michal Hocko
2016-03-08 12:29 ` Vlastimil Babka
2016-03-08 9:58 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Sergey Senozhatsky
2016-03-08 13:57 ` Michal Hocko
2016-03-08 10:36 ` Hugh Dickins
2016-03-08 13:42 ` [PATCH 0/2] oom rework: high order enahncements Michal Hocko
2016-03-08 13:42 ` [PATCH 1/3] mm, compaction: change COMPACT_ constants into enum Michal Hocko
2016-03-08 14:19 ` Vlastimil Babka
2016-03-09 3:55 ` Hillf Danton
2016-03-08 13:42 ` [PATCH 2/3] mm, compaction: cover all compaction mode in compact_zone Michal Hocko
2016-03-08 14:22 ` Vlastimil Babka
2016-03-09 3:57 ` Hillf Danton
2016-03-08 13:42 ` [PATCH 3/3] mm, oom: protect !costly allocations some more Michal Hocko
2016-03-08 14:34 ` Vlastimil Babka
2016-03-08 14:48 ` Michal Hocko
2016-03-08 15:03 ` Vlastimil Babka
2016-03-09 11:11 ` Michal Hocko
2016-03-09 14:07 ` Vlastimil Babka
2016-03-11 12:17 ` Hugh Dickins
2016-03-11 13:06 ` Michal Hocko
2016-03-11 19:08 ` Hugh Dickins
2016-03-14 16:21 ` Michal Hocko
2016-03-08 15:19 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Joonsoo Kim
2016-03-08 16:05 ` Michal Hocko
2016-03-08 17:03 ` Joonsoo Kim
2016-03-09 10:41 ` Michal Hocko
2016-03-11 14:53 ` Joonsoo Kim
2016-03-11 15:20 ` Michal Hocko
2016-02-29 20:35 ` [PATCH 0/3] OOM detection rework v4 Michal Hocko
2016-03-01 7:29 ` Hugh Dickins
2016-03-01 13:38 ` Michal Hocko
2016-03-01 14:40 ` Michal Hocko
2016-03-01 18:14 ` Vlastimil Babka
2016-03-02 2:55 ` Joonsoo Kim
2016-03-02 12:37 ` Michal Hocko
2016-03-02 14:06 ` Joonsoo Kim
2016-03-02 12:24 ` Michal Hocko
2016-03-02 13:00 ` Michal Hocko
2016-03-02 13:22 ` Vlastimil Babka
2016-03-02 2:28 ` Joonsoo Kim
2016-03-02 12:39 ` Michal Hocko
2016-03-03 9:54 ` Hugh Dickins
2016-03-03 12:32 ` Michal Hocko
2016-03-03 20:57 ` Hugh Dickins
2016-03-04 7:41 ` Vlastimil Babka
2016-03-04 7:53 ` Joonsoo Kim
2016-03-04 12:28 ` Michal Hocko
2016-03-11 10:45 ` Tetsuo Handa
2016-03-11 13:08 ` Michal Hocko
2016-03-11 13:32 ` Tetsuo Handa
2016-03-11 15:28 ` Michal Hocko
2016-03-11 16:49 ` Tetsuo Handa
2016-03-11 17:00 ` Michal Hocko
2016-03-11 17:20 ` Tetsuo Handa
2016-03-12 4:08 ` Tetsuo Handa
2016-03-13 14:41 ` Tetsuo Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160204125700.GA14425@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=hannes@cmpxchg.org \
--cc=hillf.zj@alibaba-inc.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=rientjes@google.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox