From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
To: mhocko@kernel.org
Cc: rientjes@google.com, akpm@linux-foundation.org,
torvalds@linux-foundation.org, hannes@cmpxchg.org,
mgorman@suse.de, hillf.zj@alibaba-inc.com,
kamezawa.hiroyu@jp.fujitsu.com, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 0/3] OOM detection rework v4
Date: Sun, 7 Feb 2016 13:09:33 +0900 [thread overview]
Message-ID: <201602071309.EJD59750.FOVMSFOOFHtJQL@I-love.SAKURA.ne.jp> (raw)
In-Reply-To: <20160204133905.GB14425@dhcp22.suse.cz>
Michal Hocko wrote:
> On Thu 04-02-16 22:10:54, Tetsuo Handa wrote:
> > Michal Hocko wrote:
> > > I am not sure we can fix these pathological loads where we hit the
> > > higher order depletion and there is a chance that one of the thousands
> > > tasks terminates in an unpredictable way which happens to race with the
> > > OOM killer.
> >
> > When I hit this problem on Dec 24th, I didn't run thousands of tasks.
> > I think there were less than one hundred tasks in the system and only
> > a few tasks were running. Not a pathological load at all.
>
> But as the OOM report clearly stated there were no > order-1 pages
> available in that particular case. And that happened after the direct
> reclaim and compaction were already invoked.
>
> As I've mentioned in the referenced email, we can try to do multiple
> retries e.g. do not give up on the higher order requests until we hit
> the maximum number of retries but I consider it quite ugly to be honest.
> I think that a proper communication with compaction is a more
> appropriate way to go long term. E.g. I find it interesting that
> try_to_compact_pages doesn't even care about PAGE_ALLOC_COSTLY_ORDER
> and treat is as any other high order request.
>
FYI, I again hit unexpected OOM-killer during genxref on linux-4.5-rc2 source.
I think current patchset is too fragile to merge.
----------------------------------------
[ 3101.626995] smbd invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0
[ 3101.629148] smbd cpuset=/ mems_allowed=0
[ 3101.630332] CPU: 1 PID: 3941 Comm: smbd Not tainted 4.5.0-rc2-next-20160205 #293
[ 3101.632335] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[ 3101.634567] 0000000000000286 000000005784a8f9 ffff88007c47bad0 ffffffff8139abbd
[ 3101.636533] 0000000000000000 ffff88007c47bd00 ffff88007c47bb70 ffffffff811bdc6c
[ 3101.638381] 0000000000000206 ffffffff81810b30 ffff88007c47bb10 ffffffff810be079
[ 3101.640215] Call Trace:
[ 3101.641169] [<ffffffff8139abbd>] dump_stack+0x85/0xc8
[ 3101.642560] [<ffffffff811bdc6c>] dump_header+0x5b/0x3b0
[ 3101.643983] [<ffffffff810be079>] ? trace_hardirqs_on_caller+0xf9/0x1c0
[ 3101.645616] [<ffffffff810be14d>] ? trace_hardirqs_on+0xd/0x10
[ 3101.647081] [<ffffffff81143fb6>] oom_kill_process+0x366/0x550
[ 3101.648631] [<ffffffff811443df>] out_of_memory+0x1ef/0x5a0
[ 3101.650081] [<ffffffff8114449d>] ? out_of_memory+0x2ad/0x5a0
[ 3101.651624] [<ffffffff81149d0d>] __alloc_pages_nodemask+0xbad/0xd90
[ 3101.653207] [<ffffffff8114a0ac>] alloc_kmem_pages_node+0x4c/0xc0
[ 3101.654767] [<ffffffff8106d5c1>] copy_process.part.31+0x131/0x1b40
[ 3101.656381] [<ffffffff8111d9ea>] ? __audit_syscall_entry+0xaa/0xf0
[ 3101.657952] [<ffffffff810e8119>] ? current_kernel_time64+0xa9/0xc0
[ 3101.659492] [<ffffffff8106f19b>] _do_fork+0xdb/0x5d0
[ 3101.660814] [<ffffffff810030c1>] ? do_audit_syscall_entry+0x61/0x70
[ 3101.662305] [<ffffffff81003254>] ? syscall_trace_enter_phase1+0x134/0x150
[ 3101.663988] [<ffffffff81703d2c>] ? return_from_SYSCALL_64+0x2d/0x7a
[ 3101.665572] [<ffffffff810035ec>] ? do_syscall_64+0x1c/0x180
[ 3101.667067] [<ffffffff8106f714>] SyS_clone+0x14/0x20
[ 3101.668510] [<ffffffff8100362d>] do_syscall_64+0x5d/0x180
[ 3101.669931] [<ffffffff81703cff>] entry_SYSCALL64_slow_path+0x25/0x25
[ 3101.671642] Mem-Info:
[ 3101.672612] active_anon:46842 inactive_anon:2094 isolated_anon:0
active_file:108974 inactive_file:131350 isolated_file:0
unevictable:0 dirty:1174 writeback:0 unstable:0
slab_reclaimable:107536 slab_unreclaimable:14287
mapped:4199 shmem:2166 pagetables:1524 bounce:0
free:6260 free_pcp:31 free_cma:0
[ 3101.681294] Node 0 DMA free:6884kB min:44kB low:52kB high:64kB active_anon:3488kB inactive_anon:100kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:100kB slab_reclaimable:3852kB slab_unreclaimable:444kB kernel_stack:80kB pagetables:112kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 3101.691319] lowmem_reserve[]: 0 1714 1714 1714
[ 3101.692847] Node 0 DMA32 free:18156kB min:5172kB low:6464kB high:7756kB active_anon:183880kB inactive_anon:8276kB active_file:435896kB inactive_file:525396kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080640kB managed:1759480kB mlocked:0kB dirty:4696kB writeback:0kB mapped:16792kB shmem:8564kB slab_reclaimable:426292kB slab_unreclaimable:56704kB kernel_stack:3328kB pagetables:5984kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 3101.704239] lowmem_reserve[]: 0 0 0 0
[ 3101.705887] Node 0 DMA: 75*4kB (UME) 69*8kB (UME) 43*16kB (UM) 23*32kB (UME) 8*64kB (UM) 4*128kB (UME) 2*256kB (UM) 0*512kB 1*1024kB (U) 1*2048kB (M) 0*4096kB = 6884kB
[ 3101.710581] Node 0 DMA32: 4513*4kB (UME) 15*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18172kB
[ 3101.713857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 3101.716332] 242517 total pagecache pages
[ 3101.717878] 0 pages in swap cache
[ 3101.719332] Swap cache stats: add 0, delete 0, find 0/0
[ 3101.721577] Free swap = 0kB
[ 3101.722980] Total swap = 0kB
[ 3101.724364] 524157 pages RAM
[ 3101.725697] 0 pages HighMem/MovableOnly
[ 3101.727165] 80311 pages reserved
[ 3101.728482] 0 pages hwpoisoned
[ 3101.729754] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 3101.732071] [ 492] 0 492 9206 975 20 4 0 0 systemd-journal
[ 3101.734357] [ 520] 0 520 10479 631 22 3 0 -1000 systemd-udevd
[ 3101.737036] [ 527] 0 527 12805 682 24 3 0 -1000 auditd
[ 3101.739505] [ 1174] 0 1174 4830 556 14 3 0 0 irqbalance
[ 3101.741876] [ 1180] 81 1180 6672 604 20 3 0 -900 dbus-daemon
[ 3101.744728] [ 1817] 0 1817 56009 880 40 4 0 0 rsyslogd
[ 3101.747164] [ 1818] 0 1818 1096 349 8 3 0 0 rngd
[ 3101.749788] [ 1820] 0 1820 52575 1074 56 3 0 0 abrtd
[ 3101.752135] [ 1821] 0 1821 80901 5160 80 4 0 0 firewalld
[ 3101.754532] [ 1823] 0 1823 6602 681 20 3 0 0 systemd-logind
[ 3101.757342] [ 1825] 70 1825 6999 458 20 3 0 0 avahi-daemon
[ 3101.759784] [ 1827] 0 1827 51995 986 55 3 0 0 abrt-watch-log
[ 3101.762465] [ 1838] 0 1838 31586 647 21 3 0 0 crond
[ 3101.764797] [ 1946] 70 1946 6999 58 19 3 0 0 avahi-daemon
[ 3101.767262] [ 2043] 0 2043 65187 858 43 3 0 0 vmtoolsd
[ 3101.769665] [ 2618] 0 2618 27631 3112 53 3 0 0 dhclient
[ 3101.772203] [ 2622] 999 2622 130827 2570 56 3 0 0 polkitd
[ 3101.774645] [ 2704] 0 2704 138263 3351 91 4 0 0 tuned
[ 3101.777114] [ 2709] 0 2709 20640 773 45 3 0 -1000 sshd
[ 3101.779428] [ 2711] 0 2711 7328 551 19 3 0 0 xinetd
[ 3101.782016] [ 3883] 0 3883 22785 827 45 3 0 0 master
[ 3101.784576] [ 3884] 89 3884 22811 924 46 4 0 0 pickup
[ 3101.786898] [ 3885] 89 3885 22828 886 44 3 0 0 qmgr
[ 3101.789287] [ 3916] 0 3916 23203 736 50 3 0 0 login
[ 3101.791666] [ 3927] 0 3927 27511 381 13 3 0 0 agetty
[ 3101.794116] [ 3930] 0 3930 79392 1063 105 3 0 0 nmbd
[ 3101.796387] [ 3941] 0 3941 96485 1544 138 4 0 0 smbd
[ 3101.798602] [ 3944] 0 3944 96485 1290 131 4 0 0 smbd
[ 3101.800783] [ 7471] 0 7471 28886 732 15 3 0 0 bash
[ 3101.803013] [ 7580] 0 7580 2380 613 10 3 0 0 makelxr.sh
[ 3101.805147] [ 7786] 0 7786 27511 395 10 3 0 0 agetty
[ 3101.807198] [ 8139] 0 8139 35888 974 72 3 0 0 sshd
[ 3101.809255] [ 8144] 0 8144 28896 761 15 4 0 0 bash
[ 3101.811335] [15286] 0 15286 38294 30474 81 3 0 0 genxref
[ 3101.813512] Out of memory: Kill process 15286 (genxref) score 66 or sacrifice child
[ 3101.815659] Killed process 15286 (genxref) total-vm:153176kB, anon-rss:117092kB, file-rss:4804kB, shmem-rss:0kB
----------------------------------------
> Something like the following:
Yes, I do think we need something like it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2016-02-07 4:09 UTC|newest]
Thread overview: 152+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-15 18:19 Michal Hocko
2015-12-15 18:19 ` [PATCH 1/3] mm, oom: rework oom detection Michal Hocko
2016-01-14 22:58 ` David Rientjes
2016-01-16 1:07 ` Tetsuo Handa
2016-01-19 22:48 ` David Rientjes
2016-01-20 11:13 ` Tetsuo Handa
2016-01-20 13:13 ` Michal Hocko
2016-04-04 8:23 ` Vladimir Davydov
2016-04-04 9:42 ` Michal Hocko
2015-12-15 18:19 ` [PATCH 2/3] mm: throttle on IO only when there are too many dirty and writeback pages Michal Hocko
2016-03-17 11:35 ` Tetsuo Handa
2016-03-17 12:01 ` Michal Hocko
2015-12-15 18:19 ` [PATCH 3/3] mm: use watermak checks for __GFP_REPEAT high order allocations Michal Hocko
2015-12-16 23:35 ` [PATCH 0/3] OOM detection rework v4 Andrew Morton
2015-12-18 12:12 ` Michal Hocko
2015-12-16 23:58 ` Andrew Morton
2015-12-18 13:15 ` Michal Hocko
2015-12-18 16:35 ` Johannes Weiner
2015-12-24 12:41 ` Tetsuo Handa
2015-12-28 12:08 ` Tetsuo Handa
2015-12-28 14:13 ` Tetsuo Handa
2016-01-06 12:44 ` Vlastimil Babka
2016-01-08 12:37 ` Michal Hocko
2015-12-29 16:32 ` Michal Hocko
2015-12-30 15:05 ` Tetsuo Handa
2016-01-02 15:47 ` Tetsuo Handa
2016-01-20 12:24 ` Michal Hocko
2016-01-27 23:18 ` David Rientjes
2016-01-28 21:19 ` Michal Hocko
2015-12-29 16:27 ` Michal Hocko
2016-01-28 20:40 ` [PATCH 4/3] mm, oom: drop the last allocation attempt before out_of_memory Michal Hocko
2016-01-28 21:36 ` Johannes Weiner
2016-01-28 23:19 ` David Rientjes
2016-01-28 23:51 ` Johannes Weiner
2016-01-29 10:39 ` Tetsuo Handa
2016-01-29 15:32 ` Michal Hocko
2016-01-30 12:18 ` Tetsuo Handa
2016-01-29 15:23 ` Michal Hocko
2016-01-29 15:24 ` Michal Hocko
2016-01-28 21:19 ` [PATCH 5/3] mm, vmscan: make zone_reclaimable_pages more precise Michal Hocko
2016-01-28 23:20 ` David Rientjes
2016-01-29 3:41 ` Hillf Danton
2016-01-29 10:35 ` Tetsuo Handa
2016-01-29 15:17 ` Michal Hocko
2016-01-29 21:30 ` Tetsuo Handa
2016-02-03 13:27 ` [PATCH 0/3] OOM detection rework v4 Michal Hocko
2016-02-03 22:58 ` David Rientjes
2016-02-04 12:57 ` Michal Hocko
2016-02-04 13:10 ` Tetsuo Handa
2016-02-04 13:39 ` Michal Hocko
2016-02-04 14:24 ` Michal Hocko
2016-02-07 4:09 ` Tetsuo Handa [this message]
2016-02-15 20:06 ` Michal Hocko
2016-02-16 13:10 ` Tetsuo Handa
2016-02-16 15:19 ` Michal Hocko
2016-02-25 3:47 ` Hugh Dickins
2016-02-25 6:48 ` Sergey Senozhatsky
2016-02-25 9:17 ` Hillf Danton
2016-02-25 9:27 ` Michal Hocko
2016-02-25 9:48 ` Hillf Danton
2016-02-25 11:02 ` Sergey Senozhatsky
2016-02-25 9:23 ` Michal Hocko
2016-02-26 6:32 ` Hugh Dickins
2016-02-26 7:54 ` Hillf Danton
2016-02-26 9:24 ` Michal Hocko
2016-02-26 10:27 ` Hillf Danton
2016-02-26 13:49 ` Michal Hocko
2016-02-26 9:33 ` Michal Hocko
2016-02-29 21:02 ` Michal Hocko
2016-03-02 2:19 ` Joonsoo Kim
2016-03-02 9:50 ` Michal Hocko
2016-03-02 13:32 ` Joonsoo Kim
2016-03-02 14:06 ` Michal Hocko
2016-03-02 14:34 ` Joonsoo Kim
2016-03-03 9:26 ` Michal Hocko
2016-03-03 10:29 ` Tetsuo Handa
2016-03-03 14:10 ` Joonsoo Kim
2016-03-03 15:25 ` Michal Hocko
2016-03-04 5:23 ` Joonsoo Kim
2016-03-04 15:15 ` Michal Hocko
2016-03-04 17:39 ` Michal Hocko
2016-03-07 5:23 ` Joonsoo Kim
2016-03-03 15:50 ` Vlastimil Babka
2016-03-03 16:26 ` Michal Hocko
2016-03-04 7:10 ` Joonsoo Kim
2016-03-02 15:01 ` Minchan Kim
2016-03-07 16:08 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Michal Hocko
2016-03-08 3:51 ` Sergey Senozhatsky
2016-03-08 9:08 ` Michal Hocko
2016-03-08 9:24 ` Sergey Senozhatsky
2016-03-08 9:24 ` [PATCH] mm, oom: protect !costly allocations some more Vlastimil Babka
2016-03-08 9:32 ` Sergey Senozhatsky
2016-03-08 9:46 ` Michal Hocko
2016-03-08 9:52 ` Vlastimil Babka
2016-03-08 10:10 ` Michal Hocko
2016-03-08 11:12 ` Vlastimil Babka
2016-03-08 12:22 ` Michal Hocko
2016-03-08 12:29 ` Vlastimil Babka
2016-03-08 9:58 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Sergey Senozhatsky
2016-03-08 13:57 ` Michal Hocko
2016-03-08 10:36 ` Hugh Dickins
2016-03-08 13:42 ` [PATCH 0/2] oom rework: high order enahncements Michal Hocko
2016-03-08 13:42 ` [PATCH 1/3] mm, compaction: change COMPACT_ constants into enum Michal Hocko
2016-03-08 14:19 ` Vlastimil Babka
2016-03-09 3:55 ` Hillf Danton
2016-03-08 13:42 ` [PATCH 2/3] mm, compaction: cover all compaction mode in compact_zone Michal Hocko
2016-03-08 14:22 ` Vlastimil Babka
2016-03-09 3:57 ` Hillf Danton
2016-03-08 13:42 ` [PATCH 3/3] mm, oom: protect !costly allocations some more Michal Hocko
2016-03-08 14:34 ` Vlastimil Babka
2016-03-08 14:48 ` Michal Hocko
2016-03-08 15:03 ` Vlastimil Babka
2016-03-09 11:11 ` Michal Hocko
2016-03-09 14:07 ` Vlastimil Babka
2016-03-11 12:17 ` Hugh Dickins
2016-03-11 13:06 ` Michal Hocko
2016-03-11 19:08 ` Hugh Dickins
2016-03-14 16:21 ` Michal Hocko
2016-03-08 15:19 ` [PATCH] mm, oom: protect !costly allocations some more (was: Re: [PATCH 0/3] OOM detection rework v4) Joonsoo Kim
2016-03-08 16:05 ` Michal Hocko
2016-03-08 17:03 ` Joonsoo Kim
2016-03-09 10:41 ` Michal Hocko
2016-03-11 14:53 ` Joonsoo Kim
2016-03-11 15:20 ` Michal Hocko
2016-02-29 20:35 ` [PATCH 0/3] OOM detection rework v4 Michal Hocko
2016-03-01 7:29 ` Hugh Dickins
2016-03-01 13:38 ` Michal Hocko
2016-03-01 14:40 ` Michal Hocko
2016-03-01 18:14 ` Vlastimil Babka
2016-03-02 2:55 ` Joonsoo Kim
2016-03-02 12:37 ` Michal Hocko
2016-03-02 14:06 ` Joonsoo Kim
2016-03-02 12:24 ` Michal Hocko
2016-03-02 13:00 ` Michal Hocko
2016-03-02 13:22 ` Vlastimil Babka
2016-03-02 2:28 ` Joonsoo Kim
2016-03-02 12:39 ` Michal Hocko
2016-03-03 9:54 ` Hugh Dickins
2016-03-03 12:32 ` Michal Hocko
2016-03-03 20:57 ` Hugh Dickins
2016-03-04 7:41 ` Vlastimil Babka
2016-03-04 7:53 ` Joonsoo Kim
2016-03-04 12:28 ` Michal Hocko
2016-03-11 10:45 ` Tetsuo Handa
2016-03-11 13:08 ` Michal Hocko
2016-03-11 13:32 ` Tetsuo Handa
2016-03-11 15:28 ` Michal Hocko
2016-03-11 16:49 ` Tetsuo Handa
2016-03-11 17:00 ` Michal Hocko
2016-03-11 17:20 ` Tetsuo Handa
2016-03-12 4:08 ` Tetsuo Handa
2016-03-13 14:41 ` Tetsuo Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201602071309.EJD59750.FOVMSFOOFHtJQL@I-love.SAKURA.ne.jp \
--to=penguin-kernel@i-love.sakura.ne.jp \
--cc=akpm@linux-foundation.org \
--cc=hannes@cmpxchg.org \
--cc=hillf.zj@alibaba-inc.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=rientjes@google.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox