* OOM killer doesn't kill the right task....
@ 2008-04-21 7:01 David Chinner
2008-04-21 8:24 ` KOSAKI Motohiro
0 siblings, 1 reply; 4+ messages in thread
From: David Chinner @ 2008-04-21 7:01 UTC (permalink / raw)
To: lkml; +Cc: xfs-oss, linux-mm
Running in a 512MB UML system without swap, XFSQA test 084 reliably
kills the kernel completely as the OOM killer is unable to find a
task to kill. log output is below.
I don't know when it started failing - ISTR this working just fine
on 2.6.24 kernels.
Test program is here:
http://oss.sgi.com/cgi-bin/cvsweb.cgi/xfs-cmds/xfstests/src/resvtest.c?rev=1.3
And it is invoked with two different command lines from the test
suite (not sure which one triggers the failure):
$ ./resvtest -i 20 -b $pagesize <file_on_xfs_filesystem>
and
$ ./resvtest -i 40 -b 512 <file_on_xfs_filesystem>
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
[ 1061.900000] resvtest invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=-17
[ 1061.900000] Call Trace:
[ 1061.900000] 6792bb58: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1061.900000] 6792bb68: [<60063895>] oom_kill_process+0x125/0x160
[ 1061.900000] 6792bbb8: [<60063a83>] out_of_memory+0xa3/0x140
[ 1061.900000] 6792bc08: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1061.900000] 6792bc38: [<6003310c>] __might_sleep+0xdc/0x120
[ 1061.900000] 6792bc78: [<60076a82>] anon_vma_prepare+0x32/0x120
[ 1061.900000] 6792bcb8: [<60070e5b>] do_anonymous_page+0x4b/0x1b0
[ 1061.900000] 6792bd18: [<600718db>] handle_mm_fault+0x26b/0x2e0
[ 1061.900000] 6792bd28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1061.900000] 6792bd88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1061.900000] 6792bdf8: [<600175e2>] segv+0x1b2/0x2d0
[ 1061.900000] 6792bee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1061.900000] 6792bf18: [<6002b2be>] userspace+0x22e/0x300
[ 1061.900000] 6792bfc8: [<60014992>] fork_handler+0x62/0x70
[ 1061.900000]
[ 1061.900000] Mem-info:
[ 1061.900000] Normal per-cpu:
[ 1061.900000] CPU 0: hi: 186, btch: 31 usd: 132
[ 1061.900000] Active:121370 inactive:41 dirty:0 writeback:0 unstable:0
[ 1061.900000] free:714 slab:1492 mapped:14 pagetables:416 bounce:0
[ 1061.900000] Normal free:2856kB min:2876kB low:3592kB high:4312kB active:485480kB inactive:164kB present:517120kB pages_scanned:788832 all_unreclaimable? yes
[ 1061.900000] lowmem_reserve[]: 0 0
[ 1061.900000] Normal: 2*4kB 2*8kB 1*16kB 4*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2856kB
[ 1061.900000] 121 total pagecache pages
[ 1061.900000] Swap cache: add 0, delete 0, find 0/0
[ 1061.900000] Free swap = 0kB
[ 1061.900000] Total swap = 0kB
[ 1061.900000] Free swap: 0kB
[ 1061.900000] 131072 pages of RAM
[ 1061.900000] 0 pages of HIGHMEM
[ 1061.900000] 5047 reserved pages
[ 1061.900000] 202 pages shared
[ 1061.900000] 0 pages swap cached
[ 1061.900000] Out of memory: kill process 1039 (uml_switch) score 936 or a child
[ 1061.900000] Killed process 1039 (uml_switch)
[ 1061.910000] resvtest invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=-17
[ 1061.910000] Call Trace:
[ 1061.910000] 6792bb58: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1061.910000] 6792bb68: [<60063895>] oom_kill_process+0x125/0x160
[ 1061.910000] 6792bbb8: [<60063a83>] out_of_memory+0xa3/0x140
[ 1061.910000] 6792bc08: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1061.910000] 6792bc38: [<6003310c>] __might_sleep+0xdc/0x120
[ 1061.910000] 6792bc78: [<60076a82>] anon_vma_prepare+0x32/0x120
[ 1061.910000] 6792bcb8: [<60070e5b>] do_anonymous_page+0x4b/0x1b0
[ 1061.910000] 6792bd18: [<600718db>] handle_mm_fault+0x26b/0x2e0
[ 1061.910000] 6792bd28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1061.910000] 6792bd88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1061.910000] 6792bdf8: [<600175e2>] segv+0x1b2/0x2d0
[ 1061.910000] 6792bee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1061.910000] 6792bf18: [<6002b2be>] userspace+0x22e/0x300
[ 1061.910000] 6792bfc8: [<60014992>] fork_handler+0x62/0x70
[ 1061.910000]
[ 1061.910000] Mem-info:
[ 1061.910000] Normal per-cpu:
[ 1061.910000] CPU 0: hi: 186, btch: 31 usd: 166
[ 1061.910000] Active:121348 inactive:41 dirty:0 writeback:0 unstable:0
[ 1061.910000] free:714 slab:1492 mapped:6 pagetables:409 bounce:0
[ 1061.910000] Normal free:2856kB min:2876kB low:3592kB high:4312kB active:485392kB inactive:164kB present:517120kB pages_scanned:789072 all_unreclaimable? yes
[ 1061.910000] lowmem_reserve[]: 0 0
[ 1061.910000] Normal: 2*4kB 2*8kB 1*16kB 4*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2856kB
[ 1061.910000] 121 total pagecache pages
[ 1061.910000] Swap cache: add 0, delete 0, find 0/0
[ 1061.910000] Free swap = 0kB
[ 1061.910000] Total swap = 0kB
[ 1061.910000] Free swap: 0kB
[ 1061.910000] 131072 pages of RAM
[ 1061.910000] 0 pages of HIGHMEM
[ 1061.910000] 5047 reserved pages
[ 1061.910000] 194 pages shared
[ 1061.910000] 0 pages swap cached
[ 1061.910000] Out of memory: kill process 1061 (cron) score 315 or a child
[ 1061.910000] Killed process 1061 (cron)
[ 1062.240000] init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
[ 1062.240000] Call Trace:
[ 1062.240000] 7fc239d8: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1062.240000] 7fc239e8: [<60063895>] oom_kill_process+0x125/0x160
[ 1062.240000] 7fc23a38: [<60063a83>] out_of_memory+0xa3/0x140
[ 1062.240000] 7fc23a88: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1062.240000] 7fc23af8: [<600681b1>] read_pages+0x41/0xe0
[ 1062.240000] 7fc23b38: [<60068354>] __do_page_cache_readahead+0x104/0x1d0
[ 1062.240000] 7fc23bd8: [<6006851c>] do_page_cache_readahead+0x5c/0x80
[ 1062.240000] 7fc23c08: [<60060e30>] filemap_fault+0x1a0/0x2f0
[ 1062.240000] 7fc23c68: [<60071027>] __do_fault+0x67/0x480
[ 1062.240000] 7fc23c88: [<6002a5ff>] map+0x11f/0x140
[ 1062.240000] 7fc23cf8: [<6007147c>] do_linear_fault+0x3c/0x40
[ 1062.240000] 7fc23d08: [<6031638e>] _spin_unlock_irq+0xe/0x10
[ 1062.240000] 7fc23d18: [<600717b3>] handle_mm_fault+0x143/0x2e0
[ 1062.240000] 7fc23d28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1062.240000] 7fc23d88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1062.240000] 7fc23df8: [<600175e2>] segv+0x1b2/0x2d0
[ 1062.240000] 7fc23ee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1062.240000] 7fc23f18: [<6002b2be>] userspace+0x22e/0x300
[ 1062.240000] 7fc23f58: [<60000a90>] kernel_init+0x0/0x80
[ 1062.240000] 7fc23fc8: [<60014914>] new_thread_handler+0x84/0xa0
[ 1062.240000]
[ 1062.240000] Mem-info:
[ 1062.240000] Normal per-cpu:
[ 1062.240000] CPU 0: hi: 186, btch: 31 usd: 148
[ 1062.240000] Active:121456 inactive:1 dirty:0 writeback:0 unstable:0
[ 1062.240000] free:703 slab:1461 mapped:0 pagetables:396 bounce:0
[ 1062.240000] Normal free:2812kB min:2876kB low:3592kB high:4312kB active:485824kB inactive:4kB present:517120kB pages_scanned:755399 all_unreclaimable? yes
[ 1062.240000] lowmem_reserve[]: 0 0
[ 1062.240000] Normal: 3*4kB 2*8kB 0*16kB 3*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2812kB
[ 1062.240000] 75 total pagecache pages
[ 1062.240000] Swap cache: add 0, delete 0, find 0/0
[ 1062.240000] Free swap = 0kB
[ 1062.240000] Total swap = 0kB
[ 1062.240000] Free swap: 0kB
[ 1062.240000] 131072 pages of RAM
[ 1062.240000] 0 pages of HIGHMEM
[ 1062.240000] 5047 reserved pages
[ 1062.240000] 181 pages shared
[ 1062.240000] 0 pages swap cached
[ 1062.240000] Out of memory: kill process 1029 (inetd) score 157 or a child
[ 1062.240000] Killed process 1029 (inetd)
[ 1062.490000] init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
[ 1062.490000] Call Trace:
[ 1062.490000] 7fc239d8: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1062.490000] 7fc239e8: [<60063895>] oom_kill_process+0x125/0x160
[ 1062.490000] 7fc23a38: [<60063a83>] out_of_memory+0xa3/0x140
[ 1062.490000] 7fc23a88: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1062.490000] 7fc23aa8: [<6001828b>] do_op_one_page+0x13b/0x150
[ 1062.490000] 7fc23ab8: [<60018390>] copy_chunk_from_user+0x0/0x40
[ 1062.490000] 7fc23b38: [<60068354>] __do_page_cache_readahead+0x104/0x1d0
[ 1062.490000] 7fc23bd8: [<6006851c>] do_page_cache_readahead+0x5c/0x80
[ 1062.490000] 7fc23c08: [<60060e30>] filemap_fault+0x1a0/0x2f0
[ 1062.490000] 7fc23c68: [<60071027>] __do_fault+0x67/0x480
[ 1062.490000] 7fc23c88: [<6002a5ff>] map+0x11f/0x140
[ 1062.490000] 7fc23cf8: [<6007147c>] do_linear_fault+0x3c/0x40
[ 1062.490000] 7fc23d08: [<6031638e>] _spin_unlock_irq+0xe/0x10
[ 1062.490000] 7fc23d18: [<600717b3>] handle_mm_fault+0x143/0x2e0
[ 1062.490000] 7fc23d28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1062.490000] 7fc23d88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1062.490000] 7fc23df8: [<600175e2>] segv+0x1b2/0x2d0
[ 1062.490000] 7fc23ee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1062.490000] 7fc23f18: [<6002b2be>] userspace+0x22e/0x300
[ 1062.490000] 7fc23f58: [<60000a90>] kernel_init+0x0/0x80
[ 1062.490000] 7fc23fc8: [<60014914>] new_thread_handler+0x84/0xa0
[ 1062.490000]
[ 1062.490000] Mem-info:
[ 1062.490000] Normal per-cpu:
[ 1062.490000] CPU 0: hi: 186, btch: 31 usd: 147
[ 1062.490000] Active:121400 inactive:75 dirty:0 writeback:0 unstable:0
[ 1062.490000] free:707 slab:1461 mapped:0 pagetables:387 bounce:0
[ 1062.490000] Normal free:2828kB min:2876kB low:3592kB high:4312kB active:485600kB inactive:300kB present:517120kB pages_scanned:874765 all_unreclaimable? yes
[ 1062.490000] lowmem_reserve[]: 0 0
[ 1062.490000] Normal: 3*4kB 2*8kB 1*16kB 3*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2828kB
[ 1062.490000] 75 total pagecache pages
[ 1062.490000] Swap cache: add 0, delete 0, find 0/0
[ 1062.490000] Free swap = 0kB
[ 1062.490000] Total swap = 0kB
[ 1062.490000] Free swap: 0kB
[ 1062.490000] 131072 pages of RAM
[ 1062.490000] 0 pages of HIGHMEM
[ 1062.490000] 5047 reserved pages
[ 1062.490000] 181 pages shared
[ 1062.490000] 0 pages swap cached
[ 1062.490000] Out of memory: kill process 1001 (syslogd) score 91 or a child
[ 1062.490000] Killed process 1001 (syslogd)
[ 1062.770000] klogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
[ 1062.770000] Call Trace:
[ 1062.770000] 7e9ef9d8: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1062.770000] 7e9ef9e8: [<60063895>] oom_kill_process+0x125/0x160
[ 1062.770000] 7e9efa38: [<60063a83>] out_of_memory+0xa3/0x140
[ 1062.770000] 7e9efa88: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1062.770000] 7e9efab8: [<603143ef>] schedule+0x16f/0x260
[ 1062.770000] 7e9efb38: [<60068354>] __do_page_cache_readahead+0x104/0x1d0
[ 1062.770000] 7e9efbd8: [<6006851c>] do_page_cache_readahead+0x5c/0x80
[ 1062.770000] 7e9efc08: [<60060e30>] filemap_fault+0x1a0/0x2f0
[ 1062.770000] 7e9efc68: [<60071027>] __do_fault+0x67/0x480
[ 1062.770000] 7e9efc88: [<6002a5ff>] map+0x11f/0x140
[ 1062.770000] 7e9efcf8: [<6007147c>] do_linear_fault+0x3c/0x40
[ 1062.770000] 7e9efd08: [<6031638e>] _spin_unlock_irq+0xe/0x10
[ 1062.770000] 7e9efd18: [<600717b3>] handle_mm_fault+0x143/0x2e0
[ 1062.770000] 7e9efd28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1062.770000] 7e9efd88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1062.770000] 7e9efdf8: [<600175e2>] segv+0x1b2/0x2d0
[ 1062.770000] 7e9efee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1062.770000] 7e9eff18: [<6002b2be>] userspace+0x22e/0x300
[ 1062.770000] 7e9effc8: [<60014992>] fork_handler+0x62/0x70
[ 1062.770000]
[ 1062.770000] Mem-info:
[ 1062.770000] Normal per-cpu:
[ 1062.770000] CPU 0: hi: 186, btch: 31 usd: 158
[ 1062.770000] Active:121480 inactive:12 dirty:0 writeback:0 unstable:0
[ 1062.770000] free:711 slab:1453 mapped:0 pagetables:379 bounce:0
[ 1062.770000] Normal free:2844kB min:2876kB low:3592kB high:4312kB active:485920kB inactive:48kB present:517120kB pages_scanned:850578 all_unreclaimable? yes
[ 1062.770000] lowmem_reserve[]: 0 0
[ 1062.770000] Normal: 3*4kB 2*8kB 2*16kB 3*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2844kB
[ 1062.770000] 75 total pagecache pages
[ 1062.770000] Swap cache: add 0, delete 0, find 0/0
[ 1062.770000] Free swap = 0kB
[ 1062.770000] Total swap = 0kB
[ 1062.770000] Free swap: 0kB
[ 1062.770000] 131072 pages of RAM
[ 1062.770000] 0 pages of HIGHMEM
[ 1062.770000] 5047 reserved pages
[ 1062.770000] 181 pages shared
[ 1062.770000] 0 pages swap cached
[ 1062.770000] Out of memory: kill process 1008 (klogd) score 58 or a child
[ 1062.770000] Killed process 1008 (klogd)
[ 1063.270000] resvtest invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=-17
[ 1063.270000] Call Trace:
[ 1063.270000] 6792bb58: [<60037a85>] printk_ratelimit+0x15/0x20
[ 1063.270000] 6792bb68: [<60063895>] oom_kill_process+0x125/0x160
[ 1063.270000] 6792bbb8: [<60063a83>] out_of_memory+0xa3/0x140
[ 1063.270000] 6792bc08: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1063.270000] 6792bc38: [<6003310c>] __might_sleep+0xdc/0x120
[ 1063.270000] 6792bc78: [<60076a82>] anon_vma_prepare+0x32/0x120
[ 1063.270000] 6792bcb8: [<60070e5b>] do_anonymous_page+0x4b/0x1b0
[ 1063.270000] 6792bd18: [<600718db>] handle_mm_fault+0x26b/0x2e0
[ 1063.270000] 6792bd28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1063.270000] 6792bd88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1063.270000] 6792bdf8: [<600175e2>] segv+0x1b2/0x2d0
[ 1063.270000] 6792bee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1063.270000] 6792bf18: [<6002b2be>] userspace+0x22e/0x300
[ 1063.270000] 6792bfc8: [<60014992>] fork_handler+0x62/0x70
[ 1063.270000]
[ 1063.270000] Mem-info:
[ 1063.270000] Normal per-cpu:
[ 1063.270000] CPU 0: hi: 186, btch: 31 usd: 174
[ 1063.270000] Active:121437 inactive:75 dirty:0 writeback:0 unstable:0
[ 1063.270000] free:715 slab:1453 mapped:0 pagetables:372 bounce:0
[ 1063.270000] Normal free:2860kB min:2876kB low:3592kB high:4312kB active:485748kB inactive:300kB present:517120kB pages_scanned:941856 all_unreclaimable? yes
[ 1063.270000] lowmem_reserve[]: 0 0
[ 1063.270000] Normal: 3*4kB 2*8kB 3*16kB 3*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2860kB
[ 1063.270000] 75 total pagecache pages
[ 1063.270000] Swap cache: add 0, delete 0, find 0/0
[ 1063.270000] Free swap = 0kB
[ 1063.270000] Total swap = 0kB
[ 1063.270000] Free swap: 0kB
[ 1063.270000] 131072 pages of RAM
[ 1063.270000] 0 pages of HIGHMEM
[ 1063.270000] 5047 reserved pages
[ 1063.270000] 181 pages shared
[ 1063.270000] 0 pages swap cached
[ 1063.270000] Out of memory: kill process 1081 (getty) score 58 or a child
[ 1063.270000] Killed process 1081 (getty)
[ 1063.620000] Kernel panic - not syncing: Out of memory and no killable processes...
[ 1063.620000]
[ 1063.620000]
[ 1063.620000] Pid: 5888, comm: resvtest Not tainted 2.6.25-xfs-btree
[ 1063.620000] RIP: 0033:[<000000004028d60b>]
[ 1063.620000] RSP: 0000007fbfd6b640 EFLAGS: 00010206
[ 1063.620000] RAX: 0000000000001011 RBX: 000000001dd12370 RCX: 000000001dd13380
[ 1063.620000] RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000004
[ 1063.620000] RBP: 0000000000001000 R08: 0000000000000003 R09: 0000007fbfd6b510
[ 1063.620000] R10: 0000000000000008 R11: 0000000000000206 R12: 000000004055fa00
[ 1063.620000] R13: 000000004055f9a0 R14: 0000000000016c81 R15: 0000000000001010
[ 1063.620000] Call Trace:
[ 1063.620000] 6792ba48: [<600179df>] panic_exit+0x2f/0x50
[ 1063.620000] 6792ba68: [<600514b5>] notifier_call_chain+0x45/0x90
[ 1063.620000] 6792baa8: [<600515bd>] __atomic_notifier_call_chain+0xd/0x10
[ 1063.620000] 6792bab8: [<600515d1>] atomic_notifier_call_chain+0x11/0x20
[ 1063.620000] 6792bac8: [<60035f26>] panic+0xe6/0x1a0
[ 1063.620000] 6792bb28: [<6005038c>] ktime_get_ts+0x4c/0x60
[ 1063.620000] 6792bb48: [<60063406>] select_bad_process+0x36/0x110
[ 1063.620000] 6792bbb8: [<60063b00>] out_of_memory+0x120/0x140
[ 1063.620000] 6792bc08: [<6006567e>] __alloc_pages+0x2ae/0x3d0
[ 1063.620000] 6792bc38: [<6003310c>] __might_sleep+0xdc/0x120
[ 1063.620000] 6792bc78: [<60076a82>] anon_vma_prepare+0x32/0x120
[ 1063.620000] 6792bcb8: [<60070e5b>] do_anonymous_page+0x4b/0x1b0
[ 1063.620000] 6792bd18: [<600718db>] handle_mm_fault+0x26b/0x2e0
[ 1063.620000] 6792bd28: [<6003310c>] __might_sleep+0xdc/0x120
[ 1063.620000] 6792bd88: [<6001725b>] handle_page_fault+0x18b/0x240
[ 1063.620000] 6792bdf8: [<600175e2>] segv+0x1b2/0x2d0
[ 1063.620000] 6792bee8: [<6001742b>] segv_handler+0x7b/0x80
[ 1063.620000] 6792bf18: [<6002b2be>] userspace+0x22e/0x300
[ 1063.620000] 6792bfc8: [<60014992>] fork_handler+0x62/0x70
[ 1063.620000]
Terminated
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: OOM killer doesn't kill the right task....
2008-04-21 7:01 OOM killer doesn't kill the right task David Chinner
@ 2008-04-21 8:24 ` KOSAKI Motohiro
2008-04-21 9:25 ` Zdenek Kabelac
2008-04-21 11:07 ` David Chinner
0 siblings, 2 replies; 4+ messages in thread
From: KOSAKI Motohiro @ 2008-04-21 8:24 UTC (permalink / raw)
To: David Chinner; +Cc: kosaki.motohiro, lkml, xfs-oss, linux-mm
Hi David
> Running in a 512MB UML system without swap, XFSQA test 084 reliably
> kills the kernel completely as the OOM killer is unable to find a
> task to kill. log output is below.
>
> I don't know when it started failing - ISTR this working just fine
> on 2.6.24 kernels.
Can you reproduce it on non UML box?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: OOM killer doesn't kill the right task....
2008-04-21 8:24 ` KOSAKI Motohiro
@ 2008-04-21 9:25 ` Zdenek Kabelac
2008-04-21 11:07 ` David Chinner
1 sibling, 0 replies; 4+ messages in thread
From: Zdenek Kabelac @ 2008-04-21 9:25 UTC (permalink / raw)
To: KOSAKI Motohiro; +Cc: David Chinner, lkml, xfs-oss, linux-mm
2008/4/21, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>:
> Hi David
>
> > Running in a 512MB UML system without swap, XFSQA test 084 reliably
> > kills the kernel completely as the OOM killer is unable to find a
> > task to kill. log output is below.
> >
> > I don't know when it started failing - ISTR this working just fine
> > on 2.6.24 kernels.
>
> Can you reproduce it on non UML box?
>
Hi
I'll add my comments to this discussion.
I'm seeing recently also very strange OOM behavior when I limit memory
in my qemu-kvm.
There are few weird things:
a) - if I call echo 3>/proc/sys/vm/drop_caches prior dselect in my
case - the operation normaly proceeds and everything is fine
b) - if the dselect is started and memory is low - either I get this
crash or dselect simple gets killed - even thought obviously the
memory is there and could be used for the task - because when the
caches are dropped it is doable.
Regards
Zdenek
Enjoy my backtrace: (Using latest 2.6.25 Linus-gits)
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 90, btch: 15 usd: 90
CPU 1: hi: 90, btch: 15 usd: 87
Active:33126 inactive:0 dirty:0 writeback:0 unstable:0
free:732 slab:17423 mapped:0 pagetables:109 bounce:0
DMA free:1052kB min:120kB low:148kB high:180kB active:14880kB
inactive:0kB present:15576kB pages_scanned:29604 all_unreclaimable?
yes
lowmem_reserve[]: 0 238 238 238
DMA32 free:1876kB min:1912kB low:2388kB high:2868kB active:117496kB
inactive:88kB present:243940kB pages_scanned:177406 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 1052kB
DMA32: 32*4kB 3*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 1864kB
117 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
66544 pages of RAM
5711 reserved pages
82 pages shared
0 pages swap cached
Out of memory: kill process 2530 (getty) score 26 or a child
Killed process 2530 (getty)
udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Pid: 507, comm: udevd Not tainted 2.6.25 #55
Call Trace:
[<ffffffff8108d631>] oom_kill_process+0x91/0x210
[<ffffffff8108d958>] ? badness+0x168/0x200
[<ffffffff8108dba5>] out_of_memory+0x1b5/0x200
[<ffffffff81090e6f>] __alloc_pages+0x3bf/0x3f0
[<ffffffff81092dce>] __do_page_cache_readahead+0x11e/0x260
[<ffffffff81092f71>] do_page_cache_readahead+0x61/0x90
[<ffffffff8108ad42>] filemap_fault+0x372/0x510
[<ffffffff81099e28>] __do_fault+0x68/0x490
[<ffffffff8105fd0e>] ? put_lock_stats+0xe/0x30
[<ffffffff8109bd96>] handle_mm_fault+0x1c6/0x7c0
[<ffffffff812ef49c>] do_page_fault+0x3fc/0xad0
[<ffffffff81087b94>] ? call_rcu+0x74/0xb0
[<ffffffff81087ba2>] ? call_rcu+0x82/0xb0
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff810d1a90>] ? mntput_no_expire+0x20/0x90
[<ffffffff810b7d39>] ? __fput+0x179/0x1b0
[<ffffffff812eb931>] ? trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff812ecc4d>] error_exit+0x0/0xa9
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 90, btch: 15 usd: 86
CPU 1: hi: 90, btch: 15 usd: 87
Active:33097 inactive:0 dirty:0 writeback:0 unstable:0
free:725 slab:17432 mapped:1 pagetables:109 bounce:0
DMA free:1052kB min:120kB low:148kB high:180kB active:14848kB
inactive:0kB present:15576kB pages_scanned:29796 all_unreclaimable?
yes
lowmem_reserve[]: 0 238 238 238
DMA32 free:1848kB min:1912kB low:2388kB high:2868kB active:117540kB
inactive:0kB present:243940kB pages_scanned:54533 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 1052kB
DMA32: 30*4kB 4*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 1864kB
116 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
66544 pages of RAM
5711 reserved pages
81 pages shared
0 pages swap cached
Out of memory: kill process 2531 (getty) score 26 or a child
Killed process 2531 (getty)
udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Pid: 2848, comm: udevd Not tainted 2.6.25 #55
Call Trace:
[<ffffffff8108d631>] oom_kill_process+0x91/0x210
[<ffffffff8108d958>] ? badness+0x168/0x200
[<ffffffff8108dba5>] out_of_memory+0x1b5/0x200
[<ffffffff81090e6f>] __alloc_pages+0x3bf/0x3f0
[<ffffffff81092dce>] __do_page_cache_readahead+0x11e/0x260
[<ffffffff81092f71>] do_page_cache_readahead+0x61/0x90
[<ffffffff8108ad42>] filemap_fault+0x372/0x510
[<ffffffff81099e28>] __do_fault+0x68/0x490
[<ffffffff8105fd0e>] ? put_lock_stats+0xe/0x30
[<ffffffff8109bd96>] handle_mm_fault+0x1c6/0x7c0
[<ffffffff812ef49c>] do_page_fault+0x3fc/0xad0
[<ffffffff810b3f3f>] ? kmem_cache_free+0xaf/0x110
[<ffffffff810ca8fc>] ? d_free+0x6c/0x80
[<ffffffff810ca94b>] ? d_kill+0x3b/0x60
[<ffffffff810d1a90>] ? mntput_no_expire+0x20/0x90
[<ffffffff810b7d39>] ? __fput+0x179/0x1b0
[<ffffffff812eb931>] ? trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff812ecc4d>] error_exit+0x0/0xa9
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 90, btch: 15 usd: 83
CPU 1: hi: 90, btch: 15 usd: 79
Active:33104 inactive:45 dirty:0 writeback:0 unstable:0
free:728 slab:17406 mapped:0 pagetables:103 bounce:0
DMA free:1052kB min:120kB low:148kB high:180kB active:14864kB
inactive:0kB present:15576kB pages_scanned:30052 all_unreclaimable?
yes
lowmem_reserve[]: 0 238 238 238
DMA32 free:1860kB min:1912kB low:2388kB high:2868kB active:117680kB
inactive:52kB present:243940kB pages_scanned:218359 all_unreclaimable?
yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 1052kB
DMA32: 39*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 1876kB
119 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
66544 pages of RAM
5711 reserved pages
80 pages shared
0 pages swap cached
Out of memory: kill process 2532 (getty) score 26 or a child
Killed process 2532 (getty)
init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Pid: 1, comm: init Not tainted 2.6.25 #55
Call Trace:
[<ffffffff8108d631>] oom_kill_process+0x91/0x210
[<ffffffff8108d958>] ? badness+0x168/0x200
[<ffffffff8108dba5>] out_of_memory+0x1b5/0x200
[<ffffffff81090e6f>] __alloc_pages+0x3bf/0x3f0
[<ffffffff81092dce>] __do_page_cache_readahead+0x11e/0x260
[<ffffffff81092f71>] do_page_cache_readahead+0x61/0x90
[<ffffffff8108ad42>] filemap_fault+0x372/0x510
[<ffffffff81099e28>] __do_fault+0x68/0x490
[<ffffffff8105fd0e>] ? put_lock_stats+0xe/0x30
[<ffffffff8109bd96>] handle_mm_fault+0x1c6/0x7c0
[<ffffffff812ef49c>] do_page_fault+0x3fc/0xad0
[<ffffffff812ec2a0>] ? _spin_unlock+0x30/0x60
[<ffffffff810d1a90>] ? mntput_no_expire+0x20/0x90
[<ffffffff810bf4ac>] ? path_put+0x2c/0x40
[<ffffffff810b54fa>] ? sys_faccessat+0x1ba/0x1c0
[<ffffffff812eb931>] ? trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff812ecc4d>] error_exit+0x0/0xa9
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 90, btch: 15 usd: 86
CPU 1: hi: 90, btch: 15 usd: 77
Active:33185 inactive:12 dirty:0 writeback:0 unstable:0
free:741 slab:17370 mapped:0 pagetables:106 bounce:0
DMA free:1052kB min:120kB low:148kB high:180kB active:14848kB
inactive:0kB present:15576kB pages_scanned:27115 all_unreclaimable?
yes
lowmem_reserve[]: 0 238 238 238
DMA32 free:1912kB min:1912kB low:2388kB high:2868kB active:117940kB
inactive:0kB present:243940kB pages_scanned:222562 all_unreclaimable?
yes
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 1052kB
DMA32: 48*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 1912kB
110 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
66544 pages of RAM
5711 reserved pages
129 pages shared
0 pages swap cached
Out of memory: kill process 2533 (getty) score 26 or a child
Killed process 2533 (getty)
udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Pid: 2848, comm: udevd Not tainted 2.6.25 #55
Call Trace:
[<ffffffff8108d631>] oom_kill_process+0x91/0x210
[<ffffffff8108d958>] ? badness+0x168/0x200
[<ffffffff8108dba5>] out_of_memory+0x1b5/0x200
[<ffffffff81090e6f>] __alloc_pages+0x3bf/0x3f0
[<ffffffff81092dce>] __do_page_cache_readahead+0x11e/0x260
[<ffffffff81092f71>] do_page_cache_readahead+0x61/0x90
[<ffffffff8108ad42>] filemap_fault+0x372/0x510
[<ffffffff81099e28>] __do_fault+0x68/0x490
[<ffffffff8105fd0e>] ? put_lock_stats+0xe/0x30
[<ffffffff8109bd96>] handle_mm_fault+0x1c6/0x7c0
[<ffffffff812ef49c>] do_page_fault+0x3fc/0xad0
[<ffffffff8105fd0e>] ? put_lock_stats+0xe/0x30
[<ffffffff812ec245>] ? _spin_unlock_irqrestore+0x65/0x90
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff812ec225>] ? _spin_unlock_irqrestore+0x45/0x90
[<ffffffff81036d7a>] ? hrtick_set+0x7a/0x130
[<ffffffff812e8ebd>] ? thread_return+0x9e/0x571
[<ffffffff810d1a90>] ? mntput_no_expire+0x20/0x90
[<ffffffff812eb931>] ? trace_hardirqs_on_thunk+0x35/0x3a
[<ffffffff81061ba1>] ? trace_hardirqs_on+0x131/0x190
[<ffffffff812ecc4d>] error_exit+0x0/0xa9
Mem-info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 90, btch: 15 usd: 82
CPU 1: hi: 90, btch: 15 usd: 86
Active:33076 inactive:110 dirty:0 writeback:0 unstable:0
free:743 slab:17354 mapped:4 pagetables:103 bounce:0
DMA free:1052kB min:120kB low:148kB high:180kB active:14864kB
inactive:0kB present:15576kB pages_scanned:27435 all_unreclaimable?
yes
lowmem_reserve[]: 0 238 238 238
DMA32 free:1884kB min:1912kB low:2388kB high:2868kB active:117568kB
inactive:292kB present:243940kB pages_scanned:101687
all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 1052kB
DMA32: 41*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 1884kB
113 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
66544 pages of RAM
5711 reserved pages
140 pages shared
0 pages swap cached
Out of memory: kill process 2534 (getty) score 26 or a child
Killed process 2534 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: OOM killer doesn't kill the right task....
2008-04-21 8:24 ` KOSAKI Motohiro
2008-04-21 9:25 ` Zdenek Kabelac
@ 2008-04-21 11:07 ` David Chinner
1 sibling, 0 replies; 4+ messages in thread
From: David Chinner @ 2008-04-21 11:07 UTC (permalink / raw)
To: KOSAKI Motohiro; +Cc: David Chinner, lkml, xfs-oss, linux-mm
On Mon, Apr 21, 2008 at 05:24:04PM +0900, KOSAKI Motohiro wrote:
> Hi David
>
> > Running in a 512MB UML system without swap, XFSQA test 084 reliably
> > kills the kernel completely as the OOM killer is unable to find a
> > task to kill. log output is below.
> >
> > I don't know when it started failing - ISTR this working just fine
> > on 2.6.24 kernels.
>
> Can you reproduce it on non UML box?
Not exactly. On a 64k page size ia64 box it kills my ssh session and all
it's children which includes the errant process (log below).
It doesn't kill the machine, but if I cared enough I'd argue that
even that is killing the wrong process because it's pretty damn
clear that the only process on the box using more than a couple of
MB of memory is the resvtest program....
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
resvtest invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Call Trace:
[<a0000001000125e0>] show_stack+0x40/0xa0
sp=e000003022f0fb20 bsp=e000003022f01168
[<a000000100012670>] dump_stack+0x30/0x60
sp=e000003022f0fcf0 bsp=e000003022f01150
[<a000000100104060>] oom_kill_process+0x80/0x3a0
sp=e000003022f0fcf0 bsp=e000003022f010f8
[<a000000100104ce0>] out_of_memory+0x4e0/0x660
sp=e000003022f0fd00 bsp=e000003022f010b0
[<a00000010010bf00>] __alloc_pages+0x500/0x620
sp=e000003022f0fd90 bsp=e000003022f01040
[<a000000100145020>] alloc_page_vma+0x1c0/0x200
sp=e000003022f0fda0 bsp=e000003022f01008
[<a000000100122d00>] handle_mm_fault+0x3a0/0xe60
sp=e000003022f0fda0 bsp=e000003022f00f88
[<a00000010085b700>] ia64_do_page_fault+0x2a0/0xaa0
sp=e000003022f0fda0 bsp=e000003022f00f30
[<a000000100009e20>] ia64_leave_kernel+0x0/0x270
sp=e000003022f0fe30 bsp=e000003022f00f30
Mem-info:
Node 0 Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Active:5044 inactive:64 dirty:0 writeback:0 unstable:0
free:34 slab:1110 mapped:6 pagetables:104 bounce:0
Node 0 Normal free:2176kB min:2880kB low:3584kB high:4288kB active:322816kB inactive:4096kB present:523776kB pages_scanned:32980 all_unreclaimable? yes
lowmem_reserve[]: 0 0
Node 0 Normal: 2*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB 0*1048576kB 0*2097152kB 0*4194304kB = 2432kB
76 total pagecache pages
Swap cache: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
Node memory in pages:
Node 0: RAM: 8192, rsvd: 1327, shrd: 129, swpd: 0
Node 1: RAM: 0, rsvd: 0, shrd: 0, swpd: 0
8192 pages of RAM
1327 reserved pages
129 pages shared
0 pages swap cached
Total of 45 pages in page table cache
8117 free buffer pages
Out of memory: kill process 2908 (sshd) score 379 or a child
Killed process 2909 (bash)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2008-04-21 11:07 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-04-21 7:01 OOM killer doesn't kill the right task David Chinner
2008-04-21 8:24 ` KOSAKI Motohiro
2008-04-21 9:25 ` Zdenek Kabelac
2008-04-21 11:07 ` David Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox