From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by rv-out-0506.google.com with SMTP id g37so843565rvb.26 for ; Mon, 21 Apr 2008 02:25:35 -0700 (PDT) Message-ID: Date: Mon, 21 Apr 2008 11:25:35 +0200 From: "Zdenek Kabelac" Subject: Re: OOM killer doesn't kill the right task.... In-Reply-To: <20080421172255.C45A.KOSAKI.MOTOHIRO@jp.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080421070123.GM108924158@sgi.com> <20080421172255.C45A.KOSAKI.MOTOHIRO@jp.fujitsu.com> Sender: owner-linux-mm@kvack.org Return-Path: To: KOSAKI Motohiro Cc: David Chinner , lkml , xfs-oss , linux-mm List-ID: 2008/4/21, KOSAKI Motohiro : > Hi David > > > Running in a 512MB UML system without swap, XFSQA test 084 reliably > > kills the kernel completely as the OOM killer is unable to find a > > task to kill. log output is below. > > > > I don't know when it started failing - ISTR this working just fine > > on 2.6.24 kernels. > > Can you reproduce it on non UML box? > Hi I'll add my comments to this discussion. I'm seeing recently also very strange OOM behavior when I limit memory in my qemu-kvm. There are few weird things: a) - if I call echo 3>/proc/sys/vm/drop_caches prior dselect in my case - the operation normaly proceeds and everything is fine b) - if the dselect is started and memory is low - either I get this crash or dselect simple gets killed - even thought obviously the memory is there and could be used for the task - because when the caches are dropped it is doable. Regards Zdenek Enjoy my backtrace: (Using latest 2.6.25 Linus-gits) Mem-info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 90, btch: 15 usd: 90 CPU 1: hi: 90, btch: 15 usd: 87 Active:33126 inactive:0 dirty:0 writeback:0 unstable:0 free:732 slab:17423 mapped:0 pagetables:109 bounce:0 DMA free:1052kB min:120kB low:148kB high:180kB active:14880kB inactive:0kB present:15576kB pages_scanned:29604 all_unreclaimable? yes lowmem_reserve[]: 0 238 238 238 DMA32 free:1876kB min:1912kB low:2388kB high:2868kB active:117496kB inactive:88kB present:243940kB pages_scanned:177406 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1052kB DMA32: 32*4kB 3*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1864kB 117 total pagecache pages Swap cache: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 66544 pages of RAM 5711 reserved pages 82 pages shared 0 pages swap cached Out of memory: kill process 2530 (getty) score 26 or a child Killed process 2530 (getty) udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17 Pid: 507, comm: udevd Not tainted 2.6.25 #55 Call Trace: [] oom_kill_process+0x91/0x210 [] ? badness+0x168/0x200 [] out_of_memory+0x1b5/0x200 [] __alloc_pages+0x3bf/0x3f0 [] __do_page_cache_readahead+0x11e/0x260 [] do_page_cache_readahead+0x61/0x90 [] filemap_fault+0x372/0x510 [] __do_fault+0x68/0x490 [] ? put_lock_stats+0xe/0x30 [] handle_mm_fault+0x1c6/0x7c0 [] do_page_fault+0x3fc/0xad0 [] ? call_rcu+0x74/0xb0 [] ? call_rcu+0x82/0xb0 [] ? trace_hardirqs_on+0x131/0x190 [] ? mntput_no_expire+0x20/0x90 [] ? __fput+0x179/0x1b0 [] ? trace_hardirqs_on_thunk+0x35/0x3a [] ? trace_hardirqs_on+0x131/0x190 [] error_exit+0x0/0xa9 Mem-info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 90, btch: 15 usd: 86 CPU 1: hi: 90, btch: 15 usd: 87 Active:33097 inactive:0 dirty:0 writeback:0 unstable:0 free:725 slab:17432 mapped:1 pagetables:109 bounce:0 DMA free:1052kB min:120kB low:148kB high:180kB active:14848kB inactive:0kB present:15576kB pages_scanned:29796 all_unreclaimable? yes lowmem_reserve[]: 0 238 238 238 DMA32 free:1848kB min:1912kB low:2388kB high:2868kB active:117540kB inactive:0kB present:243940kB pages_scanned:54533 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1052kB DMA32: 30*4kB 4*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1864kB 116 total pagecache pages Swap cache: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 66544 pages of RAM 5711 reserved pages 81 pages shared 0 pages swap cached Out of memory: kill process 2531 (getty) score 26 or a child Killed process 2531 (getty) udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17 Pid: 2848, comm: udevd Not tainted 2.6.25 #55 Call Trace: [] oom_kill_process+0x91/0x210 [] ? badness+0x168/0x200 [] out_of_memory+0x1b5/0x200 [] __alloc_pages+0x3bf/0x3f0 [] __do_page_cache_readahead+0x11e/0x260 [] do_page_cache_readahead+0x61/0x90 [] filemap_fault+0x372/0x510 [] __do_fault+0x68/0x490 [] ? put_lock_stats+0xe/0x30 [] handle_mm_fault+0x1c6/0x7c0 [] do_page_fault+0x3fc/0xad0 [] ? kmem_cache_free+0xaf/0x110 [] ? d_free+0x6c/0x80 [] ? d_kill+0x3b/0x60 [] ? mntput_no_expire+0x20/0x90 [] ? __fput+0x179/0x1b0 [] ? trace_hardirqs_on_thunk+0x35/0x3a [] ? trace_hardirqs_on+0x131/0x190 [] error_exit+0x0/0xa9 Mem-info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 90, btch: 15 usd: 83 CPU 1: hi: 90, btch: 15 usd: 79 Active:33104 inactive:45 dirty:0 writeback:0 unstable:0 free:728 slab:17406 mapped:0 pagetables:103 bounce:0 DMA free:1052kB min:120kB low:148kB high:180kB active:14864kB inactive:0kB present:15576kB pages_scanned:30052 all_unreclaimable? yes lowmem_reserve[]: 0 238 238 238 DMA32 free:1860kB min:1912kB low:2388kB high:2868kB active:117680kB inactive:52kB present:243940kB pages_scanned:218359 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1052kB DMA32: 39*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1876kB 119 total pagecache pages Swap cache: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 66544 pages of RAM 5711 reserved pages 80 pages shared 0 pages swap cached Out of memory: kill process 2532 (getty) score 26 or a child Killed process 2532 (getty) init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Pid: 1, comm: init Not tainted 2.6.25 #55 Call Trace: [] oom_kill_process+0x91/0x210 [] ? badness+0x168/0x200 [] out_of_memory+0x1b5/0x200 [] __alloc_pages+0x3bf/0x3f0 [] __do_page_cache_readahead+0x11e/0x260 [] do_page_cache_readahead+0x61/0x90 [] filemap_fault+0x372/0x510 [] __do_fault+0x68/0x490 [] ? put_lock_stats+0xe/0x30 [] handle_mm_fault+0x1c6/0x7c0 [] do_page_fault+0x3fc/0xad0 [] ? _spin_unlock+0x30/0x60 [] ? mntput_no_expire+0x20/0x90 [] ? path_put+0x2c/0x40 [] ? sys_faccessat+0x1ba/0x1c0 [] ? trace_hardirqs_on_thunk+0x35/0x3a [] ? trace_hardirqs_on+0x131/0x190 [] error_exit+0x0/0xa9 Mem-info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 90, btch: 15 usd: 86 CPU 1: hi: 90, btch: 15 usd: 77 Active:33185 inactive:12 dirty:0 writeback:0 unstable:0 free:741 slab:17370 mapped:0 pagetables:106 bounce:0 DMA free:1052kB min:120kB low:148kB high:180kB active:14848kB inactive:0kB present:15576kB pages_scanned:27115 all_unreclaimable? yes lowmem_reserve[]: 0 238 238 238 DMA32 free:1912kB min:1912kB low:2388kB high:2868kB active:117940kB inactive:0kB present:243940kB pages_scanned:222562 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1052kB DMA32: 48*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1912kB 110 total pagecache pages Swap cache: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 66544 pages of RAM 5711 reserved pages 129 pages shared 0 pages swap cached Out of memory: kill process 2533 (getty) score 26 or a child Killed process 2533 (getty) udevd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17 Pid: 2848, comm: udevd Not tainted 2.6.25 #55 Call Trace: [] oom_kill_process+0x91/0x210 [] ? badness+0x168/0x200 [] out_of_memory+0x1b5/0x200 [] __alloc_pages+0x3bf/0x3f0 [] __do_page_cache_readahead+0x11e/0x260 [] do_page_cache_readahead+0x61/0x90 [] filemap_fault+0x372/0x510 [] __do_fault+0x68/0x490 [] ? put_lock_stats+0xe/0x30 [] handle_mm_fault+0x1c6/0x7c0 [] do_page_fault+0x3fc/0xad0 [] ? put_lock_stats+0xe/0x30 [] ? _spin_unlock_irqrestore+0x65/0x90 [] ? trace_hardirqs_on+0x131/0x190 [] ? _spin_unlock_irqrestore+0x45/0x90 [] ? hrtick_set+0x7a/0x130 [] ? thread_return+0x9e/0x571 [] ? mntput_no_expire+0x20/0x90 [] ? trace_hardirqs_on_thunk+0x35/0x3a [] ? trace_hardirqs_on+0x131/0x190 [] error_exit+0x0/0xa9 Mem-info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 90, btch: 15 usd: 82 CPU 1: hi: 90, btch: 15 usd: 86 Active:33076 inactive:110 dirty:0 writeback:0 unstable:0 free:743 slab:17354 mapped:4 pagetables:103 bounce:0 DMA free:1052kB min:120kB low:148kB high:180kB active:14864kB inactive:0kB present:15576kB pages_scanned:27435 all_unreclaimable? yes lowmem_reserve[]: 0 238 238 238 DMA32 free:1884kB min:1912kB low:2388kB high:2868kB active:117568kB inactive:292kB present:243940kB pages_scanned:101687 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1052kB DMA32: 41*4kB 1*8kB 3*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1884kB 113 total pagecache pages Swap cache: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB Free swap: 0kB 66544 pages of RAM 5711 reserved pages 140 pages shared 0 pages swap cached Out of memory: kill process 2534 (getty) score 26 or a child Killed process 2534 (getty) Kernel panic - not syncing: Out of memory and no killable processes... -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org