From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with ESMTP id 3A95F6B004F for ; Wed, 15 Jul 2009 03:40:35 -0400 (EDT) Received: from spaceape11.eur.corp.google.com (spaceape11.eur.corp.google.com [172.28.16.145]) by smtp-out.google.com with ESMTP id n6F8I7lk029361 for ; Wed, 15 Jul 2009 09:18:07 +0100 Received: from pzk36 (pzk36.prod.google.com [10.243.19.164]) by spaceape11.eur.corp.google.com with ESMTP id n6F8Halo013221 for ; Wed, 15 Jul 2009 01:18:05 -0700 Received: by pzk36 with SMTP id 36so4057853pzk.29 for ; Wed, 15 Jul 2009 01:18:04 -0700 (PDT) Date: Wed, 15 Jul 2009 01:18:02 -0700 (PDT) From: David Rientjes Subject: Re: What to do with this message (2.6.30.1) ? In-Reply-To: <20090715084754.36ff73bf.skraw@ithnet.com> Message-ID: References: <20090713134621.124aa18e.skraw@ithnet.com> <4807377b0907132240g6f74c9cbnf1302d354a0e0a72@mail.gmail.com> <20090715084754.36ff73bf.skraw@ithnet.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org To: Stephan von Krawczynski Cc: Jesse Brandeburg , linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Rafael J. Wysocki" , Justin Piszcz List-ID: On Wed, 15 Jul 2009, Stephan von Krawczynski wrote: > Jul 15 03:01:26 backup kernel: swapper: page allocation failure. order:1, mode:0x4020 > Jul 15 03:01:26 backup kernel: Pid: 0, comm: swapper Not tainted 2.6.30.1 #4 > Jul 15 03:01:26 backup kernel: Call Trace: > Jul 15 03:01:26 backup kernel: [] ? __alloc_pages_internal+0x3df/0x3ff > Jul 15 03:01:26 backup kernel: [] ? __slab_alloc+0x175/0x4ba > Jul 15 03:01:26 backup kernel: [] ? __netdev_alloc_skb+0x15/0x2f > Jul 15 03:01:26 backup kernel: [] ? __netdev_alloc_skb+0x15/0x2f > Jul 15 03:01:26 backup kernel: [] ? __kmalloc_track_caller+0x8f/0xb6 > Jul 15 03:01:26 backup kernel: [] ? __alloc_skb+0x61/0x12f > Jul 15 03:01:26 backup kernel: [] ? __netdev_alloc_skb+0x15/0x2f > Jul 15 03:01:26 backup kernel: [] ? e1000_alloc_rx_buffers+0x8c/0x248 [e1000e] > Jul 15 03:01:26 backup kernel: [] ? e1000_clean_rx_irq+0x2a2/0x2db [e1000e] > Jul 15 03:01:26 backup kernel: [] ? e1000_clean+0x70/0x219 [e1000e] > Jul 15 03:01:27 backup kernel: [] ? net_rx_action+0x69/0x11f > Jul 15 03:01:27 backup kernel: [] ? __do_softirq+0x66/0xf7 > Jul 15 03:01:27 backup kernel: [] ? call_softirq+0x1c/0x28 > Jul 15 03:01:27 backup kernel: [] ? do_softirq+0x2c/0x68 > Jul 15 03:01:27 backup kernel: [] ? do_IRQ+0xa9/0xbf > Jul 15 03:01:27 backup kernel: [] ? ret_from_intr+0x0/0xa > Jul 15 03:01:27 backup kernel: [] ? mwait_idle+0x6e/0x73 > Jul 15 03:01:27 backup kernel: [] ? mwait_idle+0x6e/0x73 > Jul 15 03:01:27 backup kernel: [] ? cpu_idle+0x40/0x7c > Jul 15 03:01:28 backup kernel: [] ? start_kernel+0x31e/0x32a > Jul 15 03:01:28 backup kernel: [] ? x86_64_start_kernel+0xe5/0xeb > Jul 15 03:01:28 backup kernel: DMA per-cpu: > Jul 15 03:01:28 backup kernel: CPU 0: hi: 0, btch: 1 usd: 0 > Jul 15 03:01:28 backup kernel: CPU 1: hi: 0, btch: 1 usd: 0 > Jul 15 03:01:28 backup kernel: CPU 2: hi: 0, btch: 1 usd: 0 > Jul 15 03:01:28 backup kernel: CPU 3: hi: 0, btch: 1 usd: 0 > Jul 15 03:01:28 backup kernel: DMA32 per-cpu: > Jul 15 03:01:28 backup kernel: CPU 0: hi: 186, btch: 31 usd: 157 > Jul 15 03:01:28 backup kernel: CPU 1: hi: 186, btch: 31 usd: 62 > Jul 15 03:01:28 backup kernel: CPU 2: hi: 186, btch: 31 usd: 84 > Jul 15 03:01:28 backup kernel: CPU 3: hi: 186, btch: 31 usd: 41 > Jul 15 03:01:28 backup kernel: Normal per-cpu: > Jul 15 03:01:28 backup kernel: CPU 0: hi: 186, btch: 31 usd: 175 > Jul 15 03:01:28 backup kernel: CPU 1: hi: 186, btch: 31 usd: 73 > Jul 15 03:01:28 backup kernel: CPU 2: hi: 186, btch: 31 usd: 33 > Jul 15 03:01:28 backup kernel: CPU 3: hi: 186, btch: 31 usd: 56 > Jul 15 03:01:28 backup kernel: Active_anon:32502 active_file:111663 inactive_anon:8167 > Jul 15 03:01:28 backup kernel: inactive_file:1332510 unevictable:0 dirty:39449 writeback:1586 unstable:0 > Jul 15 03:01:28 backup kernel: free:10034 slab:546449 mapped:1841 pagetables:1189 bounce:0 > Jul 15 03:01:28 backup kernel: DMA free:11704kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:10752kB pages_scanned:0 all_unreclaimable? yes > Jul 15 03:01:28 backup kernel: lowmem_reserve[]: 0 3767 8059 8059 > Jul 15 03:01:28 backup kernel: DMA32 free:22052kB min:5364kB low:6704kB high:8044kB active_anon:19216kB inactive_anon:4032kB active_file:113380kB inactive_file:2196508kB unevictable:0kB present:3857440kB pages_scanned:0 all_unreclaimable? no > Jul 15 03:01:28 backup kernel: lowmem_reserve[]: 0 0 4292 4292 > Jul 15 03:01:28 backup kernel: Normal free:6380kB min:6112kB low:7640kB high:9168kB active_anon:110792kB inactive_anon:28636kB active_file:333272kB inactive_file:3133532kB unevictable:0kB present:4395520kB pages_scanned:0 all_unreclaimable? no > Jul 15 03:01:28 backup kernel: lowmem_reserve[]: 0 0 0 0 > Jul 15 03:01:28 backup kernel: DMA: 6*4kB 6*8kB 3*16kB 2*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 0*2048kB 2*4096kB = 11704kB > Jul 15 03:01:28 backup kernel: DMA32: 5283*4kB 93*8kB 2*16kB 0*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 22036kB > Jul 15 03:01:28 backup kernel: Normal: 1310*4kB 99*8kB 6*16kB 0*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6256kB > Jul 15 03:01:29 backup kernel: 1444268 total pagecache pages > Jul 15 03:01:29 backup kernel: 34 pages in swap cache > Jul 15 03:01:29 backup kernel: Swap cache stats: add 118, delete 84, find 0/2 > Jul 15 03:01:29 backup kernel: Free swap = 2104080kB > Jul 15 03:01:29 backup kernel: Total swap = 2104488kB > I added Justin Piszcz to the cc since he was having the same problem as described in http://bugzilla.kernel.org/show_bug.cgi?id=13648. He was unable to get slabtop -o output when this was happening, though, so maybe you could grab a snapshot of that when you get these failures? It will help us figure out what cache the slab leak is in (assuming there is one, >1G of slab on this machine is egregious). Justin, were you using e1000e in your bug report? If you have some additional time, it would also be helpful to get a bisection of when the problem started occurring (it appears to be sometime between 2.6.29 and 2.6.30). -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org