2.6.23-rc4-mm1 memory controller BUG_ON()

2.6.23-rc4-mm1 memory controller BUG_ON()

[Dave Hansen]

Looks like somebody is holding a lock while trying to do a
mem_container_charge(), and the mem_container_charge() call is doing an
allocation.  Naughty.

I'm digging into it a bit more, but thought I'd report it, first.

.config: http://sr71.net/~dave/linux/memory-controller-bug.config

BUG: sleeping function called from invalid context at /home/dave/work/linux/2.6/23/rc4/mm1/lxc/mm/slab.c:3052
in_atomic():1, irqs_disabled():0
 [<c01029c1>] show_trace_log_lvl+0x19/0x2e
 [<c01029e8>] show_trace+0x12/0x14
 [<c0102ad3>] dump_stack+0x13/0x15
 [<c010f223>] __might_sleep+0xe4/0xea
 [<c014cdc0>] kmem_cache_alloc+0x25/0xae
 [<c014e1dd>] mem_container_charge+0xc9/0x2cd
 [<c014e403>] mem_container_cache_charge+0x22/0x28
 [<c0131839>] add_to_page_cache+0x35/0xd7
 [<c01318f0>] add_to_page_cache_lru+0x15/0x29
 [<c0131c43>] find_or_create_page+0x75/0x93
 [<c016cc71>] grow_dev_page+0x32/0x125
 [<c016ce15>] grow_buffers+0xb1/0xd4
 [<c016ceef>] __getblk_slow+0xb7/0xcf
 [<c016d2b9>] __getblk+0x44/0x4f
 [<c018c721>] ext3_getblk+0xca/0x19c
 [<c018fc48>] ext3_find_entry+0x127/0x325
 [<c0190063>] ext3_lookup+0x2c/0xe1
 [<c015635a>] real_lookup+0x54/0xc5
 [<c01565bd>] do_lookup+0x59/0xa0
 [<c0156824>] __link_path_walk+0x220/0xa4f
 [<c0157094>] link_path_walk+0x41/0xa5
 [<c0157110>] path_walk+0x18/0x1a
 [<c01573de>] do_path_lookup+0x165/0x182
 [<c01574a0>] __path_lookup_intent_open+0x44/0x75
 [<c01574f2>] path_lookup_open+0x21/0x27
 [<c0157c5a>] open_namei+0x7f/0x4c4
 [<c014f6c0>] do_filp_open+0x26/0x3b
 [<c014f975>] do_sys_open+0x43/0xc7
 [<c014fa13>] sys_open+0x1a/0x1c
 [<c010010e>] init_post+0x45/0xe7
 [<c03f6ab6>] kernel_init+0x8a/0x8e
 [<c010287b>] kernel_thread_helper+0x7/0x10
 =======================


-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.23-rc4-mm1 memory controller BUG_ON()

[Dave Hansen]

On Thu, 2007-09-13 at 12:34 -0700, Dave Hansen wrote:
> Looks like somebody is holding a lock while trying to do a
> mem_container_charge(), and the mem_container_charge() call is doing an
> allocation.  Naughty.
> 
> I'm digging into it a bit more, but thought I'd report it, first.
> 
> .config: http://sr71.net/~dave/linux/memory-controller-bug.config

I'm now thinking this is because the add_to_page_cache() functions have
a gfp_mask passed in, and the mem_container_charge() functions don't
take that mask.  So, even if the add_to_page_cache() user specified !
__GFP_WAIT, the mem_container_charge() function can sleep on its
kmalloc.

I'll try passing gfp_flags through to it and see what happens.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.23-rc4-mm1 memory controller BUG_ON()

[Balbir Singh]

Dave Hansen wrote:
> Looks like somebody is holding a lock while trying to do a
> mem_container_charge(), and the mem_container_charge() call is doing an
> allocation.  Naughty.
> 
> I'm digging into it a bit more, but thought I'd report it, first.
> 

Hi, Dave,

Thanks for reporting this. I sent out a patch to fix this problem
(suggested by Nick Piggin). The patch is available at

http://lkml.org/lkml/2007/9/12/113

Could you try the patch and check if the problem goes away?

Any pointers on how to reproduce the problem would be extremely useful.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.23-rc4-mm1 memory controller BUG_ON()

[Balbir Singh]

Dave Hansen wrote:
> On Thu, 2007-09-13 at 12:34 -0700, Dave Hansen wrote:
>> Looks like somebody is holding a lock while trying to do a
>> mem_container_charge(), and the mem_container_charge() call is doing an
>> allocation.  Naughty.
>>
>> I'm digging into it a bit more, but thought I'd report it, first.
>>
>> .config: http://sr71.net/~dave/linux/memory-controller-bug.config
> 
> I'm now thinking this is because the add_to_page_cache() functions have
> a gfp_mask passed in, and the mem_container_charge() functions don't
> take that mask.  So, even if the add_to_page_cache() user specified !
> __GFP_WAIT, the mem_container_charge() function can sleep on its
> kmalloc.
> 
> I'll try passing gfp_flags through to it and see what happens.
> 

Please see my patch at

http://lkml.org/lkml/2007/9/12/113 and some more details as a reply
to your earlier email.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.23-rc4-mm1 memory controller BUG_ON()

[Dave Hansen]

On Fri, 2007-09-14 at 01:51 +0530, Balbir Singh wrote:
> Dave Hansen wrote:
> > Looks like somebody is holding a lock while trying to do a
> > mem_container_charge(), and the mem_container_charge() call is doing
> an
> > allocation.  Naughty.
> > 
> > I'm digging into it a bit more, but thought I'd report it, first.
> > 
> 
> Hi, Dave,
> 
> Thanks for reporting this. I sent out a patch to fix this problem
> (suggested by Nick Piggin). The patch is available at
> 
> http://lkml.org/lkml/2007/9/12/113
> 
> Could you try the patch and check if the problem goes away? 

Balbir and I had a chat about this on IRC.  Those patches don't seem to
fix it.  But, I'm getting Balbir hooked up with the kvm instance that I
ran this in along with my .config.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.23-rc4-mm1 memory controller BUG_ON()

[Balbir Singh]

On Thu, Sep 13, 2007 at 01:54:09PM -0700, Dave Hansen wrote:
> On Fri, 2007-09-14 at 01:51 +0530, Balbir Singh wrote:
> > Dave Hansen wrote:
> > > Looks like somebody is holding a lock while trying to do a
> > > mem_container_charge(), and the mem_container_charge() call is doing
> > an
> > > allocation.  Naughty.
> > > 
> > > I'm digging into it a bit more, but thought I'd report it, first.
> > > 
> > 
> > Hi, Dave,
> > 
> > Thanks for reporting this. I sent out a patch to fix this problem
> > (suggested by Nick Piggin). The patch is available at
> > 
> > http://lkml.org/lkml/2007/9/12/113
> > 
> > Could you try the patch and check if the problem goes away? 
> 
> Balbir and I had a chat about this on IRC.  Those patches don't seem to
> fix it.  But, I'm getting Balbir hooked up with the kvm instance that I
> ran this in along with my .config.
>

Hi, Dave,

Here's a fix for the problem, it works nicely on the qemu setup that
you helped me with. I am also testing on another box. Could you please
verify if the patch helps?

Description of the patch
------------------------
 
Move mem_controller_cache_charge() above radix_tree_preload().
radix_tree_preload() disables preemption, even though the gfp_mask passed
contains __GFP_WAIT, we cannot really do __GFP_WAIT allocations, thus we hit
a BUG_ON() in kmem_cache_alloc().

This patch moves mem_controller_cache_charge() to above radix_tree_preload()
for cache charging.

Signed-off-by: Balbir Singh <balbir@linux.vnet.ibm.com>
---

 mm/filemap.c    |   13 ++++++-------
 mm/swap_state.c |   13 +++++++------
 2 files changed, 13 insertions(+), 13 deletions(-)

diff -puN mm/filemap.c~memory-controller-make-charging-gfpmask-aware-fixes mm/filemap.c
--- linux-2.6.23-rc4/mm/filemap.c~memory-controller-make-charging-gfpmask-aware-fixes	2007-09-14 13:20:44.000000000 +0530
+++ linux-2.6.23-rc4-balbir/mm/filemap.c	2007-09-14 13:23:50.000000000 +0530
@@ -441,14 +441,12 @@ int filemap_write_and_wait_range(struct 
 int add_to_page_cache(struct page *page, struct address_space *mapping,
 		pgoff_t offset, gfp_t gfp_mask)
 {
-	int error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM);
+	int error = mem_container_cache_charge(page, current->mm, gfp_mask);
+	if (error)
+		goto out;
 
+	error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM);
 	if (error == 0) {
-
-		error = mem_container_cache_charge(page, current->mm, gfp_mask);
-		if (error)
-			goto out;
-
 		write_lock_irq(&mapping->tree_lock);
 		error = radix_tree_insert(&mapping->page_tree, offset, page);
 		if (!error) {
@@ -463,7 +461,8 @@ int add_to_page_cache(struct page *page,
 
 		write_unlock_irq(&mapping->tree_lock);
 		radix_tree_preload_end();
-	}
+	} else
+		mem_container_uncharge_page(page);
 out:
 	return error;
 }
diff -puN mm/swap_state.c~memory-controller-make-charging-gfpmask-aware-fixes mm/swap_state.c
--- linux-2.6.23-rc4/mm/swap_state.c~memory-controller-make-charging-gfpmask-aware-fixes	2007-09-14 13:20:44.000000000 +0530
+++ linux-2.6.23-rc4-balbir/mm/swap_state.c	2007-09-14 13:26:14.000000000 +0530
@@ -78,13 +78,13 @@ static int __add_to_swap_cache(struct pa
 	BUG_ON(!PageLocked(page));
 	BUG_ON(PageSwapCache(page));
 	BUG_ON(PagePrivate(page));
-	error = radix_tree_preload(gfp_mask);
-	if (!error) {
 
-		error = mem_container_cache_charge(page, current->mm, gfp_mask);
-		if (error)
-			goto out;
+	error = mem_container_cache_charge(page, current->mm, gfp_mask);
+	if (error)
+		goto out;
 
+	error = radix_tree_preload(gfp_mask);
+	if (!error) {
 		write_lock_irq(&swapper_space.tree_lock);
 		error = radix_tree_insert(&swapper_space.page_tree,
 						entry.val, page);
@@ -99,7 +99,8 @@ static int __add_to_swap_cache(struct pa
 
 		write_unlock_irq(&swapper_space.tree_lock);
 		radix_tree_preload_end();
-	}
+	} else
+		mem_container_uncharge_page(page);
 out:
 	return error;
 }
_

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>