2.6.21-rc7-mm1 on test.kernel.org

2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

An amd64 machine is crashing badly.

http://test.kernel.org/abat/84767/debug/console.log

VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 308k freed
INIT: version 2.86 booting
Bad page state in process 'init'
page:ffff81007e492628 flags:0x0100000000000000 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:

Call Trace:
 [<ffffffff80250d3c>] bad_page+0x74/0x10d
 [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
 [<ffffffff802531cb>] free_hot_page+0xb/0xd
 [<ffffffff8025a9b7>] free_pgd_range+0x274/0x467
 [<ffffffff8025ac2a>] free_pgtables+0x80/0x8f
 [<ffffffff8026139c>] exit_mmap+0x90/0x11a
 [<ffffffff802270dd>] mmput+0x29/0x98
Bad page state in process 'hotplug'
page:ffff81017e458bb0 flags:0x0a00000000000000 mapping:0000000000000000 mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:

Call Trace:
 [<ffffffff80250d3c>] bad_page+0x74/0x10d
 [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
 [<ffffffff802531cb>] free_hot_page+0xb/0xd
 [<ffffffff80227074>] __mmdrop+0x68/0xa8
 [<ffffffff80222911>] schedule_tail+0x48/0x86
 [<ffffffff8020960c>] ret_from_fork+0xc/0x25


So free_pgd_range() is freeing a refcount=1 page.  Can anyone see what
might be causing this?  The quicklist code impacts this area more than
anything else..

Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
be wonderful.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
> be wonderful.

Cannot compile a UP x86_64 kernel

  LD      arch/x86_64/kernel/pcspeaker.o
  LD      arch/x86_64/kernel/built-in.o
  AS      arch/x86_64/kernel/head.o
  CC      arch/x86_64/kernel/head64.o
arch/x86_64/kernel/head64.c: In function 'x86_64_start_kernel':
arch/x86_64/kernel/head64.c:70: error: size of array 'type name' is 
negative
make[1]: *** [arch/x86_64/kernel/head64.o] Error 1
make: *** [arch/x86_64/kernel] Error 2
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 13:21:25 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> On Tue, 24 Apr 2007, Andrew Morton wrote:
> 
> > Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
> > be wonderful.
> 
> Cannot compile a UP x86_64 kernel
> 
>   LD      arch/x86_64/kernel/pcspeaker.o
>   LD      arch/x86_64/kernel/built-in.o
>   AS      arch/x86_64/kernel/head.o
>   CC      arch/x86_64/kernel/head64.o
> arch/x86_64/kernel/head64.c: In function 'x86_64_start_kernel':
> arch/x86_64/kernel/head64.c:70: error: size of array 'type name' is 
> negative

That's a BUILD_BUG_ON.  Check the source...


	/*
	 * Make sure kernel is aligned to 2MB address. Catching it at compile
	 * time is better. Change your config file and compile the kernel
	 * for a 2MB aligned address (CONFIG_PHYSICAL_START)
	 */
	BUILD_BUG_ON(CONFIG_PHYSICAL_START & (__KERNEL_ALIGN - 1));

You need to change your CONFIG_PHYSICAL_START so it is a multiple of 2MB.


(The test.kernel.org config uses SMP?)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> On Tue, 24 Apr 2007 13:21:25 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
> 
> > On Tue, 24 Apr 2007, Andrew Morton wrote:
> > 
> > > Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
> > > be wonderful.
> > 
> > Cannot compile a UP x86_64 kernel
> > 
> >   LD      arch/x86_64/kernel/pcspeaker.o
> >   LD      arch/x86_64/kernel/built-in.o
> >   AS      arch/x86_64/kernel/head.o
> >   CC      arch/x86_64/kernel/head64.o
> > arch/x86_64/kernel/head64.c: In function 'x86_64_start_kernel':
> > arch/x86_64/kernel/head64.c:70: error: size of array 'type name' is 
> > negative
> 
> That's a BUILD_BUG_ON.  Check the source...
> 
> 
> 	/*
> 	 * Make sure kernel is aligned to 2MB address. Catching it at compile
> 	 * time is better. Change your config file and compile the kernel
> 	 * for a 2MB aligned address (CONFIG_PHYSICAL_START)
> 	 */
> 	BUILD_BUG_ON(CONFIG_PHYSICAL_START & (__KERNEL_ALIGN - 1));
> 
> You need to change your CONFIG_PHYSICAL_START so it is a multiple of 2MB.

???? My old .config wont work anymore.

> (The test.kernel.org config uses SMP?)

Also broke with SMP.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 13:32:32 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> On Tue, 24 Apr 2007, Andrew Morton wrote:
> 
> > On Tue, 24 Apr 2007 13:21:25 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
> > 
> > > On Tue, 24 Apr 2007, Andrew Morton wrote:
> > > 
> > > > Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
> > > > be wonderful.
> > > 
> > > Cannot compile a UP x86_64 kernel
> > > 
> > >   LD      arch/x86_64/kernel/pcspeaker.o
> > >   LD      arch/x86_64/kernel/built-in.o
> > >   AS      arch/x86_64/kernel/head.o
> > >   CC      arch/x86_64/kernel/head64.o
> > > arch/x86_64/kernel/head64.c: In function 'x86_64_start_kernel':
> > > arch/x86_64/kernel/head64.c:70: error: size of array 'type name' is 
> > > negative
> > 
> > That's a BUILD_BUG_ON.  Check the source...
> > 
> > 
> > 	/*
> > 	 * Make sure kernel is aligned to 2MB address. Catching it at compile
> > 	 * time is better. Change your config file and compile the kernel
> > 	 * for a 2MB aligned address (CONFIG_PHYSICAL_START)
> > 	 */
> > 	BUILD_BUG_ON(CONFIG_PHYSICAL_START & (__KERNEL_ALIGN - 1));
> > 
> > You need to change your CONFIG_PHYSICAL_START so it is a multiple of 2MB.
> 
> ???? My old .config wont work anymore.

kexec requries 2MB alignment.  I think your old config would have just
crashed.  Now you got told about it at compile time.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> kexec requries 2MB alignment.  I think your old config would have just
> crashed.  Now you got told about it at compile time.

Old config worked great so far.

I compiled and booted 2.6.21-rc7-mm1 just fine. Nothing special apart from 
the usual problem with serial not accepting characters that we had for 
awhile now.

Could we get a .config?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Christoph Lameter wrote:
> On Tue, 24 Apr 2007, Andrew Morton wrote:
> 
>> kexec requries 2MB alignment.  I think your old config would have just
>> crashed.  Now you got told about it at compile time.
> 
> Old config worked great so far.
> 
> I compiled and booted 2.6.21-rc7-mm1 just fine. Nothing special apart from 
> the usual problem with serial not accepting characters that we had for 
> awhile now.
> 
> Could we get a .config?

http://test.kernel.org/abat/84767/build/dotconfig

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 13:52:44 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> On Tue, 24 Apr 2007, Andrew Morton wrote:
> 
> > kexec requries 2MB alignment.  I think your old config would have just
> > crashed.  Now you got told about it at compile time.
> 
> Old config worked great so far.

Only if you don't use sata, scsi or wireless.  They break oldconfig
regularly.

> I compiled and booted 2.6.21-rc7-mm1 just fine.

uh.  me too :(

> Nothing special apart from 
> the usual problem with serial not accepting characters that we had for 
> awhile now.

I wasn't aware of that one.

> Could we get a .config?

test.kernel.org configs are subtly hidden on the front page.  Go to
test.kernel.org, click on the "amd64" or "numaq" links in the title row
there.

The offending machine is elm3b6.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> > Could we get a .config?
> 
> test.kernel.org configs are subtly hidden on the front page.  Go to
> test.kernel.org, click on the "amd64" or "numaq" links in the title row
> there.
> 
> The offending machine is elm3b6.

My x86_64 box boots fine with the indicated .config. Hardware related?


 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> On Tue, 24 Apr 2007, Andrew Morton wrote:
> 
> > > Could we get a .config?
> > 
> > test.kernel.org configs are subtly hidden on the front page.  Go to
> > test.kernel.org, click on the "amd64" or "numaq" links in the title row
> > there.
> > 
> > The offending machine is elm3b6.
> 
> My x86_64 box boots fine with the indicated .config.

So do both of mine.

> Hardware related?

Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Andrew Morton wrote:
> On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
> 
>> On Tue, 24 Apr 2007, Andrew Morton wrote:
>>
>>>> Could we get a .config?
>>> test.kernel.org configs are subtly hidden on the front page.  Go to
>>> test.kernel.org, click on the "amd64" or "numaq" links in the title row
>>> there.
>>>
>>> The offending machine is elm3b6.
>> My x86_64 box boots fine with the indicated .config.
> 
> So do both of mine.
> 
>> Hardware related?
> 
> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?

Yep real NUMA box.  Will try and get hold of the box to test.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Andy Whitcroft wrote:
> Andrew Morton wrote:
>> On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
>>
>>> On Tue, 24 Apr 2007, Andrew Morton wrote:
>>>
>>>>> Could we get a .config?
>>>> test.kernel.org configs are subtly hidden on the front page.  Go to
>>>> test.kernel.org, click on the "amd64" or "numaq" links in the title row
>>>> there.
>>>>
>>>> The offending machine is elm3b6.
>>> My x86_64 box boots fine with the indicated .config.
>> So do both of mine.
>>
>>> Hardware related?
>> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?
> 
> Yep real NUMA box.  Will try and get hold of the box to test.
> 
> -apw

git bisect points to:

    quicklist-support-for-x86_64

Reverting just this patch sorts this problem on the x86_64.

-apw


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Badari Pulavarty]

On Wed, 2007-04-25 at 01:16 +0100, Andy Whitcroft wrote:
> Andy Whitcroft wrote:
> > Andrew Morton wrote:
> >> On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
> >>
> >>> On Tue, 24 Apr 2007, Andrew Morton wrote:
> >>>
> >>>>> Could we get a .config?
> >>>> test.kernel.org configs are subtly hidden on the front page.  Go to
> >>>> test.kernel.org, click on the "amd64" or "numaq" links in the title row
> >>>> there.
> >>>>
> >>>> The offending machine is elm3b6.
> >>> My x86_64 box boots fine with the indicated .config.
> >> So do both of mine.
> >>
> >>> Hardware related?
> >> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?
> > 
> > Yep real NUMA box.  Will try and get hold of the box to test.
> > 
> > -apw
> 
> git bisect points to:
> 
>     quicklist-support-for-x86_64
> 
> Reverting just this patch sorts this problem on the x86_64.

Hmm.. I narrowed it further down to ..

quicklists-for-page-table-pages-avoid-useless-virt_to_page-
conversion.patch

Andy, can you try backing out only this and enable QUICK_LIST
on your machine ?

Thanks,
Badari

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Badari Pulavarty wrote:
> On Wed, 2007-04-25 at 01:16 +0100, Andy Whitcroft wrote:
>> Andy Whitcroft wrote:
>>> Andrew Morton wrote:
>>>> On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
>>>>
>>>>> On Tue, 24 Apr 2007, Andrew Morton wrote:
>>>>>
>>>>>>> Could we get a .config?
>>>>>> test.kernel.org configs are subtly hidden on the front page.  Go to
>>>>>> test.kernel.org, click on the "amd64" or "numaq" links in the title row
>>>>>> there.
>>>>>>
>>>>>> The offending machine is elm3b6.
>>>>> My x86_64 box boots fine with the indicated .config.
>>>> So do both of mine.
>>>>
>>>>> Hardware related?
>>>> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?
>>> Yep real NUMA box.  Will try and get hold of the box to test.
>>>
>>> -apw
>> git bisect points to:
>>
>>     quicklist-support-for-x86_64
>>
>> Reverting just this patch sorts this problem on the x86_64.
> 
> Hmm.. I narrowed it further down to ..
> 
> quicklists-for-page-table-pages-avoid-useless-virt_to_page-
> conversion.patch
> 
> Andy, can you try backing out only this and enable QUICK_LIST
> on your machine ?

Yep confirmed that reverting that one is enough to fix this machine.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Badari Pulavarty]

On Wed, 2007-04-25 at 01:44 +0100, Andy Whitcroft wrote:
> Badari Pulavarty wrote:
> > On Wed, 2007-04-25 at 01:16 +0100, Andy Whitcroft wrote:
> >> Andy Whitcroft wrote:
> >>> Andrew Morton wrote:
> >>>> On Tue, 24 Apr 2007 14:30:16 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:
> >>>>
> >>>>> On Tue, 24 Apr 2007, Andrew Morton wrote:
> >>>>>
> >>>>>>> Could we get a .config?
> >>>>>> test.kernel.org configs are subtly hidden on the front page.  Go to
> >>>>>> test.kernel.org, click on the "amd64" or "numaq" links in the title row
> >>>>>> there.
> >>>>>>
> >>>>>> The offending machine is elm3b6.
> >>>>> My x86_64 box boots fine with the indicated .config.
> >>>> So do both of mine.
> >>>>
> >>>>> Hardware related?
> >>>> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?
> >>> Yep real NUMA box.  Will try and get hold of the box to test.
> >>>
> >>> -apw
> >> git bisect points to:
> >>
> >>     quicklist-support-for-x86_64
> >>
> >> Reverting just this patch sorts this problem on the x86_64.
> > 
> > Hmm.. I narrowed it further down to ..
> > 
> > quicklists-for-page-table-pages-avoid-useless-virt_to_page-
> > conversion.patch
> > 
> > Andy, can you try backing out only this and enable QUICK_LIST
> > on your machine ?
> 
> Yep confirmed that reverting that one is enough to fix this machine.
> 
> -apw


Here is the patch to fix it (against -mm) ? 
Works on my machine :)

Thanks,
Badari

Signed-off-by: Badari Pulavarty <pbadari@us.ibm.com>
 include/linux/quicklist.h |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-2.6.21-rc7/include/linux/quicklist.h
===================================================================
--- linux-2.6.21-rc7.orig/include/linux/quicklist.h	2007-04-24 19:10:09.000000000 -0700
+++ linux-2.6.21-rc7/include/linux/quicklist.h	2007-04-24 19:10:57.000000000 -0700
@@ -61,7 +61,8 @@ static inline void __quicklist_free(int 
 	if (unlikely(nid != numa_node_id())) {
 		if (dtor)
 			dtor(p);
-		free_hot_page(page);
+		if (put_page_testzero(page))
+			free_hot_page(page);
 		return;
 	}
 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Badari Pulavarty wrote:

> quicklists-for-page-table-pages-avoid-useless-virt_to_page-
> conversion.patch
> 
> Andy, can you try backing out only this and enable QUICK_LIST
> on your machine ?

Ahh. Right..... The free that we switched to there to avoid the 
virt_to_page conversion does not decrement the refcount and thus
is not equivalent.

Does this patch fix it?

---
 include/linux/quicklist.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.21-rc7-mm1/include/linux/quicklist.h
===================================================================
--- linux-2.6.21-rc7-mm1.orig/include/linux/quicklist.h	2007-04-24 23:35:11.000000000 -0700
+++ linux-2.6.21-rc7-mm1/include/linux/quicklist.h	2007-04-24 23:35:59.000000000 -0700
@@ -61,7 +61,7 @@ static inline void __quicklist_free(int 
 	if (unlikely(nid != numa_node_id())) {
 		if (dtor)
 			dtor(p);
-		free_hot_page(page);
+		__free_page(page);
 		return;
 	}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Christoph Lameter wrote:
> On Tue, 24 Apr 2007, Badari Pulavarty wrote:
> 
>> quicklists-for-page-table-pages-avoid-useless-virt_to_page-
>> conversion.patch
>>
>> Andy, can you try backing out only this and enable QUICK_LIST
>> on your machine ?
> 
> Ahh. Right..... The free that we switched to there to avoid the 
> virt_to_page conversion does not decrement the refcount and thus
> is not equivalent.
> 
> Does this patch fix it?
> 
> ---
>  include/linux/quicklist.h |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> Index: linux-2.6.21-rc7-mm1/include/linux/quicklist.h
> ===================================================================
> --- linux-2.6.21-rc7-mm1.orig/include/linux/quicklist.h	2007-04-24 23:35:11.000000000 -0700
> +++ linux-2.6.21-rc7-mm1/include/linux/quicklist.h	2007-04-24 23:35:59.000000000 -0700
> @@ -61,7 +61,7 @@ static inline void __quicklist_free(int 
>  	if (unlikely(nid != numa_node_id())) {
>  		if (dtor)
>  			dtor(p);
> -		free_hot_page(page);
> +		__free_page(page);
>  		return;
>  	}

Confirmed, this fixes the machine.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Wed, 25 Apr 2007 09:21:36 +0100 Andy Whitcroft <apw@shadowen.org> wrote:

> >  	if (unlikely(nid != numa_node_id())) {
> >  		if (dtor)
> >  			dtor(p);
> > -		free_hot_page(page);
> > +		__free_page(page);
> >  		return;
> >  	}
> 
> Confirmed, this fixes the machine.

OK, thanks guys - another one for the hot-fixes directory.

Do we know where the extra refcount on that page is coming from?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Wed, 25 Apr 2007, Andrew Morton wrote:

> Do we know where the extra refcount on that page is coming from?

>From the allocation of the page via quicklist_alloc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> Well it's AMD64, presumably real NUMA.  Maybe try numa=fake=4?

That is on by default on my box.
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Badari Pulavarty]

On Tue, 2007-04-24 at 13:06 -0700, Andrew Morton wrote:
> An amd64 machine is crashing badly.
> 
> http://test.kernel.org/abat/84767/debug/console.log
> 
> VFS: Mounted root (ext3 filesystem) readonly.
> Freeing unused kernel memory: 308k freed
> INIT: version 2.86 booting
> Bad page state in process 'init'
> page:ffff81007e492628 flags:0x0100000000000000 mapping:0000000000000000 mapcount:0 count:1
> Trying to fix it up, but a reboot is needed
> Backtrace:
> 
> Call Trace:
>  [<ffffffff80250d3c>] bad_page+0x74/0x10d
>  [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
>  [<ffffffff802531cb>] free_hot_page+0xb/0xd
>  [<ffffffff8025a9b7>] free_pgd_range+0x274/0x467
>  [<ffffffff8025ac2a>] free_pgtables+0x80/0x8f
>  [<ffffffff8026139c>] exit_mmap+0x90/0x11a
>  [<ffffffff802270dd>] mmput+0x29/0x98
> Bad page state in process 'hotplug'
> page:ffff81017e458bb0 flags:0x0a00000000000000 mapping:0000000000000000 mapcount:0 count:1
> Trying to fix it up, but a reboot is needed
> Backtrace:
> 
> Call Trace:
>  [<ffffffff80250d3c>] bad_page+0x74/0x10d
>  [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
>  [<ffffffff802531cb>] free_hot_page+0xb/0xd
>  [<ffffffff80227074>] __mmdrop+0x68/0xa8
>  [<ffffffff80222911>] schedule_tail+0x48/0x86
>  [<ffffffff8020960c>] ret_from_fork+0xc/0x25
> 
> 
> So free_pgd_range() is freeing a refcount=1 page.  Can anyone see what
> might be causing this?  The quicklist code impacts this area more than
> anything else..
> 
> Naturally, I can't reproduce it (no amd64 boxen).  A bisection search would
> be wonderful.


I am able to reproduce this on my amd64 box also, I will take a look ..
but feel free to beat me to it :)

Bad page state in process 'boot'
page:ffff8101df9550a0 flags:0x0e00000000000000 mapping:0000000000000000
mapcount:0 count:1
Trying to fix it up, but a reboot is needed
Backtrace:

Call Trace:
 [<ffffffff8025dbda>] filemap_fault+0x1ba/0x420
 [<ffffffff8025fdf0>] bad_page+0x70/0x120
 [<ffffffff80260c26>] free_hot_cold_page+0x1b6/0x1d0
 [<ffffffff80260cab>] free_hot_page+0xb/0x10
 [<ffffffff8026ec0d>] free_pgd_range+0x4dd/0x4f0
 [<ffffffff8026ef09>] free_pgtables+0xa9/0xe0
 [<ffffffff802701a6>] exit_mmap+0x96/0x130
 [<ffffffff8022d974>] mmput+0x44/0xc0
 [<ffffffff80231f10>] exit_mm+0x90/0x100
 [<ffffffff80233821>] do_exit+0x151/0x970
 [<ffffffff80234077>] do_group_exit+0x37/0x90
 [<ffffffff802340e2>] sys_exit_group+0x12/0x20
 [<ffffffff80209bae>] system_call+0x7e/0x83




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

Andy, I'm looking at the power4 build:

http://test.kernel.org/abat/84751/debug/test.log.0

which has

  LD      init/built-in.o
  LD      .tmp_vmlinux1
init/built-in.o(.init.text+0x32e4): In function `.rd_load_image':
: undefined reference to `.__kmalloc_size_too_large'
fs/built-in.o(.text+0xa60f0): In function `.ext3_fill_super':
: undefined reference to `.__kmalloc_size_too_large'
fs/built-in.o(.text+0xbe934): In function `.ext2_fill_super':
: undefined reference to `.__kmalloc_size_too_large'
fs/built-in.o(.text+0xf3370): In function `.nfs4_proc_lookup':

something has gone stupid with kmalloc there, and I cannot reproduce it
with my compiler and with your (very old) .config at
http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/power4

So I'm a bit stumped.  Does autotest just do `yes "" | make oldconfig' or
what?  When I do that, I get SLUB, but no compile errors.

And do you know what compiler version is being used there?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Andrew Morton wrote:
> Andy, I'm looking at the power4 build:
> 
> http://test.kernel.org/abat/84751/debug/test.log.0
> 
> which has
> 
>   LD      init/built-in.o
>   LD      .tmp_vmlinux1
> init/built-in.o(.init.text+0x32e4): In function `.rd_load_image':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xa60f0): In function `.ext3_fill_super':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xbe934): In function `.ext2_fill_super':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xf3370): In function `.nfs4_proc_lookup':
> 
> something has gone stupid with kmalloc there, and I cannot reproduce it
> with my compiler and with your (very old) .config at
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/power4
> 
> So I'm a bit stumped.  Does autotest just do `yes "" | make oldconfig' or
> what?  When I do that, I get SLUB, but no compile errors.

Yes, exactly that.

> 
> And do you know what compiler version is being used there?

gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)

I am bisecting for the bad page bug right now, will let you know where
it points.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andy Whitcroft]

Andy Whitcroft wrote:
> Andrew Morton wrote:
>> Andy, I'm looking at the power4 build:
>>
>> http://test.kernel.org/abat/84751/debug/test.log.0
>>
>> which has
>>
>>   LD      init/built-in.o
>>   LD      .tmp_vmlinux1
>> init/built-in.o(.init.text+0x32e4): In function `.rd_load_image':
>> : undefined reference to `.__kmalloc_size_too_large'
>> fs/built-in.o(.text+0xa60f0): In function `.ext3_fill_super':
>> : undefined reference to `.__kmalloc_size_too_large'
>> fs/built-in.o(.text+0xbe934): In function `.ext2_fill_super':
>> : undefined reference to `.__kmalloc_size_too_large'
>> fs/built-in.o(.text+0xf3370): In function `.nfs4_proc_lookup':
>>
>> something has gone stupid with kmalloc there, and I cannot reproduce it
>> with my compiler and with your (very old) .config at
>> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/power4
>>
>> So I'm a bit stumped.  Does autotest just do `yes "" | make oldconfig' or
>> what?  When I do that, I get SLUB, but no compile errors.
> 
> Yes, exactly that.
> 
>> And do you know what compiler version is being used there?
> 
> gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13sarge1)

Sorry I misslead, that is the x84_64, the below is the correct version:

gcc version 3.3.3 (SuSE Linux)

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Badari Pulavarty]

On Tue, 2007-04-24 at 15:51 -0700, Andrew Morton wrote:
> Andy, I'm looking at the power4 build:
> 
> http://test.kernel.org/abat/84751/debug/test.log.0
> 
> which has
> 
>   LD      init/built-in.o
>   LD      .tmp_vmlinux1
> init/built-in.o(.init.text+0x32e4): In function `.rd_load_image':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xa60f0): In function `.ext3_fill_super':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xbe934): In function `.ext2_fill_super':
> : undefined reference to `.__kmalloc_size_too_large'
> fs/built-in.o(.text+0xf3370): In function `.nfs4_proc_lookup':
> 
> something has gone stupid with kmalloc there, and I cannot reproduce it
> with my compiler and with your (very old) .config at
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/power4
> 
> So I'm a bit stumped.  Does autotest just do `yes "" | make oldconfig' or
> what?  When I do that, I get SLUB, but no compile errors.
> 
> And do you know what compiler version is being used there?

include/linux/slub_def.h:

static inline struct kmem_cache *kmalloc_slab(size_t size)
{
        int index = kmalloc_index(size);

        if (index == 0)
                return NULL;

        if (index < 0) {
                /*
                 * Generate a link failure. Would be great if we could
                 * do something to stop the compile here.
                 */
                extern void __kmalloc_size_too_large(void);
                __kmalloc_size_too_large();
        }
        return &kmalloc_caches[index];
}

hmm.. 

gcc version 3.3.3 -- generates those link failures
gcc version 4.1.0 -- doesn't generate this error

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 17:51:27 -0700 Badari Pulavarty <pbadari@gmail.com> wrote:

> On Tue, 2007-04-24 at 15:51 -0700, Andrew Morton wrote:
> > Andy, I'm looking at the power4 build:
> > 
> > http://test.kernel.org/abat/84751/debug/test.log.0
> > 
> > which has
> > 
> >   LD      init/built-in.o
> >   LD      .tmp_vmlinux1
> > init/built-in.o(.init.text+0x32e4): In function `.rd_load_image':
> > : undefined reference to `.__kmalloc_size_too_large'
> > fs/built-in.o(.text+0xa60f0): In function `.ext3_fill_super':
> > : undefined reference to `.__kmalloc_size_too_large'
> > fs/built-in.o(.text+0xbe934): In function `.ext2_fill_super':
> > : undefined reference to `.__kmalloc_size_too_large'
> > fs/built-in.o(.text+0xf3370): In function `.nfs4_proc_lookup':
> > 
> > something has gone stupid with kmalloc there, and I cannot reproduce it
> > with my compiler and with your (very old) .config at
> > http://ftp.kernel.org/pub/linux/kernel/people/mbligh/config/abat/power4
> > 
> > So I'm a bit stumped.  Does autotest just do `yes "" | make oldconfig' or
> > what?  When I do that, I get SLUB, but no compile errors.
> > 
> > And do you know what compiler version is being used there?
> 
> include/linux/slub_def.h:
> 
> static inline struct kmem_cache *kmalloc_slab(size_t size)
> {
>         int index = kmalloc_index(size);
> 
>         if (index == 0)
>                 return NULL;
> 
>         if (index < 0) {
>                 /*
>                  * Generate a link failure. Would be great if we could
>                  * do something to stop the compile here.
>                  */
>                 extern void __kmalloc_size_too_large(void);
>                 __kmalloc_size_too_large();
>         }
>         return &kmalloc_caches[index];
> }
> 
> hmm.. 
> 
> gcc version 3.3.3 -- generates those link failures
> gcc version 4.1.0 -- doesn't generate this error

My power box is 3.4.4 and it doesn't do that either.  I guess it's just a
gcc buglet.

Poor Christoph ;)

I wonder why slab doesn't hit that problem.

I wonder whether slub should use kmalloc-sizes.h.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> > gcc version 4.1.0 -- doesn't generate this error
> 
> My power box is 3.4.4 and it doesn't do that either.  I guess it's just a
> gcc buglet.

Its a constant folding issue. Yuck. I went through a series of these last 
year and I thought I had this under constrol.
> 
> Poor Christoph ;)
> 
> I wonder why slab doesn't hit that problem.

It relies less on constant folding.
 
> I wonder whether slub should use kmalloc-sizes.h.

I tried to avoid that mess.

What is the lowest gcc version we currently support?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 23:23:49 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> What is the lowest gcc version we currently support?

Documentation/Changes has the list.  gcc-3.2

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Andrew Morton wrote:

> > gcc version 3.3.3 -- generates those link failures
> > gcc version 4.1.0 -- doesn't generate this error
> 
> My power box is 3.4.4 and it doesn't do that either.  I guess it's just a
> gcc buglet.
> 
> Poor Christoph ;)
> 
> I wonder why slab doesn't hit that problem.
> 
> I wonder whether slub should use kmalloc-sizes.h.

Builds fine here with gcc 3.3 on x86_64. Maybe a problem with the arch 
specific backend? Maybe it does not inline by default?

Does forcing inlining fix the problem?

Index: linux-2.6.21-rc7/include/linux/slub_def.h
===================================================================
--- linux-2.6.21-rc7.orig/include/linux/slub_def.h	2007-04-24 23:50:27.000000000 -0700
+++ linux-2.6.21-rc7/include/linux/slub_def.h	2007-04-24 23:51:08.000000000 -0700
@@ -84,7 +84,7 @@ extern struct kmem_cache kmalloc_caches[
  * Sorry that the following has to be that ugly but some versions of GCC
  * have trouble with constant propagation and loops.
  */
-static inline int kmalloc_index(int size)
+static __always_inline int kmalloc_index(int size)
 {
 	if (size == 0)
 		return 0;
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Christoph Lameter]

On Tue, 24 Apr 2007, Badari Pulavarty wrote:

> static inline struct kmem_cache *kmalloc_slab(size_t size)
> {
>         int index = kmalloc_index(size);
> 
>         if (index == 0)
>                 return NULL;
> 
>         if (index < 0) {
>                 /*
>                  * Generate a link failure. Would be great if we could
>                  * do something to stop the compile here.
>                  */
>                 extern void __kmalloc_size_too_large(void);
>                 __kmalloc_size_too_large();
>         }
>         return &kmalloc_caches[index];
> }
> 
> hmm.. 
> 
> gcc version 3.3.3 -- generates those link failures
> gcc version 4.1.0 -- doesn't generate this error

Likely an issue with constant folding.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Badari Pulavarty]

On Tue, 2007-04-24 at 13:06 -0700, Andrew Morton wrote:
> An amd64 machine is crashing badly.
> 
> http://test.kernel.org/abat/84767/debug/console.log
> 
> VFS: Mounted root (ext3 filesystem) readonly.
> Freeing unused kernel memory: 308k freed
> INIT: version 2.86 booting
> Bad page state in process 'init'
> page:ffff81007e492628 flags:0x0100000000000000 mapping:0000000000000000 mapcount:0 count:1
> Trying to fix it up, but a reboot is needed
> Backtrace:
> 
> Call Trace:
>  [<ffffffff80250d3c>] bad_page+0x74/0x10d
>  [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
...
> 
> So free_pgd_range() is freeing a refcount=1 page.  Can anyone see what
> might be causing this?  The quicklist code impacts this area more than
> anything else..
> 

Yep. quicklist patches are causing these.

making CONFIG_QUICKLIST=n didn't solve the problem. I had
to back out all quicklist patches to make my machine boot.

Thanks,
Badari


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 2.6.21-rc7-mm1 on test.kernel.org

[Andrew Morton]

On Tue, 24 Apr 2007 16:59:29 -0700 Badari Pulavarty <pbadari@gmail.com> wrote:

> On Tue, 2007-04-24 at 13:06 -0700, Andrew Morton wrote:
> > An amd64 machine is crashing badly.
> > 
> > http://test.kernel.org/abat/84767/debug/console.log
> > 
> > VFS: Mounted root (ext3 filesystem) readonly.
> > Freeing unused kernel memory: 308k freed
> > INIT: version 2.86 booting
> > Bad page state in process 'init'
> > page:ffff81007e492628 flags:0x0100000000000000 mapping:0000000000000000 mapcount:0 count:1
> > Trying to fix it up, but a reboot is needed
> > Backtrace:
> > 
> > Call Trace:
> >  [<ffffffff80250d3c>] bad_page+0x74/0x10d
> >  [<ffffffff80253090>] free_hot_cold_page+0x8d/0x172
> ...
> > 
> > So free_pgd_range() is freeing a refcount=1 page.  Can anyone see what
> > might be causing this?  The quicklist code impacts this area more than
> > anything else..
> > 
> 
> Yep. quicklist patches are causing these.
> 
> making CONFIG_QUICKLIST=n didn't solve the problem. I had
> to back out all quicklist patches to make my machine boot.
> 

Great, thanks for working that out.  If people start reporting this I'll
drop 'em and do an -rc2, but things are awful quiet out there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>