git-slub crashes on the t16p

git-slub crashes on the t16p

[Andrew Morton]

It's the tree I pulled about 12 hours ago.  Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch
applied.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Hi,

On 4/10/2008, "Andrew Morton" <akpm@linux-foundation.org> wrote:
> It's the tree I pulled about 12 hours ago.  Quite early in boot.
> 
> crash: http://userweb.kernel.org/~akpm/p4105087.jpg
> config: http://userweb.kernel.org/~akpm/config-t61p.txt
> git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch
> 
> A t61p is a dual-core x86_64.
> 
> I was testing with all of the -mm series up to and including git-slub.patch
> applied.

You have CONFIG_NUMA enabled, so we check for NULL in inc_slabs_node():

        if (!NUMA_BUILD || n) {
                atomic_long_inc(&n->nr_slabs);
                atomic_long_add(objects, &n->total_objects);

I think I hit the same problem and it went away after make clean. Hmm...

                                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka J Enberg]

On Thu, 10 Apr 2008, Andrew Morton wrote:
> It's the tree I pulled about 12 hours ago.  Quite early in boot.
> 
> crash: http://userweb.kernel.org/~akpm/p4105087.jpg
> config: http://userweb.kernel.org/~akpm/config-t61p.txt
> git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch
> 
> A t61p is a dual-core x86_64.
> 
> I was testing with all of the -mm series up to and including git-slub.patch
> applied.

Does the following patch fix it?

diff --git a/mm/slub.c b/mm/slub.c
index 4b694a7..3916b4d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -895,7 +895,7 @@ static inline void inc_slabs_node(struct kmem_cache *s, int node, int objects)
 	 * dilemma by deferring the increment of the count during
 	 * bootstrap (see early_kmem_cache_node_alloc).
 	 */
-	if (!NUMA_BUILD || n) {
+	if (n) {
 		atomic_long_inc(&n->nr_slabs);
 		atomic_long_add(objects, &n->total_objects);
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Hi Andrew,

Pekka J Enberg wrote:
>> It's the tree I pulled about 12 hours ago.  Quite early in boot.
>>
>> crash: http://userweb.kernel.org/~akpm/p4105087.jpg
>> config: http://userweb.kernel.org/~akpm/config-t61p.txt
>> git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch
>>
>> A t61p is a dual-core x86_64.
>>
>> I was testing with all of the -mm series up to and including git-slub.patch
>> applied.

On Thu, 10 Apr 2008, Andrew Morton wrote:
> Does the following patch fix it?

Okay, forget the patch. Looking at disassembly of the oops:

0000000000000000 <.text>:
    0:   eb ce                   jmp    0xffffffffffffffd0
    2:   48 89 de                mov    %rbx,%rsi
    5:   4c 89 e7                mov    %r12,%rdi
    8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
    d:   b8 01 00 00 00          mov    $0x1,%eax
   12:   5b                      pop    %rbx
   13:   41 5c                   pop    %r12
   15:   c9                      leaveq
   16:   c3                      retq
   17:   c3                      retq
   18:   48 63 f6                movslq %esi,%rsi
   1b:   55                      push   %rbp
   1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
   23:   00
   24:   48 89 e5                mov    %rsp,%rbp
   27:   48 85 c9                test   %rcx,%rcx
   2a:   74 0d                   je     0x39
   2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
   31:   48 63 c2                movslq %edx,%rax
   34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
   39:   c9                      leaveq
   3a:   c3                      retq
   3b:   48 8b 07                mov    (%rdi),%rax
   3e:   55                      push   %rbp
   3f:   48                      rex.W
   40:   89                      .byte 0x89

Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my 
logs and I hit the exact same problem but it went away with "make 
clean". Andrew, can you please try that as well?

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Pekka Enberg wrote:
>>> It's the tree I pulled about 12 hours ago.  Quite early in boot.
>>>
>>> crash: http://userweb.kernel.org/~akpm/p4105087.jpg
>>> config: http://userweb.kernel.org/~akpm/config-t61p.txt
>>> git-slub.patch: 
>>> http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch
>>>
>>> A t61p is a dual-core x86_64.
>>>
>>> I was testing with all of the -mm series up to and including 
>>> git-slub.patch
>>> applied.
> 
> On Thu, 10 Apr 2008, Andrew Morton wrote:
>> Does the following patch fix it?
> 
> Okay, forget the patch. Looking at disassembly of the oops:
> 
> 0000000000000000 <.text>:
>    0:   eb ce                   jmp    0xffffffffffffffd0
>    2:   48 89 de                mov    %rbx,%rsi
>    5:   4c 89 e7                mov    %r12,%rdi
>    8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
>    d:   b8 01 00 00 00          mov    $0x1,%eax
>   12:   5b                      pop    %rbx
>   13:   41 5c                   pop    %r12
>   15:   c9                      leaveq
>   16:   c3                      retq
>   17:   c3                      retq
>   18:   48 63 f6                movslq %esi,%rsi
>   1b:   55                      push   %rbp
>   1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
>   23:   00
>   24:   48 89 e5                mov    %rsp,%rbp
>   27:   48 85 c9                test   %rcx,%rcx
>   2a:   74 0d                   je     0x39
>   2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
>   31:   48 63 c2                movslq %edx,%rax
>   34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
>   39:   c9                      leaveq
>   3a:   c3                      retq
>   3b:   48 8b 07                mov    (%rdi),%rax
>   3e:   55                      push   %rbp
>   3f:   48                      rex.W
>   40:   89                      .byte 0x89
> 
> Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my 
> logs and I hit the exact same problem but it went away with "make 
> clean". Andrew, can you please try that as well?

I can't reproduce this on my 2-way with CONFIG_NUMA with or without 
numa=fake enabled and Mel Gorman tells me my tree boots on their monster 
numa boxen so I'm a wee bit at of a loss here.... Hmm

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Andrew Morton]

On Thu, 10 Apr 2008 19:35:58 +0300 Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> Pekka Enberg wrote:
> >>> It's the tree I pulled about 12 hours ago.  Quite early in boot.
> >>>
> >>> crash: http://userweb.kernel.org/~akpm/p4105087.jpg
> >>> config: http://userweb.kernel.org/~akpm/config-t61p.txt
> >>> git-slub.patch: 
> >>> http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch
> >>>
> >>> A t61p is a dual-core x86_64.
> >>>
> >>> I was testing with all of the -mm series up to and including 
> >>> git-slub.patch
> >>> applied.
> > 
> > On Thu, 10 Apr 2008, Andrew Morton wrote:
> >> Does the following patch fix it?
> > 
> > Okay, forget the patch. Looking at disassembly of the oops:
> > 
> > 0000000000000000 <.text>:
> >    0:   eb ce                   jmp    0xffffffffffffffd0
> >    2:   48 89 de                mov    %rbx,%rsi
> >    5:   4c 89 e7                mov    %r12,%rdi
> >    8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
> >    d:   b8 01 00 00 00          mov    $0x1,%eax
> >   12:   5b                      pop    %rbx
> >   13:   41 5c                   pop    %r12
> >   15:   c9                      leaveq
> >   16:   c3                      retq
> >   17:   c3                      retq
> >   18:   48 63 f6                movslq %esi,%rsi
> >   1b:   55                      push   %rbp
> >   1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
> >   23:   00
> >   24:   48 89 e5                mov    %rsp,%rbp
> >   27:   48 85 c9                test   %rcx,%rcx
> >   2a:   74 0d                   je     0x39
> >   2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
> >   31:   48 63 c2                movslq %edx,%rax
> >   34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
> >   39:   c9                      leaveq
> >   3a:   c3                      retq
> >   3b:   48 8b 07                mov    (%rdi),%rax
> >   3e:   55                      push   %rbp
> >   3f:   48                      rex.W
> >   40:   89                      .byte 0x89
> > 
> > Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my 
> > logs and I hit the exact same problem but it went away with "make 
> > clean". Andrew, can you please try that as well?
> 
> I can't reproduce this on my 2-way with CONFIG_NUMA with or without 
> numa=fake enabled and Mel Gorman tells me my tree boots on their monster 
> numa boxen so I'm a wee bit at of a loss here.... Hmm

I retested with just origin.patch and git-slub.patch.  Did a `make mproper'
first.  Similar crash happens: http://userweb.kernel.org/~akpm/p4105088.jpg

That's within the call to atomic64_inc(), from the inc_slabs_node() here:

0xffffffff8109c348 is in new_slab (mm/slub.c:1159).
1154                    flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node);
1155            if (!page)
1156                    goto out;
1157    
1158            inc_slabs_node(s, node, page->objects);
1159            page->slab = s;
1160            page->flags |= 1 << PG_slab;
1161            if (s->flags & (SLAB_DEBUG_FREE | SLAB_RED_ZONE | SLAB_POISON |
1162                            SLAB_STORE_USER | SLAB_TRACE))
1163                    SetSlabDebug(page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Christoph Lameter]

On Thu, 10 Apr 2008, Andrew Morton wrote:

> That's within the call to atomic64_inc(), from the inc_slabs_node() here:

Right. The slab counter cleanup patch did a non equivalent transformation 
here.

Index: linux-2.6/mm/slub.c
===================================================================
--- linux-2.6.orig/mm/slub.c	2008-04-10 10:27:29.000000000 -0700
+++ linux-2.6/mm/slub.c	2008-04-10 10:28:02.000000000 -0700
@@ -1174,6 +1174,8 @@
 	if (!page)
 		goto out;
 
+	/* Must use the node that the page allocator determined for us. */
+	node = page_to_nid(page);
 	inc_slabs_node(s, node, page->objects);
 	page->slab = s;
 	page->flags |= 1 << PG_slab;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Christoph Lameter wrote:
> On Thu, 10 Apr 2008, Andrew Morton wrote:
> 
>> That's within the call to atomic64_inc(), from the inc_slabs_node() here:
> 
> Right. The slab counter cleanup patch did a non equivalent transformation 
> here.
> 
> Index: linux-2.6/mm/slub.c
> ===================================================================
> --- linux-2.6.orig/mm/slub.c	2008-04-10 10:27:29.000000000 -0700
> +++ linux-2.6/mm/slub.c	2008-04-10 10:28:02.000000000 -0700
> @@ -1174,6 +1174,8 @@
>  	if (!page)
>  		goto out;
>  
> +	/* Must use the node that the page allocator determined for us. */
> +	node = page_to_nid(page);
>  	inc_slabs_node(s, node, page->objects);
>  	page->slab = s;
>  	page->flags |= 1 << PG_slab;

Actually, that's fixed in my tree since Saturday. So unfortunately I 
don't think this is the problem...

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Pekka Enberg wrote:
> Actually, that's fixed in my tree since Saturday. So unfortunately I 
> don't think this is the problem...

Aah, it is, Andrew has this:

+	inc_slabs_node(s, node, page->objects);

Did I mess up my git tree or something? At least git clone gives me the 
correct results...

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Christoph Lameter]

On Thu, 10 Apr 2008, Pekka Enberg wrote:

> Pekka Enberg wrote:
> > Actually, that's fixed in my tree since Saturday. So unfortunately I don't
> > think this is the problem...
> 
> Aah, it is, Andrew has this:
> 
> +	inc_slabs_node(s, node, page->objects);
> 
> Did I mess up my git tree or something? At least git clone gives me the
> correct results...

I guess he was pulling my tree and not yours.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Christoph Lameter]

One thing that does not make sense is that there was 0x64 in there. All 
unused node pointers should be NULL (they are zapped in 
kmem_cache_open()). So there may still be something else at play.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Pekka Enberg]

Hi Christoph,

Christoph Lameter wrote:
> One thing that does not make sense is that there was 0x64 in there. All 
> unused node pointers should be NULL (they are zapped in 
> kmem_cache_open()). So there may still be something else at play.

One thing that looks fishy to me is this the compilation of:

   return s->node[node];

to this for Andrew:

   1b:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx

For me, the offset of ->node is 0x140 and _not_ 0x120 even with Andrew's 
config.

			Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Andrew Morton]

On Thu, 10 Apr 2008 10:30:05 -0700 (PDT) Christoph Lameter <clameter@sgi.com> wrote:

> On Thu, 10 Apr 2008, Andrew Morton wrote:
> 
> > That's within the call to atomic64_inc(), from the inc_slabs_node() here:
> 
> Right. The slab counter cleanup patch did a non equivalent transformation 
> here.
> 
> Index: linux-2.6/mm/slub.c
> ===================================================================
> --- linux-2.6.orig/mm/slub.c	2008-04-10 10:27:29.000000000 -0700
> +++ linux-2.6/mm/slub.c	2008-04-10 10:28:02.000000000 -0700
> @@ -1174,6 +1174,8 @@
>  	if (!page)
>  		goto out;
>  
> +	/* Must use the node that the page allocator determined for us. */
> +	node = page_to_nid(page);
>  	inc_slabs_node(s, node, page->objects);
>  	page->slab = s;
>  	page->flags |= 1 << PG_slab;

That fixed it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: git-slub crashes on the t16p

[Christoph Lameter]

On Thu, 10 Apr 2008, Pekka Enberg wrote:

> Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my logs
> and I hit the exact same problem but it went away with "make clean". Andrew,
> can you please try that as well?

Memory corruption? 0x64 is 100 btw.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>