Hi,

Full description of the problem:

Kernel version: 2.6.32.36

Oops information:

[9638271.695663] BUG: unable to handle kernel paging request at 00000000= 00a3ad90
[9638271.695685] IP: [<ffffffff800cddfa>] kfree+0x5a/0x200
[9638271.695701] PGD f94ff067 PUD fd652067 PMD 0
[9638271.695707] Oops: 0000 [#1] SMP
[9638271.695712] last sysfs file: /sys/devices/xen-backend/vbd-415-51776/st= atistics/wr_sect

Trap number:14, message:Oops
Error num: 0
Sigal Num:11_SIGSEGV
Event ID:DIE_OOPS
RIP: e030:[<ffffffff800cddfa>]
<ffffffff800cddfa>{kfree+0x5a}
RSP: e02b:ffff88001ce65da8 EFLAGS: 00010006
RAX: 0000000000a3ad90 RBX: 0000000000000000 RCX: 00000000000002eb
RDX: 00000000001761f0 RSI: 00000000000002eb RDI: ffff88002ec3e3e0
RBP: fffffffffffffffe R08: 0000000000000000 R09: ffff88002ec3e3e0
R10: ffffffffffffffff R11: ffffffff801b0e50 R12: 0000000000008001
R13: 0000000000000024 R14: 00000000ffffff9c R15: ffff88001ce65e48
FS: 00007fbe05e71700(0000) GS:ffff880002008000(0000) knlGS:0000000000= 000000
CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000a3ad90 CR3: 00000000f9009000 CR4: 0000000000002620
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<kernel_trace>
       <ffffffff80009b05>{dump_trace= 3;0x65}
       <ffffffff8037d897>{notifier_call= _chain+0x37}
       <ffffffff8005a1ed>{notify_die= 3;0x2d}
       <ffffffff8037bd0b>{__die+0x8= b}
       <ffffffff8001bed1>{no_context= 3;0xd1}
       <ffffffff8001c1f5>{__bad_area_no= semaphore+0x175}
       <ffffffff8037b298>{page_fault= 3;0x28}
       <ffffffff800cddfa>{kfree+0x5= a}
       <ffffffff800da03d>{put_filp+= 0x1d}
       <ffffffff800e7133>{do_filp_open&= #43;0x723}
       <ffffffff800d62b7>{do_sys_open&#= 43;0x97}
       <ffffffff80007378>{system_call_f= astpath+0x16}
       [<00007fbe059c8040>]
</kernel_trace>

Following is my own preliminary analysis:

crash> dis kfree
0xffffffff800cdda0 <kfree>:     push   = %r15
0xffffffff800cdda2 <kfree+2>:   push   %r14 0xffffffff800cdda4 <kfree+4>:   push   %r13 0xffffffff800cdda6 <kfree+6>:   push   %r12 0xffffffff800cdda8 <kfree+8>:   push   %rbp 0xffffffff800cdda9 <kfree+9>:   push   %rbx 0xffffffff800cddaa <kfree+10>: sub    $0x18,= %rsp
0xffffffff800cddae <kfree+14>: cmp    $0x10,= %rdi
0xffffffff800cddb2 <kfree+18>: mov    %rdi,0= x8(%rsp)
0xffffffff800cddb7 <kfree+23>: jbe    0xffff= ffff800cde7c <kfree+220>
0xffffffff800cddbd <kfree+29>: mov    %gs:0x= 67c1,%al
0xffffffff800cddc5 <kfree+37>: movb   $0x1,%gs:0x= 67c1
0xffffffff800cddce <kfree+46>: mov    %al,0x= 17(%rsp)
0xffffffff800cddd2 <kfree+50>: mov    0x8(%r= sp),%rdi
0xffffffff800cddd7 <kfree+55>: mov    0x7588= 72(%rip),%rbx        # 0xffffffff8082665= 0
0xffffffff800cddde <kfree+62>: callq 0xffffffff800228= e0 <__phys_addr>
0xffffffff800cdde3 <kfree+67>: shr    $0xc,%= rax
0xffffffff800cdde7 <kfree+71>: lea    0x0(,%= rax,8),%rdx
0xffffffff800cddef <kfree+79>: shl    $0x6,%= rax
0xffffffff800cddf3 <kfree+83>: sub    %rdx,%= rax
0xffffffff800cddf6 <kfree+86>: lea    (%rbx,= %rax,1),%rax
0xffffffff800cddfa <kfree+90>: mov    (%rax)= ,%rdx
0xffffffff800cddfd <kfree+93>: test   $0x20000,%e= dx
0xffffffff800cde03 <kfree+99>: je     0= xffffffff800cde1b <kfree+123>
0xffffffff800cde05 <kfree+101>: mov    0x10(%rax),= %rax
0xffffffff800cde09 <kfree+105>: mov    (%rax),%rdx=
0xffffffff800cde0c <kfree+108>: test   $0x20000,%edx 0xffffffff800cde12 <kfree+114>: je     0xffff= ffff800cde1b <kfree+123>
......

Normally %rbx should be the value of mem_map which is a fixed value in m= y system, the address of the mem_map is 0xffffffff80826650, and the value o= f mem_map is 0xffff880004802000.

But here, %rbx was changed to 0x0000000000000000, in my opinion, the pos= sible reason is below:

1. mem_map was changed with an unknown reason, led to %rbx is wrong= .
2. mem_map is right, but %rip is wrong, led to %rbx is wrong.
3. mem_map is right, and %rip is also right, but %rbx was changed afte= r later.

I changed the mem_map value to 0x0000000000000000, kernel is panic immed= iately, but it can=A1=AFt produce the vmcore, this problem has the vmcore(s= ad to say, vmcore was gone because of carelessness).

So we can exclude the reason one, the rest of the reason is two and thre= e, but i don=A1=AFt know how they can happen.

I don't do anything before the system panic, and i can=A1=AFt reproduce = this problem.