Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell <lists2009@fnarfbargle.com>]

On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults.  I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)?  (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/

Ok, here we go then.

http://www.fnarfbargle.com/private/004_kernel_oops/

The permissions are right this time.
2.6.39 + KSM

[  694.227866] general protection fault: 0000 [#1] SMP
[  694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[  694.228050] CPU 3
[  694.228091] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle 
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate 
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid 
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303 
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd 
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last 
unloaded: scsi_wait_scan]
[  694.230897]
[  694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[  694.231111] RIP: 0010:[<ffffffff810db878>]  [<ffffffff810db878>] 
dup_fd+0x168/0x300
[  694.231210] RSP: 0018:ffff8802f524fdd0  EFLAGS: 00010206
[  694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX: 
bfffffffffffffff
[  694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI: 
0000000000000800
[  694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09: 
0000000000000003
[  694.231407] R10: 0000000000000001 R11: 4000000000000000 R12: 
0000000000000100
[  694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15: 
ffff8803bd8c1600
[  694.231507] FS:  00007f02cfc32700(0000) GS:ffff88041fcc0000(0000) 
knlGS:0000000000000000
[  694.231560] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4: 
00000000000006e0
[  694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[  694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[  694.231757] Process keepalive (pid: 11841, threadinfo 
ffff8802f524e000, task ffff8802f5143690)
[  694.231809] Stack:
[  694.231852]  ffff8802f5143690 0000000000000020 ffff8802f56badc0 
ffff8802f5721b90
[  694.232050]  ffff880417aa54e0 0000000001200011 ffff880417aa54e0 
0000000000000000
[  694.232248]  00007f02cfc329d0 ffff8802f5143690 0000000000000000 
ffffffff81037645
[  694.232448] Call Trace:
[  694.232499]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.232549]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.232599]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.232651]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.232699]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24 
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44 
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[  694.235190] RIP  [<ffffffff810db878>] dup_fd+0x168/0x300
[  694.235282]  RSP <ffff8802f524fdd0>
[  694.235379] ---[ end trace 949fad05591fcdb3 ]---
[  694.235428] Kernel panic - not syncing: Fatal exception
[  694.235478] Pid: 11841, comm: keepalive Tainted: G      D     2.6.39 #3
[  694.235525] Call Trace:
[  694.235573]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[  694.235624]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[  694.235676]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[  694.235726]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[  694.235778]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
[  694.235827]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.235877]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.235926]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.235978]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.236028]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.236083] Rebooting in 60 seconds..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>