Re: Splat during resume

Re: Splat during resume

[Borislav Petkov]

On Sat, Mar 25, 2017 at 07:58:55PM +0100, Borislav Petkov wrote:
> Hey Rafael,
> 
> have you seen this already (partial splat photo attached)? Happens
> during resume from s2d. Judging by the timestamps, this looks like the
> resume kernel before we switch to the original, boot one but I could be
> mistaken.
> 
> This is -rc3+tip/master.
> 
> I can't catch a full splat because this is a laptop and it doesn't have
> serial. netconsole is helping me for shit so we'd need some guess work.
> 
> So I'm open to suggestions.
> 
> Please don't say "bisect" yet ;-)))

No need, I found it. Reverting

  ea3b5e60ce80 ("x86/mm/ident_map: Add 5-level paging support")

makes the machine suspend and resume just fine again. Lemme add people to CC.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: Splat during resume

[Borislav Petkov]

On Sat, Mar 25, 2017 at 10:46:15PM +0100, Borislav Petkov wrote:
> On Sat, Mar 25, 2017 at 07:58:55PM +0100, Borislav Petkov wrote:
> > Hey Rafael,
> > 
> > have you seen this already (partial splat photo attached)? Happens
> > during resume from s2d. Judging by the timestamps, this looks like the
> > resume kernel before we switch to the original, boot one but I could be
> > mistaken.
> > 
> > This is -rc3+tip/master.
> > 
> > I can't catch a full splat because this is a laptop and it doesn't have
> > serial. netconsole is helping me for shit so we'd need some guess work.
> > 
> > So I'm open to suggestions.
> > 
> > Please don't say "bisect" yet ;-)))
> 
> No need, I found it. Reverting
> 
>   ea3b5e60ce80 ("x86/mm/ident_map: Add 5-level paging support")
> 
> makes the machine suspend and resume just fine again. Lemme add people to CC.

So I see rIP pointing to ident_pmd_init() and the stack trace has
load_image_and_restore() so if I try to connect the dots, I get:

load_image_and_restore
|-> hibernation_restore
 |-> resume_target_kernel
  |-> swsusp_arch_resume
   |-> set_up_temporary_mappings
    |-> kernel_ident_mapping_init
     |-> ... ident_pmd_init

I'll let you folks make sense of what's going on.

Thanks.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Splat during resume

[Kirill A. Shutemov]

On Sat, Mar 25, 2017 at 10:50:12PM +0100, Borislav Petkov wrote:
> > No need, I found it. Reverting
> > 
> >   ea3b5e60ce80 ("x86/mm/ident_map: Add 5-level paging support")
> > 
> > makes the machine suspend and resume just fine again. Lemme add people to CC.
> 
> So I see rIP pointing to ident_pmd_init() and the stack trace has
> load_image_and_restore() so if I try to connect the dots, I get:
> 
> load_image_and_restore
> |-> hibernation_restore
>  |-> resume_target_kernel
>   |-> swsusp_arch_resume
>    |-> set_up_temporary_mappings
>     |-> kernel_ident_mapping_init
>      |-> ... ident_pmd_init
> 
> I'll let you folks make sense of what's going on.

Sorry for this. See

http://lkml.kernel.org/r/20170324120458.nw3fwpmdptjtj5qb@node.shutemov.name

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Splat during resume

[Linus Torvalds]

I think this is the same as the kexec issue that also hit -tip.

It's *probably* fixed by the final series to actually enable 5-level
paging (which I don't think is in -tip yet), but even if that is the
case this is obviously a nasty bisectability problem.

You migth want to verify, though. The second batch starts here:

  https://marc.info/?l=linux-mm&m=148977696117208&w=2

Hmm?

In the meantime, this is currently -tip only, so I will stack back
from this thread unless you can reproduce it in mainline too.

               Linus

On Sat, Mar 25, 2017 at 2:50 PM, Borislav Petkov <bp@alien8.de> wrote:
>
> So I see rIP pointing to ident_pmd_init() and the stack trace has
> load_image_and_restore() so if I try to connect the dots, I get:
>
> load_image_and_restore
> |-> hibernation_restore
>  |-> resume_target_kernel
>   |-> swsusp_arch_resume
>    |-> set_up_temporary_mappings
>     |-> kernel_ident_mapping_init
>      |-> ... ident_pmd_init
>
> I'll let you folks make sense of what's going on.
>
> Thanks.
>
> --
> Regards/Gruss,
>     Boris.
>
> Good mailing practices for 400: avoid top-posting and trim the reply.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Splat during resume

[Borislav Petkov]

On Sat, Mar 25, 2017 at 03:05:41PM -0700, Linus Torvalds wrote:
> I think this is the same as the kexec issue that also hit -tip.
> 
> It's *probably* fixed by the final series to actually enable 5-level
> paging (which I don't think is in -tip yet), but even if that is the
> case this is obviously a nasty bisectability problem.

It being -tip only for now, I'm guessing that can still be addressed...?

> You migth want to verify, though. The second batch starts here:
> 
>   https://marc.info/?l=linux-mm&m=148977696117208&w=2
> 
> Hmm?
> 
> In the meantime, this is currently -tip only, so I will stack back
> from this thread unless you can reproduce it in mainline too.

I could try mainline, just in case and if you want me to but

* considering the patch which broke this is in tip only and
* after git-am'ing the 6 part-2 patches you and Kirill pointed me at, the resume issue is fixed,

it probably is not really needed.

Thanks guys.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: Splat during resume

[Mike Galbraith]

On Sat, 2017-03-25 at 22:46 +0100, Borislav Petkov wrote:
> On Sat, Mar 25, 2017 at 07:58:55PM +0100, Borislav Petkov wrote:
> > Hey Rafael,
> > 
> > have you seen this already (partial splat photo attached)? Happens
> > during resume from s2d. Judging by the timestamps, this looks like the
> > resume kernel before we switch to the original, boot one but I could be
> > mistaken.
> > 
> > This is -rc3+tip/master.
> > 
> > I can't catch a full splat because this is a laptop and it doesn't have
> > serial. netconsole is helping me for shit so we'd need some guess work.
> > 
> > So I'm open to suggestions.
> > 
> > Please don't say "bisect" yet ;-)))
> 
> No need, I found it. Reverting
> 
>   ea3b5e60ce80 ("x86/mm/ident_map: Add 5-level paging support")
> 
> makes the machine suspend and resume just fine again. Lemme add people to CC.

To be filed under "_maybe_ interesting", my tip-rt tree hits the below
on boot (survives), ONLY on vaporite (kvm), silicon boots clean, works
fine, hibernate/suspend gripe free.  The revert fixed up vaporite.

[   16.566554] BUG: unable to handle kernel paging request at ffffc753f000f000
[   16.566562] IP: ident_pmd_init.isra.4+0x56/0xb0
[   16.566563] PGD 0 

[   16.566565] Oops: 0000 [#1] PREEMPT SMP
[   16.566569] Dumping ftrace buffer:
[   16.566593]    (ftrace buffer empty)
[   16.566593] Modules linked in: nf_conntrack_ipv6(E) nf_defrag_ipv6(E) ip6table_raw(E) ipt_REJECT(E) iptable_raw(E) xt_CT(E) iptable_filter(E) ip6table_mangle(E) nf_conntrack_netbios_ns(E) nf_conntrack_broadcast(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) ip_tables(E) xt_conntrack(E) nf_conntrack(E) libcrc32c(E) ip6table_filter(E) ip6_tables(E) x_tables(E) joydev(E) snd_hda_codec_generic(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) snd_pcm(E) snd_timer(E) snd(E) soundcore(E) 8139too(E) i2c_piix4(E) virtio_balloon(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) pcbc(E) ppdev(E) aesni_intel(E) serio_raw(E) pcspkr(E) aes_x86_64(E) parport_pc(E) crypto_simd(E) parport(E) acpi_cpufreq(E) glue_helper(E) button(E) cryptd(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E)
[   16.566611]  sunrpc(E) ext4(E) crc16(E) jbd2(E) mbcache(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) ata_generic(E) virtio_rng(E) virtio_blk(E) virtio_console(E) ata_piix(E) qxl(E) crc32c_intel(E) drm_kms_helper(E) syscopyarea(E) uhci_hcd(E) ehci_pci(E) sysfillrect(E) sysimgblt(E) ehci_hcd(E) fb_sys_fops(E) ahci(E) virtio_pci(E) libahci(E) ttm(E) virtio_ring(E) 8139cp(E) virtio(E) usbcore(E) drm(E) libata(E) mii(E) floppy(E) sg(E) scsi_mod(E) autofs4(E)
[   16.566625] CPU: 6 PID: 1295 Comm: kexec Tainted: G            E   4.11.0-rt12-tip-rt #80
[   16.566626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20161202_174313-build11a 04/01/2014
[   16.566626] task: ffff88022a3daf40 task.stack: ffffc90002520000
[   16.566628] RIP: 0010:ident_pmd_init.isra.4+0x56/0xb0
[   16.566628] RSP: 0018:ffffc90002523da0 EFLAGS: 00010286
[   16.566629] RAX: ffffc000001fffff RBX: 0000000000000000 RCX: 0000000000000000
[   16.566629] RDX: ffffc753f000f000 RSI: ffffc90002523e90 RDI: ffffc90002523e88
[   16.566629] RBP: 0000000040000000 R08: 0000000040000000 R09: 0000000035ff6fff
[   16.566630] R10: 0000000026000000 R11: 000000000009f000 R12: ffffc000001fffff
[   16.566630] R13: ffffc00000000fff R14: ffffc753f000f000 R15: ffffc90002523e88
[   16.566631] FS:  00007f7ad2486700(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
[   16.566631] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   16.566632] CR2: ffffc753f000f000 CR3: 000000023fd68000 CR4: 00000000001406e0
[   16.566634] Call Trace:
[   16.566639]  ? ident_pud_init+0x7a/0x180
[   16.566641]  ? kernel_ident_mapping_init+0x152/0x1f0
[   16.566643]  ? machine_kexec_prepare+0xa7/0x470
[   16.566644]  ? kexec_mark_crashkres+0x70/0x70
[   16.566647]  ? SyS_kexec_file_load+0x2e4/0x6b0
[   16.566651]  ? do_sys_open+0x182/0x1e0
[   16.566655]  ? entry_SYSCALL_64_fastpath+0x1a/0xa5
[   16.566656] Code: 53 48 89 cb 48 81 e3 00 00 e0 ff 48 83 ec 08 4c 39 c3 48 89 34 24 73 54 48 89 da 4c 89 e0 48 c1 ea 12 81 e2 f8 0f 00 00 4c 01 f2 <48> 8b 0a f6 c1 80 49 0f 44 c5 48 21 c8 a9 81 01 00 00 75 21 48 
[   16.566665] RIP: ident_pmd_init.isra.4+0x56/0xb0 RSP: ffffc90002523da0
[   16.566665] CR2: ffffc753f000f000

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Splat during resume

[Borislav Petkov]

On Sun, Mar 26, 2017 at 10:25:43AM +0200, Mike Galbraith wrote:
> To be filed under "_maybe_ interesting", my tip-rt tree hits the below
> on boot (survives), ONLY on vaporite (kvm), silicon boots clean, works
> fine, hibernate/suspend gripe free.

Strange - I did boot fine but resume shit in its pants.

> The revert fixed up vaporite.

vaporite, haha, good one. I like that, let's do
s/vaporitization/virtualization/g from now on :-)

Btw, try the 6 patches here: https://marc.info/?l=linux-mm&m=148977696117208&w=2
ontop of tip. Should fix your vaporite too.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Splat during resume

[Mike Galbraith]

On Sun, 2017-03-26 at 10:41 +0200, Borislav Petkov wrote:

> Btw, try the 6 patches here: 
> https://marc.info/?l=linux-mm&m=148977696117208&w=2
> ontop of tip. Should fix your vaporite too.

Yeah, silicon is still happy, vaporite boots gripe free.  Trying to
hibernate vaporite was a bad idea, but is unrelated to this thread.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>