From: Frank Dinoff <fdinoff@google.com>
To: Miklos Szeredi <miklos@szeredi.hu>
Cc: linux-fsdevel@vger.kernel.org, linux-mm <linux-mm@kvack.org>,
Johannes Weiner <hannes@cmpxchg.org>
Subject: Re: fuse: kernel panic while using splice (lru corruption)
Date: Wed, 20 Oct 2021 17:26:59 -0400 [thread overview]
Message-ID: <CAAmZXrtiJcmLzf6eb90RKdCs3Q=mFNCqAD86nZQJmVwr6YwEmA@mail.gmail.com> (raw)
In-Reply-To: <CAJfpeguXW=Xz-sRUjwOhwinRKpEo8tyxfe_ofhhRPsZreBoQSw@mail.gmail.com>
On Thu, Oct 7, 2021 at 8:54 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> Adding linux-mm/Johannes to Cc.
>
> On Wed, 6 Oct 2021 at 21:13, Frank Dinoff <fdinoff@google.com> wrote:
> >
> > I'm experiencing a kernel panic while using fuse related to SPLICE_F_MOVE.
> >
> > Some stack traces
> >
> > [ 52.864466] CPU: 1 PID: 10619 Comm: cp Not tainted 5.15.0-upstream-DEV #7
> > [ 52.879137] Hardware name: Google Google Compute Engine/Google
> > Compute Engine, BIOS Google 01/01/2011
> > [ 52.888490] RIP: 0010:__list_del_entry_valid+0x69/0x80
> > [ 52.893907] Code: 7f 12 84 31 c0 e8 2d 42 55 00 0f 0b 48 c7 c7 37
> > 8e 10 84 31 c0 e8 1d 42 55 00 0f 0b 48 c7 c7 19 52 19 84 31 c0 e8 0d
> > 42 55 00 <0f> 0b 48 c7 c7 8e e7 15 84 31 c0 e8 fd 41 55 00 0f 0b 00 00
> > 00 cc
> > [ 52.980251] RSP: 0018:ffff8938ea093978 EFLAGS: 00010046
> > [ 52.994508] RAX: 0000000000000054 RBX: ffffd8d7c5914ec0 RCX: 45fe15c1d0642d00
> > [ 53.001774] RDX: ffff893939d230b8 RSI: ffff893939d17510 RDI: ffff893939d17510
> > [ 53.009039] RBP: ffff8938ea093978 R08: 0000000000000000 R09: ffffffff8492dbf0
> > [ 53.016312] R10: 00000000ffff7fff R11: 0000000000000000 R12: ffff8938ea093a98
> > [ 53.023575] R13: ffff8938ced23400 R14: 000000000000000d R15: ffff8938ced23400
> > [ 53.030843] FS: 00007f8bd4b1d740(0000) GS:ffff893939d00000(0000)
> > knlGS:0000000000000000
> > [ 53.039064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 53.044939] CR2: 000000000020c86e CR3: 0000000164572003 CR4: 00000000003706e0
> > [ 53.087945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 53.105377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [ 53.119206] Call Trace:
> > [ 53.121786] release_pages+0x1d0/0x490
> > [ 53.125856] __pagevec_release+0x4f/0x60
> > [ 53.129914] invalidate_inode_pages2_range+0x5c4/0x600
> > [ 53.135186] ? kmem_cache_free+0x7c/0x100
> > [ 53.139330] invalidate_inode_pages2+0x17/0x20
> > [ 53.143907] fuse_finish_open+0x75/0x150
> > [ 53.147976] fuse_open_common+0x113/0x120
> > [ 53.152117] fuse_open+0x10/0x20
> > [ 53.155487] do_dentry_open+0x263/0x360
> > [ 53.167370] vfs_open+0x2d/0x30
> > [ 53.173633] path_openat+0xa0f/0xd90
> > [ 53.177353] ? mntput+0x23/0x40
> > [ 53.180635] ? path_put+0x1e/0x30
> > [ 53.184104] do_filp_open+0xc7/0x170
> > [ 53.187933] do_sys_openat2+0x91/0x170
> > [ 53.195012] __x64_sys_openat+0x7e/0xa0
> > [ 53.198989] do_syscall_64+0x44/0xa0
> > [ 53.202714] ? exc_page_fault+0x71/0x160
> > [ 53.211294] entry_SYSCALL_64_after_hwframe+0x44/0xae
> > [ 53.219256] RIP: 0033:0x7f8bd4c4bec2
> > [ 53.231375] Code: 8d 48 08 48 89 4d d8 8b 18 48 8b 05 90 8d 07 00
> > 83 38 00 75 30 b8 01 01 00 00 41 89 da bf 9c ff ff ff 4c 89 f6 44 89
> > fa 0f 05 <48> 89 c3 48 3d 00 f0 ff ff 77 42 89 d8 48 81 c4 c8 00 00 00
> > 5b 41
> > [ 53.261026] RSP: 002b:00007ffd4ae55570 EFLAGS: 00000246 ORIG_RAX:
> > 0000000000000101
> > [ 53.268738] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8bd4c4bec2
> > [ 53.284714] RDX: 0000000000000000 RSI: 00007ffd4ae57bbc RDI: 00000000ffffff9c
> > [ 53.305198] RBP: 00007ffd4ae55650 R08: 0000000000000000 R09: 00007ffd4ae55baf
> > [ 53.312469] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> > [ 53.319736] R13: 00000000000081a4 R14: 00007ffd4ae57bbc R15: 0000000000000000
> > [ 53.327003] Modules linked in: 9p 9pnet_virtio 9pnet vfat fat
> > virtio_net net_failover failover
> > [ 53.335755] ---[ end trace 086000a6a6747ea3 ]---
> >
> > With CONFIG_DEBUG_VM it looks like we are trying to add a page to the lru that
> > is already on the lru.
>
> Code in question is in fs/fuse/dev.c:fuse_try_move_page() after the
> call to replace_page_cache_page().
>
> Looks like PIPE_BUF_FLAG_LRU isn't reliable., which means it's
> completely useless.
>
> Johannes, any idea how to fix this?
Ping, any thoughts on how to fix this?
>
> >
> > [ 133.194168] invalid opcode: 0000 [#1] SMP PTI
> > [ 133.197918] CPU: 1 PID: 10035 Comm: fusexmp Not tainted 5.15.0-dbg-DEV #5
> > [ 133.213134] Hardware name: Google Google Compute Engine/Google
> > Compute Engine, BIOS Google 01/01/2011
> > [ 133.213134] RIP: 0010:lru_cache_add+0x1be/0x1f0
> > [ 133.213134] Code: c7 c6 1c 6c 01 8c e8 41 8d 03 00 0f 0b 48 c7 c6
> > 1c 6c 01 8c e8 33 8d 03 00 0f 0b 48 89 df 48 c7 c6 60 12 fe 8b e8 22
> > 8d 03 00 <0f> 0b 48 c7 c6 d8 6b ff 8b e8 14 8d 03 00 0f 0b 48 c7 c6 1c
> > 6c 01
> > [ 133.213134] RSP: 0018:ffff9eb7c319fb88 EFLAGS: 00010296
> > [ 133.213134] RAX: bbc197b826b34400 RBX: ffffeb5405464e80 RCX: bbc197b826b34400
> > [ 133.213134] RDX: c0000000ffff7fff RSI: 0000000000000004 RDI: ffff8e52b9d17868
> > [ 133.213134] RBP: ffff9eb7c319fb90 R08: 0000000000000000 R09: ffffffff8c73ef80
> > [ 133.213134] R10: 00000000ffff7fff R11: 0000000000000000 R12: 0000000000000000
> > [ 133.303035] R13: ffffeb5405464e80 R14: ffff9eb7c319fd70 R15: 0000000000001000
> > [ 133.303035] FS: 00007fe9b5aef700(0000) GS:ffff8e52b9d00000(0000)
> > knlGS:0000000000000000
> > [ 133.303035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 133.303035] CR2: 000000000020c86e CR3: 0000000106ab2006 CR4: 00000000003706e0
> > [ 133.303035] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 133.303035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [ 133.303035] Call Trace:
> > [ 133.303035] fuse_copy_page+0x60e/0xa80
> > [ 133.303035] fuse_copy_args+0xd1/0x1e0
> > [ 133.303035] fuse_dev_do_write+0x14f6/0x1b20
> > [ 133.303035] ? __kmalloc_node+0x3c/0x50
> > [ 133.303035] ? __mutex_unlock_slowpath+0x3d/0x230
> > [ 133.303035] fuse_dev_splice_write+0x364/0x410
> > [ 133.303035] do_splice+0x551/0x870
> > [ 133.303035] ? rcu_lock_release+0x10/0x20
> > [ 133.303035] ? __fget_files+0x15a/0x170
> > [ 133.303035] __se_sys_splice+0x15e/0x210
> > [ 133.303035] __x64_sys_splice+0x29/0x30
> > [ 133.303035] do_syscall_64+0x44/0xa0
> > [ 133.303035] entry_SYSCALL_64_after_hwframe+0x44/0xae
> > [ 133.303035] RIP: 0033:0x7fe9bab60953
> > [ 133.303035] Code: 49 89 ca b8 13 01 00 00 0f 05 48 3d 01 f0 ff ff
> > 73 34 c3 48 83 ec 08 e8 0b 0b 00 00 48 89 04 24 49 89 ca b8 13 01 00
> > 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 51 0b 00 00 48 89 d0 48 83 c4 08 48
> > 3d 01
> > [ 133.553661] RSP: 002b:00007fe9b5aee8c0 EFLAGS: 00000297 ORIG_RAX:
> > 0000000000000113
> > [ 133.553661] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe9bab60953
> > [ 133.553661] RDX: 0000000000000003 RSI: 0000000000000000 RDI: 00000000000000cd
> > [ 133.553661] RBP: 00007fe9b5aee9d0 R08: 0000000000040010 R09: 0000000000000001
> > [ 133.553661] R10: 0000000000000000 R11: 0000000000000297 R12: 0000000000040010
> > [ 133.553661] R13: 00007fe9b5aee9e0 R14: 00007fe9b5aeea00 R15: 00000624ffc22440
> > [ 133.553661] Modules linked in: 9p 9pnet_virtio 9pnet vfat fat
> > virtio_net net_failover failover
> > [ 133.625460] ---[ end trace bef475c49a5ebe99 ]---
> >
> > The following reproduces the panic.
> >
> > In libfuse modify and build example/passthrough.c:
> > $ git clone https://github.com/libfuse/libfuse
> > $ cd libfuse
> > # Enable FUSE_CAP_SLICE_WRITE and FUSE_CAP_SPLICE_MOVE in xmp_init.
> > $ sed -i 's/(void) conn;/conn->want |= FUSE_CAP_SPLICE_WRITE |
> > FUSE_CAP_SPLICE_MOVE;/' example/passthrough.c
> > $ mkdir build; cd build; meson ..; ninja
> >
> > $ mkdir /tmp/fuse
> > $ example/passthrough /tmp/fuse
> > $ dd if=/dev/zero of=/tmp/zeros bs=1M count=100
> > $ while true; do /tmp/fuse/bin/cp /tmp/fuse/tmp/zeros /dev/null & done
>
> Yes, I can reproduce this too.
>
> Thanks,
> Miklos
next prev parent reply other threads:[~2021-10-20 21:27 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAAmZXrsGg2xsP1CK+cbuEMumtrqdvD-NKnWzhNcvn71RV3c1yw@mail.gmail.com>
2021-10-07 12:54 ` Miklos Szeredi
2021-10-20 21:26 ` Frank Dinoff [this message]
2021-10-28 9:45 ` Miklos Szeredi
2021-10-28 14:09 ` Miklos Szeredi
2021-11-11 22:18 ` Frank Dinoff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAAmZXrtiJcmLzf6eb90RKdCs3Q=mFNCqAD86nZQJmVwr6YwEmA@mail.gmail.com' \
--to=fdinoff@google.com \
--cc=hannes@cmpxchg.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=miklos@szeredi.hu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox