* [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) @ 2023-04-21 14:33 syzbot 2023-04-21 14:43 ` Dmitry Vyukov 0 siblings, 1 reply; 8+ messages in thread From: syzbot @ 2023-04-21 14:33 UTC (permalink / raw) To: adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso Hello, syzbot found the following issue on: HEAD commit: 76f598ba7d8e Merge tag 'for-linus' of git://git.kernel.org.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=133bfbedc80000 kernel config: https://syzkaller.appspot.com/x/.config?x=9c5d44636e91081b dashboard link: https://syzkaller.appspot.com/bug?extid=c2de99a72baaa06d31f3 compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 Unfortunately, I don't have any reproducer for this issue yet. Downloadable assets: disk image: https://storage.googleapis.com/syzbot-assets/a3654f5f77b9/disk-76f598ba.raw.xz vmlinux: https://storage.googleapis.com/syzbot-assets/abfb4aaa5772/vmlinux-76f598ba.xz kernel image: https://storage.googleapis.com/syzbot-assets/789fb5546551/bzImage-76f598ba.xz IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com ================================================================== BUG: KCSAN: data-race in strscpy / strscpy write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: strscpy+0xa9/0x170 lib/string.c:165 strscpy_pad+0x27/0x80 lib/string_helpers.c:835 __set_task_comm+0x46/0x140 fs/exec.c:1232 set_task_comm include/linux/sched.h:1984 [inline] __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 __ext4_fill_super fs/ext4/super.c:5480 [inline] ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 vfs_get_tree+0x51/0x190 fs/super.c:1510 do_new_mount+0x200/0x650 fs/namespace.c:3042 path_mount+0x498/0xb40 fs/namespace.c:3372 do_mount fs/namespace.c:3385 [inline] __do_sys_mount fs/namespace.c:3594 [inline] __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x63/0xcd read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: strscpy+0xde/0x170 lib/string.c:174 ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] __bpf_prog_run include/linux/filter.h:601 [inline] bpf_prog_run include/linux/filter.h:608 [inline] __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 trace_sched_switch include/trace/events/sched.h:222 [inline] __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 schedule+0x51/0x80 kernel/sched/core.c:6701 schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 kthread+0x11c/0x1e0 kernel/kthread.c:369 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 value changed: 0x72 -> 0x34 Reported by Kernel Concurrency Sanitizer on: CPU: 0 PID: 16161 Comm: ext4lazyinit Not tainted 6.3.0-rc5-syzkaller-00022-g76f598ba7d8e #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023 ================================================================== --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 14:33 [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) syzbot @ 2023-04-21 14:43 ` Dmitry Vyukov 2023-04-21 14:58 ` Matthew Wilcox 2023-04-21 15:15 ` Alexei Starovoitov 0 siblings, 2 replies; 8+ messages in thread From: Dmitry Vyukov @ 2023-04-21 14:43 UTC (permalink / raw) To: syzbot Cc: adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh On Fri, 21 Apr 2023 at 16:33, syzbot <syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following issue on: > > HEAD commit: 76f598ba7d8e Merge tag 'for-linus' of git://git.kernel.org.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=133bfbedc80000 > kernel config: https://syzkaller.appspot.com/x/.config?x=9c5d44636e91081b > dashboard link: https://syzkaller.appspot.com/bug?extid=c2de99a72baaa06d31f3 > compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 > > Unfortunately, I don't have any reproducer for this issue yet. > > Downloadable assets: > disk image: https://storage.googleapis.com/syzbot-assets/a3654f5f77b9/disk-76f598ba.raw.xz > vmlinux: https://storage.googleapis.com/syzbot-assets/abfb4aaa5772/vmlinux-76f598ba.xz > kernel image: https://storage.googleapis.com/syzbot-assets/789fb5546551/bzImage-76f598ba.xz > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com +bpf maintainers If I am reading this correctly, this can cause a leak of kernel memory and/or crash via bpf_get_current_comm helper. strcpy() can temporary leave comm buffer non-0 terminated as it terminates it only after the copy: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/string.c?id=76f598ba7d8e2bfb4855b5298caedd5af0c374a8#n184 If bpf_get_current_comm() observes such non-0-terminated comm, it will start reading off bounds. > ================================================================== > BUG: KCSAN: data-race in strscpy / strscpy > > write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: > strscpy+0xa9/0x170 lib/string.c:165 > strscpy_pad+0x27/0x80 lib/string_helpers.c:835 > __set_task_comm+0x46/0x140 fs/exec.c:1232 > set_task_comm include/linux/sched.h:1984 [inline] > __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 > kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 > ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] > ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 > __ext4_fill_super fs/ext4/super.c:5480 [inline] > ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 > get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 > ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 > vfs_get_tree+0x51/0x190 fs/super.c:1510 > do_new_mount+0x200/0x650 fs/namespace.c:3042 > path_mount+0x498/0xb40 fs/namespace.c:3372 > do_mount fs/namespace.c:3385 [inline] > __do_sys_mount fs/namespace.c:3594 [inline] > __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 > __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 > do_syscall_x64 arch/x86/entry/common.c:50 [inline] > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 > entry_SYSCALL_64_after_hwframe+0x63/0xcd > > read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: > strscpy+0xde/0x170 lib/string.c:174 > ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] > bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 > ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 > __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 > bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] > __bpf_prog_run include/linux/filter.h:601 [inline] > bpf_prog_run include/linux/filter.h:608 [inline] > __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] > bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 > __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 > trace_sched_switch include/trace/events/sched.h:222 [inline] > __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 > schedule+0x51/0x80 kernel/sched/core.c:6701 > schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 > kthread+0x11c/0x1e0 kernel/kthread.c:369 > ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 > > value changed: 0x72 -> 0x34 > > Reported by Kernel Concurrency Sanitizer on: > CPU: 0 PID: 16161 Comm: ext4lazyinit Not tainted 6.3.0-rc5-syzkaller-00022-g76f598ba7d8e #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023 > ================================================================== > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 14:43 ` Dmitry Vyukov @ 2023-04-21 14:58 ` Matthew Wilcox 2023-04-21 17:40 ` Mike Christie 2023-04-21 15:15 ` Alexei Starovoitov 1 sibling, 1 reply; 8+ messages in thread From: Matthew Wilcox @ 2023-04-21 14:58 UTC (permalink / raw) To: Dmitry Vyukov Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh, Ingo Molnar, Peter Zijlstra, Mike Christie On Fri, Apr 21, 2023 at 04:43:19PM +0200, Dmitry Vyukov wrote: > On Fri, 21 Apr 2023 at 16:33, syzbot > <syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com> wrote: > > > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: 76f598ba7d8e Merge tag 'for-linus' of git://git.kernel.org.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=133bfbedc80000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=9c5d44636e91081b > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2de99a72baaa06d31f3 > > compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/a3654f5f77b9/disk-76f598ba.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/abfb4aaa5772/vmlinux-76f598ba.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/789fb5546551/bzImage-76f598ba.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com > > +bpf maintainers > > If I am reading this correctly, this can cause a leak of kernel memory > and/or crash via bpf_get_current_comm helper. > > strcpy() can temporary leave comm buffer non-0 terminated as it > terminates it only after the copy: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/string.c?id=76f598ba7d8e2bfb4855b5298caedd5af0c374a8#n184 > > If bpf_get_current_comm() observes such non-0-terminated comm, it will > start reading off bounds. Just to be clear, this isn't ext4 at all; ext4 happens to be calling kthread_create(), but it's actually a generic kthread problem, right? I'm not sure how it is that bpf is able to see the task before comm is initialised; that seems to be the real race here, that comm is not set before the kthread is a schedulable entity? Adding the scheduler people. > > ================================================================== > > BUG: KCSAN: data-race in strscpy / strscpy > > > > write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: > > strscpy+0xa9/0x170 lib/string.c:165 > > strscpy_pad+0x27/0x80 lib/string_helpers.c:835 > > __set_task_comm+0x46/0x140 fs/exec.c:1232 > > set_task_comm include/linux/sched.h:1984 [inline] > > __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 > > kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 > > ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] > > ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 > > __ext4_fill_super fs/ext4/super.c:5480 [inline] > > ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 > > get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 > > ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 > > vfs_get_tree+0x51/0x190 fs/super.c:1510 > > do_new_mount+0x200/0x650 fs/namespace.c:3042 > > path_mount+0x498/0xb40 fs/namespace.c:3372 > > do_mount fs/namespace.c:3385 [inline] > > __do_sys_mount fs/namespace.c:3594 [inline] > > __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 > > __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 > > do_syscall_x64 arch/x86/entry/common.c:50 [inline] > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 > > entry_SYSCALL_64_after_hwframe+0x63/0xcd > > > > read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: > > strscpy+0xde/0x170 lib/string.c:174 > > ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] > > bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 > > ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 > > __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 > > bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] > > __bpf_prog_run include/linux/filter.h:601 [inline] > > bpf_prog_run include/linux/filter.h:608 [inline] > > __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] > > bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 > > __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 > > trace_sched_switch include/trace/events/sched.h:222 [inline] > > __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 > > schedule+0x51/0x80 kernel/sched/core.c:6701 > > schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 > > kthread+0x11c/0x1e0 kernel/kthread.c:369 > > ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 > > > > value changed: 0x72 -> 0x34 > > > > Reported by Kernel Concurrency Sanitizer on: > > CPU: 0 PID: 16161 Comm: ext4lazyinit Not tainted 6.3.0-rc5-syzkaller-00022-g76f598ba7d8e #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023 > > ================================================================== > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 14:58 ` Matthew Wilcox @ 2023-04-21 17:40 ` Mike Christie 2023-04-21 18:20 ` Theodore Ts'o 2023-04-24 7:59 ` Christian Brauner 0 siblings, 2 replies; 8+ messages in thread From: Mike Christie @ 2023-04-21 17:40 UTC (permalink / raw) To: Matthew Wilcox, Dmitry Vyukov Cc: syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh, Ingo Molnar, Peter Zijlstra, Christian Brauner cc'ing Christian, because I might have fixed this with a patch in his tree. On 4/21/23 9:58 AM, Matthew Wilcox wrote: > I'm not sure how it is that bpf is able to see the task before comm is > initialised; that seems to be the real race here, that comm is not set > before the kthread is a schedulable entity? Adding the scheduler people. > >>> ================================================================== >>> BUG: KCSAN: data-race in strscpy / strscpy >>> >>> write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: >>> strscpy+0xa9/0x170 lib/string.c:165 >>> strscpy_pad+0x27/0x80 lib/string_helpers.c:835 >>> __set_task_comm+0x46/0x140 fs/exec.c:1232 >>> set_task_comm include/linux/sched.h:1984 [inline] >>> __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 >>> kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 >>> ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] >>> ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 >>> __ext4_fill_super fs/ext4/super.c:5480 [inline] >>> ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 >>> get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 >>> ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 >>> vfs_get_tree+0x51/0x190 fs/super.c:1510 >>> do_new_mount+0x200/0x650 fs/namespace.c:3042 >>> path_mount+0x498/0xb40 fs/namespace.c:3372 >>> do_mount fs/namespace.c:3385 [inline] >>> __do_sys_mount fs/namespace.c:3594 [inline] >>> __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 >>> __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 >>> do_syscall_x64 arch/x86/entry/common.c:50 [inline] >>> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 >>> entry_SYSCALL_64_after_hwframe+0x63/0xcd >>> >>> read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: >>> strscpy+0xde/0x170 lib/string.c:174 >>> ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] >>> bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 >>> ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 >>> __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 >>> bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] >>> __bpf_prog_run include/linux/filter.h:601 [inline] >>> bpf_prog_run include/linux/filter.h:608 [inline] >>> __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] >>> bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 >>> __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 >>> trace_sched_switch include/trace/events/sched.h:222 [inline] >>> __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 >>> schedule+0x51/0x80 kernel/sched/core.c:6701 >>> schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 >>> kthread+0x11c/0x1e0 kernel/kthread.c:369 >>> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 >>> I didn't see the beginning of this thread and I think the part of the sysbot report that lists the patches/trees being used got cut off so I'm not 100% sure what's in the kernel. In Linus's current tree we do set_task_comm in __kthread_create_on_node after waiting on the kthread_create_info completion which is completed by threadd(). At this time, kthread() has already done the complete() on the kthread_create_info completion and started to run the threadfn function and that could be running. So we can hit the race that way. In linux next, from https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=kernel.user_worker we have: commit cf587db2ee0261c74d04f61f39783db88a0b65e4 Author: Mike Christie <michael.christie@oracle.com> Date: Fri Mar 10 16:03:23 2023 -0600 kernel: Allow a kernel thread's name to be set in copy_process and so now copy_process() sets the name before the taskfn is started, so we shouldn't hit any races like above. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 17:40 ` Mike Christie @ 2023-04-21 18:20 ` Theodore Ts'o 2023-04-24 7:59 ` Christian Brauner 1 sibling, 0 replies; 8+ messages in thread From: Theodore Ts'o @ 2023-04-21 18:20 UTC (permalink / raw) To: Mike Christie Cc: Matthew Wilcox, Dmitry Vyukov, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh, Ingo Molnar, Peter Zijlstra, Christian Brauner On Fri, Apr 21, 2023 at 12:40:45PM -0500, Mike Christie wrote: > > I didn't see the beginning of this thread and I think the part of the > sysbot report that lists the patches/trees being used got cut off so > I'm not 100% sure what's in the kernel. Syzbot reported this on commit 76f598ba7d8e which is upstream after v6.3-rc6. Cheers, - Ted ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 17:40 ` Mike Christie 2023-04-21 18:20 ` Theodore Ts'o @ 2023-04-24 7:59 ` Christian Brauner 2023-04-24 8:25 ` Dmitry Vyukov 1 sibling, 1 reply; 8+ messages in thread From: Christian Brauner @ 2023-04-24 7:59 UTC (permalink / raw) To: Mike Christie Cc: Matthew Wilcox, Dmitry Vyukov, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh, Ingo Molnar, Peter Zijlstra On Fri, Apr 21, 2023 at 12:40:45PM -0500, Mike Christie wrote: > cc'ing Christian, because I might have fixed this with a patch in > his tree. > > On 4/21/23 9:58 AM, Matthew Wilcox wrote: > > I'm not sure how it is that bpf is able to see the task before comm is > > initialised; that seems to be the real race here, that comm is not set > > before the kthread is a schedulable entity? Adding the scheduler people. > > > >>> ================================================================== > >>> BUG: KCSAN: data-race in strscpy / strscpy > >>> > >>> write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: > >>> strscpy+0xa9/0x170 lib/string.c:165 > >>> strscpy_pad+0x27/0x80 lib/string_helpers.c:835 > >>> __set_task_comm+0x46/0x140 fs/exec.c:1232 > >>> set_task_comm include/linux/sched.h:1984 [inline] > >>> __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 > >>> kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 > >>> ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] > >>> ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 > >>> __ext4_fill_super fs/ext4/super.c:5480 [inline] > >>> ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 > >>> get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 > >>> ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 > >>> vfs_get_tree+0x51/0x190 fs/super.c:1510 > >>> do_new_mount+0x200/0x650 fs/namespace.c:3042 > >>> path_mount+0x498/0xb40 fs/namespace.c:3372 > >>> do_mount fs/namespace.c:3385 [inline] > >>> __do_sys_mount fs/namespace.c:3594 [inline] > >>> __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 > >>> __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 > >>> do_syscall_x64 arch/x86/entry/common.c:50 [inline] > >>> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 > >>> entry_SYSCALL_64_after_hwframe+0x63/0xcd > >>> > >>> read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: > >>> strscpy+0xde/0x170 lib/string.c:174 > >>> ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] > >>> bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 > >>> ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 > >>> __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 > >>> bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] > >>> __bpf_prog_run include/linux/filter.h:601 [inline] > >>> bpf_prog_run include/linux/filter.h:608 [inline] > >>> __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] > >>> bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 > >>> __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 > >>> trace_sched_switch include/trace/events/sched.h:222 [inline] > >>> __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 > >>> schedule+0x51/0x80 kernel/sched/core.c:6701 > >>> schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 > >>> kthread+0x11c/0x1e0 kernel/kthread.c:369 > >>> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 > >>> > > > I didn't see the beginning of this thread and I think the part of the > sysbot report that lists the patches/trees being used got cut off so > I'm not 100% sure what's in the kernel. > > In Linus's current tree we do set_task_comm in __kthread_create_on_node > after waiting on the kthread_create_info completion which is completed by > threadd(). At this time, kthread() has already done the complete() on the > kthread_create_info completion and started to run the threadfn function and > that could be running. So we can hit the race that way. > > > In linux next, from > https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=kernel.user_worker > we have: > > commit cf587db2ee0261c74d04f61f39783db88a0b65e4 > Author: Mike Christie <michael.christie@oracle.com> > Date: Fri Mar 10 16:03:23 2023 -0600 > > kernel: Allow a kernel thread's name to be set in copy_process > > and so now copy_process() sets the name before the taskfn is started, so we > shouldn't hit any races like above. Yeah, that looks like it should fix it. Afaict, this has no reproducer so there's no point in letting syzbot test on this. I've sent the pull request for the kernel user worker series on Friday. So I guess we'll see whether it's reproducible on v6.4-rc1. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-24 7:59 ` Christian Brauner @ 2023-04-24 8:25 ` Dmitry Vyukov 0 siblings, 0 replies; 8+ messages in thread From: Dmitry Vyukov @ 2023-04-24 8:25 UTC (permalink / raw) To: Christian Brauner Cc: Mike Christie, Matthew Wilcox, syzbot, adilger.kernel, linux-ext4, linux-fsdevel, linux-kernel, linux-mm, llvm, nathan, ndesaulniers, syzkaller-bugs, trix, tytso, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, martin.lau, bpf, KP Singh, Ingo Molnar, Peter Zijlstra On Mon, 24 Apr 2023 at 09:59, Christian Brauner <brauner@kernel.org> wrote: > > On Fri, Apr 21, 2023 at 12:40:45PM -0500, Mike Christie wrote: > > cc'ing Christian, because I might have fixed this with a patch in > > his tree. > > > > On 4/21/23 9:58 AM, Matthew Wilcox wrote: > > > I'm not sure how it is that bpf is able to see the task before comm is > > > initialised; that seems to be the real race here, that comm is not set > > > before the kthread is a schedulable entity? Adding the scheduler people. > > > > > >>> ================================================================== > > >>> BUG: KCSAN: data-race in strscpy / strscpy > > >>> > > >>> write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: > > >>> strscpy+0xa9/0x170 lib/string.c:165 > > >>> strscpy_pad+0x27/0x80 lib/string_helpers.c:835 > > >>> __set_task_comm+0x46/0x140 fs/exec.c:1232 > > >>> set_task_comm include/linux/sched.h:1984 [inline] > > >>> __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 > > >>> kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 > > >>> ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] > > >>> ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 > > >>> __ext4_fill_super fs/ext4/super.c:5480 [inline] > > >>> ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 > > >>> get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 > > >>> ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 > > >>> vfs_get_tree+0x51/0x190 fs/super.c:1510 > > >>> do_new_mount+0x200/0x650 fs/namespace.c:3042 > > >>> path_mount+0x498/0xb40 fs/namespace.c:3372 > > >>> do_mount fs/namespace.c:3385 [inline] > > >>> __do_sys_mount fs/namespace.c:3594 [inline] > > >>> __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 > > >>> __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 > > >>> do_syscall_x64 arch/x86/entry/common.c:50 [inline] > > >>> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 > > >>> entry_SYSCALL_64_after_hwframe+0x63/0xcd > > >>> > > >>> read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: > > >>> strscpy+0xde/0x170 lib/string.c:174 > > >>> ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] > > >>> bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 > > >>> ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 > > >>> __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 > > >>> bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] > > >>> __bpf_prog_run include/linux/filter.h:601 [inline] > > >>> bpf_prog_run include/linux/filter.h:608 [inline] > > >>> __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] > > >>> bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 > > >>> __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 > > >>> trace_sched_switch include/trace/events/sched.h:222 [inline] > > >>> __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 > > >>> schedule+0x51/0x80 kernel/sched/core.c:6701 > > >>> schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 > > >>> kthread+0x11c/0x1e0 kernel/kthread.c:369 > > >>> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 > > >>> > > > > > > I didn't see the beginning of this thread and I think the part of the > > sysbot report that lists the patches/trees being used got cut off so > > I'm not 100% sure what's in the kernel. > > > > In Linus's current tree we do set_task_comm in __kthread_create_on_node > > after waiting on the kthread_create_info completion which is completed by > > threadd(). At this time, kthread() has already done the complete() on the > > kthread_create_info completion and started to run the threadfn function and > > that could be running. So we can hit the race that way. > > > > > > In linux next, from > > https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=kernel.user_worker > > we have: > > > > commit cf587db2ee0261c74d04f61f39783db88a0b65e4 > > Author: Mike Christie <michael.christie@oracle.com> > > Date: Fri Mar 10 16:03:23 2023 -0600 > > > > kernel: Allow a kernel thread's name to be set in copy_process > > > > and so now copy_process() sets the name before the taskfn is started, so we > > shouldn't hit any races like above. > > Yeah, that looks like it should fix it. > > Afaict, this has no reproducer so there's no point in letting syzbot > test on this. I've sent the pull request for the kernel user worker > series on Friday. So I guess we'll see whether it's reproducible on > v6.4-rc1. To see if it still happens to tell syzbot about the fix, then it will remind if it still happens with the fix or not. Otherwise everybody will forget about this tomorrow ;) #syz fix: kernel: Allow a kernel thread's name to be set in copy_process Btw, a similar race will still be possible b/c it's possible to change the name at any point by writing to /proc/self/task/[tid]/comm. But I am not sure how provably dangerous it will be and there is a different attitude towards fixing races proactively for different kernel subsystems, so we will probably not report it. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) 2023-04-21 14:43 ` Dmitry Vyukov 2023-04-21 14:58 ` Matthew Wilcox @ 2023-04-21 15:15 ` Alexei Starovoitov 1 sibling, 0 replies; 8+ messages in thread From: Alexei Starovoitov @ 2023-04-21 15:15 UTC (permalink / raw) To: Dmitry Vyukov Cc: syzbot, adilger.kernel, linux-ext4, Linux-Fsdevel, LKML, linux-mm, clang-built-linux, Nathan Chancellor, Nick Desaulniers, syzkaller-bugs, Tom Rix, Theodore Ts'o, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau, bpf, KP Singh On Fri, Apr 21, 2023 at 7:43 AM Dmitry Vyukov <dvyukov@google.com> wrote: > > On Fri, 21 Apr 2023 at 16:33, syzbot > <syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com> wrote: > > > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: 76f598ba7d8e Merge tag 'for-linus' of git://git.kernel.org.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=133bfbedc80000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=9c5d44636e91081b > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2de99a72baaa06d31f3 > > compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/a3654f5f77b9/disk-76f598ba.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/abfb4aaa5772/vmlinux-76f598ba.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/789fb5546551/bzImage-76f598ba.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+c2de99a72baaa06d31f3@syzkaller.appspotmail.com > > +bpf maintainers > > If I am reading this correctly, this can cause a leak of kernel memory > and/or crash via bpf_get_current_comm helper. > > strcpy() can temporary leave comm buffer non-0 terminated as it > terminates it only after the copy: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/string.c?id=76f598ba7d8e2bfb4855b5298caedd5af0c374a8#n184 > > If bpf_get_current_comm() observes such non-0-terminated comm, it will > start reading off bounds. fyi it's using strscpy_pad() in bpf-next. See commit f3f213497797 ("bpf: ensure all memory is initialized in bpf_get_current_comm") but this race might still exist. not sure. > > ================================================================== > > BUG: KCSAN: data-race in strscpy / strscpy > > > > write to 0xffff88812ed8b730 of 8 bytes by task 16157 on cpu 1: > > strscpy+0xa9/0x170 lib/string.c:165 > > strscpy_pad+0x27/0x80 lib/string_helpers.c:835 > > __set_task_comm+0x46/0x140 fs/exec.c:1232 > > set_task_comm include/linux/sched.h:1984 [inline] > > __kthread_create_on_node+0x2b2/0x320 kernel/kthread.c:474 > > kthread_create_on_node+0x8a/0xb0 kernel/kthread.c:512 > > ext4_run_lazyinit_thread fs/ext4/super.c:3848 [inline] > > ext4_register_li_request+0x407/0x650 fs/ext4/super.c:3983 > > __ext4_fill_super fs/ext4/super.c:5480 [inline] > > ext4_fill_super+0x3f4a/0x43f0 fs/ext4/super.c:5637 > > get_tree_bdev+0x2b1/0x3a0 fs/super.c:1303 > > ext4_get_tree+0x1c/0x20 fs/ext4/super.c:5668 > > vfs_get_tree+0x51/0x190 fs/super.c:1510 > > do_new_mount+0x200/0x650 fs/namespace.c:3042 > > path_mount+0x498/0xb40 fs/namespace.c:3372 > > do_mount fs/namespace.c:3385 [inline] > > __do_sys_mount fs/namespace.c:3594 [inline] > > __se_sys_mount+0x27f/0x2d0 fs/namespace.c:3571 > > __x64_sys_mount+0x67/0x80 fs/namespace.c:3571 > > do_syscall_x64 arch/x86/entry/common.c:50 [inline] > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80 > > entry_SYSCALL_64_after_hwframe+0x63/0xcd > > > > read to 0xffff88812ed8b733 of 1 bytes by task 16161 on cpu 0: > > strscpy+0xde/0x170 lib/string.c:174 > > ____bpf_get_current_comm kernel/bpf/helpers.c:260 [inline] > > bpf_get_current_comm+0x45/0x70 kernel/bpf/helpers.c:252 > > ___bpf_prog_run+0x281/0x3050 kernel/bpf/core.c:1822 > > __bpf_prog_run32+0x74/0xa0 kernel/bpf/core.c:2043 > > bpf_dispatcher_nop_func include/linux/bpf.h:1124 [inline] > > __bpf_prog_run include/linux/filter.h:601 [inline] > > bpf_prog_run include/linux/filter.h:608 [inline] > > __bpf_trace_run kernel/trace/bpf_trace.c:2263 [inline] > > bpf_trace_run4+0x9f/0x140 kernel/trace/bpf_trace.c:2304 > > __traceiter_sched_switch+0x3a/0x50 include/trace/events/sched.h:222 > > trace_sched_switch include/trace/events/sched.h:222 [inline] > > __schedule+0x7e7/0x8e0 kernel/sched/core.c:6622 > > schedule+0x51/0x80 kernel/sched/core.c:6701 > > schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:6760 > > kthread+0x11c/0x1e0 kernel/kthread.c:369 > > ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308 > > > > value changed: 0x72 -> 0x34 > > > > Reported by Kernel Concurrency Sanitizer on: > > CPU: 0 PID: 16161 Comm: ext4lazyinit Not tainted 6.3.0-rc5-syzkaller-00022-g76f598ba7d8e #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/30/2023 > > ================================================================== > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-04-24 8:25 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-04-21 14:33 [syzbot] [ext4?] [mm?] KCSAN: data-race in strscpy / strscpy (3) syzbot 2023-04-21 14:43 ` Dmitry Vyukov 2023-04-21 14:58 ` Matthew Wilcox 2023-04-21 17:40 ` Mike Christie 2023-04-21 18:20 ` Theodore Ts'o 2023-04-24 7:59 ` Christian Brauner 2023-04-24 8:25 ` Dmitry Vyukov 2023-04-21 15:15 ` Alexei Starovoitov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox