* [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one @ 2025-02-05 11:41 syzbot 2025-02-05 15:00 ` Jann Horn 2025-02-05 15:07 ` Lorenzo Stoakes 0 siblings, 2 replies; 14+ messages in thread From: syzbot @ 2025-02-05 11:41 UTC (permalink / raw) To: Liam.Howlett, akpm, jannh, linux-kernel, linux-mm, lorenzo.stoakes, syzkaller-bugs, vbabka Hello, syzbot found the following issue on: HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 Unfortunately, I don't have any reproducer for this issue yet. Downloadable assets: disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com ================================================================== BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1: vm_flags_init include/linux/mm.h:875 [inline] vm_flags_reset include/linux/mm.h:887 [inline] mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 __do_sys_mprotect mm/mprotect.c:861 [inline] __se_sys_mprotect mm/mprotect.c:858 [inline] __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 try_to_migrate+0x11f/0x150 migrate_folio_unmap mm/migrate.c:1320 [inline] migrate_pages_batch+0x786/0x1930 mm/migrate.c:1866 migrate_pages_sync mm/migrate.c:1989 [inline] migrate_pages+0xf02/0x1840 mm/migrate.c:2098 do_mbind mm/mempolicy.c:1394 [inline] kernel_mbind mm/mempolicy.c:1537 [inline] __do_sys_mbind mm/mempolicy.c:1611 [inline] __se_sys_mbind+0xfd1/0x11c0 mm/mempolicy.c:1607 __x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1607 x64_sys_call+0x2662/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:238 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f value changed: 0x0000000000102077 -> 0x0000000000102071 Reported by Kernel Concurrency Sanitizer on: CPU: 0 UID: 0 PID: 6418 Comm: syz.0.1339 Not tainted 6.14.0-rc1-syzkaller-00026-gd009de7d5428 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024 ================================================================== --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. If the report is already addressed, let syzbot know by replying with: #syz fix: exact-commit-title If you want to overwrite report's subsystems, reply with: #syz set subsystems: new-subsystem (See the list of subsystem names on the web dashboard) If the report is a duplicate of another one, reply with: #syz dup: exact-subject-of-another-report If you want to undo deduplication, reply with: #syz undup ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 11:41 [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one syzbot @ 2025-02-05 15:00 ` Jann Horn 2025-02-05 15:11 ` Lorenzo Stoakes 2025-02-05 15:47 ` Liam R. Howlett 2025-02-05 15:07 ` Lorenzo Stoakes 1 sibling, 2 replies; 14+ messages in thread From: Jann Horn @ 2025-02-05 15:00 UTC (permalink / raw) To: syzbot Cc: Liam.Howlett, akpm, linux-kernel, linux-mm, lorenzo.stoakes, syzkaller-bugs, vbabka On Wed, Feb 5, 2025 at 12:41 PM syzbot <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > syzbot found the following issue on: > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > Unfortunately, I don't have any reproducer for this issue yet. > > Downloadable assets: > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > ================================================================== > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > > write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1: > vm_flags_init include/linux/mm.h:875 [inline] > vm_flags_reset include/linux/mm.h:887 [inline] > mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 > do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 This is one side changing the VMA flags under the mmap lock in write mode... > __do_sys_mprotect mm/mprotect.c:861 [inline] > __se_sys_mprotect mm/mprotect.c:858 [inline] > __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 > x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 ... while the other side comes through the rmap, which does not involve the mmap lock. Yes, that does not have any mutual locking by design, I think. The comments in the VMA flags code incorrectly assume that no concurrency is possible here; and I think the comment in mprotect_fixup() about protection by the mmap_lock has also been kinda wrong since the beginning of git history. The VM_LOCKED check in the migration code was added by Hugh in commit b74355078b655, but that's just one example syzbot stumbled over; we have similar racy vm_flags reads through the rmap on other paths like: unmap_mapping_range_tree -> unmap_mapping_range_vma -> zap_page_range_single -> unmap_single_vma -> unmap_page_range -> ... -> zap_pte_range -> zap_present_ptes -> vm_normal_page I think the right fix might just be to make sure that we use WRITE_ONCE() for these vm_flags updates, and READ_ONCE() around ->vm_flags reads that can happen in rmap walk paths, though we should think about the consequences of concurrently changing flags in every place that gets a READ_ONCE()... > try_to_migrate+0x11f/0x150 > migrate_folio_unmap mm/migrate.c:1320 [inline] > migrate_pages_batch+0x786/0x1930 mm/migrate.c:1866 > migrate_pages_sync mm/migrate.c:1989 [inline] > migrate_pages+0xf02/0x1840 mm/migrate.c:2098 > do_mbind mm/mempolicy.c:1394 [inline] > kernel_mbind mm/mempolicy.c:1537 [inline] > __do_sys_mbind mm/mempolicy.c:1611 [inline] > __se_sys_mbind+0xfd1/0x11c0 mm/mempolicy.c:1607 > __x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1607 > x64_sys_call+0x2662/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:238 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > value changed: 0x0000000000102077 -> 0x0000000000102071 > > Reported by Kernel Concurrency Sanitizer on: > CPU: 0 UID: 0 PID: 6418 Comm: syz.0.1339 Not tainted 6.14.0-rc1-syzkaller-00026-gd009de7d5428 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024 > ================================================================== > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > If the report is already addressed, let syzbot know by replying with: > #syz fix: exact-commit-title > > If you want to overwrite report's subsystems, reply with: > #syz set subsystems: new-subsystem > (See the list of subsystem names on the web dashboard) > > If the report is a duplicate of another one, reply with: > #syz dup: exact-subject-of-another-report > > If you want to undo deduplication, reply with: > #syz undup ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:00 ` Jann Horn @ 2025-02-05 15:11 ` Lorenzo Stoakes 2025-02-05 15:14 ` Jann Horn 2025-02-05 15:46 ` Marco Elver 2025-02-05 15:47 ` Liam R. Howlett 1 sibling, 2 replies; 14+ messages in thread From: Lorenzo Stoakes @ 2025-02-05 15:11 UTC (permalink / raw) To: Jann Horn Cc: syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka On Wed, Feb 05, 2025 at 04:00:06PM +0100, Jann Horn wrote: > On Wed, Feb 5, 2025 at 12:41 PM syzbot > <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > > syzbot found the following issue on: > > > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > > > ================================================================== > > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > > > > write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1: > > vm_flags_init include/linux/mm.h:875 [inline] > > vm_flags_reset include/linux/mm.h:887 [inline] > > mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 > > do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 > > This is one side changing the VMA flags under the mmap lock in write mode... > > > __do_sys_mprotect mm/mprotect.c:861 [inline] > > __se_sys_mprotect mm/mprotect.c:858 [inline] > > __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 > > x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > > rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 > > ... while the other side comes through the rmap, which does not > involve the mmap lock. Yes, that does not have any mutual locking by > design, I think. > > The comments in the VMA flags code incorrectly assume that no > concurrency is possible here; and I think the comment in > mprotect_fixup() about protection by the mmap_lock has also been kinda > wrong since the beginning of git history. > > The VM_LOCKED check in the migration code was added by Hugh in commit > b74355078b655, but that's just one example syzbot stumbled over; we > have similar racy vm_flags reads through the rmap on other paths like: > > unmap_mapping_range_tree -> unmap_mapping_range_vma -> > zap_page_range_single -> unmap_single_vma -> unmap_page_range -> ... > -> zap_pte_range -> zap_present_ptes -> vm_normal_page > > I think the right fix might just be to make sure that we use > WRITE_ONCE() for these vm_flags updates, and READ_ONCE() around > ->vm_flags reads that can happen in rmap walk paths, though we should > think about the consequences of concurrently changing flags in every > place that gets a READ_ONCE()... Yup cool similar to my thread on this. I hate that we have these landmines waiting for us. Be good to find a way to explicitly annotate this, or at least comment somehow. But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at least for the proximate thing. It's a wonder these things don't trigger more, except you need probably very precise timing to do it... I can do a quick cheeky patch. > > > > try_to_migrate+0x11f/0x150 > > migrate_folio_unmap mm/migrate.c:1320 [inline] > > migrate_pages_batch+0x786/0x1930 mm/migrate.c:1866 > > migrate_pages_sync mm/migrate.c:1989 [inline] > > migrate_pages+0xf02/0x1840 mm/migrate.c:2098 > > do_mbind mm/mempolicy.c:1394 [inline] > > kernel_mbind mm/mempolicy.c:1537 [inline] > > __do_sys_mbind mm/mempolicy.c:1611 [inline] > > __se_sys_mbind+0xfd1/0x11c0 mm/mempolicy.c:1607 > > __x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1607 > > x64_sys_call+0x2662/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:238 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > value changed: 0x0000000000102077 -> 0x0000000000102071 > > > > Reported by Kernel Concurrency Sanitizer on: > > CPU: 0 UID: 0 PID: 6418 Comm: syz.0.1339 Not tainted 6.14.0-rc1-syzkaller-00026-gd009de7d5428 #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024 > > ================================================================== > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > If the report is already addressed, let syzbot know by replying with: > > #syz fix: exact-commit-title > > > > If you want to overwrite report's subsystems, reply with: > > #syz set subsystems: new-subsystem > > (See the list of subsystem names on the web dashboard) > > > > If the report is a duplicate of another one, reply with: > > #syz dup: exact-subject-of-another-report > > > > If you want to undo deduplication, reply with: > > #syz undup ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:11 ` Lorenzo Stoakes @ 2025-02-05 15:14 ` Jann Horn 2025-02-05 15:46 ` Marco Elver 1 sibling, 0 replies; 14+ messages in thread From: Jann Horn @ 2025-02-05 15:14 UTC (permalink / raw) To: Lorenzo Stoakes Cc: syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka On Wed, Feb 5, 2025 at 4:11 PM Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote: > On Wed, Feb 05, 2025 at 04:00:06PM +0100, Jann Horn wrote: > > On Wed, Feb 5, 2025 at 12:41 PM syzbot > > <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > > > syzbot found the following issue on: > > > > > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > > > git tree: upstream > > > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > Downloadable assets: > > > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > > > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > > > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > > > > > ================================================================== > > > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > > > > > > write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1: > > > vm_flags_init include/linux/mm.h:875 [inline] > > > vm_flags_reset include/linux/mm.h:887 [inline] > > > mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 > > > do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 > > > > This is one side changing the VMA flags under the mmap lock in write mode... > > > > > __do_sys_mprotect mm/mprotect.c:861 [inline] > > > __se_sys_mprotect mm/mprotect.c:858 [inline] > > > __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 > > > x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 > > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > > > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > > > rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 > > > > ... while the other side comes through the rmap, which does not > > involve the mmap lock. Yes, that does not have any mutual locking by > > design, I think. > > > > The comments in the VMA flags code incorrectly assume that no > > concurrency is possible here; and I think the comment in > > mprotect_fixup() about protection by the mmap_lock has also been kinda > > wrong since the beginning of git history. > > > > The VM_LOCKED check in the migration code was added by Hugh in commit > > b74355078b655, but that's just one example syzbot stumbled over; we > > have similar racy vm_flags reads through the rmap on other paths like: > > > > unmap_mapping_range_tree -> unmap_mapping_range_vma -> > > zap_page_range_single -> unmap_single_vma -> unmap_page_range -> ... > > -> zap_pte_range -> zap_present_ptes -> vm_normal_page > > > > I think the right fix might just be to make sure that we use > > WRITE_ONCE() for these vm_flags updates, and READ_ONCE() around > > ->vm_flags reads that can happen in rmap walk paths, though we should > > think about the consequences of concurrently changing flags in every > > place that gets a READ_ONCE()... > > Yup cool similar to my thread on this. > > I hate that we have these landmines waiting for us. Be good to find a way > to explicitly annotate this, or at least comment somehow. > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > least for the proximate thing. > > It's a wonder these things don't trigger more, except you need probably > very precise timing to do it... > > I can do a quick cheeky patch. Thanks! ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:11 ` Lorenzo Stoakes 2025-02-05 15:14 ` Jann Horn @ 2025-02-05 15:46 ` Marco Elver 2025-02-05 15:51 ` Lorenzo Stoakes 1 sibling, 1 reply; 14+ messages in thread From: Marco Elver @ 2025-02-05 15:46 UTC (permalink / raw) To: Lorenzo Stoakes Cc: Jann Horn, syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka, Paul E. McKenney On Wed, 5 Feb 2025 at 16:11, 'Lorenzo Stoakes' via syzkaller-bugs <syzkaller-bugs@googlegroups.com> wrote: > > On Wed, Feb 05, 2025 at 04:00:06PM +0100, Jann Horn wrote: > > On Wed, Feb 5, 2025 at 12:41 PM syzbot > > <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > > > syzbot found the following issue on: > > > > > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > > > git tree: upstream > > > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > Downloadable assets: > > > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > > > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > > > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > > > > > ================================================================== > > > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one [...] > I hate that we have these landmines waiting for us. Be good to find a way > to explicitly annotate this, or at least comment somehow. > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > least for the proximate thing. > > It's a wonder these things don't trigger more, except you need probably > very precise timing to do it... They do trigger, but we don't send all of them to LKML. When we first introduced KCSAN, the notion of "data race" was still poorly understood. At the time we decided to pre-review a number of them (but our time to do so has been going down :-/), or let willing maintainers deal with them directly. A number of articles followed, such as: - https://lwn.net/Articles/816850/ - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/access-marking.txt And I think much of the community has indeed been "Calibrating your fear of big bad optimizing compilers" [https://lwn.net/Articles/799218/]. :-) If you want to see more reports (you can try to search for ones relevant to you): https://syzkaller.appspot.com/upstream?manager=ci2-upstream-kcsan-gce (see "moderation") ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:46 ` Marco Elver @ 2025-02-05 15:51 ` Lorenzo Stoakes 2025-02-05 16:25 ` Marco Elver 0 siblings, 1 reply; 14+ messages in thread From: Lorenzo Stoakes @ 2025-02-05 15:51 UTC (permalink / raw) To: Marco Elver Cc: Jann Horn, syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka, Paul E. McKenney On Wed, Feb 05, 2025 at 04:46:40PM +0100, Marco Elver wrote: > On Wed, 5 Feb 2025 at 16:11, 'Lorenzo Stoakes' via syzkaller-bugs > <syzkaller-bugs@googlegroups.com> wrote: > > > > On Wed, Feb 05, 2025 at 04:00:06PM +0100, Jann Horn wrote: > > > On Wed, Feb 5, 2025 at 12:41 PM syzbot > > > <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > > > > syzbot found the following issue on: > > > > > > > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > > > > git tree: upstream > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > Downloadable assets: > > > > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > > > > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > > > > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > > > > > > > ================================================================== > > > > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > [...] > > I hate that we have these landmines waiting for us. Be good to find a way > > to explicitly annotate this, or at least comment somehow. > > > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > > least for the proximate thing. > > > > It's a wonder these things don't trigger more, except you need probably > > very precise timing to do it... > > They do trigger, but we don't send all of them to LKML. > When we first introduced KCSAN, the notion of "data race" was still > poorly understood. At the time we decided to pre-review a number of > them (but our time to do so has been going down :-/), or let willing > maintainers deal with them directly. A number of articles followed, We very much appreciate your efforts :) We are definitely willing to see these in mm, and as you can see from the discussion here, the interaction between the rmap locks and other locks is complicated (see also the docs I wrote on them at [0]). So it'd be really good to pick up on these kinds of races. Obviously if there are spurious reports, better to filter those out, and again, your efforts at doing so and enabling this are hugely appreicated! [0]:https://origin.kernel.org/doc/html/latest/mm/process_addrs.html > such as: > - https://lwn.net/Articles/816850/ > - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/access-marking.txt > Thanks :) > And I think much of the community has indeed been "Calibrating your > fear of big bad optimizing compilers" > [https://lwn.net/Articles/799218/]. :-) I personally appreciate all the help I can get in calbirating said fear :P > > If you want to see more reports (you can try to search for ones > relevant to you): > https://syzkaller.appspot.com/upstream?manager=ci2-upstream-kcsan-gce > (see "moderation") Thanks! ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:51 ` Lorenzo Stoakes @ 2025-02-05 16:25 ` Marco Elver 2025-02-05 18:56 ` Liam R. Howlett 2025-02-05 18:59 ` Lorenzo Stoakes 0 siblings, 2 replies; 14+ messages in thread From: Marco Elver @ 2025-02-05 16:25 UTC (permalink / raw) To: Lorenzo Stoakes Cc: Jann Horn, syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka, Paul E. McKenney On Wed, 5 Feb 2025 at 16:51, Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote: [...] > > [...] > > > I hate that we have these landmines waiting for us. Be good to find a way > > > to explicitly annotate this, or at least comment somehow. > > > > > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > > > least for the proximate thing. > > > > > > It's a wonder these things don't trigger more, except you need probably > > > very precise timing to do it... > > > > They do trigger, but we don't send all of them to LKML. > > When we first introduced KCSAN, the notion of "data race" was still > > poorly understood. At the time we decided to pre-review a number of > > them (but our time to do so has been going down :-/), or let willing > > maintainers deal with them directly. A number of articles followed, > > We very much appreciate your efforts :) > > We are definitely willing to see these in mm, and as you can see from the > discussion here, the interaction between the rmap locks and other locks is > complicated (see also the docs I wrote on them at [0]). Tangentially, I've been trying to work out how to bring this [1] Clang feature to the kernel: it's more or less a simple "capability system" [2] to express "acquire this before doing that / don't hold this thing here / etc.". Locking rules are an obvious application. It's been on a number of people's radar over the years, but nothing materialized. Sparse's locking analysis is much weaker, nor easy (i.e. quick) to use. [1] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html [2] https://www.cs.cornell.edu/talc/papers/capabilities.pdf The current work-in-progress is here: https://git.kernel.org/pub/scm/linux/kernel/git/melver/linux.git/log/?h=cap-analysis It lacks documentation, and proper commit messages, but is otherwise usable (see example enablements for kfence, kcov, and stackdepot and lib/test_capability-analysis.c). An official RFC will follow, but the hard part of writing documentation is in the works. ;-) There are also other questions, such as: can a subset of the analysis be applied tree-wide (vs. current selective enablement), as it would help find more bugs faster. However, the reality of it is that using this system would be opting into a "dialect of C with capability analysis" with its own set of restrictions, and I don't know if everyone is willing to pay this cost. What I'd be curious about is, if some of the complex rules you mention above can be expressed so that Clang's "capability analysis" can point out some bugs. I suspect not everything can be expressed, but even if we get 50% there, we could catch a huge amount of bugs statically at compile-time. I let this cat out the bag, because this thread seems like a good way to get super-early high-level feedback. :-) It'll be a while before the first RFC. Thanks, -- Marco ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 16:25 ` Marco Elver @ 2025-02-05 18:56 ` Liam R. Howlett 2025-02-05 18:59 ` Lorenzo Stoakes 1 sibling, 0 replies; 14+ messages in thread From: Liam R. Howlett @ 2025-02-05 18:56 UTC (permalink / raw) To: Marco Elver Cc: Lorenzo Stoakes, Jann Horn, syzbot, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka, Paul E. McKenney * Marco Elver <elver@google.com> [250205 11:29]: > On Wed, 5 Feb 2025 at 16:51, Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote: > [...] > > > [...] > > > > I hate that we have these landmines waiting for us. Be good to find a way > > > > to explicitly annotate this, or at least comment somehow. > > > > > > > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > > > > least for the proximate thing. > > > > > > > > It's a wonder these things don't trigger more, except you need probably > > > > very precise timing to do it... > > > > > > They do trigger, but we don't send all of them to LKML. > > > When we first introduced KCSAN, the notion of "data race" was still > > > poorly understood. At the time we decided to pre-review a number of > > > them (but our time to do so has been going down :-/), or let willing > > > maintainers deal with them directly. A number of articles followed, > > > > We very much appreciate your efforts :) > > > > We are definitely willing to see these in mm, and as you can see from the > > discussion here, the interaction between the rmap locks and other locks is > > complicated (see also the docs I wrote on them at [0]). > > Tangentially, I've been trying to work out how to bring this [1] Clang > feature to the kernel: it's more or less a simple "capability system" > [2] to express "acquire this before doing that / don't hold this thing > here / etc.". Locking rules are an obvious application. It's been on a > number of people's radar over the years, but nothing materialized. > Sparse's locking analysis is much weaker, nor easy (i.e. quick) to > use. > > [1] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html > [2] https://www.cs.cornell.edu/talc/papers/capabilities.pdf > > The current work-in-progress is here: > https://git.kernel.org/pub/scm/linux/kernel/git/melver/linux.git/log/?h=cap-analysis > It lacks documentation, and proper commit messages, but is otherwise > usable (see example enablements for kfence, kcov, and stackdepot and > lib/test_capability-analysis.c). > An official RFC will follow, but the hard part of writing > documentation is in the works. ;-) > > There are also other questions, such as: can a subset of the analysis > be applied tree-wide (vs. current selective enablement), as it would > help find more bugs faster. You will get so many false positives from my code in the maple tree alone that it will not be very useful, due to rcu usage. My main issue in the maple tree is that I have a pointer that guards the other data as valid, so I read data and then check this guard pointer (the parent pointer in the node) to ensure what I've seen is indeed valid. > However, the reality of it is that using this system would be opting > into a "dialect of C with capability analysis" with its own set of > restrictions, and I don't know if everyone is willing to pay this > cost. > > What I'd be curious about is, if some of the complex rules you mention > above can be expressed so that Clang's "capability analysis" can point > out some bugs. I suspect not everything can be expressed, but even if > we get 50% there, we could catch a huge amount of bugs statically at > compile-time. The locking of the vma is difficult because we have the per-vma lock, the mmap read/write lock, the rmap lock, and the rcu read lock. The vmas are also transitioning to be rcu type-safe. You can also find the vma through the mm's vma tree, and the rmap. Also, other tasks can look up vmas by finding them through the rmap and gup, maybe others. The path taken to find the vma will dictate which locks ensure it to be safe (for a lower cost than taking ALL of the locks or THE SAME lock). And like this path, many of the races are benign by design. I'm not sure how you could verify this with a compiler - without annotation, as Jann is suggesting here. > > I let this cat out the bag, because this thread seems like a good way > to get super-early high-level feedback. :-) > It'll be a while before the first RFC. My initial thought is that you don't want to start with vmas, but if you do start with vmas and can make it work, then most other places will be easier. Thanks, Liam ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 16:25 ` Marco Elver 2025-02-05 18:56 ` Liam R. Howlett @ 2025-02-05 18:59 ` Lorenzo Stoakes 1 sibling, 0 replies; 14+ messages in thread From: Lorenzo Stoakes @ 2025-02-05 18:59 UTC (permalink / raw) To: Marco Elver Cc: Jann Horn, syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka, Paul E. McKenney On Wed, Feb 05, 2025 at 05:25:16PM +0100, Marco Elver wrote: > On Wed, 5 Feb 2025 at 16:51, Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote: > [...] > > > [...] > > > > I hate that we have these landmines waiting for us. Be good to find a way > > > > to explicitly annotate this, or at least comment somehow. > > > > > > > > But agreed, probably adding a READ_ONCE()/WRITE_ONCE() is appropriate at > > > > least for the proximate thing. > > > > > > > > It's a wonder these things don't trigger more, except you need probably > > > > very precise timing to do it... > > > > > > They do trigger, but we don't send all of them to LKML. > > > When we first introduced KCSAN, the notion of "data race" was still > > > poorly understood. At the time we decided to pre-review a number of > > > them (but our time to do so has been going down :-/), or let willing > > > maintainers deal with them directly. A number of articles followed, > > > > We very much appreciate your efforts :) > > > > We are definitely willing to see these in mm, and as you can see from the > > discussion here, the interaction between the rmap locks and other locks is > > complicated (see also the docs I wrote on them at [0]). > > Tangentially, I've been trying to work out how to bring this [1] Clang > feature to the kernel: it's more or less a simple "capability system" > [2] to express "acquire this before doing that / don't hold this thing > here / etc.". Locking rules are an obvious application. It's been on a > number of people's radar over the years, but nothing materialized. > Sparse's locking analysis is much weaker, nor easy (i.e. quick) to > use. > > [1] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html > [2] https://www.cs.cornell.edu/talc/papers/capabilities.pdf > > The current work-in-progress is here: > https://git.kernel.org/pub/scm/linux/kernel/git/melver/linux.git/log/?h=cap-analysis > It lacks documentation, and proper commit messages, but is otherwise > usable (see example enablements for kfence, kcov, and stackdepot and > lib/test_capability-analysis.c). > An official RFC will follow, but the hard part of writing > documentation is in the works. ;-) > > There are also other questions, such as: can a subset of the analysis > be applied tree-wide (vs. current selective enablement), as it would > help find more bugs faster. > However, the reality of it is that using this system would be opting > into a "dialect of C with capability analysis" with its own set of > restrictions, and I don't know if everyone is willing to pay this > cost. > > What I'd be curious about is, if some of the complex rules you mention > above can be expressed so that Clang's "capability analysis" can point > out some bugs. I suspect not everything can be expressed, but even if > we get 50% there, we could catch a huge amount of bugs statically at > compile-time. > > I let this cat out the bag, because this thread seems like a good way > to get super-early high-level feedback. :-) > It'll be a while before the first RFC. By all means, let cats out of the bag, I am a cat enthusiast and prefer them loose about the hoose :P It's interesting, and I'm a big believer in declarative means of expressing attributes like this such a preconditions. I mean we have some sparse stuff for that scattered around, as well as lockdep obviously, so it's not something that'd be crazily out of line. It'd be nice to have something to say 'when we access the VMA this is the VMA under an rmap lock', perhaps a means of casting a VMA to an equivalent type or tagging that somehow. I'm not sure if some semantic means of saying 'we must have the rmap lock or a VMA lock here' is useful, as this is implicit with _any_ VMA access, and perhaps anything that'd ultimately be all that useful would need more complicated semantics. It'd be good to explicitly pick up on a lack of READ_ONCE()/WRITE_ONCE() in circumstances where an rmap lock could be involved, but again I'm not sure if we could express it this way. This is one I need to go off and think about, but I really appreciate the suggestion and I for one at least am very open-minded to any tooling -whatsoever- that can help us document, express, assert on and manage this complicated locking logic. > > Thanks, > -- Marco ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:00 ` Jann Horn 2025-02-05 15:11 ` Lorenzo Stoakes @ 2025-02-05 15:47 ` Liam R. Howlett 2025-02-05 15:52 ` Jann Horn 1 sibling, 1 reply; 14+ messages in thread From: Liam R. Howlett @ 2025-02-05 15:47 UTC (permalink / raw) To: Jann Horn Cc: syzbot, akpm, linux-kernel, linux-mm, lorenzo.stoakes, syzkaller-bugs, vbabka * Jann Horn <jannh@google.com> [250205 10:00]: > On Wed, Feb 5, 2025 at 12:41 PM syzbot > <syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com> wrote: > > syzbot found the following issue on: > > > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > > > ================================================================== > > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > > > > write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1: > > vm_flags_init include/linux/mm.h:875 [inline] > > vm_flags_reset include/linux/mm.h:887 [inline] > > mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 > > do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 > > This is one side changing the VMA flags under the mmap lock in write mode... > > > __do_sys_mprotect mm/mprotect.c:861 [inline] > > __se_sys_mprotect mm/mprotect.c:858 [inline] > > __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 > > x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > > rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 > > ... while the other side comes through the rmap, which does not > involve the mmap lock. Yes, that does not have any mutual locking by > design, I think. > > The comments in the VMA flags code incorrectly assume that no > concurrency is possible here; and I think the comment in > mprotect_fixup() about protection by the mmap_lock has also been kinda > wrong since the beginning of git history. > > The VM_LOCKED check in the migration code was added by Hugh in commit > b74355078b655, but that's just one example syzbot stumbled over; we > have similar racy vm_flags reads through the rmap on other paths like: > > unmap_mapping_range_tree -> unmap_mapping_range_vma -> > zap_page_range_single -> unmap_single_vma -> unmap_page_range -> ... > -> zap_pte_range -> zap_present_ptes -> vm_normal_page I think we need a list of vm_area_struct parts that are OK to access without the read/write/vma lock. It seems flags is not one of those as it could be racy. > > I think the right fix might just be to make sure that we use > WRITE_ONCE() for these vm_flags updates, and READ_ONCE() around > ->vm_flags reads that can happen in rmap walk paths, though we should > think about the consequences of concurrently changing flags in every > place that gets a READ_ONCE()... ...But it's okay here, for this one. I think - maybe. Everything is fine. > > > > try_to_migrate+0x11f/0x150 > > migrate_folio_unmap mm/migrate.c:1320 [inline] > > migrate_pages_batch+0x786/0x1930 mm/migrate.c:1866 > > migrate_pages_sync mm/migrate.c:1989 [inline] > > migrate_pages+0xf02/0x1840 mm/migrate.c:2098 > > do_mbind mm/mempolicy.c:1394 [inline] > > kernel_mbind mm/mempolicy.c:1537 [inline] > > __do_sys_mbind mm/mempolicy.c:1611 [inline] > > __se_sys_mbind+0xfd1/0x11c0 mm/mempolicy.c:1607 > > __x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1607 > > x64_sys_call+0x2662/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:238 > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > value changed: 0x0000000000102077 -> 0x0000000000102071 > > > > Reported by Kernel Concurrency Sanitizer on: > > CPU: 0 UID: 0 PID: 6418 Comm: syz.0.1339 Not tainted 6.14.0-rc1-syzkaller-00026-gd009de7d5428 #0 > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024 > > ================================================================== > > > > > > --- > > This report is generated by a bot. It may contain errors. > > See https://goo.gl/tpsmEJ for more information about syzbot. > > syzbot engineers can be reached at syzkaller@googlegroups.com. > > > > syzbot will keep track of this issue. See: > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > > > If the report is already addressed, let syzbot know by replying with: > > #syz fix: exact-commit-title > > > > If you want to overwrite report's subsystems, reply with: > > #syz set subsystems: new-subsystem > > (See the list of subsystem names on the web dashboard) > > > > If the report is a duplicate of another one, reply with: > > #syz dup: exact-subject-of-another-report > > > > If you want to undo deduplication, reply with: > > #syz undup ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:47 ` Liam R. Howlett @ 2025-02-05 15:52 ` Jann Horn 0 siblings, 0 replies; 14+ messages in thread From: Jann Horn @ 2025-02-05 15:52 UTC (permalink / raw) To: Liam R. Howlett, Jann Horn, syzbot, akpm, linux-kernel, linux-mm, lorenzo.stoakes, syzkaller-bugs, vbabka On Wed, Feb 5, 2025 at 4:47 PM Liam R. Howlett <Liam.Howlett@oracle.com> wrote: > * Jann Horn <jannh@google.com> [250205 10:00]: > > The comments in the VMA flags code incorrectly assume that no > > concurrency is possible here; and I think the comment in > > mprotect_fixup() about protection by the mmap_lock has also been kinda > > wrong since the beginning of git history. > > > > The VM_LOCKED check in the migration code was added by Hugh in commit > > b74355078b655, but that's just one example syzbot stumbled over; we > > have similar racy vm_flags reads through the rmap on other paths like: > > > > unmap_mapping_range_tree -> unmap_mapping_range_vma -> > > zap_page_range_single -> unmap_single_vma -> unmap_page_range -> ... > > -> zap_pte_range -> zap_present_ptes -> vm_normal_page > > I think we need a list of vm_area_struct parts that are OK to access > without the read/write/vma lock. It seems flags is not one of those as > it could be racy. We do have this big table in these nice docs that Lorenzo spent quite some effort on: https://kernel.org/doc/html/latest/mm/process_addrs.html#vma-fields Though that does not currently call out this possible concurrency of vm_flags. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 11:41 [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one syzbot 2025-02-05 15:00 ` Jann Horn @ 2025-02-05 15:07 ` Lorenzo Stoakes 2025-02-05 15:14 ` Jann Horn 1 sibling, 1 reply; 14+ messages in thread From: Lorenzo Stoakes @ 2025-02-05 15:07 UTC (permalink / raw) To: syzbot Cc: Liam.Howlett, akpm, jannh, linux-kernel, linux-mm, syzkaller-bugs, vbabka On Wed, Feb 05, 2025 at 03:41:20AM -0800, syzbot wrote: > Hello, > > syzbot found the following issue on: > > HEAD commit: d009de7d5428 Merge tag 'livepatching-for-6.14-rc2' of git:.. > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=12b678a4580000 > kernel config: https://syzkaller.appspot.com/x/.config?x=9e757e3762bd630b > dashboard link: https://syzkaller.appspot.com/bug?extid=c2e5712cbb14c95d4847 > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > Unfortunately, I don't have any reproducer for this issue yet. > > Downloadable assets: > disk image: https://storage.googleapis.com/syzbot-assets/9235000a1b88/disk-d009de7d.raw.xz > vmlinux: https://storage.googleapis.com/syzbot-assets/098ef82f8ab3/vmlinux-d009de7d.xz > kernel image: https://storage.googleapis.com/syzbot-assets/4f51f5eb5782/bzImage-d009de7d.xz > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+c2e5712cbb14c95d4847@syzkaller.appspotmail.com > > ================================================================== > BUG: KCSAN: data-race in mprotect_fixup / try_to_migrate_one > > write to 0xffff888114b41700 of 8 bytes by task 6432 on cpu 1 This is vma->vm_flags: static inline void vm_flags_init(struct vm_area_struct *vma, vm_flags_t flags) { ACCESS_PRIVATE(vma, __vm_flags) = flags; } : > vm_flags_init include/linux/mm.h:875 [inline] > vm_flags_reset include/linux/mm.h:887 [inline] > mprotect_fixup+0x419/0x5e0 mm/mprotect.c:679 > do_mprotect_pkey+0x6cc/0x9a0 mm/mprotect.c:840 > __do_sys_mprotect mm/mprotect.c:861 [inline] > __se_sys_mprotect mm/mprotect.c:858 [inline] > __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:858 > x64_sys_call+0x2770/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:11 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 Be super nice if these reports could show the line of code! It's: if (vma->vm_flags & VM_LOCKED) mprotect() can't change VM_LOCKED, but maybe we need a READ_ONCE() / WRITE_ONCE() or _something_ to deal with tearing...? > rmap_walk_anon+0x28f/0x440 mm/rmap.c:2646 > try_to_migrate+0x11f/0x150 > migrate_folio_unmap mm/migrate.c:1320 [inline] > migrate_pages_batch+0x786/0x1930 mm/migrate.c:1866 > migrate_pages_sync mm/migrate.c:1989 [inline] > migrate_pages+0xf02/0x1840 mm/migrate.c:2098 > do_mbind mm/mempolicy.c:1394 [inline] > kernel_mbind mm/mempolicy.c:1537 [inline] > __do_sys_mbind mm/mempolicy.c:1611 [inline] > __se_sys_mbind+0xfd1/0x11c0 mm/mempolicy.c:1607 > __x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1607 > x64_sys_call+0x2662/0x2dc0 arch/x86/include/generated/asm/syscalls_64.h:238 > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > value changed: 0x0000000000102077 -> 0x0000000000102071 VM_READ VM_WRITE VM_EXEC VM_MAYREAD VM_MAYWRITE VM_MAYEXEC VM_LOCKED VM_ACCOUNT -> VM_READ VM_MAYREAD VM_MAYWRITE VM_MAYEXEC VM_LOCKED VM_ACCOUNT i.e. the mprotect() went from RWX -> R. I see Jann's replied so will leave analysis there for now :P > > Reported by Kernel Concurrency Sanitizer on: > CPU: 0 UID: 0 PID: 6418 Comm: syz.0.1339 Not tainted 6.14.0-rc1-syzkaller-00026-gd009de7d5428 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024 > ================================================================== > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > If the report is already addressed, let syzbot know by replying with: > #syz fix: exact-commit-title > > If you want to overwrite report's subsystems, reply with: > #syz set subsystems: new-subsystem > (See the list of subsystem names on the web dashboard) > > If the report is a duplicate of another one, reply with: > #syz dup: exact-subject-of-another-report > > If you want to undo deduplication, reply with: > #syz undup ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:07 ` Lorenzo Stoakes @ 2025-02-05 15:14 ` Jann Horn 2025-02-05 15:34 ` Lorenzo Stoakes 0 siblings, 1 reply; 14+ messages in thread From: Jann Horn @ 2025-02-05 15:14 UTC (permalink / raw) To: Lorenzo Stoakes, syzkaller Cc: syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka On Wed, Feb 5, 2025 at 4:07 PM Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote: > On Wed, Feb 05, 2025 at 03:41:20AM -0800, syzbot wrote: > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > > Be super nice if these reports could show the line of code! > > It's: > > if (vma->vm_flags & VM_LOCKED) fwiw if you follow the "dashboard link" in the bug report, you'll get a version of this where the source locations are clickable links to git.kernel.org at the right commit. Also note the "syzbot engineers can be reached at syzkaller@googlegroups.com" at the bottom of the message. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one 2025-02-05 15:14 ` Jann Horn @ 2025-02-05 15:34 ` Lorenzo Stoakes 0 siblings, 0 replies; 14+ messages in thread From: Lorenzo Stoakes @ 2025-02-05 15:34 UTC (permalink / raw) To: Jann Horn Cc: syzkaller, syzbot, Liam.Howlett, akpm, linux-kernel, linux-mm, syzkaller-bugs, vbabka On Wed, Feb 05, 2025 at 04:14:03PM +0100, Jann Horn wrote: > On Wed, Feb 5, 2025 at 4:07 PM Lorenzo Stoakes > <lorenzo.stoakes@oracle.com> wrote: > > On Wed, Feb 05, 2025 at 03:41:20AM -0800, syzbot wrote: > > > read to 0xffff888114b41700 of 8 bytes by task 6418 on cpu 0: > > > try_to_migrate_one+0xb5a/0x12e0 mm/rmap.c:2321 > > > > Be super nice if these reports could show the line of code! > > > > It's: > > > > if (vma->vm_flags & VM_LOCKED) > > fwiw if you follow the "dashboard link" in the bug report, you'll get > a version of this where the source locations are clickable links to > git.kernel.org at the right commit. > > Also note the "syzbot engineers can be reached at > syzkaller@googlegroups.com" at the bottom of the message. But but but... I like shouting into the void! :P Anyway I see that they're cc'd and sometimes reply, this was more a meandering thought if somebody happened to notice rather than urgent. A 'nice to have' if you will... ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2025-02-05 18:59 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-02-05 11:41 [syzbot] [mm?] KCSAN: data-race in mprotect_fixup / try_to_migrate_one syzbot 2025-02-05 15:00 ` Jann Horn 2025-02-05 15:11 ` Lorenzo Stoakes 2025-02-05 15:14 ` Jann Horn 2025-02-05 15:46 ` Marco Elver 2025-02-05 15:51 ` Lorenzo Stoakes 2025-02-05 16:25 ` Marco Elver 2025-02-05 18:56 ` Liam R. Howlett 2025-02-05 18:59 ` Lorenzo Stoakes 2025-02-05 15:47 ` Liam R. Howlett 2025-02-05 15:52 ` Jann Horn 2025-02-05 15:07 ` Lorenzo Stoakes 2025-02-05 15:14 ` Jann Horn 2025-02-05 15:34 ` Lorenzo Stoakes
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox