From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70715C3ABC3 for ; Mon, 12 May 2025 17:09:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C5EC6B018F; Mon, 12 May 2025 13:09:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 573746B0190; Mon, 12 May 2025 13:09:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43BA06B0191; Mon, 12 May 2025 13:09:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 26D7B6B018F for ; Mon, 12 May 2025 13:09:28 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 19E08C0512 for ; Mon, 12 May 2025 17:09:29 +0000 (UTC) X-FDA: 83434892058.23.264A706 Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) by imf21.hostedemail.com (Postfix) with ESMTP id 2E20E1C0002 for ; Mon, 12 May 2025 17:09:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZyGMrBoS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of jannh@google.com designates 209.85.208.51 as permitted sender) smtp.mailfrom=jannh@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747069767; a=rsa-sha256; cv=none; b=XfeQ6M0xf+tBJ+iDv7QPZYJXzFZFnglGhr/rMblzMvrs5rZ2rM6FPXR0Pd0WPb7hyW2uYj erZmegAVaTXpkIPb7/jGPXqK960mOAsfkbrTnuqIi3NjLcKeYkDKxhJtjSBLuN6htegZMz WypBvMtuVk5pG1ff9I0sov+6CjiXgZE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZyGMrBoS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of jannh@google.com designates 209.85.208.51 as permitted sender) smtp.mailfrom=jannh@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747069767; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6xA4hXDiJTOqrw9v7C63HCsGIrY+T2CW3IJC/Gos96Q=; b=pVUmx5zACcApMnHrisOPxSWJYR7BQw7fr/f57mF85iCZHRBzPcVIhY/7b8IhTk7CuT/2oc LddrdzotRfljGM3V9zuciXEHPEEelp4IDjRyFTknAPGvPfN5P7hyKIKcvYM8g/5oyCQU99 Nqn6NXWQL227aCTud0DOx0uiI/voXJQ= Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-5fc4fc27983so495a12.1 for ; Mon, 12 May 2025 10:09:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1747069765; x=1747674565; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=6xA4hXDiJTOqrw9v7C63HCsGIrY+T2CW3IJC/Gos96Q=; b=ZyGMrBoSr/gPGXKcuGOzOOz2RkipIvsOS+xvb/VZwH9YNkL/KdxDpbKLoMUBL4f2SP a59QdHtUuMkhoT11PxZcSXrmN4dHbr1/AUwPAkF/obbUknwmAkY360vv1Z5+C/nOV36U HwdlfqZS4SBClu6oeu0U1CPTinDBHPi7cDVZS/Xz2rtaVzUkZB4jHK+v/XTZIa/S7k1+ 56xdCEIaJBG6WwN1A4fi6ysMdDcU2VTTyO4adVDIkHNpHtVujTFuzP3XAKKnze0GU07c ywxGAKn6z2soupTm70S7bs7fU7Bdf71Qu8lnWMFcSTTIIjxCsNhTi+EepYLhIR2QJBAy gy0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747069765; x=1747674565; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6xA4hXDiJTOqrw9v7C63HCsGIrY+T2CW3IJC/Gos96Q=; b=djsK4PXVTEppvjst/tKWlozy2zj11oKA1MI1j+zEIDQXZ6gd27ggU+gI3mSPxW8Qzl TdQ9OUCs8dEvHzkrZy45jb+dRSVZiPnqS0x5z1jSj3HcJxny6l7sngSIZhF/bhIYmiHb W/jKdvaChwVWultGg1EUcdnC0pbhWneI6ihQ6uChlqPiGDkb/l2fJAhnVYsyqJHR26vW mx4KKgP1lKp2kxVIb/Y6rC28l1cAY/icsRz07t/EDmyUTAL/eqszMF7fqI3udlzYhrsk 0J9xwX/fD8hYGJqEg5mf5VScGJuDqi6FOZfmxvwBHkMTsO8gxr9D6LWHp8dPN8KGknQK gcTw== X-Forwarded-Encrypted: i=1; AJvYcCU5rjNxvz3Dh2dOk8qaGOyu3FTi0e3bzowfXHMwpycRDJ5q6CWKKJCXug2AznGg7saT0/K1zNBtYg==@kvack.org X-Gm-Message-State: AOJu0YzeS5G0czKqWwsVo9KbojcDvBhLT5o1pbdCnU9ACcLR4B2h4VcX sDIT78A3pqVWhmpUifnq3aB7x/N7O2pJ1vYjECyQ7Ck6TEsn1Q+4jSyn1NQ05QjmyYdnn1NuA58 Knuxclop7KP87gi9+m1q571t4s70ma+3+cZ8L X-Gm-Gg: ASbGnctamUHjcVyhqzzLet8BFZ/GR+Akyhbb1Ybd4+zibqrcs3m2l1HW7UhVNhqumPG eC7uv1N3u3pPg6kOgpXZrO1B6jewIAfYPI6Eh3QNwMSu9C87PsHQ6oEhmxy3d6JyAUZBY9+1abE ZzRwWEJdsh8KdbkUIPH0UQH6lzRmt2KmKuVVZcZsmFZF27ZHiu5E+i3DKi1No= X-Google-Smtp-Source: AGHT+IHcOgWNDsuVhEhzn0EYH1WFbiw9zSlOfSNM8iF5SvBUQFz6ojSSaePK3bgKsjQtCwbXMxK3NhJBcJSkE6DzCgE= X-Received: by 2002:a50:cd08:0:b0:5fd:6065:7bbc with SMTP id 4fb4d7f45d1cf-5fd60657bf5mr95281a12.0.1747069765111; Mon, 12 May 2025 10:09:25 -0700 (PDT) MIME-Version: 1.0 References: <682196ed.050a0220.f2294.0053.GAE@google.com> <81a0d2da-d863-4a6e-8d3b-b899d38ea605@kernel.dk> In-Reply-To: <81a0d2da-d863-4a6e-8d3b-b899d38ea605@kernel.dk> From: Jann Horn Date: Mon, 12 May 2025 19:08:49 +0200 X-Gm-Features: AX0GCFuq_g4AfY0PHmir37iGT2R5rx3E-VYkDkABiEeWv2bw0Ptt1mi7L2zvqoc Message-ID: Subject: Re: [syzbot] [io-uring] KCSAN: data-race in copy_mm / percpu_counter_destroy_many To: Jens Axboe , Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Pedro Falcato Cc: syzbot , asml.silence@gmail.com, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com, Linux-MM , dennis@kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 2E20E1C0002 X-Stat-Signature: 7io9urxdjskr9da5zybydgy65kg5djkt X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1747069766-554714 X-HE-Meta: U2FsdGVkX1+hpYQOB7vEjSpmOTtGSGVojYJ4ODtog940KnIVQsRViqhhxHS5PfA8JGCOKRvrjwzfUVsRGupgGrefdHi3OPkIAhFHsMBimSpLEPYGJxI8tH3CKxEVZzxLIX8oe1pU+zdOpEmtPzBaF+XHtSWO6fzUAmfAutDQLi5zn1nNfDvlAXex03IQyVK38t9s2Lr9QiFYcR0MUMvOS0IBS0fNO5Uqq+Wo8S8CpsJZFnzt6qGr8QwTEasWQikILrK/s1ZIeBT+8mEDNei8aylupwpIEqGStIvZAF76klxfDR73kut769/3+JXK8D9UiZIKDc0pWb0kCGd0ikuvHHSEPmz6V8d5sNHIXsXOFRyF1+2JZ6sT0Uj/qPGRh0OdE4vAqgkEyekn65WQGw/APyOQZ3JiLlUiIG8qtndqOh5AYioowRVGH0z7urntNZMRE2a0rOjKvslJ5CMhmovvCoO8TLHdfFon8Nph3e55sv/fHQ1HHX826ApMxjNzn7EQT3G2ZC/3KeVknrQVHg8H4MQ4hJurmNWYm/ezzSy3cMuB5UAdfK3htLmUU8qXIScOPv1rPGEH2yu+q5dOFmw9lzoZQrLWCItQHJYZ0oI4pI3GUJoBaZNKANBLGpH96zTep00HdDe8J67zSjpbhwbS2M458ybCP2GneoWZIwUtv5NEca0m8SKzP5dYJSEJUHSF4RwbnnYQGHzzQUtA/BCfAMqH2SNRST11NXKWqgFDzznlfhAVAXscQrXZpgC89Dx05BnHLn8M39JX0/yvRXCNg5bQiVpIrQ/B81IoL3ADwKs8F/jThiRbLtYppK2azl4JYfgF/hgvKghkOY1jy9xeXm8V2+cKf3x2FPf7xL+MMcXAnSjvu8HOGPB7X1eEFU+N1uma4tvmjZAeb4MLokZyvmDET52ffy91FqAdw6T1ejnyPf2Jh6pzt/U1bCx4s2t5x3YaVhBRlwQYPbD3TN2 9WaifnKi gsV5KyR4KsIL9639CsPYRj0c1e7SroqvhybScxPB5c7kU1wjIGHR3P+wpRcBkK/+KibRLzTDIlQ8jrYYs9NXVsmlX4G+U9cJPZ/oQglG3ZnEk9ZBB0439dyxJ/024h3f0fDkXDllibhO9OZTddylUl4XL99RcUPKsmuVdBx/2tM+3FubZZpZiCAWL6xZUqUNUgh09MjLouec2Ah7+Ld/lZvLG5LwG2IF7XftXYsVS3hzJHpfZWiuioNesdaI4vOnJZb1mYZKkiIfKhtv/nvwVY5zQGUWVxON99Xdl6cxHlWOBPrelBLFDueZFLgakdlWSy04tvpFHo2y30aKIZRstJFTOSUrVWHriD12x+7Bkt/iEitGe4GvGWi+5+/h+bv0nnQhKh2o995FTt3H1otsmcLj93wvi+v8jEaZZta+O/nSU8uBvkSuNcRx/MKGRLpM22igC6g1wP4PmgrJdwoQ2q+ywtcJLYA7Jq6L8nqGUg1CJj64u6DC/xkUGUK5sLe8z/WHrID+vRZRiutIeYz73c0ud1/b4WB6RdAYPnwjMiQzx8wbGIO2hF5hcCwLMcK+Ri0ziCodRrURgqkZO/BFcxyTVQuj9jPY3KGURvFMcggA75WRD2nTdX8u7c60eCWRIpa+49oPY1LM2q21Mqp5lx2AlzRwKv5UwnsnUHaxeV/tAHMZ3C+3PKAFTIs0RM+bR1/V7QZOtO9mVYtlQJPFqc+vLBwfoZr7okyqzRUBwrv5GDx8riEbUJ+nVsIsVj6OTLDOtq0AcOcd3BQ6qrDoEOOYZBgzJml5w4YZSae9xPc+hNWsLEkjjups8xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: +mmap maintainers On Mon, May 12, 2025 at 3:51=E2=80=AFPM Jens Axboe wrote: > On 5/12/25 12:36 AM, syzbot wrote: > > Hello, > > > > syzbot found the following issue on: > > > > HEAD commit: 3ce9925823c7 Merge tag 'mm-hotfixes-stable-2025-05-10-1= 4-2.. > > git tree: upstream > > console output: https://syzkaller.appspot.com/x/log.txt?x=3D14ff74d4580= 000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=3D6154604431d= 9aaf9 > > dashboard link: https://syzkaller.appspot.com/bug?extid=3D8be9bf36c3cf5= 74426c8 > > compiler: Debian clang version 20.1.2 (++20250402124445+58df0ef89= dd6-1~exp1~20250402004600.97), Debian LLD 20.1.2 > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > Downloadable assets: > > disk image: https://storage.googleapis.com/syzbot-assets/afdc6302fc05/d= isk-3ce99258.raw.xz > > vmlinux: https://storage.googleapis.com/syzbot-assets/fc7f98d3c420/vmli= nux-3ce99258.xz > > kernel image: https://storage.googleapis.com/syzbot-assets/ea7ca2da2258= /bzImage-3ce99258.xz > > > > IMPORTANT: if you fix the issue, please add the following tag to the co= mmit: > > Reported-by: syzbot+8be9bf36c3cf574426c8@syzkaller.appspotmail.com > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > BUG: KCSAN: data-race in copy_mm / percpu_counter_destroy_many > > > > write to 0xffff8881045e19d8 of 8 bytes by task 2123 on cpu 0: > > __list_del include/linux/list.h:195 [inline] > > __list_del_entry include/linux/list.h:218 [inline] > > list_del include/linux/list.h:229 [inline] > > percpu_counter_destroy_many+0xc7/0x2b0 lib/percpu_counter.c:244 > > __mmdrop+0x22e/0x350 kernel/fork.c:947 > > mmdrop include/linux/sched/mm.h:55 [inline] > > io_ring_ctx_free+0x31e/0x360 io_uring/io_uring.c:2740 > > io_ring_exit_work+0x529/0x560 io_uring/io_uring.c:2962 > > process_one_work kernel/workqueue.c:3238 [inline] > > process_scheduled_works+0x4cb/0x9d0 kernel/workqueue.c:3319 > > worker_thread+0x582/0x770 kernel/workqueue.c:3400 > > kthread+0x486/0x510 kernel/kthread.c:464 > > ret_from_fork+0x4b/0x60 arch/x86/kernel/process.c:153 > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 > > > > read to 0xffff8881045e1600 of 1344 bytes by task 5051 on cpu 1: > > dup_mm kernel/fork.c:1728 [inline] > > copy_mm+0xfb/0x1310 kernel/fork.c:1786 > > copy_process+0xcf1/0x1f90 kernel/fork.c:2429 > > kernel_clone+0x16c/0x5b0 kernel/fork.c:2844 > > __do_sys_clone kernel/fork.c:2987 [inline] > > __se_sys_clone kernel/fork.c:2971 [inline] > > __x64_sys_clone+0xe6/0x120 kernel/fork.c:2971 > > x64_sys_call+0x2c59/0x2fb0 arch/x86/include/generated/asm/syscalls_64.= h:57 > > do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] > > do_syscall_64+0xd0/0x1a0 arch/x86/entry/syscall_64.c:94 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > Reported by Kernel Concurrency Sanitizer on: > > CPU: 1 UID: 0 PID: 5051 Comm: syz.1.494 Not tainted 6.15.0-rc5-syzkalle= r-00300-g3ce9925823c7 #0 PREEMPT(voluntary) > > This doesn't look like an io_uring issue, it's successive setup and teard= own > of a percpu counter. Adding some relevant folks. This is an intentional-but-dodgy data race: dup_mm() does memcpy() of the entire old mm_struct, while the old mm_struct is not yet locked, in order to initialize some parts of the new mm_struct that are expected to be stable (I guess?); and then afterwards re-initializes all the important fields with proper locking. (What happens in the KCSAN report is that one mm is being forked while an **unrelated** mm is being destroyed; and the unrelated mm's destruction involves a linked list deletion at mm.rss_stat[].list, and the two mm's happen to be adjacent on this linked list, so destroying the unrelated mm updates a linked list element embedded in the first mm.) This design is pretty dirty, and it would be a good idea to get rid of that memcpy() by going through every mm_struct field, checking whether it is already initialized after the memcpy(), and figuring out proper initialization if it isn't; another easier, less clean approach would be to just slap a data_race() annotation around the memcpy() for now. >From a quick look, I think these are the fields that might not be reinitial= ized: Fields that are safe to copy without lock because they're immutable: - mmap_{compat_,}{legacy_,}base - task_size - binfmt Fields which could actually race but only if CRIU-specific APIs raced with execve: - {start,end}_{code,data} - start_brk, start_stack - {arg,env}_{start,end} - saved_auxv Fields which actually look like they might be data racing in practice (but with most of these, probably not much bad stuff actually results from that): - membarrier_state - mm_cid_next_scan - brk - numa_scan_offset - tlb_flush_batched - ksm_merging_pages, ksm_rmap_items, ksm_zero_pages And then there's arch-specific stuff in mm_context_t, I haven't looked at that for all architectures. But there is some pretty dodgy stuff in there too, for example on X86 the mm->context.vdso_image pointer can be updated through the CRIU API concurrently with the memcpy(), and it doesn't look like that pointer is re-initialized later, so I think that could theoretically result in a torn pointer being accessed later on in the child. (Note that memcpy() is very much not guaranteed to copy pointer-sized fields atomically, though it tends to often do that in practice.)