From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CF9FC5AE59 for ; Sat, 31 May 2025 06:22:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0BC46B0150; Sat, 31 May 2025 02:22:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E3266B0154; Sat, 31 May 2025 02:22:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F9196B0155; Sat, 31 May 2025 02:22:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6D4036B0150 for ; Sat, 31 May 2025 02:22:56 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0D3EE1D5291 for ; Sat, 31 May 2025 06:22:56 +0000 (UTC) X-FDA: 83502209952.01.1907F90 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by imf13.hostedemail.com (Postfix) with ESMTP id 549F020008 for ; Sat, 31 May 2025 06:22:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DHB5Zpcy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748672574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8z6mh7pdJDvN8yUkwBxqHC+Y8n4XgRQIeuqir52nwPE=; b=5hPBuNoYwFzddD9urc6epCWn8OeWBzDiXXvBlNQCceoN98Gh30kgX7hf4pZhqEDya/R20X MUYxv8nNtC9972z6tNEkZNHglSDcObSzcGf824/Ri8rWmqWlKyWcH9HiQzVIU9iKdVX6ge XPrCVNbIdyOZbuFKgtmtV231jWFKfHo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748672574; a=rsa-sha256; cv=none; b=G19/VQkagAibrCkjEag+gAmXALO2jkiLBBO9UVpNp4ftVt4shpHrXu8Bas8QEk9IgOTcKV 74gXm+KGRoydXastCSXm+sFbblQ/9H/x0d2LWrTngmK3+8fySA/OnBsd7tE4Xa3UViL8iq GIuG1Ud63HoTRhdozyMajS2eImYZmtg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DHB5Zpcy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf13.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com Received: by mail-lj1-f177.google.com with SMTP id 38308e7fff4ca-32a9e5b625eso2717441fa.2 for ; Fri, 30 May 2025 23:22:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748672572; x=1749277372; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8z6mh7pdJDvN8yUkwBxqHC+Y8n4XgRQIeuqir52nwPE=; b=DHB5Zpcyc/SX0eOr6lKQDHLHcFhXEMZvb5u+E+uzPoNZnISZa/tQkkkZY8MpmLj0FV 1F1q3tiK0GCkitpaqG6Ye9z1PM+T3Tpoz3OdpITbaOHvX4Je5BJaJ+IBjB1LrST92Jo1 MwxtT9xcY/MF4/aH8dVwe2Sgw4aMw4QmMk1DKyZjlGCazzaGof1vMrKulWSDnlWKV1Qf XMEHRG1JK72xk5DbTBWrZDCd1KGS2fSTl1wjcuW1bHn8W6rzDdomfPRYr8LYrr+n+pVh 1WgP2h3cTSpgPc5S7BSHFams/x58bpen5KkDYVXyqyx64vU0UuQfqXPVCyomZnusCGXi zkXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748672572; x=1749277372; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8z6mh7pdJDvN8yUkwBxqHC+Y8n4XgRQIeuqir52nwPE=; b=OJjt6fUr1+eI8x9eJjnI/IqxfVxY9LTeYm49HtizSLIZeZIDdRSlVFHHNE+yAS+zj+ GTUpTtT6G/Hvtql0VVfV7KcnJkMOfp5OOua0GsTCrE9MQXOcZzhERIGlSagyeWquBOmT mE/7Ux+cMWnllhxGb47rzRmJTuOTdc6ksxbXSK/c3T3MVQoQMSIOlz8nTp/+Je0O7h1B BIsVRKZ2fU8QhBk1zg9PpBITV8DXAuAY6I646GhB4PKVEzaQdhXjorrsomNgrIUNRz0x liN+RORaaxLlbohphQnG3BH+gJyAbNK9BB1ld8sRTQnKAaWalziaxjlqUH4mKj1uFTHi O7MQ== X-Gm-Message-State: AOJu0Yz8071XHl5vElRiZs/EH4n/RaTHIdJlajsCAb0dqsLFtUUkpiNi JLFQFbOYNLGv5DMunyvX6fXNGyqrad/J6f7ug5I8T9Kq3RScXIRevUT+K62cyfKPx5XiAeo7bZ5 JfjJOF1BpJiMA18W+W6KTjlvYKniMnhY= X-Gm-Gg: ASbGncvet6i6AT8MBUnncLTJWJ/qxzNHBd1iuh06u3ZeZ4XYznITjp++dhZs/BLA9XU LIxaEQaVqMTAo8lT1Lx4nSIOoQbSyaAvYn6yLGZSmgjBD1cQjsEYfD+YDV253egenK0ojZqoQG3 ld03JVWB8i41JuIta7V5PelVZ+b5rDI7MT X-Google-Smtp-Source: AGHT+IHS8XLKQ5NmPv0l2ZJX5RYz0dEFA4DWAVhyuy7guXysiJ85JCN16IFjvHRjGXXy7OsItWIIp4P8s8h7CeYnOrs= X-Received: by 2002:a2e:a986:0:b0:32a:710f:5a0 with SMTP id 38308e7fff4ca-32a906d76a0mr15468041fa.11.1748672572114; Fri, 30 May 2025 23:22:52 -0700 (PDT) MIME-Version: 1.0 References: <20250530201710.81365-1-ryncsn@gmail.com> In-Reply-To: From: Kairui Song Date: Sat, 31 May 2025 14:22:34 +0800 X-Gm-Features: AX0GCFs1kvLiRCh7TzrXTKfCqNtzaev34lKPFyF-_wQf4bInaissLB-H6Q8jHs4 Message-ID: Subject: Re: [PATCH] mm: userfaultfd: fix race of userfaultfd_move and swap cache To: Lokesh Gidra Cc: linux-mm@kvack.org, Andrew Morton , Barry Song <21cnbao@gmail.com>, Peter Xu , Suren Baghdasaryan , Andrea Arcangeli , David Hildenbrand , stable@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 549F020008 X-Stat-Signature: qhosi5cs3bhsmxpbcxdofnjgs69c7jfn X-Rspam-User: X-HE-Tag: 1748672574-366732 X-HE-Meta: U2FsdGVkX18z4Sc/Hz0qs+zVfkL9R5+4ugExxLEnAaldz9gxH1A+azGJw0QnFVRkR1cis7iV+U/+lP76Gje2RgNXLU4hb7q3zlXKeludcp3G57e5dTaPOQU5b3EZdym5yQwvuwDmG32jQdnVzDCX5PE8GYkHTODRv9yhjfYjS82Eradvm3aPmRaIylWNwl226e+fKk5j640eELXd0iLwWaaS29dBhODk9k06WJb9mv6322GQvY9qW7wQVUGh5cs6N8RRitC/lYFi6pX9CkNVAtsiMN59WJYu21B8rvGgkxV8AERI2nw2f6NzfH6a3EN6t/iIaI3b8JlB8SRtUcz18sumaI2LoXFV2Q7eQ1UDl3a+Dm1j3k+dpK/1nHi/aGPUx/AQwUZABph3SzQFsKAK1Pb0YwoL+CJeTEKsjaftoQweJyWDlid0M5hM1CJ1PFrKCCSO4IiDyEL1p1Xq+zqF3DpMTh+N1jPqRjXsDHJ6ZAwIXGHzX5x6yMKCu1turp6jCKuPV8rPoP78JTN7oqWGXc4jVKnk30hwN27AeIHxoz2kYsAN3qp+JK4skV2Ng9nlk0SE0MlV5JRZOP2SoMVaFGV8JMrjFXs5XzR5+0FnGXF/4DNZMVF/nSCpfaPpSDyp3XPN/giNF2w5CxHdT9IM6TPL6e+moVnmBWntVwTgF/3x4kXUqAybRIi/MZUIO6E10gseJ/XrFkn/hPvb7/O0xzl1q16jvwNK10p4FIi4bJ1hruuJiUW3gLf0i7t7UcJaCRn97iZAsOiZWWEhIh5RCWKDWz9XnO+zuPRi5esfbzmUyuvRg94Z4GIOs//D22BHrOjCyXgG7dO8gkPOAyQbYNXcq5IyKUVeWooPSIo5nEUwnOSjtQ+mC/0Id6jsoylClTBFxB1Nvd+UXryzYZAKLf+om+rCzf+M6mpwdY/ZHj8CkbOaPgBCiLxgGjJcds0o2D6vY2N3ns3H0WrdInm jBcmawf8 9NYOQIsjga8VxGu+rVsz1JBJ8YA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, May 31, 2025 at 7:42=E2=80=AFAM Lokesh Gidra wrote: > > On Fri, May 30, 2025 at 1:17=E2=80=AFPM Kairui Song wr= ote: > > > > From: Kairui Song > > > > On seeing a swap entry PTE, userfaultfd_move does a lockless swap cache > > lookup, and try to move the found folio to the faulting vma when. > > Currently, it relies on the PTE value check to ensure the moved folio > > still belongs to the src swap entry, which turns out is not reliable. > > > > While working and reviewing the swap table series with Barry, following > > existing race is observed and reproduced [1]: > > > > ( move_pages_pte is moving src_pte to dst_pte, where src_pte is a > > swap entry PTE holding swap entry S1, and S1 isn't in the swap cache.) > > > > CPU1 CPU2 > > userfaultfd_move > > move_pages_pte() > > entry =3D pte_to_swp_entry(orig_src_pte); > > // Here it got entry =3D S1 > > ... < Somehow interrupted> ... > > > > // folio A is just a new allocated f= olio > > // and get installed into src_pte > > > > // src_pte now points to folio A, S1 > > // has swap count =3D=3D 0, it can b= e freed > > // by folio_swap_swap or swap > > // allocator's reclaim. > > > > // folio B is a folio in another VMA= . > > > > // S1 is freed, folio B could use it > > // for swap out with no problem. > > ... > > folio =3D filemap_get_folio(S1) > > // Got folio B here !!! > > ... < Somehow interrupted again> ... > > > > // Now S1 is free to be used again. > > > > // Now src_pte is a swap entry pte > > // holding S1 again. > > folio_trylock(folio) > > move_swap_pte > > double_pt_lock > > is_pte_pages_stable > > // Check passed because src_pte =3D=3D S1 > > folio_move_anon_rmap(...) > > // Moved invalid folio B here !!! > > > > The race window is very short and requires multiple collisions of > > multiple rare events, so it's very unlikely to happen, but with a > > deliberately constructed reproducer and increased time window, it can b= e > > reproduced [1]. > > Thanks for catching and fixing this. Just to clarify a few things > about your reproducer: > 1. Is it necessary for the 'race' mapping to be MAP_SHARED, or > MAP_PRIVATE will work as well? Hi, I used MAP_SHARED just to prevent vma merging, so folio B and folio A belong to different vma, so the folio_move_anon_rmap will cause a real problem. The race is not related to the map flag. > 2. You mentioned that the 'current dir is on a block device'. Are you > indicating that if we are using zram for swap then it doesn't > reproduce? ZRAM may also trigger the race, with a slightly different race window, but the race window is even shorter so I can't reproduce with that. > > > > > It's also possible that folio (A) is swapped in, and swapped out again > > after the filemap_get_folio lookup, in such case folio (A) may stay in > > swap cache so it needs to be moved too. In this case we should also try > > again so kernel won't miss a folio move. > > > > Fix this by checking if the folio is the valid swap cache folio after > > acquiring the folio lock, and checking the swap cache again after > > acquiring the src_pte lock. > > > > SWP_SYNCRHONIZE_IO path does make the problem more complex, but so far > > we don't need to worry about that since folios only might get exposed t= o > > swap cache in the swap out path, and it's covered in this patch too by > > checking the swap cache again after acquiring src_pte lock. > > > > Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI") > > Closes: https://lore.kernel.org/linux-mm/CAMgjq7B1K=3D6OOrK2OUZ0-tqCzi+= EJt+2_K97TPGoSt=3D9+JwP7Q@mail.gmail.com/ [1] > > Signed-off-by: Kairui Song > > --- > > mm/userfaultfd.c | 26 ++++++++++++++++++++++++++ > > 1 file changed, 26 insertions(+) > > > > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > > index bc473ad21202..a1564d205dfb 100644 > > --- a/mm/userfaultfd.c > > +++ b/mm/userfaultfd.c > > @@ -15,6 +15,7 @@ > > #include > > #include > > #include > > +#include > I guess you mistakenly left it from your reproducer code :) Ah, yes, I'll drop this :) > > #include > > #include > > #include "internal.h" > > @@ -1086,6 +1087,8 @@ static int move_swap_pte(struct mm_struct *mm, st= ruct vm_area_struct *dst_vma, > > spinlock_t *dst_ptl, spinlock_t *src_ptl, > > struct folio *src_folio) > > { > > + swp_entry_t entry; > > + > > double_pt_lock(dst_ptl, src_ptl); > > > > if (!is_pte_pages_stable(dst_pte, src_pte, orig_dst_pte, orig_s= rc_pte, > > @@ -1102,6 +1105,19 @@ static int move_swap_pte(struct mm_struct *mm, s= truct vm_area_struct *dst_vma, > > if (src_folio) { > > folio_move_anon_rmap(src_folio, dst_vma); > > src_folio->index =3D linear_page_index(dst_vma, dst_add= r); > > + } else { > > + /* > > + * Check again after acquiring the src_pte lock. Or we = might > > + * miss a new loaded swap cache folio. > > + */ > > + entry =3D pte_to_swp_entry(orig_src_pte); > > + src_folio =3D filemap_get_folio(swap_address_space(entr= y), > > + swap_cache_index(entry)); > > Given the non-trivial overhead of filemap_get_folio(), do you think it > will work if filemap_get_filio() was only once after locking src_ptl? > Please correct me if my assumption about the overhead is wrong. No we can't do the filemap_get_filio after locking src_ptl, moving the folio requires locking it, lock a folio inside a PTE lock looks a bad idea... So I just added a lockless lookup and fallback to try again if found one. The overhead should be low I think, it's a lockless xarray lookup and when I was doing many profiling of SWAP performance optimizations, Xarray look up is not a heavy burden. > > > + if (!IS_ERR_OR_NULL(src_folio)) { > > + double_pt_unlock(dst_ptl, src_ptl); > > + folio_put(src_folio); > > + return -EAGAIN; > > + } > > } > > > > orig_src_pte =3D ptep_get_and_clear(mm, src_addr, src_pte); > > @@ -1409,6 +1425,16 @@ static int move_pages_pte(struct mm_struct *mm, = pmd_t *dst_pmd, pmd_t *src_pmd, > > folio_lock(src_folio); > > goto retry; > > } > > + /* > > + * Check if the folio still belongs to the targ= et swap entry after > > + * acquiring the lock. Folio can be freed in th= e swap cache while > > + * not locked. > > + */ > > + if (unlikely(!folio_test_swapcache(folio) || > > + entry.val !=3D folio->swap.val)) { > > + err =3D -EAGAIN; > > + goto out; > > + } > > To avoid further increasing move_pages_pte() size, I recommend moving > the entire 'pte not present' case into move_swap_pte(), and maybe > returning some positive integer (or something more appropriate) to > handle the retry case. And then in move_swap_pte(), as suggested > above, you can do filemap_get_folio only once after locking ptl. > > I think this will fix the bug as well as improve the code's organization. > > > } > > err =3D move_swap_pte(mm, dst_vma, dst_addr, src_addr, = dst_pte, src_pte, > > orig_dst_pte, orig_src_pte, dst_pmd, ds= t_pmdval, > > -- > > 2.49.0 > > >