From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FA9AC5B543 for ; Thu, 5 Jun 2025 08:02:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE02D6B00A0; Thu, 5 Jun 2025 04:02:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB7886B0124; Thu, 5 Jun 2025 04:02:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACD7E6B0125; Thu, 5 Jun 2025 04:02:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8AC006B00A0 for ; Thu, 5 Jun 2025 04:02:36 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3F45C121CA1 for ; Thu, 5 Jun 2025 08:02:36 +0000 (UTC) X-FDA: 83520605112.11.D4D36DB Received: from mail-vk1-f173.google.com (mail-vk1-f173.google.com [209.85.221.173]) by imf02.hostedemail.com (Postfix) with ESMTP id 6964C8000A for ; Thu, 5 Jun 2025 08:02:34 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=N8DEJN+s; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749110554; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kMcINj7Ifsivjw4DYRfFWHrcDAFZixnKbjiSAQ5jK08=; b=17Mg6N8svNWEPWEyDIppnttZUOiO8OQM1iwa7tx75R0d3guFm7/sduDS0gcOCaEhRxaLti i6I1ahBrwbSL/tz7cKazet7xvPMhHXMCRVspdK8NWcJspfVzcc4YB4aB3arJsyW/3GipwH 8/D0220dCHDoHp/SDwzW6zO2cOskdW0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749110554; a=rsa-sha256; cv=none; b=SdapB7RIx65YPE0lCMmUGHoI2QRV8D0ar3azOEU5kfdumThKDgiMFUXCPhSmhWuKX3ZZ4K 5p3Ktl60LxN+sC8bA1b8KhZjZL+St1xlksQhNX6o44yHDv1C/dJLmGQZaAyeW3ets1C18G hUIZ+cIVmqxM/qPu6qQ3GrxPhyMLkcI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=N8DEJN+s; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-vk1-f173.google.com with SMTP id 71dfb90a1353d-5308d2d76f4so271692e0c.0 for ; Thu, 05 Jun 2025 01:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749110553; x=1749715353; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=kMcINj7Ifsivjw4DYRfFWHrcDAFZixnKbjiSAQ5jK08=; b=N8DEJN+s1DetXPPEZBkCn+FOOVjT53u57gtBQJApbKm0gSvKd5pvs37l4r9wqZHWG3 d1Fs94LWIar+YWFDWF47Loxp3ADzFqiXeIQZzv/3gUssJdpX2b7h4dRvlf0CJM2tq0y6 4SRxAJbxuMs5O2fTvOcPxktqeBfvFGzWdgPegb950WCl5F+9CP0PEGAQj0LOJsBdf+G5 IatiLkKfJ8Y2edrGZoSVJcRdmCP1ufvNc9ETMirZGDtBvoqesnWeSu5zOEHuxKB75ZVM UAj0xvncNlBwHZP9j7Da5lM5uHtv38o1OSYStSIL4RlaK6dH4H4te6GsNzq0JkZUHNK3 mVGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749110553; x=1749715353; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kMcINj7Ifsivjw4DYRfFWHrcDAFZixnKbjiSAQ5jK08=; b=gBQbyOFfZYHuW3eiQ6phKIhiR4HTc7bCZaEyEFpZc6iVol5WnGf+8NJcIU9tyzwPUH vUJ+duddNpcyRlPGNsLYNVRr3C1hJVYT/AVySX12ETw39v1rRXwnoDhbwZZ8PX6kYJuE KS9SIy79Dfc/+TyQn2LidR6P/7pJL8bbDEs8/ml7jqQDEyPPwXgvDf3pvT0ls/Bzkx74 5GuVrGgaJbDu96ZbkXOzDcV9qY/R1opG9NX9Rj/sSDfPXLDWX4sWQpeFL2B887z0aahX 82GxmSRdNYOlNcWketXt0PliDPdq58Osds9ZglrOYf8HRIzHlXymPYYcT/o7HfqLulKQ uN8w== X-Gm-Message-State: AOJu0YwpyGl9stFdbDPGwm9lx0dS4VqUtbZyMf2/ke+TpQMsHZnks76m hlNueVQaeL3aku2s6+19X6Gh2hHuWviuE0uWeEdGSiuIEQzNgiUir3km4vOiBuWzJlEpaaUXPmD Y0C0fLQUaCfa4exNCZfQ7EJsaT3Ec0pk= X-Gm-Gg: ASbGncvBX2cEp+RUEg9kxXZQnWIZiXCkDhJWd3UineY4Irg/4K10M1Co92iGLDrTZtH WemFaJFj3iGqS8jTGsX61lLaPd0uk/sjQ8RM9Wnd1jppfhmytWwQKXaMYrsB51iECMjmBdZlhyG Awi699qLlTk3yV0bykDmYxUnVr3l6ZEXsuzQ== X-Google-Smtp-Source: AGHT+IEX+HHyUIRP44E1txAP6qEbFK08W1HJ+COXPJhPWVLVrfGqK7Fe5xaZxSc/hgJSZd33iLpUWgJtiaVb5kZJMm4= X-Received: by 2002:a05:6122:828d:b0:52f:4680:1c75 with SMTP id 71dfb90a1353d-530c739f782mr5502331e0c.6.1749110553036; Thu, 05 Jun 2025 01:02:33 -0700 (PDT) MIME-Version: 1.0 References: <20250604151038.21968-1-ryncsn@gmail.com> In-Reply-To: <20250604151038.21968-1-ryncsn@gmail.com> From: Barry Song <21cnbao@gmail.com> Date: Thu, 5 Jun 2025 20:02:22 +1200 X-Gm-Features: AX0GCFvFCmpEOUvT9F2g6UK_BGIPE032pznjFE_9uuzAxLQJQJPYSC62dQa0oKY Message-ID: Subject: Re: [PATCH v4] mm: userfaultfd: fix race of userfaultfd_move and swap cache To: Kairui Song Cc: linux-mm@kvack.org, Andrew Morton , Peter Xu , Suren Baghdasaryan , Andrea Arcangeli , David Hildenbrand , Lokesh Gidra , stable@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6964C8000A X-Stat-Signature: ha5azyxn43acf4fto1mnju38ogz5pn6d X-Rspam-User: X-HE-Tag: 1749110554-30470 X-HE-Meta: U2FsdGVkX1/SAyN26fRYsRYJfX1CESOQHwne0Rt0wYorJsDND9p7Y9aqB1UGFpsV9k+9VE6F4JA9Iwi0v8Bvm8v9T2C9BTrBvjQOIOk44yEIZOnxC9fZc05Nr5SuhojCDN6wKwP48x9rwbvG0VPepzRVng9wcfB2fNhNbj+w4BsEg9e2hBzw8KRg4kuWVvnK5IBRho7lMsoUY3zxrFZJ/vXryYxDwXmjr1srHJtNAAPjhpF7Gh+jwVZ0Gs4kd0E2CaPV2zTnCABpM4lTf+QU2pXmHfX1phQL1lj/8gZHWCZy/z1gJ35SIA7oG/mbArDMhjbolfH+rIUwJQlp9R2zRWthx5WK88gvtKhwNwyVM+V49gflpdGNaBGcCu+3pmTdHVfc3xt5//ZY3kubaxfIgXBZkljg4sQulY2IDL9TiWGlbTul/hhFz2p/YcPK6qzryEpHbHGVVIV4WIgc8LmeGFZRXQA93r/1tfG9B5APHPvl1UoktwwL2OR5tewhNibhQ1CE8NCo8gAxq9lkd9r0s/WN9Z2lfhwr8GPSilHLFiSeKM9hKNO7L4nCmPqrCt6wydmI0dkQwSYvoFIJbdVDXF0SzesWJPEtqBTNlYNaTdn4gPr6+tfdfkImNfcpRT0bWk2YgzLAGt2Jr/BxYfVbjkShn7KAS8QKoCKYebcLMb1o9v1fmX8htqpW9Xvg2dcBgOAaO7/NrbHwV7NXIT7CkxpVB1ZOxcGAS9p3LGgeyzqE+YKPUO4VjFQP9hPNmeTPiwxsSVQDH2gKPxNkt2e8U7RhGmFt/B5xbxVM13D5bYblKQWHLjeCLfD/LQ2EFlOq95dQkp/uYrO5GzJOnAylOnCrRvnlsSqmH2AzErmrnBOMiyP5RiOb3YvzrJmPbtp/MCpOYYmd0rNNJx/NnmF/jKAqVYx9pxU2fsHX2g/rBWc0y6jqhQIe2LEU5n6UC6MGnLYrcxxA8gcqtpvaiSO uuFihIad EAmdOy1u+wzGYN1NNjopNzaAV6QpqCLuwHjHU+BGZQpafVXHeumNlnwiFIHQQUI23mqkz2Vaj1phlBQSYmvHycx8+vTaF7n56M2hmhUmgLmKqWtYRBJsU3rcCq+kj7dNZxthyouoNPe3sE0FiWp20d3V2Gk7FT6LBX4V5sC+vZLvUuHtggQ7ZxBpW8J2wDGfmSE5D7owLY4NFROyeU6QquaHwM2YPfPRGsVL55hixWaeW1ETVokTyabawxL57GbMyDjs5UMgm0oSrfjmVKCqDoDPjnx05+8ry38mJ3nbgeRWNzBf5XXvzBvvOPq3K4XdgWi4x6jmaixBbdXie0g+UU3aila5326sHG7x2TpfMX3P+0YISUcBUU1AoiSZlhdY7j3MTbF03n50JDDv2bUaXNkgdwMq5slBYPfphAzCmJz3ZMff+QCDBAsXiGyyCUm+2yVgTeUAIjwhZgGo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jun 5, 2025 at 3:10=E2=80=AFAM Kairui Song wrote= : > > From: Kairui Song > > On seeing a swap entry PTE, userfaultfd_move does a lockless swap > cache lookup, and tries to move the found folio to the faulting vma. > Currently, it relies on checking the PTE value to ensure that the moved > folio still belongs to the src swap entry and that no new folio has > been added to the swap cache, which turns out to be unreliable. > > While working and reviewing the swap table series with Barry, following > existing races are observed and reproduced [1]: > > In the example below, move_pages_pte is moving src_pte to dst_pte, > where src_pte is a swap entry PTE holding swap entry S1, and S1 > is not in the swap cache: > > CPU1 CPU2 > userfaultfd_move > move_pages_pte() > entry =3D pte_to_swp_entry(orig_src_pte); > // Here it got entry =3D S1 > ... < interrupted> ... > > // folio A is a new allocated folio > // and get installed into src_pte > > // src_pte now points to folio A, S1 > // has swap count =3D=3D 0, it can be = freed > // by folio_swap_swap or swap > // allocator's reclaim. > > // folio B is a folio in another VMA. > > // S1 is freed, folio B can use it > // for swap out with no problem. > ... > folio =3D filemap_get_folio(S1) > // Got folio B here !!! > ... < interrupted again> ... > > // Now S1 is free to be used again. > > // Now src_pte is a swap entry PTE > // holding S1 again. > folio_trylock(folio) > move_swap_pte > double_pt_lock > is_pte_pages_stable > // Check passed because src_pte =3D=3D S1 > folio_move_anon_rmap(...) > // Moved invalid folio B here !!! > > The race window is very short and requires multiple collisions of > multiple rare events, so it's very unlikely to happen, but with a > deliberately constructed reproducer and increased time window, it > can be reproduced easily. > > This can be fixed by checking if the folio returned by filemap is the > valid swap cache folio after acquiring the folio lock. > > Another similar race is possible: filemap_get_folio may return NULL, but > folio (A) could be swapped in and then swapped out again using the same > swap entry after the lookup. In such a case, folio (A) may remain in the > swap cache, so it must be moved too: > > CPU1 CPU2 > userfaultfd_move > move_pages_pte() > entry =3D pte_to_swp_entry(orig_src_pte); > // Here it got entry =3D S1, and S1 is not in swap cache > folio =3D filemap_get_folio(S1) > // Got NULL > ... < interrupted again> ... > > > move_swap_pte > double_pt_lock > is_pte_pages_stable > // Check passed because src_pte =3D=3D S1 > folio_move_anon_rmap(...) > // folio A is ignored !!! > > Fix this by checking the swap cache again after acquiring the src_pte > lock. And to avoid the filemap overhead, we check swap_map directly [2]. > > The SWP_SYNCHRONOUS_IO path does make the problem more complex, but so > far we don't need to worry about that, since folios can only be exposed > to the swap cache in the swap out path, and this is covered in this > patch by checking the swap cache again after acquiring the src_pte lock. > > Testing with a simple C program that allocates and moves several GB of > memory did not show any observable performance change. > > Cc: > Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI") > Closes: https://lore.kernel.org/linux-mm/CAMgjq7B1K=3D6OOrK2OUZ0-tqCzi+EJ= t+2_K97TPGoSt=3D9+JwP7Q@mail.gmail.com/ [1] > Link: https://lore.kernel.org/all/CAGsJ_4yJhJBo16XhiC-nUzSheyX-V3-nFE+tAi= =3D8Y560K8eT=3DA@mail.gmail.com/ [2] > Signed-off-by: Kairui Song Many thanks=EF=BC=81 Reviewed-by: Barry Song > Reviewed-by: Lokesh Gidra > > --- > > V1: https://lore.kernel.org/linux-mm/20250530201710.81365-1-ryncsn@gmail.= com/ > Changes: > - Check swap_map instead of doing a filemap lookup after acquiring the > PTE lock to minimize critical section overhead [ Barry Song, Lokesh Gid= ra ] > > V2: https://lore.kernel.org/linux-mm/20250601200108.23186-1-ryncsn@gmail.= com/ > Changes: > - Move the folio and swap check inside move_swap_pte to avoid skipping > the check and potential overhead [ Lokesh Gidra ] > - Add a READ_ONCE for the swap_map read to ensure it reads a up to dated > value. > > V3: https://lore.kernel.org/all/20250602181419.20478-1-ryncsn@gmail.com/ > Changes: > - Add more comments and more context in commit message. > > mm/userfaultfd.c | 33 +++++++++++++++++++++++++++++++-- > 1 file changed, 31 insertions(+), 2 deletions(-) > > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > index bc473ad21202..8253978ee0fb 100644 > --- a/mm/userfaultfd.c > +++ b/mm/userfaultfd.c > @@ -1084,8 +1084,18 @@ static int move_swap_pte(struct mm_struct *mm, str= uct vm_area_struct *dst_vma, > pte_t orig_dst_pte, pte_t orig_src_pte, > pmd_t *dst_pmd, pmd_t dst_pmdval, > spinlock_t *dst_ptl, spinlock_t *src_ptl, > - struct folio *src_folio) > + struct folio *src_folio, > + struct swap_info_struct *si, swp_entry_t entry) > { > + /* > + * Check if the folio still belongs to the target swap entry afte= r > + * acquiring the lock. Folio can be freed in the swap cache while > + * not locked. > + */ > + if (src_folio && unlikely(!folio_test_swapcache(src_folio) || > + entry.val !=3D src_folio->swap.val)) > + return -EAGAIN; > + > double_pt_lock(dst_ptl, src_ptl); > > if (!is_pte_pages_stable(dst_pte, src_pte, orig_dst_pte, orig_src= _pte, > @@ -1102,6 +1112,25 @@ static int move_swap_pte(struct mm_struct *mm, str= uct vm_area_struct *dst_vma, > if (src_folio) { > folio_move_anon_rmap(src_folio, dst_vma); > src_folio->index =3D linear_page_index(dst_vma, dst_addr)= ; > + } else { > + /* > + * Check if the swap entry is cached after acquiring the = src_pte > + * lock. Otherwise, we might miss a newly loaded swap cac= he folio. > + * > + * Check swap_map directly to minimize overhead, READ_ONC= E is sufficient. > + * We are trying to catch newly added swap cache, the onl= y possible case is > + * when a folio is swapped in and out again staying in sw= ap cache, using the > + * same entry before the PTE check above. The PTL is acqu= ired and released > + * twice, each time after updating the swap_map's flag. S= o holding > + * the PTL here ensures we see the updated value. False p= ositive is possible, > + * e.g. SWP_SYNCHRONOUS_IO swapin may set the flag withou= t touching the > + * cache, or during the tiny synchronization window betwe= en swap cache and > + * swap_map, but it will be gone very quickly, worst resu= lt is retry jitters. > + */ > + if (READ_ONCE(si->swap_map[swp_offset(entry)]) & SWAP_HAS= _CACHE) { > + double_pt_unlock(dst_ptl, src_ptl); > + return -EAGAIN; > + } > } > > orig_src_pte =3D ptep_get_and_clear(mm, src_addr, src_pte); > @@ -1412,7 +1441,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd= _t *dst_pmd, pmd_t *src_pmd, > } > err =3D move_swap_pte(mm, dst_vma, dst_addr, src_addr, ds= t_pte, src_pte, > orig_dst_pte, orig_src_pte, dst_pmd, dst_= pmdval, > - dst_ptl, src_ptl, src_folio); > + dst_ptl, src_ptl, src_folio, si, entry); > } > > out: > -- > 2.49.0 >