From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38DBCC48BF6 for ; Sun, 18 Feb 2024 18:37:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 482E76B0078; Sun, 18 Feb 2024 13:37:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 40C3C6B007B; Sun, 18 Feb 2024 13:37:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2852C6B007D; Sun, 18 Feb 2024 13:37:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 139186B0078 for ; Sun, 18 Feb 2024 13:37:23 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9D8C140176 for ; Sun, 18 Feb 2024 18:37:22 +0000 (UTC) X-FDA: 81805782324.09.D911793 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf30.hostedemail.com (Postfix) with ESMTP id B70E58000E for ; Sun, 18 Feb 2024 18:37:19 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MmmDrkzz; spf=pass (imf30.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708281439; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3wpiB7WqJ1VnIyDpCVSItyO4ulnuZ6JOGYbVejHIO5s=; b=sUWd8/n3D0lY+bgEh79kL2H9Ldm6aNmUZkLP56X+JDVRVchNrIa3dI9nQ9qOpmOJ2uzPKR DWiBzqEoPAtkS3sI0zT99/Q3c3MZRjdUEr9vGRU7sT6tNEMpZkAxfId3eecn6pMQG0dH6Y EgqTyF6CfbvR+n2TTIhectQV8Ix2M50= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708281439; a=rsa-sha256; cv=none; b=mXdZWiKSUHaa4bgrQ9cZ99T6UC7OPmvrXPkTAOOpKUp/TDmhnIqOb9aKHsBPH6qyjWUarE zd08q4Es8vpneHlyNaWj2P1mQ5XsVRJNbibtSqhKqXmlfcQDRgDEgr4K17Fsp2ewOTpXQy pxgvrDAfSpdrPkhc/6+jOdSv9odVbLg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MmmDrkzz; spf=pass (imf30.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id CA5D460BC1 for ; Sun, 18 Feb 2024 18:37:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23464C4166B for ; Sun, 18 Feb 2024 18:37:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708281438; bh=avXJgoEJCe9mxWv/f7Hz3oObHvGvHLsq1pSi10T5wPY=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=MmmDrkzzkjEizPaGKr7qA9Mmcw+AZjaVF2BUQkoUSnSTNoUDig7JAbvMmmQg5NGQ+ 2e17KmhdLCcuhE009hlQgXSLxJqvQunRUUHqHeqHv1WKzmDsWMTRutwE/M0fHchk7N k5eeU7zkatYs1nBYXHpQmVQrmc6p6s2pXa6DQjwtrjGNFHZeAGghUBAmdBCFAEgMqb qwin2Ui+SrvzXQ3cGtl34YN0sSCR1kHo88qlU7p7GVG9Jt/mSQ8orjR28oNDA0ro7q HUfmRwB9Kyn2DlPpN+cLpPGWZJO5hyMR+2AEh5ZLEzbfK8VxbF9sVZ2ClCzZT/lvRl I7ulVL2NA57OQ== Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7c7476a11e0so33057139f.3 for ; Sun, 18 Feb 2024 10:37:18 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXL1l5KLrEMvRiBfXCkdHhNrIH3rir+mIy2BHkZvA1F+9h+DVcwfT5uP0eP3dDg8O5CxK+xcRLAE60szDexNMbj8Tk= X-Gm-Message-State: AOJu0YyiMiBmT5ATvCHjM/FrNrn1DFk7Zn/Wy7wQE+2YE2ENx81wcWWx KUjM1q0/pGFIsb2U2x43qlLHqk07lD6HfRy3eCi+tUe+PVRCzK8jRmiBSHdoAn/GpxNk5b5wYZN zC15Je2uiOY49SbU73HaqhRXRuRESo2crUi7X X-Google-Smtp-Source: AGHT+IFPkHmWeH9mHJ3CxuXj/PnFJU1gwfRSDIcSamShkS2OZXHLItiD2SvdIicEJrhAh+37W2Er3pGicGV+OwM0Uus= X-Received: by 2002:a92:d989:0:b0:365:1459:e1ed with SMTP id r9-20020a92d989000000b003651459e1edmr6017618iln.4.1708281436980; Sun, 18 Feb 2024 10:37:16 -0800 (PST) MIME-Version: 1.0 References: <20240216095105.14502-1-ryncsn@gmail.com> <871q9atd6o.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf1qrwsw.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87sf1qrwsw.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Chris Li Date: Sun, 18 Feb 2024 10:37:02 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v3] mm/swap: fix race when skipping swapcache To: "Huang, Ying" Cc: Barry Song <21cnbao@gmail.com>, David Hildenbrand , Kairui Song , linux-mm@kvack.org, Andrew Morton , Minchan Kim , Yu Zhao , Barry Song , SeongJae Park , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , stable@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: fpomj5t5ogbfwsjfej15jq1st9pbge5y X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B70E58000E X-Rspam-User: X-HE-Tag: 1708281439-140749 X-HE-Meta: U2FsdGVkX19rg+KVqbMubgiDA2pdlJfWfrLTp/nyEDzq6vmSsPV+DLJOWmu2WVKtND2VWIiq6Jwq+cbXpCRvZaXQwpHAOI/XYTcw5MEJ+0DWpLf76XtYZa6TsAzsXy7dxtVP9xs3SpnQzWT1tAyzACh/yNcj8bcpoWvX0fcc9Rll2NYsfngEtsDVqv/7k7gmBjshkT/rhCZ8m20ecslUHR5BMLJsdKNUvqfaA2nGA1Q1cszoycSTQ8MR5mtj3utIQKWyeDqi7/Po0vqabkbkW+RoBuZYei2o94JkTIl2DfxbRtMM35u0lcuOw5mqoi+DiluMRCuYd1CWRi0O0L8bBqyyuSr3MgnhFeBzxzGhRIonRsfNn9q7JnDu9oDaADcS29X/cHmzo3BMQ5dqfwLADPRWEpoEiVwQg4C5oG+Js3iEf+sUAXDR7q15AhWsTjV6t9wk2lLkuXVwJPpQ1aayYZO7KltZlT8LOd2TlNl+i7gUTGtNYiDstb8HrZxhfUEJQfwF6VnxsD0Yp1l8OjJAdu/caf/XJaMSuXmfUo9CUK+LGFoCbb/CliGetFHyMCrUUPzEdTOisSIZXaA6zVwTl/EeYNRyt9onXKP8GuANf4cX8w7ue9e1rHhJF7OHjF4UBEiTfmKUmzjeZKePCrOf7WhSCmPKC4pTjWs5D/E1qF5yUTMj2/Hg2S3GGxNdpncWa457m3e2X8iUHCpm4Yfsx5ndnCPeGb8cXXz3wlWc8/dykSu0wuWVffkkyDH5s5fR3s0FYmPJxdnxWeTlTu2y7q69WN+rkfS7pQR2tnSwJx2wwAWNiCNuXqEeVDN7FyoQkfQg3OxlC67L6XIPTOle6aEOjxTwj7js88QLUJ6c/3DvbaJ+pFC6LB2rtSjMJey3rHCcqAx9h1wTjUD36ISF29nTfWBBfmd0HTyTONAJLqSANgyH7jGIphqRvXTiGAK2yFPCJihOYvKkhg0H/tS lJAp3NPS oB+NUeRxfFLDvi27hlBBDP9DDaDLY/m8Olux/Nl1FxgA7H9SZl3LD+qlx1kGU+JPznGmL6ysEfR3Lw8w6K9UcJwYpm5/Rh1ESORfwh/HzB8x81bRHr5yPMeud2HtM+7JbDYB8GHsgkLJovChtfOgtnbZziJGgRUzlmxn/Nk+dNbYljqnAp/uzDFr43y7vU93tqHT8eiYYZVXiTEM8JboZMQrTpwwa43Cm1L/SrfuJJLng/Xiw7lNjB5CfIkjy5QLJjQFCCm3ooLkj4l/V9CefqtICbN9C7HfKufJWba9RzE/gU3MIkmwKvVc6bWRRnVgYDs+XQgEEa8ibeEUVo6I2vYIHkG6B2vy1cgg7ruEKpAHPLX40NmheumNxgLfkes3BU6517jOsjOYWtBm85/scRukaqy9FgSYiJgfEfUgWSWVzc9RFApU3ilLIbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Feb 18, 2024 at 12:41=E2=80=AFAM Huang, Ying = wrote: > > Barry Song <21cnbao@gmail.com> writes: > > > On Sun, Feb 18, 2024 at 9:02=E2=80=AFPM Huang, Ying wrote: > >> > >> David Hildenbrand writes: > >> > >> > On 16.02.24 10:51, Kairui Song wrote: > >> >> From: Kairui Song > >> >> When skipping swapcache for SWP_SYNCHRONOUS_IO, if two or more > >> >> threads > >> >> swapin the same entry at the same time, they get different pages (A= , B). > >> >> Before one thread (T0) finishes the swapin and installs page (A) > >> >> to the PTE, another thread (T1) could finish swapin of page (B), > >> >> swap_free the entry, then swap out the possibly modified page > >> >> reusing the same entry. It breaks the pte_same check in (T0) becaus= e > >> >> PTE value is unchanged, causing ABA problem. Thread (T0) will > >> >> install a stalled page (A) into the PTE and cause data corruption. > >> >> One possible callstack is like this: > >> >> CPU0 CPU1 > >> >> ---- ---- > >> >> do_swap_page() do_swap_page() with same entry > >> >> > >> >> > >> >> swap_read_folio() <- read to page A swap_read_folio() <- read to p= age B > >> >> > >> >> ... set_pte_at() > >> >> swap_free() <- entry is free > >> >> > >> >> > >> >> pte_same() <- Check pass, PTE seems > >> >> unchanged, but page A > >> >> is stalled! > >> >> swap_free() <- page B content lost! > >> >> set_pte_at() <- staled page A installed! > >> >> And besides, for ZRAM, swap_free() allows the swap device to discar= d > >> >> the entry content, so even if page (B) is not modified, if > >> >> swap_read_folio() on CPU0 happens later than swap_free() on CPU1, > >> >> it may also cause data loss. > >> >> To fix this, reuse swapcache_prepare which will pin the swap entry > >> >> using > >> >> the cache flag, and allow only one thread to pin it. Release the pi= n > >> >> after PT unlocked. Racers will simply wait since it's a rare and ve= ry > >> >> short event. A schedule() call is added to avoid wasting too much C= PU > >> >> or adding too much noise to perf statistics > >> >> Other methods like increasing the swap count don't seem to be a goo= d > >> >> idea after some tests, that will cause racers to fall back to use t= he > >> >> swap cache again. Parallel swapin using different methods leads to > >> >> a much more complex scenario. > >> >> Reproducer: > >> >> This race issue can be triggered easily using a well constructed > >> >> reproducer and patched brd (with a delay in read path) [1]: > >> >> With latest 6.8 mainline, race caused data loss can be observed > >> >> easily: > >> >> $ gcc -g -lpthread test-thread-swap-race.c && ./a.out > >> >> Polulating 32MB of memory region... > >> >> Keep swapping out... > >> >> Starting round 0... > >> >> Spawning 65536 workers... > >> >> 32746 workers spawned, wait for done... > >> >> Round 0: Error on 0x5aa00, expected 32746, got 32743, 3 data los= s! > >> >> Round 0: Error on 0x395200, expected 32746, got 32743, 3 data lo= ss! > >> >> Round 0: Error on 0x3fd000, expected 32746, got 32737, 9 data lo= ss! > >> >> Round 0 Failed, 15 data loss! > >> >> This reproducer spawns multiple threads sharing the same memory > >> >> region > >> >> using a small swap device. Every two threads updates mapped pages o= ne by > >> >> one in opposite direction trying to create a race, with one dedicat= ed > >> >> thread keep swapping out the data out using madvise. > >> >> The reproducer created a reproduce rate of about once every 5 > >> >> minutes, > >> >> so the race should be totally possible in production. > >> >> After this patch, I ran the reproducer for over a few hundred round= s > >> >> and no data loss observed. > >> >> Performance overhead is minimal, microbenchmark swapin 10G from 32G > >> >> zram: > >> >> Before: 10934698 us > >> >> After: 11157121 us > >> >> Non-direct: 13155355 us (Dropping SWP_SYNCHRONOUS_IO flag) > >> >> Fixes: 0bcac06f27d7 ("mm, swap: skip swapcache for swapin of > >> >> synchronous device") > >> >> Link: https://github.com/ryncsn/emm-test-project/tree/master/swap-s= tress-race [1] > >> >> Reported-by: "Huang, Ying" > >> >> Closes: https://lore.kernel.org/lkml/87bk92gqpx.fsf_-_@yhuang6-desk= 2.ccr.corp.intel.com/ > >> >> Signed-off-by: Kairui Song > >> >> Cc: stable@vger.kernel.org > >> >> --- > >> >> Update from V2: > >> >> - Add a schedule() if raced to prevent repeated page faults wasting= CPU > >> >> and add noise to perf statistics. > >> >> - Use a bool to state the special case instead of reusing existing > >> >> variables fixing error handling [Minchan Kim]. > >> >> V2: > >> >> https://lore.kernel.org/all/20240206182559.32264-1-ryncsn@gmail.com= / > >> >> Update from V1: > >> >> - Add some words on ZRAM case, it will discard swap content on swap= _free so the race window is a bit different but cure is the same. [Barry So= ng] > >> >> - Update comments make it cleaner [Huang, Ying] > >> >> - Add a function place holder to fix CONFIG_SWAP=3Dn built [SeongJa= e Park] > >> >> - Update the commit message and summary, refer to SWP_SYNCHRONOUS_I= O instead of "direct swapin path" [Yu Zhao] > >> >> - Update commit message. > >> >> - Collect Review and Acks. > >> >> V1: > >> >> https://lore.kernel.org/all/20240205110959.4021-1-ryncsn@gmail.com/ > >> >> include/linux/swap.h | 5 +++++ > >> >> mm/memory.c | 20 ++++++++++++++++++++ > >> >> mm/swap.h | 5 +++++ > >> >> mm/swapfile.c | 13 +++++++++++++ > >> >> 4 files changed, 43 insertions(+) > >> >> diff --git a/include/linux/swap.h b/include/linux/swap.h > >> >> index 4db00ddad261..8d28f6091a32 100644 > >> >> --- a/include/linux/swap.h > >> >> +++ b/include/linux/swap.h > >> >> @@ -549,6 +549,11 @@ static inline int swap_duplicate(swp_entry_t s= wp) > >> >> return 0; > >> >> } > >> >> +static inline int swapcache_prepare(swp_entry_t swp) > >> >> +{ > >> >> + return 0; > >> >> +} > >> >> + > >> >> static inline void swap_free(swp_entry_t swp) > >> >> { > >> >> } > >> >> diff --git a/mm/memory.c b/mm/memory.c > >> >> index 7e1f4849463a..7059230d0a54 100644 > >> >> --- a/mm/memory.c > >> >> +++ b/mm/memory.c > >> >> @@ -3799,6 +3799,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > >> >> struct page *page; > >> >> struct swap_info_struct *si =3D NULL; > >> >> rmap_t rmap_flags =3D RMAP_NONE; > >> >> + bool need_clear_cache =3D false; > >> >> bool exclusive =3D false; > >> >> swp_entry_t entry; > >> >> pte_t pte; > >> >> @@ -3867,6 +3868,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf= ) > >> >> if (!folio) { > >> >> if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && > >> >> __swap_count(entry) =3D=3D 1) { > >> >> + /* > >> >> + * Prevent parallel swapin from proceeding wit= h > >> >> + * the cache flag. Otherwise, another thread m= ay > >> >> + * finish swapin first, free the entry, and sw= apout > >> >> + * reusing the same entry. It's undetectable a= s > >> >> + * pte_same() returns true due to entry reuse. > >> >> + */ > >> >> + if (swapcache_prepare(entry)) { > >> >> + /* Relax a bit to prevent rapid repeat= ed page faults */ > >> >> + schedule(); > >> >> + goto out; > >> >> + } > >> >> + need_clear_cache =3D true; > >> >> + > >> > > >> > I took a closer look at __read_swap_cache_async() and it essentially > >> > does something similar. > >> > > >> > Instead of returning, it keeps retrying until it finds that > >> > swapcache_prepare() fails for another reason than -EEXISTS (e.g., > >> > freed concurrently) or it finds the entry in the swapcache. > >> > > >> > So if you would succeed here on a freed+reused swap entry, > >> > __read_swap_cache_async() would simply retry. > >> > > >> > It spells that out: > >> > > >> > /* > >> > * We might race against __delete_from_swap_cache(), a= nd > >> > * stumble across a swap_map entry whose SWAP_HAS_CACH= E > >> > * has not yet been cleared. Or race against another > >> > * __read_swap_cache_async(), which has set SWAP_HAS_C= ACHE > >> > * in swap_map, but not yet added its folio to swap ca= che. > >> > */ > >> > > >> > Whereby we could not race against this code here as well where we > >> > speculatively set SWAP_HAS_CACHE and might never add something to th= e swap > >> > cache. > >> > > >> > > >> > I'd probably avoid the wrong returns and do something even closer to > >> > __read_swap_cache_async(). > >> > > >> > while (true) { > >> > /* > >> > * Fake that we are trying to insert a page into the swapcache= , to > >> > * serialize against concurrent threads wanting to do the same= . > >> > * [more from your description] > >> > */ > >> > ret =3D swapcache_prepare(entry); > >> > if (likely(!ret) > >> > /* > >> > * Move forward with swapin, we'll recheck if the PTE = hasn't > >> > * changed later. > >> > */ > >> > break; > >> > else if (ret !=3D -EEXIST) > >> > goto out; > >> > >> The swap entry may be kept in swap cache for long time. For example, = it > >> may be read into swap cache via MADV_WILLNEED. I am trying to find the alternative path which can cause the swap fault to bring in the swap cache in page fault while the SYNC IO is looping for HAS_SWAP_CACHE. Kairui was able to identify the in the current code in do_page_fault() path, the rmap and fork case wouldn't be able to modify the swap cache causing a problem. The MADV_WILLNEED is an excellent example. Thank you for finding this example. > > > > This seems fine. > > > > if swapcache has data from WILLNEED, the new page fault will hit it. Th= us, > > we won't go into the SYNC_IO path any more? > > They may happen in parallel. That is, one task is busy looping, while > another task read the swap entry into swap cache. Agree. Chris