From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89A86C02198 for ; Fri, 14 Feb 2025 17:59:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 280EE28000A; Fri, 14 Feb 2025 12:59:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20882280002; Fri, 14 Feb 2025 12:59:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 082C728000A; Fri, 14 Feb 2025 12:59:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D5E55280002 for ; Fri, 14 Feb 2025 12:59:00 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 913E31C7F0C for ; Fri, 14 Feb 2025 17:59:00 +0000 (UTC) X-FDA: 83119311240.07.EB60319 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf23.hostedemail.com (Postfix) with ESMTP id A4BC0140017 for ; Fri, 14 Feb 2025 17:58:58 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A3u4QK+p; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739555938; a=rsa-sha256; cv=none; b=hH9/4IyqBlt7acw7DmqAyXZ0pniQcHii6yG0glwgPyOULETodATe3xUXWNXBfORtuIX/bx 8hEb33MW5xf3sf8UNmRyoLQAWutoqayGqtOxFeOVf0NehjzXHpEnSZZ7chMiuJ3WrTzasA ohJgHkUJDRto/JncH3wEPy3KHwNtS6k= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A3u4QK+p; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739555938; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g131YEqRMfLTVZhTMtUAtQy5KjWaVYxNxWGQgGzsAzk=; b=A3up3Y8OAOkMhX7uDFnvXgEhRBlkHo49fsSe5JoD4rFDxmRO7QrNC+fe2VgxqP6dYMYqDt GEi5W59T15qnxvYjNzyXOXzpExiJCriFYsQ2kNiV2ebPOvoCfdIV+8pTq8joSy1X0Mrgh3 ecxbZoxVJdNhUr3WUuQO4Iu++rDbpmE= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220c8eb195aso47559755ad.0 for ; Fri, 14 Feb 2025 09:58:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739555937; x=1740160737; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=g131YEqRMfLTVZhTMtUAtQy5KjWaVYxNxWGQgGzsAzk=; b=A3u4QK+pzVYgqg4ydOHxmaAjq89HR493kJcCCCYc/qgl8lx2AsjmLxh/RlseFgKgJp 4njoNN9T0cb62DJE/gt937sZ1AdnP14pTWpL9iegSWaY8Dnp8QC+L9GRNugtDOKEK2om OXC5wsEr/4ck5SUdno7gtYMUteMxrdZ4oY/dVkHx+IDKrzYEEiJtb8N6/MMJ8iXZ8KOE 2ELRUaL7QbaPZ+g0Ve7CSHhuKK9MWMsxBKGZlC0D591ItEBCwYbeQJl6UqURJA15dsJy H+qaMR+7wMrXHsEYoT3SC6pNoi7l7M1e+M/9Y/XnDtrHzuE/WrwPIerJFuPz6bhbuv82 +0Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739555937; x=1740160737; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=g131YEqRMfLTVZhTMtUAtQy5KjWaVYxNxWGQgGzsAzk=; b=L+j6LTPucjMFdsIIcyJSR7/pk5u3yCG680RTNhideS9LDps5KnulfVA5FCEwQADAKd 6WCEZPB5zcjAd1EZCxfcyIG+xGTSephqs+iZaAV9BbkT/8cXXCGSfj8zEpwmtx+aSBPe Db2hnBy6XrgBgg/X1a4ZcKICP4VNQ0zKjwuKZ10I+lFGHXXEqaQhnAh4y+TsmFRLRvVP dqV75dUOHM5+D2qVuUPe3SE1IjEbhZfmx15pl4fTpxlnCwUSaU+HdS7cQX02K6cZ39n0 5y2wYp6AkkfBpdeOwkSN8nNQOSerckctUh0Rqn8Idtcw8wy/o872F6kU9ok6LQZszDcw 9aMA== X-Gm-Message-State: AOJu0YwRSuy+3XpY2Cwng0KV7a1lo8AiUJdyQIjjAy2nkwxwL10W5Yb/ CrNIIOI/qV4183Hoj7+mn7foMNKLU+Krttm9S5crUbUIdK2cJW2+1plxte2uI+g= X-Gm-Gg: ASbGncuEVRDeE/ibBp9n/g+HnOvVmgTC61teBkASoWaglVpvL9qw4pyxrQWln2mkpFW G+PzhhLQw3NKETS9nSiFpgFKd2qOiK/KqCDpRL7PknEmjVwX8luxzb1CniT2YnFro5lP05vbgBh cemDz/bprZzQKH6mjg/9WO6Rr3x/bwqWhWc5Vi1bEKQa91ezcQQL/8qKIQSHD4JgzREAzaJplSR v+EBTIOOEQtbf3e2AR4e0pTo85DOlFxusULVT4wb9Ye+SQyv2eqPqXf5vVEJMpPdA2wKut/vXVt gkRZQAEXaWiGrsnDvO7O6UVvR5ehQzTUbwJg X-Google-Smtp-Source: AGHT+IEBFWRSKG7OwZIbmlb93b7xdfMca6gf2NNi7p6B/FLu0eRP37//Pl09ZSVseFox+/txo2Nx6Q== X-Received: by 2002:a17:902:d483:b0:215:6e01:ad07 with SMTP id d9443c01a7336-22103efb5b1mr2464155ad.6.1739555937010; Fri, 14 Feb 2025 09:58:57 -0800 (PST) Received: from KASONG-MC4.tencent.com ([106.37.120.226]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d55943b5sm31216605ad.246.2025.02.14.09.58.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 14 Feb 2025 09:58:56 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Baoquan He , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 7/7] mm, swap: simplify folio swap allocation Date: Sat, 15 Feb 2025 01:57:09 +0800 Message-ID: <20250214175709.76029-8-ryncsn@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214175709.76029-1-ryncsn@gmail.com> References: <20250214175709.76029-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A4BC0140017 X-Stat-Signature: 9nzrsc9s5rwhehzddtd3zems3tzu7y95 X-Rspam-User: X-HE-Tag: 1739555938-739516 X-HE-Meta: U2FsdGVkX18y8tduqN0MjtT4Xi9nEa0ZyZn2RIBvNRbar6/t9OLjYjbGY0eFRlzFQy7vhEkfnmEqyKGbjxoVXSXuYd4UojrTYv50rcbS9s0q+8kI2aJG8roefpKUwRXKe2BteLtNOx0ugrKAUXScl9APxTkJnoZqZmnqxI63wcZDGgYZPyT1DDMXfeE8mvAtyUWo2P1jhejyuySccrdC0De7X8CG34ejXSFklVRPu1SGal9cGz1sZ1rBC/CGqMmns0UFnzcBQlYdfr4TW30My3st2GoTURfvimsPB+wufHm19PVFpv4g9GywBbEXICq+b3PXfB+R2aW7jEjzMzKlSIiclYIQwWj8v9laB9h5O6lRkSrjoQ/D0qoHNzs351VBTQ+H1YU5NA0kHxYVtAxeKvVsbAgoYtxnpGXnMO5383sfoKE4r1ugeuXvVy3dKx4jfhFhaki32Z/jGVCwnBEaH/vHbYwB1UYaRxDstQ1g88SY4B7TgIu2eyVO7sf9J+Q7TW0aCq64oOYvYKvYPjfdWR4ReYF7eYN89xmSh5gDN5b/afT80ifWeC2Usm0OxMxapwx9o1HuXfWhyVYVcAsSdDTZuHMAcbNwYaPGTkqVQVMA0SpvaL4sNKHCMBfjmz5hVpmXBVB3i10Br9mEMx9/2MA8r9r+gUlhZi7wZPCP1sLveEFQliFcUCh0V7lKvsIF/Dpl2FHuiko9TPZGthdxIExdkCOu9yq61sSHLso6C2VuaRXe6e8oaHijTgqV3LJKQ1FL22ksjAtrnefYrXmd0JcXvdggJsiDTIIXFnXQZDqyyXdcO8bBmgxDD1k1W40d0benRlKV/1SYX/lVRPWtJ2AoMQox/inIsMIvMWOEZa45B6fw5AyG43TRAO0WeMc7++0a44HOrB11BFj8kV59XD26XHtXx7281Q3ZqrnmiGjZafRgaa4rp0KW/NvhjaNY0/wSFrsm99EYXZk4wV7 wBEI0Pd3 OHoqJ9lG+W7CMHB6Ll5M384hCKMywF28RGiS3lnzm59QF7u84nUmc7vcri3V47irizLUaLTm7LtlLU1seVs9OjLs7xBq1B+IyWY9xXrtGsr8dyj+3b6CThoRXfseIIUHYvpfXOqhD553fbtnFvKOWiKK268jp6D3Etsby8zEYk5Zg/id7I5D5msgHqa/he5mLZ/nd8Tb1Y9ZYp4iIx8hpEPfZmHZFOU5kMH8oK+rDQj2pYTGay4EPgnn6e8EMtxV9MSshjHC6onp9NK+hmnrT11E0aNC0jbwcGmDPwPvxq/ggoIgj8JUG3VXHg5kbUHrxUY4/4goGadow09/by/wUQijCP2IbdXNlWQFn900LcVlXXOhphNSvWM+bPAo7VydtDnCf+qRAYktAtA4JcaHmhH9/78QZbsgFof8Jc014aCr68LNhT3233n5udFTr+KtRDgXLWs+d6rcFbHXgEvxRZJSQ6HFWoHYHlxYbqR5G9rOfiFl3bvnjLJTaV2RAv2X972PRqbQnfwvoZDu9XsGBmUQbMQebeKfEcG2pQcc+IITvVA4MXgy16qFAnQuGNr1boQTj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song With slot cache gone, clean up the allocation helpers even more. folio_alloc_swap will be the only entry for allocation and adding the folio to swap cache (except suspend), making it opposite of folio_free_swap. Signed-off-by: Kairui Song --- include/linux/swap.h | 8 ++-- mm/shmem.c | 21 +++------ mm/swap.h | 6 --- mm/swap_state.c | 57 ---------------------- mm/swapfile.c | 110 ++++++++++++++++++++++++++++--------------- mm/vmscan.c | 16 ++++++- 6 files changed, 94 insertions(+), 124 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 456833705ea0..e799e965dac8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -478,7 +478,7 @@ static inline long get_nr_swap_pages(void) } extern void si_swapinfo(struct sysinfo *); -swp_entry_t folio_alloc_swap(struct folio *folio); +bool folio_alloc_swap(struct folio *folio, gfp_t gfp_mask); bool folio_free_swap(struct folio *folio); void put_swap_folio(struct folio *folio, swp_entry_t entry); extern swp_entry_t get_swap_page_of_type(int); @@ -587,11 +587,9 @@ static inline int swp_swapcount(swp_entry_t entry) return 0; } -static inline swp_entry_t folio_alloc_swap(struct folio *folio) +static bool folio_alloc_swap(struct folio *folio, gfp_t gfp_mask); { - swp_entry_t entry; - entry.val = 0; - return entry; + return false; } static inline bool folio_free_swap(struct folio *folio) diff --git a/mm/shmem.c b/mm/shmem.c index b35ba250c53d..2aa206b52ff2 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1546,7 +1546,6 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) struct inode *inode = mapping->host; struct shmem_inode_info *info = SHMEM_I(inode); struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); - swp_entry_t swap; pgoff_t index; int nr_pages; bool split = false; @@ -1628,14 +1627,6 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) folio_mark_uptodate(folio); } - swap = folio_alloc_swap(folio); - if (!swap.val) { - if (nr_pages > 1) - goto try_split; - - goto redirty; - } - /* * Add inode to shmem_unuse()'s list of swapped-out inodes, * if it's not already there. Do it now before the folio is @@ -1648,20 +1639,20 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(folio, swap, - __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, - NULL) == 0) { + if (folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN)) { shmem_recalc_inode(inode, 0, nr_pages); - swap_shmem_alloc(swap, nr_pages); - shmem_delete_from_page_cache(folio, swp_to_radix_entry(swap)); + swap_shmem_alloc(folio->swap, nr_pages); + shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap)); mutex_unlock(&shmem_swaplist_mutex); BUG_ON(folio_mapped(folio)); return swap_writepage(&folio->page, wbc); } + list_del_init(&info->swaplist); mutex_unlock(&shmem_swaplist_mutex); - put_swap_folio(folio, swap); + if (nr_pages > 1) + goto try_split; redirty: folio_mark_dirty(folio); if (wbc->for_reclaim) diff --git a/mm/swap.h b/mm/swap.h index ad2f121de970..0abb68091b4f 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -50,7 +50,6 @@ static inline pgoff_t swap_cache_index(swp_entry_t entry) } void show_swap_cache_info(void); -bool add_to_swap(struct folio *folio); void *get_shadow_from_swap_cache(swp_entry_t entry); int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp, void **shadowp); @@ -163,11 +162,6 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, return filemap_get_folio(mapping, index); } -static inline bool add_to_swap(struct folio *folio) -{ - return false; -} - static inline void *get_shadow_from_swap_cache(swp_entry_t entry) { return NULL; diff --git a/mm/swap_state.c b/mm/swap_state.c index 2b5744e211cd..68fd981b514f 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -166,63 +166,6 @@ void __delete_from_swap_cache(struct folio *folio, __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr); } -/** - * add_to_swap - allocate swap space for a folio - * @folio: folio we want to move to swap - * - * Allocate swap space for the folio and add the folio to the - * swap cache. - * - * Context: Caller needs to hold the folio lock. - * Return: Whether the folio was added to the swap cache. - */ -bool add_to_swap(struct folio *folio) -{ - swp_entry_t entry; - int err; - - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); - - entry = folio_alloc_swap(folio); - if (!entry.val) - return false; - - /* - * XArray node allocations from PF_MEMALLOC contexts could - * completely exhaust the page allocator. __GFP_NOMEMALLOC - * stops emergency reserves from being allocated. - * - * TODO: this could cause a theoretical memory reclaim - * deadlock in the swap out path. - */ - /* - * Add it to the swap cache. - */ - err = add_to_swap_cache(folio, entry, - __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN, NULL); - if (err) - goto fail; - /* - * Normally the folio will be dirtied in unmap because its - * pte should be dirty. A special case is MADV_FREE page. The - * page's pte could have dirty bit cleared but the folio's - * SwapBacked flag is still set because clearing the dirty bit - * and SwapBacked flag has no lock protected. For such folio, - * unmap will not set dirty bit for it, so folio reclaim will - * not write the folio out. This can cause data corruption when - * the folio is swapped in later. Always setting the dirty flag - * for the folio solves the problem. - */ - folio_mark_dirty(folio); - - return true; - -fail: - put_swap_folio(folio, entry); - return false; -} - /* * This must be called only on folios that have * been verified to be in the swap cache and locked. diff --git a/mm/swapfile.c b/mm/swapfile.c index 66c8869ef346..8449bd703bd8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1181,9 +1181,9 @@ static bool get_swap_device_info(struct swap_info_struct *si) * Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static int swap_alloc_fast(swp_entry_t *entry, - unsigned char usage, - int order) +static bool swap_alloc_fast(swp_entry_t *entry, + unsigned char usage, + int order) { struct swap_cluster_info *ci; struct swap_info_struct *si; @@ -1203,47 +1203,31 @@ static int swap_alloc_fast(swp_entry_t *entry, return !!found; } -swp_entry_t folio_alloc_swap(struct folio *folio) +/* Rotate the device and switch to a new cluster */ +static bool swap_alloc_rotate(swp_entry_t *entry, + unsigned char usage, + int order) { - unsigned int order = folio_order(folio); - unsigned int size = 1 << order; - struct swap_info_struct *si, *next; - swp_entry_t entry = {}; - unsigned long offset; int node; + unsigned long offset; + struct swap_info_struct *si, *next; - if (order) { - /* - * Should not even be attempting large allocations when huge - * page swap is disabled. Warn and fail the allocation. - */ - if (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER) { - VM_WARN_ON_ONCE(1); - return entry; - } - } - - /* Fast path using percpu cluster */ - local_lock(&percpu_swap_cluster.lock); - if (swap_alloc_fast(&entry, SWAP_HAS_CACHE, order)) - goto out_alloced; - - /* Rotate the device and switch to a new cluster */ + node = numa_node_id(); spin_lock(&swap_avail_lock); start_over: - node = numa_node_id(); plist_for_each_entry_safe(si, next, &swap_avail_heads[node], avail_lists[node]) { + /* Rotate the device and switch to a new cluster */ plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]); spin_unlock(&swap_avail_lock); if (get_swap_device_info(si)) { offset = cluster_alloc_swap_entry(si, order, SWAP_HAS_CACHE); put_swap_device(si); if (offset) { - entry = swp_entry(si->type, offset); - goto out_alloced; + *entry = swp_entry(si->type, offset); + return true; } if (order) - goto out_failed; + return false; } spin_lock(&swap_avail_lock); @@ -1262,20 +1246,68 @@ swp_entry_t folio_alloc_swap(struct folio *folio) goto start_over; } spin_unlock(&swap_avail_lock); -out_failed: + return false; +} + +/** + * folio_alloc_swap - allocate swap space for a folio + * @folio: folio we want to move to swap + * @gfp: gfp mask for shadow nodes + * + * Allocate swap space for the folio and add the folio to the + * swap cache. + * + * Context: Caller needs to hold the folio lock. + * Return: Whether the folio was added to the swap cache. + */ +bool folio_alloc_swap(struct folio *folio, gfp_t gfp) +{ + unsigned int order = folio_order(folio); + unsigned int size = 1 << order; + swp_entry_t entry = {}; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); + + /* + * Should not even be attempting large allocations when huge + * page swap is disabled. Warn and fail the allocation. + */ + if (order && (!IS_ENABLED(CONFIG_THP_SWAP) || size > SWAPFILE_CLUSTER)) { + VM_WARN_ON_ONCE(1); + return false; + } + + local_lock(&percpu_swap_cluster.lock); + if (swap_alloc_fast(&entry, SWAP_HAS_CACHE, order)) + goto out_alloced; + if (swap_alloc_rotate(&entry, SWAP_HAS_CACHE, order)) + goto out_alloced; local_unlock(&percpu_swap_cluster.lock); - return entry; + return false; out_alloced: local_unlock(&percpu_swap_cluster.lock); - if (mem_cgroup_try_charge_swap(folio, entry)) { - put_swap_folio(folio, entry); - entry.val = 0; - } else { - atomic_long_sub(size, &nr_swap_pages); - } + if (mem_cgroup_try_charge_swap(folio, entry)) + goto out_free; - return entry; + /* + * XArray node allocations from PF_MEMALLOC contexts could + * completely exhaust the page allocator. __GFP_NOMEMALLOC + * stops emergency reserves from being allocated. + * + * TODO: this could cause a theoretical memory reclaim + * deadlock in the swap out path. + */ + if (add_to_swap_cache(folio, entry, gfp | __GFP_NOMEMALLOC, NULL)) + goto out_free; + + atomic_long_sub(size, &nr_swap_pages); + return true; + +out_free: + put_swap_folio(folio, entry); + return false; } static struct swap_info_struct *_swap_info_get(swp_entry_t entry) diff --git a/mm/vmscan.c b/mm/vmscan.c index fcca38bc640f..71a6b597e469 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1289,7 +1289,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, split_folio_to_list(folio, folio_list)) goto activate_locked; } - if (!add_to_swap(folio)) { + if (!folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOWARN)) { int __maybe_unused order = folio_order(folio); if (!folio_test_large(folio)) @@ -1305,9 +1305,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, } #endif count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK); - if (!add_to_swap(folio)) + if (!folio_alloc_swap(folio, __GFP_HIGH | __GFP_NOWARN)) goto activate_locked_split; } + /* + * Normally the folio will be dirtied in unmap because its + * pte should be dirty. A special case is MADV_FREE page. The + * page's pte could have dirty bit cleared but the folio's + * SwapBacked flag is still set because clearing the dirty bit + * and SwapBacked flag has no lock protected. For such folio, + * unmap will not set dirty bit for it, so folio reclaim will + * not write the folio out. This can cause data corruption when + * the folio is swapped in later. Always setting the dirty flag + * for the folio solves the problem. + */ + folio_mark_dirty(folio); } } -- 2.48.1