From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33C20C3ABDA for ; Wed, 14 May 2025 20:19:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7023C6B00B6; Wed, 14 May 2025 16:19:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B1036B00B7; Wed, 14 May 2025 16:19:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 500E26B00B8; Wed, 14 May 2025 16:19:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 274CB6B00B6 for ; Wed, 14 May 2025 16:19:32 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 23FAC160AEA for ; Wed, 14 May 2025 20:19:32 +0000 (UTC) X-FDA: 83442628584.09.7994523 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf05.hostedemail.com (Postfix) with ESMTP id 3CAEE100008 for ; Wed, 14 May 2025 20:19:30 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CS67fJos; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747253970; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SAUgAKhRtBA+T6vbn3SCUdXrNKCIX35HSqyOu0l+ing=; b=3mISAaIcmicrX8cCIN7W2z4l90ZoBYjQ9CDuEVSTFa4Lw/Oz3IK4eAD49IedEUttzpk8SK iW9dcoBoOfGBS9OvRY4MPup4M7YuzUmveNqzEGaKFD7BuKzwCye4Cj5PqsDdYEHW3zTlcM ylNKP771mQLocYKxQUsy0EZbiuS5QcE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CS67fJos; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747253970; a=rsa-sha256; cv=none; b=WU8aHmMvF++oYLx+gGYWQc60duDS1pR1dmD+mva5ThGd2qYa4jr6pei1FvClvFUsgogu3A DoFByjX2RtprDM+hfMih0sy87LhqCKO7lctvfXUImErBEy1NSFsR2YqOUdSKHAiFW3zXUJ xgvkC8NAbUUCAD3T0LIVjy/Y1JKK9Pg= Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7fd581c2bf4so102165a12.3 for ; Wed, 14 May 2025 13:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747253968; x=1747858768; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=SAUgAKhRtBA+T6vbn3SCUdXrNKCIX35HSqyOu0l+ing=; b=CS67fJosdUXWm6Zt9U8ecrxV2bSKlEvW8OpJaVW6AZfERGnIuUdw1PJaITxZ9dKkGR MXr1At++crESfz4xYxWR1BgWuvl+Ov+jwi7qlQtsFMQXK/K95qdSU48IL0brgSRKCBtB 75WOXJBTARAIBSVbF4+HC65TPux9mukE0GAbM62pD6aUFNmslK+ZjmetmKev55LhPrIn r6d3EKYNjedq/NblZ5MYnc8Xbcmyw2w0fzibBVIN6vUXgYm4uKuZWtwDtI8434gDGawn G/R/8/3cpAwLqpBNl5r79nDVcjyTBj+0h4HYVtp+pWg1liDvQMUxITsUj2sxyrstO0VF ma3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747253968; x=1747858768; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=SAUgAKhRtBA+T6vbn3SCUdXrNKCIX35HSqyOu0l+ing=; b=KmrHOQSAUeWPIBI0qL+Y8zBoNyCHgxRajMjUAI5nLeEUQ25lWMAa1L5tZNWJDckzgC 4RGebHnSXfG2AHHtCm74Je9Tswn7bOpuFeCYdR2QwWJ8o5avChabXpjKxoZkZOiWqJKb rPv5iLiJa3Z+SyTquWhrAfSTR4KxKkQL9DdiGbZP+h8B0KJG0fJPTsUu20aktof9dJ7U OFLJYwTd70HViEzBsYEQZgwJn2fsYVIVTZIiuVI8N2tuRlhxXwJLUEUn6iX5nuFbLivd L1d5/oP6tnY26XyrQQ/+E80WQ0dkNhilrjX9P7BKzZecWFt8eJ0Bmfru9UfpmFSVgoP4 qGvA== X-Gm-Message-State: AOJu0Yw8t5uZgAXEvopmxhUZyDS9FVDF/Tub8wIJKXssNKavaf821Lfn PeP4kXUlPer0CFExnuiskqvFj+ZYJVbN9IpcjffXIFW9Dv7YE1L4Tp0cCR6mDWE= X-Gm-Gg: ASbGncu8/AEU/i5MNDpGAAXKPbvLw/f8Z44gHlSg09Kt0K+NXh/u9XTwmN+G/d1ifr1 YFTtftD5DXeI+499un2Apl/4f03gkPj4F5SI+YdB5eNbZAYtPAezuimuazRob14HVHSB8CvMsel na9FYIg/GYHhulYqgZbBpPl0CE/l8yDrhMiYkOtZQAeGxiQUXpof0l42sVn3eUGQ3JJ7kZ3luZD i1UjXT+eGVZ7urwwnysJ6RnKg7+nulpd8qgmM0KHhEuNcsYzUKcXCt/f5C2Bpoyk1ctKUpLjXl1 37oPBWOuN8OyMfWa5O6ISTNda6IG7iK38vJqWnM7hGqjYI/a+zzIaUWmJL9C2YGgaQSTnGGQ X-Google-Smtp-Source: AGHT+IEtQBiYOo/yRXuzGXXOkEu6LpuMIZhrEF4/ummoZk9ecubjYBuIrdy9u6QyT1pLoiZEuAUNXw== X-Received: by 2002:a17:902:e810:b0:224:10a2:cae1 with SMTP id d9443c01a7336-23198145208mr59404535ad.37.1747253967808; Wed, 14 May 2025 13:19:27 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-30e33401934sm2003692a91.9.2025.05.14.13.19.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 14 May 2025 13:19:27 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Hugh Dickins , Chris Li , David Hildenbrand , Yosry Ahmed , "Huang, Ying" , Nhat Pham , Johannes Weiner , Baolin Wang , Baoquan He , Barry Song , Kalesh Singh , Kemeng Shi , Tim Chen , Ryan Roberts , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 21/28] mm, swap: add folio to swap cache directly on allocation Date: Thu, 15 May 2025 04:17:21 +0800 Message-ID: <20250514201729.48420-22-ryncsn@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250514201729.48420-1-ryncsn@gmail.com> References: <20250514201729.48420-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 3CAEE100008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: dm1gx73g3wzikzj8eyp3mheabx7sintk X-HE-Tag: 1747253970-543286 X-HE-Meta: U2FsdGVkX1/aLbcJSd3oykXoAh0MVR3CcLmJwm+0j0l82HWjGbn1D5KrXxdLqb9FUskHKKDd+SLlrGaaBTxbKZQRGPDt0YdiaLDkAJ0UzLDb+Zr6RkaUV7/4FCfbpRxtB7i8bd9PGUkvim/GGm501Q1aq/YSB05RAp48VtO6PHWVF4vKe+cPXzblJUDQYCTny3UYVYTaujwRxJJChK2UILsGyQg27g5agUfpPV8nj0E+mBmMWrLCloaGlOtmVs4HgYryFCGJMZttINUInLQp6JAABxdQOfu1lsQiVoBXkuMdJuerZhDn/Vc5BJOvFucU3/KAflnhtUReJXwGn7r5yXWWYb81Pmkkxbr9Ibc5biuYiUHq3rxrdRqDQHmfphJFsbQeAlL9FZtXNTmAdTHGEA4A5XbX0jN6FnUqafYMDW7NbhwVVhkJvmDUOjsv3wvRz1wqihn3qSQWj2UYQ2hGF4MZcrDdo3nqyso4g0tDaX7Bnb1/7rdsQjK1L3uYgUhBeey5zZh9tR5Ll5FZb56G7NDp5rnukksZLHADj3YBtg1GEu2WeULd5aBz7F4C4Mknr9JSuzJGd8DIj3rM0iOx/a0QX57RygcdsoIZiwR118cd0q1aOYPLebf7eARdEDjBgU9qDjuMCo32mJOCPT94yAG4uDNMW/tpufRBArQ0TqLa9lEG/yreWdxLa1bs2Xl4C74lvKsNJDoIwxa7tyrYNSUJ9J/FcZcWSWrmSYb+Lr0WaJlhgwjICJtJ/AOOfj9egYyw6L2n3PdQZctLhy3BrFzsy+etQZrkTfO+undab4lK3gKorIQH+bLQEVhpETv9qu/ImqzVgTCa3r7PAwnf5GNyiJ8bH7rdqJcJ3sNJKQ0V12wEfJnMtZU60ZukKNWVBicHy4iG1V1aWeiK7RdGVMtxbxASlcuazwJStrpuWDpuj/6b8mKioJdnjVYBM7dd5b0/Yn4EMNu12LcxPlf rDL59j4W WCPZqJT1WmcJ0/k1WnRx0uUlOaapTDawnDnMgujubEMlE4/4kcNtkTig/CYOVn0uHKKxmYXv8IZQveYIZc3bAo3yIgWlI0Lfa+ZrspWSGGKwPvRS0SFuo9HthJAGKfporMtfCdbMqC6p/fEHm7LPlkUHn7EyID+W4bnIOlGMCZQ1wVpLbrGHmCcLkq8ET1jOe1q1QeInz0NwvLOJhtwHEJIqE3Dm/xKn6g9le6YCfSErnqrFu0RdyNC4eKZPCoBTOS3FRcGLb+vSm8rPQtFRJCVP8TgEw61BZJnyQL1hhp4Rgg4Us1EkT2qaQI5ERgVnPnt97+ZDyZ+94Y8wOulJXLBXIgbuhBPn2j4jXJcIMTrQLynYk7vTkPtVBj5+t5fRBxb/G6iVFiVOA0Q84d2Yo0qaLWdqNP1R96g67R/e20CFTAIrUBvl96oq/QsKeZk/9VJ5p6QNZ/mdef2sc/L6R/Qm/Cledg/GXqcqxfJNvGPBuzSEkKd9QxUjP2x2oOb7amS4RnMmYZicnqekUdhbBPHbVA8es6Tu3+IdJFfes65f+HyqC43t+LElpaIN+FDj+WXDs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song All swap allocations are folio based now (except for hibernation), and swap cache is protected by cluster lock too. So insert the folio directly in to the swap cache upon allocation while holding the cluster to avoid problems caused by dropping and re-acquiring the lock. Signed-off-by: Kairui Song --- mm/swap.h | 8 ++-- mm/swap_state.c | 48 +++++++++++++++---- mm/swapfile.c | 122 ++++++++++++++++++++---------------------------- 3 files changed, 93 insertions(+), 85 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 2ae4624a0e48..b042609e6eb2 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -185,7 +185,10 @@ static inline struct address_space *swap_address_space(swp_entry_t entry) /* Below helpers requires the caller to pin the swap device. */ extern struct folio *swap_cache_get_folio(swp_entry_t entry); extern struct folio *swap_cache_add_folio(swp_entry_t entry, struct folio *folio, - void **shadow, bool swapin); + void **shadow); +extern void __swap_cache_add_folio(swp_entry_t entry, + struct swap_cluster_info *ci, + struct folio *folio); extern bool swap_cache_check_folio(swp_entry_t entry); extern void *swap_cache_get_shadow(swp_entry_t entry); /* Below helpers requires the caller to lock the swap cluster. */ @@ -368,8 +371,7 @@ static inline struct folio *swap_cache_get_folio(swp_entry_t entry) return NULL; } -static inline int swap_cache_add_folio(swp_entry_t end, struct folio *folio, - void **shadow, bool swapin) +static inline int swap_cache_add_folio(swp_entry_t end, struct folio *folio, void **shadow) { return -EINVAL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index ea6a1741db5c..9e7d40215958 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -110,12 +110,39 @@ int __swap_cache_replace_folio(struct swap_cluster_info *ci, swp_entry_t entry, return 0; } -/* - * Return the folio being added on success, or return the existing folio - * with conflicting index on failure. - */ +/* For swap allocator's initial allocation of entries to a folio */ +void __swap_cache_add_folio(swp_entry_t entry, struct swap_cluster_info *ci, + struct folio *folio) +{ + pgoff_t offset = swp_offset(entry), end; + unsigned long nr_pages = folio_nr_pages(folio); + + /* + * Allocator should always allocate aligned entries so folio based + * operations never crossed more than one cluster. + */ + VM_WARN_ON_ONCE_FOLIO(!IS_ALIGNED(offset, nr_pages), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_uptodate(folio), folio); + + end = offset + nr_pages; + do { + WARN_ON_ONCE(!swp_te_is_null(__swap_table_get(ci, offset))); + __swap_table_set_folio(ci, offset, folio); + } while (++offset < end); + + folio_ref_add(folio, nr_pages); + folio_set_swapcache(folio); + folio->swap = entry; + + node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages); + lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages); +} + +/* For swap in or perform IO for an allocated swap entry. */ struct folio *swap_cache_add_folio(swp_entry_t entry, struct folio *folio, - void **shadow, bool swapin) + void **shadow) { swp_te_t exist; pgoff_t end, start, offset; @@ -127,9 +154,10 @@ struct folio *swap_cache_add_folio(swp_entry_t entry, struct folio *folio, start = swp_offset(entry); end = start + nr_pages; - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(folio_test_swapcache(folio), folio); - VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!folio_test_swapbacked(folio), folio); + VM_WARN_ON_ONCE_FOLIO(!IS_ALIGNED(start, nr_pages), folio); again: offset = start; existing = NULL; @@ -141,7 +169,7 @@ struct folio *swap_cache_add_folio(swp_entry_t entry, struct folio *folio, existing = swp_te_folio(exist); goto out_failed; } - if (swapin && __swap_cache_set_entry(si, ci, offset)) + if (__swap_cache_set_entry(si, ci, offset)) goto out_failed; if (shadow && swp_te_is_shadow(exist)) *shadow = swp_te_shadow(exist); @@ -381,7 +409,7 @@ static struct folio *__swapin_cache_add_prepare(swp_entry_t entry, __folio_set_locked(folio); __folio_set_swapbacked(folio); - swapcache = swap_cache_add_folio(entry, folio, &shadow, true); + swapcache = swap_cache_add_folio(entry, folio, &shadow); if (swapcache != folio) { folio_unlock(folio); return swapcache; diff --git a/mm/swapfile.c b/mm/swapfile.c index 0f2a499ff2c9..91025ba98653 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -709,18 +709,17 @@ static bool cluster_scan_range(struct swap_info_struct *si, return true; } -static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster_info *ci, - unsigned int start, unsigned char usage, - unsigned int order) +static bool cluster_alloc_range(struct swap_info_struct *si, + struct swap_cluster_info *ci, + struct folio *folio, + unsigned int offset) { - unsigned int nr_pages = 1 << order; - unsigned long offset, end = start + nr_pages; - - lockdep_assert_held(&ci->lock); + unsigned int order = folio ? folio_order(folio) : 0; + swp_entry_t entry = swp_entry(si->type, offset); + unsigned long nr_pages = 1 << order; if (!(si->flags & SWP_WRITEOK)) return false; - /* * The first allocation in a cluster makes the * cluster exclusive to this order @@ -728,28 +727,33 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster if (cluster_is_empty(ci)) ci->order = order; - for (offset = start; offset < end; offset++) { - VM_WARN_ON_ONCE(swap_count(si->swap_map[offset])); - VM_WARN_ON_ONCE(!swp_te_is_null(__swap_table_get(ci, offset))); - si->swap_map[offset] = usage; - } swap_range_alloc(si, nr_pages); ci->count += nr_pages; + if (folio) { + /* from folio_alloc_swap */ + __swap_cache_add_folio(entry, ci, folio); + memset(&si->swap_map[offset], SWAP_HAS_CACHE, nr_pages); + } else { + /* from get_swap_page_of_type */ + VM_WARN_ON_ONCE(si->swap_map[offset] || swap_cache_check_folio(entry)); + si->swap_map[offset] = 1; + } + return true; } /* Try use a new cluster for current CPU and allocate from it. */ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci, - unsigned long offset, - unsigned int order, - unsigned char usage) + struct folio *folio, + unsigned long offset) { unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER); unsigned long end = min(start + SWAPFILE_CLUSTER, si->max); - unsigned int nr_pages = 1 << order; + unsigned int order = folio ? folio_order(folio) : 0; + unsigned long nr_pages = 1 << order; bool need_reclaim, ret; lockdep_assert_held(&ci->lock); @@ -777,7 +781,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, if (!ret) continue; } - if (!cluster_alloc_range(si, ci, offset, usage, order)) + if (!cluster_alloc_range(si, ci, folio, offset)) break; found = offset; offset += nr_pages; @@ -851,10 +855,11 @@ static void swap_reclaim_work(struct work_struct *work) * Try to allocate swap entries with specified order and try set a new * cluster for current CPU too. */ -static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int order, - unsigned char usage) +static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, + struct folio *folio) { struct swap_cluster_info *ci; + unsigned int order = folio ? folio_order(folio) : 0; unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; /* @@ -874,8 +879,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset = cluster_offset(si, ci); - found = alloc_swap_scan_cluster(si, ci, offset, - order, usage); + found = alloc_swap_scan_cluster(si, ci, folio, offset); } else { swap_unlock_cluster(ci); } @@ -886,8 +890,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o new_cluster: ci = isolate_lock_cluster(si, &si->free_clusters); if (ci) { - found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), - order, usage); + found = alloc_swap_scan_cluster(si, ci, folio, cluster_offset(si, ci)); if (found) goto done; } @@ -898,8 +901,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o if (order < PMD_ORDER) { while ((ci = isolate_lock_cluster(si, &si->nonfull_clusters[order]))) { - found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), - order, usage); + found = alloc_swap_scan_cluster(si, ci, folio, cluster_offset(si, ci)); if (found) goto done; } @@ -912,8 +914,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o */ ci = isolate_lock_cluster(si, &si->frag_clusters[order]); if (ci) { - found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), - order, usage); + found = alloc_swap_scan_cluster(si, ci, folio, cluster_offset(si, ci)); if (found) goto done; } @@ -937,15 +938,13 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o * allocation, but reclaim may drop si->lock and race with another user. */ while ((ci = isolate_lock_cluster(si, &si->frag_clusters[o]))) { - found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), - 0, usage); + found = alloc_swap_scan_cluster(si, ci, folio, cluster_offset(si, ci)); if (found) goto done; } while ((ci = isolate_lock_cluster(si, &si->nonfull_clusters[o]))) { - found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), - 0, usage); + found = alloc_swap_scan_cluster(si, ci, folio, cluster_offset(si, ci)); if (found) goto done; } @@ -1138,12 +1137,12 @@ static bool get_swap_device_info(struct swap_info_struct *si) * Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static bool swap_alloc_fast(swp_entry_t *entry, - int order) +static bool swap_alloc_fast(struct folio *folio) { + unsigned int order = folio_order(folio); struct swap_cluster_info *ci; struct swap_info_struct *si; - unsigned int offset, found = SWAP_ENTRY_INVALID; + unsigned int offset; /* * Once allocated, swap_info_struct will never be completely freed, @@ -1158,24 +1157,21 @@ static bool swap_alloc_fast(swp_entry_t *entry, if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset = cluster_offset(si, ci); - found = alloc_swap_scan_cluster(si, ci, offset, order, SWAP_HAS_CACHE); - if (found) - *entry = swp_entry(si->type, found); + alloc_swap_scan_cluster(si, ci, folio, offset); } else { swap_unlock_cluster(ci); } - put_swap_device(si); - return !!found; + return folio->swap.val != SWAP_ENTRY_INVALID; } /* Rotate the device and switch to a new cluster */ -static bool swap_alloc_slow(swp_entry_t *entry, - int order) +static void swap_alloc_slow(struct folio *folio) { int node; unsigned long offset; struct swap_info_struct *si, *next; + unsigned int order = folio_order(folio); node = numa_node_id(); spin_lock(&swap_avail_lock); @@ -1185,14 +1181,12 @@ static bool swap_alloc_slow(swp_entry_t *entry, plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]); spin_unlock(&swap_avail_lock); if (get_swap_device_info(si)) { - offset = cluster_alloc_swap_entry(si, order, SWAP_HAS_CACHE); + offset = cluster_alloc_swap_entry(si, folio); put_swap_device(si); - if (offset) { - *entry = swp_entry(si->type, offset); - return true; - } + if (offset) + return; if (order) - return false; + return; } spin_lock(&swap_avail_lock); @@ -1211,7 +1205,6 @@ static bool swap_alloc_slow(swp_entry_t *entry, goto start_over; } spin_unlock(&swap_avail_lock); - return false; } /* @@ -1278,10 +1271,6 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp) { unsigned int order = folio_order(folio); unsigned int size = 1 << order; - swp_entry_t entry = {}; - - VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); if (order) { /* @@ -1302,32 +1291,21 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp) } local_lock(&percpu_swap_cluster.lock); - if (!swap_alloc_fast(&entry, order)) - swap_alloc_slow(&entry, order); + if (!swap_alloc_fast(folio)) + swap_alloc_slow(folio); local_unlock(&percpu_swap_cluster.lock); /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */ - if (mem_cgroup_try_charge_swap(folio, entry)) - goto out_free; - - if (!entry.val) + if (mem_cgroup_try_charge_swap(folio, folio->swap)) { + folio_free_swap_cache(folio); return -ENOMEM; + } - if (WARN_ON(swap_cache_add_folio(entry, folio, NULL, false) != folio)) - goto out_free; - - /* - * Allocator should always allocate aligned entries so folio based - * operations never crossed more than one cluster. - */ - VM_WARN_ON_ONCE_FOLIO(!IS_ALIGNED(folio->swap.val, size), folio); + if (!folio->swap.val) + return -ENOMEM; atomic_long_sub(size, &nr_swap_pages); return 0; - -out_free: - put_swap_folio(folio, entry); - return -ENOMEM; } /* @@ -1858,7 +1836,7 @@ swp_entry_t get_swap_page_of_type(int type) /* This is called for allocating swap entry, not cache */ if (get_swap_device_info(si)) { if (si->flags & SWP_WRITEOK) { - offset = cluster_alloc_swap_entry(si, 0, 1); + offset = cluster_alloc_swap_entry(si, NULL); if (offset) { entry = swp_entry(si->type, offset); atomic_long_dec(&nr_swap_pages); -- 2.49.0