From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F37D7C282DE for ; Thu, 13 Mar 2025 12:09:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ED74280005; Thu, 13 Mar 2025 08:09:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 280A3280009; Thu, 13 Mar 2025 08:09:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C33F928000C; Thu, 13 Mar 2025 08:09:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 36F5E280003 for ; Thu, 13 Mar 2025 08:09:34 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 469CBA9F4F for ; Thu, 13 Mar 2025 12:09:34 +0000 (UTC) X-FDA: 83216408268.03.0D9F6DF Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) by imf21.hostedemail.com (Postfix) with ESMTP id F26601C000A for ; Thu, 13 Mar 2025 12:09:29 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.56 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741867772; a=rsa-sha256; cv=none; b=L0t5LCVcXkEqV0bXXAwkmaZVpEPfga/hcGu/oPa9823YYyVeKUigAVIkoUHgpmGnS1KS+G UxDsqd4//nl44d18xd3j9kb+H8Gy3heBNHlcJFrfhZH06IXjAOU+P/7QVWnhgYMPd8MUJQ RuA87jy+H0jWR46ixFJH+g/f2EUuP7A= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.56 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741867772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2yaYCh4yJy9Hfpf2/7EP+3H2VeKj6magMhFLqWmoF0U=; b=2RVtz5YiU4k2fMnx98K4jhO+JtdpACMvNZ/83GyG+VaYjsMeP7CpUCYT8V63x5yZl9R91M g/uYfHhdf/Txwwil9232lcgnkn8+w/DRuGbFJ6z7tvR+fmHcp6BgygDIPSzvsb8Vv4lECr 4KBRwAT2YSbhMp9o1dr1pLX9MKUk7FI= Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4ZD5rb48sBz4f3jYR for ; Thu, 13 Mar 2025 20:09:03 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 065341A06D7 for ; Thu, 13 Mar 2025 20:09:26 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.101.6]) by APP2 (Coremail) with SMTP id Syh0CgCnsGPyytJnTaZeGQ--.9643S9; Thu, 13 Mar 2025 20:09:25 +0800 (CST) From: Kemeng Shi To: akpm@linux-foundation.org Cc: kasong@tencent.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 7/9] mm: swap: free each cluster individually in swap_entries_put_map_nr() Date: Fri, 14 Mar 2025 05:05:13 +0800 Message-Id: <20250313210515.9920-8-shikemeng@huaweicloud.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20250313210515.9920-1-shikemeng@huaweicloud.com> References: <20250313210515.9920-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID:Syh0CgCnsGPyytJnTaZeGQ--.9643S9 X-Coremail-Antispam: 1UD129KBjvJXoWxGry8CF4rtFy3uryUAw1rtFb_yoW5tFy8pF yagrn8tr4xZr17Jr4xAws8ArWru3y0gF1jqF9rGr1Sy3srAr18WFyvy3yYvFyUC34kur98 t3W7K347uFs8tr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUB2b4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUAVCq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAv FVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJw A2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE 3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr2 1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv 67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41l42xK82IYc2 Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s02 6x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r126r1DMIIYrxkI7VAKI48JMIIF0x vE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE 42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6x kF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxUIL05UUUUU X-CM-SenderInfo: 5vklyvpphqwq5kxd4v5lfo033gof0z/ X-Stat-Signature: tcsh9gic4gcgwe1caecqqefh3z7qo9wi X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F26601C000A X-Rspam-User: X-HE-Tag: 1741867769-303367 X-HE-Meta: U2FsdGVkX18LBB2g1xHWTNOBIgp/tS1/TCuTMofZnM0yj7uuITsoDzo2RoV0KF8pJyEMZeE5leeuMc171y74nNbU+7wUD7zuOke1jcJ0MY7ohrBr8YPDYiFzugmW+X9k7Q8zPbK1rU6cv1/6yypQ2sQE8IDzxrWzw6zCICzeyLF/unLSjYhN9VgpRpPs/4PNHxi0z4HyKW0Yjbnq4+nhsq+fERLla1Jl3hVIHSsreB1UlzQiTreE43reFyUOvEPyfLRcdD1gOnCGQGY1MEycFjSO/NRUS4LLADY7qppz6/Fp3ub04SwF4dvfrSfU600+WMvzmVf/UVOeAMJhYi3MV8xSBO979wftFoId9qpDTamPtLI53VizEb224Swx0JBdHUJomKbfqalS5xTivMhldy+xUYe04f20vvljtk7Mfu4hPin6uvQTUY1mkuBtXrhImim4lrxgv2uVefUF2cQ8vzYDH25WuKvgvaRgKVe4JdcAYdJCqIBbDjuy+b4PFDBiuIEaj39Qt6AOLnEIhCV65NiTbFLwzRkNk7VYY2keVgQtVss7nLpkbxmi6kXGtFaQu+2eKJyzrVh1pLJSskBiqVNFAq5PbaT8J11nftRSM5zKmRD+s162VqNls/J1UDIPa5P3dmkfcSWYWc/9ivwZQRQlSYo5ZwWGXbeGNavt5S3WrWtnR3WfM267Kyo9Zy/kSFusBgrHsQI4EubVTP0pmZTM297J3JF1nmgPkH5w84+bSkkHzvOl+vq9K9Tth/FIDlvEFOQ6jUO9crCb8PkIcGVXZ91lyYVHSedqJoGmFUZdSFT/usp0sKT7HH1+RtyNdtydMvX4rlA3Go8dvPXNlyvu9OCW8NzwnAGlOdh7PNTsk45r2Z62d0fmux4I6QnXkAB5XIy0zWc5QqCy61/5i7mtMiel75fMvjxGK5PjbK8h7k4cmgQ/BOoymVdAYAyxbIIcpDDdDDcy4OwHAhX BGHXcYqw P3t4GVWSoQNpHJAXaP1q8zySKcX5i0sIl+YbMA7+2ARCcmLzRqTWOhE/I46GhsA3BIi91y5dfcylt/YG2RVGo4ey9GYSESPylvSU6c5GQ6rUGy0Q39rXdEwy8Ppp+VSHOjTQ3OlXRzGsJNUv9fdiyGf2H8FC4OZ9Z9ENTNB6/lDSsgrQNNs3OzqRH0utgxFOkvQf2d6RpwzOdAUTMdBHi3E07Vw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1. Factor out general swap_entries_put_map() helper to drop entries belong to one cluster. If entries are last map, free entries in batch, otherwise put entries with cluster lock acquired and released only once. 2. Iterate and call swap_entries_put_map() for each cluster in swap_entries_put_nr() to leverage batch-remove for last map belong to one cluster and reduce lock acquire/release in fallback case. 3. As swap_entries_put_nr() won't handle SWAP_HSA_CACHE drop, rename it to swap_entries_put_map_nr(). Signed-off-by: Kemeng Shi --- mm/swapfile.c | 58 +++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 2d0f5d630211..ebac9ff74ba7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1473,25 +1473,10 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry) return NULL; } -static unsigned char swap_entry_put(struct swap_info_struct *si, - swp_entry_t entry) +static bool swap_entries_put_map(struct swap_info_struct *si, + swp_entry_t entry, int nr) { - struct swap_cluster_info *ci; unsigned long offset = swp_offset(entry); - unsigned char usage; - - ci = lock_cluster(si, offset); - usage = swap_entry_put_locked(si, ci, entry, 1); - unlock_cluster(ci); - - return usage; -} - -static bool swap_entries_put_nr(struct swap_info_struct *si, - swp_entry_t entry, int nr) -{ - unsigned long offset = swp_offset(entry); - unsigned int type = swp_type(entry); struct swap_cluster_info *ci; bool has_cache = false; unsigned char count; @@ -1502,9 +1487,6 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, count = swap_count(data_race(si->swap_map[offset])); if (count != 1 && count != SWAP_MAP_SHMEM) goto fallback; - /* cross into another cluster */ - if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER) - goto fallback; ci = lock_cluster(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { @@ -1521,15 +1503,33 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, return has_cache; fallback: - for (i = 0; i < nr; i++) { - if (data_race(si->swap_map[offset + i])) { - count = swap_entry_put(si, swp_entry(type, offset + i)); - if (count == SWAP_HAS_CACHE) - has_cache = true; - } else { - WARN_ON_ONCE(1); - } + ci = lock_cluster(si, offset); + for (i = 0; i < nr; i++, entry.val++) { + count = swap_entry_put_locked(si, ci, entry, 1); + if (count == SWAP_HAS_CACHE) + has_cache = true; } + unlock_cluster(ci); + return has_cache; + +} + +static bool swap_entries_put_map_nr(struct swap_info_struct *si, + swp_entry_t entry, int nr) +{ + int cluster_nr, cluster_rest; + unsigned long offset = swp_offset(entry); + bool has_cache = false; + + cluster_rest = SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER; + while (nr) { + cluster_nr = min(nr, cluster_rest); + has_cache |= swap_entries_put_map(si, entry, cluster_nr); + cluster_rest = SWAPFILE_CLUSTER; + nr -= cluster_nr; + entry.val += cluster_nr; + } + return has_cache; } @@ -1807,7 +1807,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr) /* * First free all entries in the range. */ - any_only_cache = swap_entries_put_nr(si, entry, nr); + any_only_cache = swap_entries_put_map_nr(si, entry, nr); /* * Short-circuit the below loop if none of the entries had their -- 2.30.0