From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8694C27C6E for ; Fri, 14 Jun 2024 23:49:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A64E76B012B; Fri, 14 Jun 2024 19:48:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DEBF6B012C; Fri, 14 Jun 2024 19:48:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CB446B012F; Fri, 14 Jun 2024 19:48:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5AA6B6B0120 for ; Fri, 14 Jun 2024 19:48:17 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 17DEB4097B for ; Fri, 14 Jun 2024 23:48:17 +0000 (UTC) X-FDA: 82231135434.05.9550C73 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf14.hostedemail.com (Postfix) with ESMTP id 5B0F610000A for ; Fri, 14 Jun 2024 23:48:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SmxxD1r2; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718408893; a=rsa-sha256; cv=none; b=WEYBpkhv4TkAuw0w8sm0W4KLRJcBBVEv8rFWNzgh5o9q9dCnP5MMKF1ZyGbf5XPAFgJUZN c9kbd6QNsniyOKob7DvaCS1uqCN+WJ8p4XHzVorkpTr7/o/aOqnXhgdVrrJphyStMHqgig 10/8oq5QzHPPsEHGussn+FkjNX7HTSg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SmxxD1r2; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718408893; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qLs1R+g3oZxYK7WRn/C8wrxvcjoD3mvE15Mj00GDyhw=; b=8MTAiziMjORz2GHZXQZowBGfQT61qRlXnaHk80gqYeSJWQE45V9SjHsD+G1pQiI1uQEEvm R2dBaA/lfboweh2zvtCfNmvSWWuMqFG9MXrWrAYQ1+cwcvKNVmq79Q6loX6CjciW2Od0EH qRF17QQc4L5Dwyt/RpYJ0kLEIZATy5c= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7B7DF62023; Fri, 14 Jun 2024 23:48:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00C7BC4AF49; Fri, 14 Jun 2024 23:48:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718408894; bh=uzaDyQbVWdU/yPUf+0A3dOmYPMQFRQEdZUccMpk/Tp8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=SmxxD1r2JehMSaGbsPEe2zZ9wuu30o7zHZp26vWiDGo59UpYQeaZs+f/DDbA/Izu9 tw81hvy8NKtK8dsGW592yHXtzwDbe/126ckg/a4ke9xJMz/v9TU/INheZk/yr15n7+ S6cJk52inMqBK2wODihGxU5Nvsk49qvS3lVOiFVwbZe+6Ex0jECxJl/scpmIZKsU9A LtjqdWrLQrAQKYTqnkEDsTK7Bll3T8Dp6CGSh+iSrpMfL6H1Z6clqlsCKAzk9orubu PB27De6gBxw1tHjjb2AENdoLOozCDVqBFE7WdZA13Kkw/a6lV/spEHdfzyk0p4dlQZ DQuqJne/ub1Uw== From: Chris Li Date: Fri, 14 Jun 2024 16:48:08 -0700 Subject: [PATCH v2 2/2] mm: swap: mTHP allocate swap entries from nonfull list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240614-swap-allocator-v2-2-2a513b4a7f2f@kernel.org> References: <20240614-swap-allocator-v2-0-2a513b4a7f2f@kernel.org> In-Reply-To: <20240614-swap-allocator-v2-0-2a513b4a7f2f@kernel.org> To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , Kairui Song , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.13.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5B0F610000A X-Stat-Signature: 8uteduosudx4mu84xttbt1xs788dkqfp X-Rspam-User: X-HE-Tag: 1718408895-734952 X-HE-Meta: U2FsdGVkX1+AYCSgO5EWraqxiCi2tGenGuPCsRLKbK8v8hq6VWyQ4JFwwuhGyFEy8VjOupv5c1v6znBe9BSiswC9MLYTRkgGHAjYHd13P8eib1jX/ek3ZbwiKta3xY0cw2/SddM8s4rl91JFo3QDnG8BXh9fbj7aCEmBPWZt6WZzAYS6in2GclKXmf6aXu+Uhd6JdjbtyoIZMDn2c6Eeliqeho+/6/U/r1SuX1ws4fUioy8o1BfyM5LfWLXyFZjPbkRYgkKaj9nYkgpJvuk1y5pmLtMgGI0fuDbbO5Ge9D99y2PCsqNCbAhRLxDecJNg6mqHBjgV4+2SV6LWRVy6MS7M3tQUnabjYXFMyo6GqS44UcYU8g6aHQ4hlal2bQ57/N6zDNExuR9agTgxouJu5ZmLNNIVPQ/RmVrTk1994YIXo5tcgAMUr8gnmajXGNCZnH7sCGgEir1pYcQKEoHdWawrjOgp1okhCdcWtz2LgoJVzYNZRuGijcFGUpJ+dD5i0yURaWAM4TituAmJljCmBenp1PmLWffi+xr3bf9LydBQSpeflu41v+xTGGfuC1QyZLj9wXPr6CAfRjJQoGEQQEXh97CX9MeJ98Pg8egKVDalu7tVptjv6F78xo15T/5oT3NLNX+7bjvj4XYLvRunXjUuyw8UznaEdjnQ1yyelzdDHC1RrpsV6ffOYOEgz1N1iHBwgzCIe0310B/FXNiKodC+7RMKBFbNJHwK3C0A+DUuveLz6SHCCt1H7orrvjf7gitD3RQw2+sJ7vy3S/KVdFO157zrTn17Y1R8hdcknL2UKcNJC1MHQi6SIog8/0ypCiNN9rnOZlZUX31pW2DwX3o97v6Kc2SVOfOVfiiQXxATSMArp3GdnFIq8lvUllg+R6lLkF7yCVxl4IygbNd8imdP8ltSnWtcwQ1+Ad+3k/Q9pxYhOxyAVRbGC1sWnyanDX8U/PuzulG0YxjXyCi RqjGsshn SBYu3kblxpcfLD5DPm/ExvDEGDCwO6HhyFpDJUx73gfZw2FvuYdWkhdYiNKQXqzoxYVQ5PV5vxiOCGLZLbuiNtrDf8/FuJWzFQg55Xr9bdJRu1ISsuiq7ZrWU+kW8NRwQIj0sgCQDVbSzUx34zvJmll1PY/12EtXoRPXDHUhxHtZCcBBTu+pjQ5KNYEpVTU67pCf11PfiemRe3xlVdkJFmOhVmgRCwuaNyLHT9yU0aw+3ndKPqQ27XwRLDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list. Allocation use the nonfull cluster list before using the the free cluster list. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. Signed-off-by: Chris Li --- include/linux/swap.h | 9 +++++++-- mm/swapfile.c | 49 ++++++++++++++++++++++++++++++++++++------------- 2 files changed, 43 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index cd9154a3e934..fcb21f9883a5 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -245,18 +245,21 @@ enum { */ struct swap_cluster_info { spinlock_t lock; /* - * Protect swap_cluster_info count and state - * field and swap_info_struct->swap_map + * Protect swap_cluster_info bitfields + * and swap_info_struct->swap_map * elements correspond to the swap * cluster */ unsigned int count:12; unsigned int state:3; + unsigned int order:4; struct list_head list; /* Protected by swap_info_struct->lock */ }; #define CLUSTER_STATE_FREE 1 /* This cluster is free */ #define CLUSTER_STATE_PER_CPU 2 /* This cluster on per_cpu_cluster */ +#define CLUSTER_STATE_SCANNED 3 /* This cluster off per_cpu_cluster */ +#define CLUSTER_STATE_NONFULL 4 /* This cluster is on nonfull list */ /* @@ -295,6 +298,8 @@ struct swap_info_struct { unsigned char *swap_map; /* vmalloc'ed array of usage counts */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 2f878b374349..85a96178fd27 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -361,8 +361,12 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (ci->state == CLUSTER_STATE_NONFULL) + list_move_tail(&ci->list, &si->free_clusters); + else + list_add_tail(&ci->list, &si->free_clusters); ci->state = CLUSTER_STATE_FREE; - list_add_tail(&ci->list, &si->free_clusters); + ci->order = 0; } /* @@ -484,7 +488,12 @@ static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluste ci->count--; if (!ci->count) - free_cluster(p, ci); + return free_cluster(p, ci); + + if (ci->state == CLUSTER_STATE_SCANNED) { + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order]); + ci->state = CLUSTER_STATE_NONFULL; + } } /* @@ -535,17 +544,25 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, unsigned int nr_pages = 1 << order; struct percpu_cluster *cluster; struct swap_cluster_info *ci; - unsigned int tmp, max; + unsigned int tmp, max, found = 0; new_cluster: cluster = this_cpu_ptr(si->percpu_cluster); tmp = cluster->next[order]; if (tmp == SWAP_NEXT_INVALID) { - if (!list_empty(&si->free_clusters)) { + if (!list_empty(&si->nonfull_clusters[order])) { + ci = list_first_entry(&si->nonfull_clusters[order], struct swap_cluster_info, list); + list_del(&ci->list); + spin_lock(&ci->lock); + ci->state = CLUSTER_STATE_PER_CPU; + spin_unlock(&ci->lock); + tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->free_clusters)) { ci = list_first_entry(&si->free_clusters, struct swap_cluster_info, list); list_del(&ci->list); spin_lock(&ci->lock); ci->state = CLUSTER_STATE_PER_CPU; + ci->order = order; spin_unlock(&ci->lock); tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; } else if (!list_empty(&si->discard_clusters)) { @@ -570,21 +587,24 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, max = min_t(unsigned long, si->max, ALIGN(tmp + 1, SWAPFILE_CLUSTER)); if (tmp < max) { ci = lock_cluster(si, tmp); - while (tmp < max) { + while (!found && tmp < max) { if (swap_range_empty(si->swap_map, tmp, nr_pages)) - break; + found = tmp; tmp += nr_pages; } + if (tmp >= max) { + ci->state = CLUSTER_STATE_SCANNED; + cluster->next[order] = SWAP_NEXT_INVALID; + } else + cluster->next[order] = tmp; + WARN_ONCE(ci->order != order, "expecting order %d got %d", order, ci->order); unlock_cluster(ci); } - if (tmp >= max) { - cluster->next[order] = SWAP_NEXT_INVALID; + if (!found) goto new_cluster; - } - *offset = tmp; - *scan_base = tmp; - tmp += nr_pages; - cluster->next[order] = tmp < max ? tmp : SWAP_NEXT_INVALID; + + *offset = found; + *scan_base = found; return true; } @@ -2896,6 +2916,9 @@ static int setup_swap_map_and_extents(struct swap_info_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); + for (i = 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i = 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr = swap_header->info.badpages[i]; if (page_nr == 0 || page_nr > swap_header->info.last_page) -- 2.45.2.627.g7a2c4fd464-goog