From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8669CC25B7D for ; Fri, 24 May 2024 17:17:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FE516B008A; Fri, 24 May 2024 13:17:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 160466B0093; Fri, 24 May 2024 13:17:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E06CF6B0092; Fri, 24 May 2024 13:17:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B3A696B008A for ; Fri, 24 May 2024 13:17:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 47575140C4A for ; Fri, 24 May 2024 17:17:37 +0000 (UTC) X-FDA: 82153946154.20.74BC85E Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf13.hostedemail.com (Postfix) with ESMTP id DFABF20023 for ; Fri, 24 May 2024 17:17:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="UX/P+vhE"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716571055; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q56V8OKpm37BAkNFTfdDCGwqAW0lZA95QPGRwWTxSlA=; b=vTdLyY7tTDEXCbesiF/WdPizgwtJdwVI0Ey6DWeEW9IrIS4WyAjNLHyJtAZLHP40BTLUq6 KFyf9t5SV/ZntjrMAcFMRufZr3mbsX3MB2as301PaJvNjtwWh5oFLFMOVTT1h4/R61pZpB XErcfBBdaa5bZX+XMcsKFAzYrUxTZvk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="UX/P+vhE"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716571055; a=rsa-sha256; cv=none; b=MaK3agWQ2gT+1QKR79EZOPB9lmHvHGNSsF7vkMv81/JDq3BLZQ17//zpUoSIOe363/0IYt U9kWYrSP6fj4Il+OYhXism4ol45ZazP4fsI1XikYZda/Nbej8YoSAT7sre0DcyqNZ+EmP9 Y4dUIcJd4vFnGAqFbBNV5EPxQArtzOg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id ED0F0CE19B3; Fri, 24 May 2024 17:17:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5A68C4AF08; Fri, 24 May 2024 17:17:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1716571051; bh=TDa3pqk4OBaY1GRmVEGGKzMmBUluuroNy+ytnClUbpo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=UX/P+vhEw9dIOFUzn2jQQTmq6HFKnwx6c0OjKQ63wRYEZeLU3Ri5wEKCzRoIbEM7q dQnyk5EqxLW3Cd7F/X0ksDLWKKL+Qiob2T13x/XAMNa1YH9EY1z43EZhtaZzlI5W2Q jKlG8D+NPX+lhGb33tNdV8rMx+mGKVBSGgBtEW+HKb/SQfBLFr6sWKYYeQdGAK2g3T jTPsz3gQ5xDWrQVNnHQDcPOrGwU6GjkstNSeRGoCh7uF0Ag8W+EjGkX8uBdfn/Xky4 6QeggWr4n83ARyedUCY5a3yWcXWXwLB1Yb8PVBX7SXyRHS7p9gUq0TvCUA4GMV3Cqj jvov4OikiN/kw== From: Chris Li Date: Fri, 24 May 2024 10:17:19 -0700 Subject: [PATCH 2/2] mm: swap: mTHP allocate swap entries from nonfull list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20240524-swap-allocator-v1-2-47861b423b26@kernel.org> References: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> In-Reply-To: <20240524-swap-allocator-v1-0-47861b423b26@kernel.org> To: Andrew Morton Cc: Kairui Song , Ryan Roberts , "Huang, Ying" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Barry Song X-Mailer: b4 0.12.4 X-Rspamd-Queue-Id: DFABF20023 X-Stat-Signature: o33mqctbc55jfx6zoxzuzxb8y61z5ynf X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1716571054-376642 X-HE-Meta: U2FsdGVkX1+bGnjSf1D9ksjRP45HWmjdugnRWWqHoIAJBBYAWohQhVlOaWKR7W/9cXbN38HNQBEDKzdSKbx9F0Stz2/ORU/f4Ytq2AMMcqQwNXtQEYODvhh21Hya9xwykZwyCVpFhQ0t8higzcYMW0KsC2YrybZ7C8DIUyB8rmpYRFHOPc6duJISoXTyE60S3BeR9f16Ua2elmbDpK/rARFuH6Dqz0XQHe9lorDOh18keGmfs2ObwrhHuSJPwFzxnYA/bFA4qd5krbxKRY16igyTD0BT1lEgV9qN+q5KQnqt58/Y0jZEsxCMs7FuQvLdsQnLiJUe9AqNoeXnYMUmEksj+R1UuM6aC5smd19V+je5Ji0gz1fp7aCOEFnmQFWCBuyHgwChphvRP8i0L1Q74CjsJNE/tVE8f2RTPlLzX9OTOeC2Crf7x6YhZ1QMGqaO78hU3+ipeopjr0mijSn8EMwawlTtzyDDnU8ex9Tx6pvUJVSu3dqXrt5Pv3i11f8m6KZjKlDQubmLsI7cEk0QYzafGGrs+fCk0JKLmi0xnZtFSsZV+sSXkA4XxYXucXj34ttETRy1tXwo0Uz4fzFG38iq9TG520NzKdDRmJsEYCjP6y0NesGhyYoqP7GZ9E4+9GsZh7MIE3qVW8enzQV7OWbBBEUy8lJmzsAMfKQ4E1YeeO/GdTZJwrHUjyR0+eRblhmDB/CTcHr8VSeVDmpAU2eygGUsp1gcfoiaCEMnnJYoC/U0rbZ2P1DTE8BSWEIhYt0T1eTnJrmX96l1Zzpm+ABho67kksQ+QDDwwsqXh8YH2k1HnyB87XHpsK0IQCMklUQuTa3+f1VXGwXyHu3HV4XmmsCNCZb0NRdC0PjX2q7C1NDCMTgHw2gfv4gZaQ+BoJ/dryipU5WjvS/QtPjW/au2KnwI/HOhWz8SMk6cm9QNVs66hRgEEBwaL9oydggg0W/82FsZAuVUuhSJsAC fJvv/bPu jX4ywGZdBfYL3jQZkJx/D5aAU2I0ksPn2RmtmM6VjHrddgOhti3PKj+4Xa615u7purDYTCVmr8+743clmwMb/mu33salS330GhFtcBO8xDZCX4dmPcSiIbYgHTqRDA4aiGKmQ/bPtBy2Of5xEXGG7K2Es79IkiksoblYaGMBVYiV0EunsK43DQ4My0pVBrK1eLzm3JRDe50eSDPiFzFTBgdoXOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Track the nonfull cluster as well as the empty cluster on lists. Each order has one nonfull cluster list. The cluster will remember which order it was used during new cluster allocation. When the cluster has free entry, add to the nonfull[order] list.  When the free cluster list is empty, also allocate from the nonempty list of that order. This improves the mTHP swap allocation success rate. There are limitations if the distribution of numbers of different orders of mTHP changes a lot. e.g. there are a lot of nonfull cluster assign to order A while later time there are a lot of order B allocation while very little allocation in order A. Currently the cluster used by order A will not reused by order B unless the cluster is 100% empty. This situation is best addressed by the longer term "swap buddy allocator", in future patches. --- include/linux/swap.h | 4 ++++ mm/swapfile.c | 25 +++++++++++++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0d3906eff3c9..1b7f0794b9bf 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -255,10 +255,12 @@ struct swap_cluster_info { * cluster */ unsigned int count:16; + unsigned int order:8; unsigned int flags:8; struct list_head next; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ /* @@ -297,6 +299,8 @@ struct swap_info_struct { unsigned char *swap_map; /* vmalloc'ed array of usage counts */ struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ struct list_head free_clusters; /* free clusters list */ + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; + /* list of cluster that contains at least one free slot */ unsigned int lowest_bit; /* index of first free in swap_map */ unsigned int highest_bit; /* index of last free in swap_map */ unsigned int pages; /* total of usable pages of swap */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 205a60c5f9cb..51923aba500e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -363,8 +363,11 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci) { + if (ci->flags & CLUSTER_FLAG_NONFULL) + list_move_tail(&ci->next, &si->free_clusters); + else + list_add_tail(&ci->next, &si->free_clusters); ci->flags = CLUSTER_FLAG_FREE; - list_add_tail(&ci->next, &si->free_clusters); } /* @@ -486,7 +489,12 @@ static void dec_cluster_info_page(struct swap_info_struct *p, struct swap_cluste ci->count--; if (!ci->count) - free_cluster(p, ci); + return free_cluster(p, ci); + + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { + list_add_tail(&ci->next, &p->nonfull_clusters[ci->order]); + ci->flags |= CLUSTER_FLAG_NONFULL; + } } /* @@ -547,6 +555,14 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, ci = list_first_entry(&si->free_clusters, struct swap_cluster_info, next); list_del(&ci->next); spin_lock(&ci->lock); + ci->order = order; + ci->flags = 0; + spin_unlock(&ci->lock); + tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; + } else if (!list_empty(&si->nonfull_clusters[order])) { + ci = list_first_entry(&si->nonfull_clusters[order], struct swap_cluster_info, next); + list_del(&ci->next); + spin_lock(&ci->lock); ci->flags = 0; spin_unlock(&ci->lock); tmp = (ci - si->cluster_info) * SWAPFILE_CLUSTER; @@ -578,6 +594,7 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, break; tmp += nr_pages; } + WARN_ONCE(ci->order != order, "expecting order %d got %d", order, ci->order); unlock_cluster(ci); } if (tmp >= max) { @@ -956,6 +973,7 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) ci = lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); ci->count = 0; + ci->order = 0; ci->flags = 0; free_cluster(si, ci); unlock_cluster(ci); @@ -2882,6 +2900,9 @@ static int setup_swap_map_and_extents(struct swap_info_struct *p, INIT_LIST_HEAD(&p->free_clusters); INIT_LIST_HEAD(&p->discard_clusters); + for (i = 0; i < SWAP_NR_ORDERS; i++) + INIT_LIST_HEAD(&p->nonfull_clusters[i]); + for (i = 0; i < swap_header->info.nr_badpages; i++) { unsigned int page_nr = swap_header->info.badpages[i]; if (page_nr == 0 || page_nr > swap_header->info.last_page) -- 2.45.1.288.g0e0cd299f1-goog