From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90957C3DA49 for ; Tue, 16 Jul 2024 22:46:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 129A96B0092; Tue, 16 Jul 2024 18:46:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D9C36B0095; Tue, 16 Jul 2024 18:46:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2FD6B0096; Tue, 16 Jul 2024 18:46:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CD6796B0092 for ; Tue, 16 Jul 2024 18:46:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4DF1C1A01BD for ; Tue, 16 Jul 2024 22:46:53 +0000 (UTC) X-FDA: 82347102306.19.C1C8683 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf27.hostedemail.com (Postfix) with ESMTP id A747E40027 for ; Tue, 16 Jul 2024 22:46:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DDL8I59d; spf=pass (imf27.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721169972; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hl3JIeWAeWzDApmCzYZXNT9/VNShRK1PzJmKHB2OO/M=; b=nz6tR3xOorhcSnwGL3sYRB/8KBQJNq8ezI1FkjK7G8WaI74ZydQbTO5T4hVXnqX/tWUQh6 M92ATtVRbvHnY2gBciN0dS5q+WM5odY3qR7VihyMcfMIkS9OOOiKz0TET4g9gAN9OlN61F /dAHRMRD9ou0HwMgeVu9lmXTvoJ5UV4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721169972; a=rsa-sha256; cv=none; b=kBiR++QenKgWJzIgVJ7E3K5O9SQkMbtMHZI/EpPj2RqKcA9Lf9hr6shnBfsfRC25G/n8mo 4IODVkKW7HbuDHq3T5pBx6iAM1k2d5TDFIQnC18W4vzW6Y6HQq5kWpKL/2b4xVDi6M6jui k9fQnqVhsmadtfw1/4Yl7px0De5oAos= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=DDL8I59d; spf=pass (imf27.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id B2FEACE13A5 for ; Tue, 16 Jul 2024 22:46:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00DB8C4AF0D for ; Tue, 16 Jul 2024 22:46:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721170006; bh=GA9SNgbQAkIoUb4fXSF3J9Gl1sMSuCtk8ODQx8kGpNg=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=DDL8I59dR/Y5Z8H4zGXdLQ8NVtkz/qfeF2FxrOvcQvGfuMgLoHUWjuFkBo4CWH5l5 mHSF9EoLkaF6AkVhbiGlJP5QjFqnPaKdYEBk8W1CZg5hCCOm+0g0jr9sIaCxVoC3A0 a1ORT5NZ7Uls+vr9w7iAqflYjSb75XJBBLpFdDHdYd2e5lX0qT4qArfMYxHkPiFaMq MBi/UIovFtt0tc8m1CwDb9iCoDtI020S8nQ5iiVG1ztxVcf/3vefvR8tghm7H0cktu /o+15MgLS2KbKzbADg4IJvJLHGn76KcTdEsQalUJxjElmnxGAyHHn+mXcNZ0qapjYW 74HSOiMCmxN/g== Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-e03a9f7c6a6so5679986276.3 for ; Tue, 16 Jul 2024 15:46:45 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCVuYqTZs/6Fdq5pcMaR1Ox+CkgKzbQjB54jOebA2kcZMG0uPt9ZY+v9NUEBh8GwUJIL1pZRQSLGvf/Xl1GL6UrjAss= X-Gm-Message-State: AOJu0Yzm546Jnpmp0PiwNPbgKOE/gf5zUj6VD8eS1Q+OvN1mEZnriqvj HMGf3A7wtLCfocVhM51anjl4WnHdG2pfDHQT+oHP8i+6HNRVkkqQUL4FGMbSXWr0MnATb2eVcCs f3l1WR+3FPol3TfUswzVmYI0MucRr9AmdnidBFQ== X-Google-Smtp-Source: AGHT+IGzlDUr83jT+R5TE01QYZtc+QLOCeCqPvcitOy/eeEH+W2VrT5O9Nvo9iItDmsJoO6h53iHxK80S7CFW/VqjB0= X-Received: by 2002:a05:6902:c0c:b0:e05:616c:5e7f with SMTP id 3f1490d57ef6-e05d56d2bd8mr5147694276.29.1721170005215; Tue, 16 Jul 2024 15:46:45 -0700 (PDT) MIME-Version: 1.0 References: <20240711-swap-allocator-v4-0-0295a4d4c7aa@kernel.org> <20240711-swap-allocator-v4-2-0295a4d4c7aa@kernel.org> In-Reply-To: From: Chris Li Date: Tue, 16 Jul 2024 15:46:34 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4 2/3] mm: swap: mTHP allocate swap entries from nonfull list To: Ryan Roberts Cc: Andrew Morton , Kairui Song , Hugh Dickins , "Huang, Ying" , Kalesh Singh , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Barry Song Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: xjocyuqkzpj4dwqankzuxrz33n8jwnhb X-Rspamd-Queue-Id: A747E40027 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1721170010-686339 X-HE-Meta: U2FsdGVkX1/KWC3Kn/qFxA5bMPGMvNEXqEN8klIsi4ECRq2g17Jh7l63G6USJlgZV+j8EAYUYUMH3Lyq+6h6KaxjEEpnSo9QHpGipfjapMpGdggvKTRYU30cpSWw5yP7Z4xsQOk6i0bu9uV9zGK3X6wZ5dbzOfdTjMxlnRmVcYl3AQMULRx/uzZ139hbyhCc2LfzlIlf+UIJED17DDbQAUdMT3FZ5TLndGLcvTgqlIbQN1cdbxchG/Jqr7VplQYnxmamFST/iFfYjT70PeyVEoJtS4o0WH4ucHZnyjnCcB/7t+E2LZZ3/Q48fVN64G/W94isAk2HpLU3aklee2oJFOwETNxu7fEykL4x6mJbS2r20a9GKjFQ3nJXYU+zaelmaTjVZR8OecX1gnbRczOuU/mWPkwBcYjYDSwKVEKm8Az9yK4+r4/wgzAL2rPDeyPW/uKQO28gF9YqpfsZjfFEtt4+tDxxQbxC7CS6RlfmP2v0oH+cK6OwKjlnXc5aXI9ZUdv2Ran024EBnhybB6XCoFhk3PPTWlSKqekX35HppjNPJFSzc6TxUmlvJDrv1GWAHmWguAaDBAN15nL1WeFuYTtaTuGXRIUGdToOxvA+GyfMMUbxw6MDTZzG3OsuHBHqVgHeOi4u1K1hZaoYAtiLYbDl9aCWIm26N3XyVjrd+wVOHp603TcIeA2ayKoFwWmLr8jd7XvkE79AAAAyRyO8UGHziJ7EVmj7eb/URjawYOyMfEFlapbGAEFQztLyj7x3veVJ1QsqAovbTk8rGnHQVPBXn75WhNBQRvZPGG0VvsLl2B+M0/JVoO0k8hawi8feRG+6XZqyO+c5xtgQNPxgJL3MxSSrx1lMdxS4I9V7bW2iRF9benQ2TbS3ArUtGbjKv3HgJgX/F+q6u+mwGHcsEuw/U9TspB3DsNJv7VWO0rx30R5yVwY+qysuJlJ6kMrtUj86OtbpTYZ5nkhr9wk c4v/BWDp jJ2t6SLMjo4RgS7YcphwGrUh4Dheash5Huz1vaf8XcBpubQiMZP9z1R0V+b08YhBK/fWWqQsoLDM7mcSQDlhoLS1FaNcYkAyO1CfIbNHDkfhunTaaaJawPNEFuNt0v5EeCXbMATKZxjjt1uF+9UjEidypmv+Yqk9HrPFIrx4qOqmDjtP6TW0rx5wnTAxtgXPs6jhx1KcsrFmEGqPXWPwsOVVjxOpm3Henu8iFwDVIHBqERdC7efpuMPh0wxRIfjsNEw0OhnddN/629MUbSI/Z6qa/iEhntc+M9ZtrNRzVS1nlt+3WzsmJvB777KiXPdlpdMi74fbs/4Eon/FnJn+eoQTOu3D9zA12zVg9kT+xnG+ErpKlRFYvaRlGYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 15, 2024 at 8:40=E2=80=AFAM Ryan Roberts = wrote: > > On 11/07/2024 08:29, Chris Li wrote: > > Track the nonfull cluster as well as the empty cluster > > on lists. Each order has one nonfull cluster list. > > > > The cluster will remember which order it was used during > > new cluster allocation. > > > > When the cluster has free entry, add to the nonfull[order] > > list. When the free cluster list is empty, also allocate > > from the nonempty list of that order. > > > > This improves the mTHP swap allocation success rate. > > > > There are limitations if the distribution of numbers of > > different orders of mTHP changes a lot. e.g. there are a lot > > of nonfull cluster assign to order A while later time there > > are a lot of order B allocation while very little allocation > > in order A. Currently the cluster used by order A will not > > reused by order B unless the cluster is 100% empty. > > > > Signed-off-by: Chris Li > > --- > > include/linux/swap.h | 4 ++++ > > mm/swapfile.c | 34 +++++++++++++++++++++++++++++++--- > > 2 files changed, 35 insertions(+), 3 deletions(-) > > > > diff --git a/include/linux/swap.h b/include/linux/swap.h > > index e9be95468fc7..db8d6000c116 100644 > > --- a/include/linux/swap.h > > +++ b/include/linux/swap.h > > @@ -254,9 +254,11 @@ struct swap_cluster_info { > > */ > > u16 count; > > u8 flags; > > + u8 order; > > struct list_head list; > > }; > > #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ > > +#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */ > > > > > > /* > > @@ -296,6 +298,8 @@ struct swap_info_struct { > > unsigned long *zeromap; /* vmalloc'ed bitmap to track zer= o pages */ > > struct swap_cluster_info *cluster_info; /* cluster info. Only for= SSD */ > > struct list_head free_clusters; /* free clusters list */ > > + struct list_head nonfull_clusters[SWAP_NR_ORDERS]; > > + /* list of cluster that contains = at least one free slot */ > > unsigned int lowest_bit; /* index of first free in swap_ma= p */ > > unsigned int highest_bit; /* index of last free in swap_map= */ > > unsigned int pages; /* total of usable pages of swap = */ > > diff --git a/mm/swapfile.c b/mm/swapfile.c > > index f70d25005d2c..e13a33664cfa 100644 > > --- a/mm/swapfile.c > > +++ b/mm/swapfile.c > > @@ -361,14 +361,21 @@ static void swap_cluster_schedule_discard(struct = swap_info_struct *si, > > memset(si->swap_map + idx * SWAPFILE_CLUSTER, > > SWAP_MAP_BAD, SWAPFILE_CLUSTER); > > > > - list_add_tail(&ci->list, &si->discard_clusters); > > + if (ci->flags) > > I'm not sure this is future proof; what happens if a flag is added in fut= ure > that does not indicate that the cluster is on a list. Perhaps explicitly = check > CLUSTER_FLAG_NONFULL? Or `if (!list_empty(&ci->list))`. Currently flags are only used to track which list it is on. BTW, this line has changed to check for explicite which list in patch 3 the big rewrite. I can move that line change to patch 2 if you want. > > > + list_move_tail(&ci->list, &si->discard_clusters); > > + else > > + list_add_tail(&ci->list, &si->discard_clusters); > > + ci->flags =3D 0; > > Bug: (I think?) the cluster ends up on the discard_clusters list and > swap_do_scheduled_discard() calls __free_cluster() which will then call swap_do_scheduled_discard() delete the entry from discard list. The flag does not track the discard list state. > list_add_tail() to put it on the free_clusters list. But since it is on t= he > discard_list at that point, shouldn't it call list_move_tail()? See above. Call list_move_tail() would be a mistake. > > > schedule_work(&si->discard_work); > > } > > > > static void __free_cluster(struct swap_info_struct *si, struct swap_cl= uster_info *ci) > > { > > + if (ci->flags & CLUSTER_FLAG_NONFULL) > > + list_move_tail(&ci->list, &si->free_clusters); > > + else > > + list_add_tail(&ci->list, &si->free_clusters); > > ci->flags =3D CLUSTER_FLAG_FREE; > > - list_add_tail(&ci->list, &si->free_clusters); > > } > > > > /* > > @@ -491,7 +498,12 @@ static void dec_cluster_info_page(struct swap_info= _struct *p, struct swap_cluste > > ci->count--; > > > > if (!ci->count) > > - free_cluster(p, ci); > > + return free_cluster(p, ci); > > nit: I'm not sure what the kernel style guide says about this, but I'm no= t a > huge fan of returning void. I'd find it clearer if you just turn the belo= w `if` > into an `else if`. I try to avoid 'else if' if possible. Changed to if (!ci->count) { free_cluster(p, ci); return; } > > > + > > + if (!(ci->flags & CLUSTER_FLAG_NONFULL)) { > > + list_add_tail(&ci->list, &p->nonfull_clusters[ci->order])= ; > > I find the transitions when you add and remove a cluster from the > nonfull_clusters list a bit strange (if I've understood correctly): It is= added > to the list whenever there is at least one free swap entry if not already= on the > list. But you take it off the list when assigning it as the current clust= er for > a cpu in scan_swap_map_try_ssd_cluster(). > > So you could have this situation: > > - cpuA allocs cluster from free list (exclusive to that cpu) > - cpuA allocs 1 swap entry from current cluster > - swap entry is freed; cluster added to nonfull_clusters > - cpuB "allocs" cluster from nonfull_clusters > > At this point both cpuA and cpuB share the same cluster as their current > cluster. So why not just put the cluster on the nonfull_clusters list at > allocation time (when removed from free_list) and only remove it from the The big rewrite on patch 3 does that, taking it off the free list and moving it into nonfull. I am only making the minimal change in this step so the big rewrite can lan= d. > nonfull_clusters list when it is completely full (or at least definitely = doesn't > have room for an `order` allocation)? Then you allow "stealing" always in= stead > of just sometimes. You would likely want to move the cluster to the end o= f the > nonfull list when selecting it in scan_swap_map_try_ssd_cluster() to redu= ce the > chances of multiple CPUs using the same cluster. For nonfull clusters it is less important to avoid multiple CPU sharing the cluster. Because the cluster already has previous swap entries allocated from the previous CPU. Those behaviors will be fine tuned after the patch 3 big rewrite. Try to make this patch simple. > Another potential optimization (which was in my hacked version IIRC) is t= o only > add/remove from nonfull list when `total - count` crosses the (1 << order= ) > boundary rather than when becoming completely full. You definitely won't = be able > to allocate order-2 if there are only 3 pages available, for example. That is in patch 3 as well. This patch is just doing the bare minimum to introduce the nonfull list. > > > + ci->flags |=3D CLUSTER_FLAG_NONFULL; > > + } > > } > > > > /* > > @@ -550,6 +562,18 @@ static bool scan_swap_map_try_ssd_cluster(struct s= wap_info_struct *si, > > if (tmp =3D=3D SWAP_NEXT_INVALID) { > > if (!list_empty(&si->free_clusters)) { > > ci =3D list_first_entry(&si->free_clusters, struc= t swap_cluster_info, list); > > + list_del(&ci->list); > > + spin_lock(&ci->lock); > > + ci->order =3D order; > > + ci->flags =3D 0; > > + spin_unlock(&ci->lock); > > + tmp =3D cluster_index(si, ci) * SWAPFILE_CLUSTER; > > + } else if (!list_empty(&si->nonfull_clusters[order])) { > > + ci =3D list_first_entry(&si->nonfull_clusters[ord= er], struct swap_cluster_info, list); > > + list_del(&ci->list); > > + spin_lock(&ci->lock); > > + ci->flags =3D 0; > > + spin_unlock(&ci->lock); > > tmp =3D cluster_index(si, ci) * SWAPFILE_CLUSTER; > > } else if (!list_empty(&si->discard_clusters)) { > > /* > > @@ -964,6 +988,7 @@ static void swap_free_cluster(struct swap_info_stru= ct *si, unsigned long idx) > > ci =3D lock_cluster(si, offset); > > memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); > > ci->count =3D 0; > > + ci->order =3D 0; > > ci->flags =3D 0; > > Wonder if it would be better to put this in __free_cluster()? Both flags and order were moved to __free_cluster() in patch 3 of this series. The order is best assigned together with flags when the cluster changes the list. Thanks for the review. The patch 3 big rewrite is the heavy lifting. Chris