From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19CBDC7115B for ; Mon, 23 Jun 2025 14:17:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 935036B0099; Mon, 23 Jun 2025 10:17:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E7458D0005; Mon, 23 Jun 2025 10:17:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DDDB6B0099; Mon, 23 Jun 2025 10:17:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 673A76B0099 for ; Mon, 23 Jun 2025 10:17:09 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1DA6059844 for ; Mon, 23 Jun 2025 14:17:09 +0000 (UTC) X-FDA: 83586867378.05.F56154D Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by imf03.hostedemail.com (Postfix) with ESMTP id 2A21920016 for ; Mon, 23 Jun 2025 14:17:06 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nUTu8od4; spf=pass (imf03.hostedemail.com: domain of bijan311@gmail.com designates 209.85.218.53 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750688227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=70tNPYqgSSY8XqkTztC+jIspGWzhAYhdeWY6QzJUGrU=; b=ahKdTEL4NiJNpvd4gdtpin426PXEChlsg2jHpVmkFuyR16URrurKtcCNrwUhnAtpI+3gbz WE9XJ6wkJ0q9gaFk80u2lcP+fP4m8+4zs7Trtp03sVCsaEduhBqBehCXLQgKQAQZSxM2JC fu61TX9Y5J0wTwqza4KuDUGCn+QO/5g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750688227; a=rsa-sha256; cv=none; b=EXCCWq0euvKZusCEQEGsN4y6xnz3DjbNQZfh7gE0bG5yUsssxgKOhhcX9Cdk3khMX5l2Yp ltdmKzEtFTHjuxmPyaEXisuei7NSULxQBNHdqX4cESQvY6L3mUvzKWjolIPRi0fRjLoN5b /8FnN49qZWrh+Y6Cev4Di55RNJvcvKw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nUTu8od4; spf=pass (imf03.hostedemail.com: domain of bijan311@gmail.com designates 209.85.218.53 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-ad93ff9f714so744320666b.2 for ; Mon, 23 Jun 2025 07:17:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750688225; x=1751293025; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=70tNPYqgSSY8XqkTztC+jIspGWzhAYhdeWY6QzJUGrU=; b=nUTu8od4Cfi3X1dxoxypQOrQ9C0KXVB/MyI0Tm63TbXXCtTk6E3khuielzU1PJTzLF QBgg2JIpx/AIFEb09a80mOJD7qrYVoaHiweWDWt0Cgw8BFAzqd2wu1QLN+9B5vuRVErf OpoBBOgt+oVQ29clH6tlYE//3tEenBzSdx0BVyTJlqqBOeu80yctiiIrkYVwX3No5Aym GlKT3WgoFBya4ILgzcINzscWflp2ccKRnzwulaKV/zat8OoY162mZRxQGjOmqmuSVT5J 12Yq6NLprX2/WXWTGU2MDVAe2Q0bd4Ag1bwaK+GnraZMX3BRhstU90Q/AyODGxjecKyo Lwdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750688225; x=1751293025; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=70tNPYqgSSY8XqkTztC+jIspGWzhAYhdeWY6QzJUGrU=; b=EkBen4GyXMVP+Askuci6D0JtZcCb3zZnv6HfLmAjsMcSGTXSWIlNOp28DE9ZB4n/2n aw9qyRectjGzeg/sO1WwlxczWY8qr5ihA/J5NWu726QJ1XzUtpNbmv24WfGrv7vm2tfJ HrOADsOnd3GgJS1ZqfCN6wqaFouAvxxn2/WNjqI3p/hqoeOpxr8Dyxv3uX/J3CZqj495 lTjzheGJMV90einoiVw/siO3sjPsCyWnxCRPfdK3V7q7X8YtZ0H9L1mEhzQy5dWb+zk0 e0T7OaSuq+KPlqd6sba7lOTqMt6i3DD88X9YXzE1DSWI36KcAY1Isgyq1fubjbSi0MsQ +npQ== X-Forwarded-Encrypted: i=1; AJvYcCVSs8SRkXRObOcWLK6vnFCwgHce7j/i2D4DIobidYOw/YlW4Z/nF5W63hReuuc7nqx8P7z7j9uWlw==@kvack.org X-Gm-Message-State: AOJu0YziUq5+qOyJ26L+HUp9Co8jKRXSNoCifNnpde1DDHeB/V8my18t C2llFnB6+1t1rT8DgNhhsLNiBKeUjI2H42Ndo4W4fUoBGolJcwTiXj8zE4IPGRcOEaeKbNPLaJC 4JudXJs+iFr+vh3YfLz2FDom+mcQqeIY= X-Gm-Gg: ASbGncu4AnRtqFPyNB9eNnXvCGL3ZXLbNVauSAG+9sKqXBzTDn0jeXxB+jHJrnCTDmp uq1ev+2En8eOZfJYX8ZcbKonObFbeajL/jX/EggbMs69CtMu0S2cq1uvp1wp+zVFrj5Xve2tdqT dBmLxXSnrREw09morGHobynnHzW6Z2seFJoca1fUWnplbpVT//kwDoK/9HASlyzd/6NysoXqHQc mA/O7GJkeUWeB0u X-Google-Smtp-Source: AGHT+IGVXdpuDSD3u1Rcg2xH/siOX+NFS+fNsYQ1G8iqkKsJn6/Ib7Lu5EOX1WPfO/C6zAEddYxGbn3tllx6tvD1qEs= X-Received: by 2002:a17:907:7e95:b0:ad8:9257:5727 with SMTP id a640c23a62f3a-ae057bd5095mr1094933066b.51.1750688225183; Mon, 23 Jun 2025 07:17:05 -0700 (PDT) MIME-Version: 1.0 References: <20250620180458.5041-3-bijan311@gmail.com> <20250621180215.36243-1-sj@kernel.org> In-Reply-To: <20250621180215.36243-1-sj@kernel.org> From: Bijan Tabatabai Date: Mon, 23 Jun 2025 09:16:53 -0500 X-Gm-Features: AX0GCFtADaqGHnbxuGoJ2rP0UQkjTI20aRfoQkXyZ3eOD4JaHNMRaVIa-DTKVX0 Message-ID: Subject: Re: [RFC PATCH v2 2/2] mm/damon/paddr: Allow multiple migrate targets To: SeongJae Park Cc: damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2A21920016 X-Stat-Signature: 147ouj7o9zbduhy3o8syhun3nozihyx6 X-Rspam-User: X-HE-Tag: 1750688226-830023 X-HE-Meta: U2FsdGVkX18vOHOD6e7aQDbb8Yh9ZnECCtDBAtdPnU76OUBvXQgi4hIE59PvaBgyJaKFvEAFQo1QyAnVk9RJmkulxFjCSRuiWMpKjxw+2KEe2b9CAa3SezAPppZXZseW2p1naAD2F9Y6H05JAVNqVZZYM2AVGoHKd8h1/cDeXId/177uBLsPK0/NYXCZgxON9anhItwe6iSWctAUQsYQck8DQkXMt3RhTzrIhnmWCKvwCQQMpEMUmW03qst2KAClWBNKqsRuGtElI4WXBUb37kFfjz6QJStQqUI+kcpkMqwqq2lnJ1qMYx1SZmkWeOX6jbmRDTelKELIEW0CCgVwy58gIIM+VOXQCjGeUTx1n2Dbm64955UA2Kyhkyf4IT1zaO7P80eAiFnxfbfz9VFJUGHWBJF0imktoC+cuqC9AoLunb02NR8U8LYPiQCmhluE8WX7na6O6Pmv4iEiixaw3fm5mFhHgcgqR4RGbB7DX4lS5m3ZRj/BEO87hRgqSCHwc2pmhUtNeXV8Tk58WArPWmxq62bzP4VN+b0du3OIVOAVa9uLPBQj3lK2eK/WSZyn1idxD96/syl3KiwiSv15GY+x2R8Cv5BL1kmIZJaXcP5vGhi0ZW3zlimURtDyWpoOYjVRZz3Iof/JzOpZmYe8gxkcWD6HY/GFWoFelwm6oosaT/brcM96Nw4ojPIuoe3Vtd2mZ1vcNZyx/l+X1+Zf1IKy2ox8QO4Je/zvpZ9PJBqpcUxmIgzxDP3O1rmLY9OSqE4If2Srru3hhkPlOBDbY+LvLor7jfrt1oqkL35WeSTDEAhDYYTAe8MrsAOJWGgTAtsWtrKn55Zag82JSRnaYOceRuX/MoZ3ZUUQpSmKsUw/UaTzOlYEWeE6Dn0R99r0/cZUBceo8jn1qsDcHHXJWPsvoB/IASMH/BZ4LChhUtS64ym5IWzf/PNJeURGYXaLqnFGbKP879f2ETc/f6O plgBGbNA RXw3ZdSfxyiKQ7Wdia3yjc0Bk0UI5xB6Wx8nttPNPvnrCw8k4t2Wjq4gQzu/AemiHow/wPSzaYgX4UShtDrygVi1vUOpdgRQG35bZdCAimuDRCpWG3vjwGI02LUgMQ2UFBQuPn7VCU4Ifk51QlHc/F0L7pl+ZmNZubNBVrIco0zgRKNR1OlMemg/KfVJyIuwEkLF9w4ldt7j1e2bsmHlK6V3x0EW1kCcDOiie0QXNf8hpYzwG8hneWdvdi1E/kx0kIVvsk3GiYjMbRFhnvrJiauEpSyUzYtcfVm1gppYw6UWo8XnGnJAkXpfJ/9eJhabwRFOHXtfoiyIQlxt3lSkdlIzk6mL9zSC0qQQPccdiQBxJIwGpjzE6rQVMrICMLF41QrjLuTZqTEQRTJfZqk+y1HHOnw+goEh0cft99Co/SkjQ5vQRsfo5BuH5kZqdXpuG1FHZF2f1IVtbNWo9iTjSSip6kBzrBuSzgKNNZwb+EM0wkcRvGVtIaiMrSABR3LpyzWvJ/ZvrPpkRUtoOw3A2vKDEoN/jlB/d3ooj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Jun 21, 2025 at 1:02=E2=80=AFPM SeongJae Park wrote= : > > Hi Bijan, > > On Fri, 20 Jun 2025 13:04:58 -0500 Bijan Tabatabai w= rote: > > > From: Bijan Tabatabai > > > > The migrate_{hot,cold} DAMONS actions take a parameter, target_nid, to > > indicate what node the actions should migrate pages to. In this patch, > > we allow passing in a list of migration targets into target_nid. When > > this is done, the mirgate_{hot, cold} actions will migrate pages betwee= n > > the specified nodes using the global interleave weights found at > > /sys/kernel/mm/mempolicy/weighted_interleave/node. This functionalit= y > > can be used to dynamically adjust how pages are interleaved in response > > to changes in bandwidth utilization to improve performance, as discusse= d > > in [1]. When only a single migration target is passed to target_nid, th= e > > migrate_{hot,cold} actions will act the same as before. > [...] > > include/linux/damon.h | 8 +-- > > mm/damon/core.c | 9 ++-- > > mm/damon/lru_sort.c | 2 +- > > mm/damon/paddr.c | 108 +++++++++++++++++++++++++++++++++++++-- > > mm/damon/reclaim.c | 2 +- > > mm/damon/sysfs-schemes.c | 14 +++-- > > samples/damon/mtier.c | 6 ++- > > samples/damon/prcl.c | 2 +- > > 8 files changed, 131 insertions(+), 20 deletions(-) > > If we keep pursuing making DAMON users be able to specify multiple migrat= ion > destination nodes and their weights[1], I think we may need only paddr.c = part > change of this patch in the final version of this great work. Sounds good to me. > [...] > > static unsigned long damon_pa_migrate(struct damon_region *r, struct d= amos *s, > > unsigned long *sz_filter_passed) > > { > > unsigned long addr, applied; > > - LIST_HEAD(folio_list); > > + struct rmap_walk_control rwc; > [...] > > > > addr =3D r->ar.start; > > while (addr < r->ar.end) { > > @@ -522,15 +599,38 @@ static unsigned long damon_pa_migrate(struct damo= n_region *r, struct damos *s, > > else > > *sz_filter_passed +=3D folio_size(folio); > > > > + /* > > + * If there is only one target node, migrate there. Other= wise, > > + * interleave across the nodes according to the global > > + * interleave weights > > + */ > > + if (nr_nodes =3D=3D 1) { > > + target_nid =3D first_node(s->target_nids); > > + } else { > > + target_nid =3D NUMA_NO_NODE; > > + /* Updates target_nid */ > > + rmap_walk(folio, &rwc); > > + } > > So we are doing rmap_walk(), which is known to be not very fast, for gett= ing > the target node id of this page, in a way very similar to that of weighte= d > interleaving, right? I don't think we really need to behave that same to > weighted interleaving with the cost. > > I'd hence suggest to implement and use a simple weights handling mechanis= m > here. It could be roud-robin way, like weighted interleaving, or probabi= listic > way, using damon_rand(). > > The round-robin way may be simpler in my opinion. For example, > > unsigned int damos_pa_nid_to_migrate(struct damos_migrate_dest *dest) > { > static unsigned int nr_migrated =3D 0; > unsigned int total_weight =3D 0; > unsigned int weights_to_ignore; > size_t i; > > for (i =3D 0; i < dest->nr_dests; i++) > total_weight +=3D dest->weight_arr[i]; > weights_to_ignore =3D nr_migrate++ % total_weight; > total_weight =3D 0; > for (i =3D 0; i < dest->nr_dests; i++) { > total_weight +=3D dest->weight_arr[i]; > if (total_weight >=3D weights_to_ignore) > return dest->node_id_arr[i]; > } > WARN_ON_ONCE(1, "I don't know what I did wrong"); > return 0; > } > > Then, we could replace the above rmap_walk() call with this one. What do= you > think? I do actually think doing the interleaving based on the VMA offset is important for a couple of reasons. 1. If also using the weighted interleaving mempolicy, and the DAMON weights are the same as the mempolicy weights, DAMON won't have to migrate newly allocated pages. This is relatively minor, but helps avoid unnecessary work. 2. More importantly, I believe this approach will cause a lot of needless ping-ponging, where the same folios are being moved around when they don't need to be. For example, let's say folios A-F are hot, and just for simplification, if they are on the same node, they will be in the same DAMON region, and only those folios are in those DAMON regions. If all the folios start in Node 0 and both nodes have a weight of 1, we have: nr_migrated =3D 0 Node 0 Node 1 ---------- ---------- A-F After the scheme is first applied nr_migrated =3D 6 Node 0 Node 1 ---------- ---------- A,C,E B,D,F This is fine, but these folios are still hot, so the scheme will be applied to them again nr_migrated =3D 12 Node 0 Node 1 ---------- ---------- A,E,D C,D,F If I am understanding your code sample correctly, this will continue to happen each time the scheme is applied, causing folios to be migrated for no reason. Using the VMA offset to determine where a page should be placed avoids this problem because it gives a folio a single node it can be in for a given set of interleave weights. This means that in steady state, no folios will be migrated. I see what you're saying about rmap_walks being expensive, but since DAMON operates off the critical path for the workload, I don't think the cost is that problematic. [...] Let me know what you think, Bijan