From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37D82C83F09 for ; Wed, 9 Jul 2025 01:01:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCF508D0009; Tue, 8 Jul 2025 21:01:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7E4F8D0001; Tue, 8 Jul 2025 21:01:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6DD48D0009; Tue, 8 Jul 2025 21:01:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A32D88D0001 for ; Tue, 8 Jul 2025 21:01:12 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 719DF1289C8 for ; Wed, 9 Jul 2025 01:01:12 +0000 (UTC) X-FDA: 83642922384.02.64559B8 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf25.hostedemail.com (Postfix) with ESMTP id 8BC09A0002 for ; Wed, 9 Jul 2025 01:01:10 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SNiReCet; spf=pass (imf25.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752022870; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BwtNm4BtmVP3B29Sr0z6Dg9gSZdTDCfEDKjBCM7yKB4=; b=uhPROI5NH3MsBBMq0lr6QtTMl7TjpIFJavCCp9+0JK5UKxOFOw6sHeFULbfrr2yhbFIsCK RRghMl3LCB6JMBYUfD/dI7IwoCWcDyNufxJ4IYIBqa9pa6AtzkrfD/I9hKQSI/GfG8JjGh bkl8er+8sHThMCbWE2ZmXcn6FumbOaQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SNiReCet; spf=pass (imf25.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.170 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752022870; a=rsa-sha256; cv=none; b=3sOQfx+57zzCn/bnH2PHgu4OYglOh0E4GKSSE5BJwIr1TxSooOTa7eDhtogs3o8aPDCwVu s06gFnSpKfdOt7syDYVd0umHXLEHgLY6g35xqUpTRu56I7SAFJF4iC3jVs2ATvXpJd3v7a O5O35hSnXcnl4vDXnjeNC8AIIgIg4oU= Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-e81cf6103a6so4647880276.3 for ; Tue, 08 Jul 2025 18:01:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752022869; x=1752627669; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BwtNm4BtmVP3B29Sr0z6Dg9gSZdTDCfEDKjBCM7yKB4=; b=SNiReCetXMRyuUa82xwmt1uC1CzzwUakkH/SG/6x1NHXgRpx2t3MPLHNdOEouK6zGw Zf6BZjwq+UJHkI81K3ovANjc800ZQUgcuEIiDrJzeRNtqYxnLPjx0PNiOQxwy4Mnub6V 2cZ15E3UKFnqlAezZtkNEChUJRoLiEMKvmCPhsvGockiKAN5l7MEoaBduDuEbH9MVbuX 62sbVCVRe30k4rsnAms9yCKlVub7wywMYDjvcCQ1YYwL+rxTpkoTm0uNYtJUNdiYl40y 9NvjZa6LAvNWAfPRTr9dA9g0oG92AbnimgHbtU/FdzkZfp7QMJGJPK9N+cWP3/ym9Bhr 9sRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752022869; x=1752627669; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BwtNm4BtmVP3B29Sr0z6Dg9gSZdTDCfEDKjBCM7yKB4=; b=BgpxDaxH3zxIF3VZbewMA9McDXA8+A849apJ43v7tDIup5esZ8vRU72EgyOVNURFz8 GMu7iRBEHJ8/qOv5LtzRZkb6Bg9mFMlTn7EMbzjLGUIx8KN/k+IUfLkxPZqLBL4XTrUJ mzzYXYlTTJlk7cRNzjK7oFdR8ThqAJO71sXECK9DV1JmV7/h73lFYLK7sdN/RIukOIxa 1ba+i4MnBsDyvSg4KMOTQ9aj8Ktn76WWUKrx9rSiBNF7+xXwXBhfLQW4Y4gTck5tdMIG WxYnxdXR85i9LprVXbfscjjil0wTKQWDgwhN2ShWl9w+SHH++5W73VqrPAyPmu/FF7PA JXnw== X-Forwarded-Encrypted: i=1; AJvYcCWKl0PU8YTUxoT2GYcqHaP3LAH8u2SvIYaoNUO8HfCvnfQP24jaXoEdiksgXn6h6KhCIGt18acCvw==@kvack.org X-Gm-Message-State: AOJu0YzSjKflDqLMYO4ZUob5dPWFG5Vway7j+IQgvJUULQMcXHszYgos HHiyVRaeHD78IgmsRQB9IlbqcwM50sFnPMBjZ8N8m86RrbSBFlyi+BhM X-Gm-Gg: ASbGnctibVjCvB8YKMTL6zO3GTQrAmyo2F4o1lpTHIMp2QO84MrvLQ2X8A5gPe1eppf 1gj8l4lfZUy3KmRmufxSwIsGwpnhzw3870F7n8t4ZRq73TVYqO8ZVXrDErwcHqdABmSnROYiusR Gey07/t1YUeICZyNWZozCUpfkVcd4cEUFJ0MAQ9UqfFrx5kFtfFgCMevQuYsBWrjsSZKEkRmMmT 4CtGFKSDjf2yornnL50/wqHJFkZZ9+GdDRowtu+Vv+OtlPEZkPjSkkLQOoNI1oNXk/fuO7zYW8O aDDy/WSFNcnac3u4Lg7zI+kenMT44XqwAz7H0f94CbVR4SGPO/Xs7+paf1WA1ngKciBbBCV4xCT YDf2RKaA= X-Google-Smtp-Source: AGHT+IEyRvCqbPeHTafmEDhZZQo8dDhtfIjvEfEuUPSnxmp83I7UH79ZdtpmrabA8PjyfakEBMidxg== X-Received: by 2002:a05:6902:2309:b0:e8b:53ba:535 with SMTP id 3f1490d57ef6-e8b6e0e2aeamr1046878276.15.1752022869325; Tue, 08 Jul 2025 18:01:09 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:235f:99bb:f36e:a060]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e899c43ed8asm3656043276.26.2025.07.08.18.01.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jul 2025 18:01:08 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, vtavarespetr@micron.com, ajayjoshi@micron.com, Ravi Shankar Jonnalagadda Subject: [PATCH v4 11/13] mm/damon/vaddr: Use damos->migrate_dests in migrate_{hot,cold} Date: Tue, 8 Jul 2025 19:59:41 -0500 Message-ID: <20250709005952.17776-12-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250709005952.17776-1-bijan311@gmail.com> References: <20250709005952.17776-1-bijan311@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8BC09A0002 X-Stat-Signature: nxtgjjj5t85z6o98nsapbb8tchq8fy74 X-Rspam-User: X-HE-Tag: 1752022870-404392 X-HE-Meta: U2FsdGVkX1+MEnk4fzQTXQevMbVkADaawiaBFeVWiPu2gbq3uhyBFdQ+bci9nLwdcgRHNNwxS4RyvhPOTXUHaDztFfNULTOs+cMEdqVzvDedZDTS2vZmBoQ/b/JFyJ05YHySnddXUSKhN5vtb8dB0D8745z84oi0KPE4mmaGBx3R7Dc3BnhHN9VbPzenTTlRvmr1XpHq3b859xfJeEjTnUjD0AC7vZV4ZhhYXSw76xCGh3xzpuo417I16wJXjZn2HXVSnLKirLK9LowMO7tsLX1cWzSkakv7m6Ax39ZJ+4ZIaKKnpH1JDftF0g1ofztYyvvrKNugOo9bFxq0TRZtVOQBOTBT/+G4hSU9dlYCJE1X5s249J1jInUkkpKEt8gcNEldneD05N0qs4NmszotuMQp6B2xGt/XIJvJlJVEPpLe8u2q1mgwgnEqpGqaRP7911mKOSdzofBYqRYqkys10PzTGpcDwq7nRL9WurKvC9DKgNWPlg3sw00ExcN2k9DiGmovtNbtG0+UBRBN2p55AXsqJtidB9ZUTA3E6cN65veh6O2ymrLgtD39m0lbamQTTb5x9OE43/1tCB4YPHU6OCsOj5gFd2ZzWbRP44oXrZ73YGkDYpAe5UdmjTHQz2QgR9f4U8cpgQeXVLNuK3APUGVmx6Sau+oYE05KxhSbbuoAl3YUB/88OVTHncpFDkI1M0mIYpiTMk0RgUXAXLbXXFvxe+bHa4mlzW/Uux0MjcguOWcadnX0ru7eiXO7+8qLcnL0n45fP+fd/bpwIqlhejUCEMVJtKe/laO63Q16P6l3zcURr4epnB9buGLsm3VR3/XxkzvjyttHaHt0lcu6b/bdslm19bOdJJmxLmD9viPcjihMeYZgYYGcRKR/W59EjPdyHPjDitRPIiUgvvlQn1lt4s1LW+xTOyJb7YVKvDzvCWd2rJhFbrV7lIA3wFSdamMFT/Uj6H3879gMRgr FWO3og+D DeVy8AnsEZ6AQssuMGM/vnqcETPsRzeR3SbtrFXQMc5kOQYZ/csZDyjHdam/JE5ifeiLFqi65+swW6jc9noPDwC24/8HncyMcJip3HhPcHIl4PF1RR78XNSic4JLPP6hyM9UmM6Xrc/s6nkuBZeMlzJ14FRRjmHVwfhwli07v0GNgc0lE+El4RMNfxcwOTgG9US7ocERZ+NPsn8OPFZRUrtbJIjOr9EO+23PwHpSpzo6xhfJtuBok+V+5vh9Z+e2Fxx9iJIdpXERwoO3yTSUaT08lz5vV8hblwoPhMgWPFvHsKnxsBgqIzR61W+tUeHvtTGlYTkM8ytuTBf4DHMFpe8RTBMsuA2+DsU+HLIT1r4mk0nJfKofqqyiUwVKG4GNVYrjpmB0EmbyVEMDWVC7aiy8ICbS42VB0iIjEjsP9dkf5i2eKLogqJMXbnCca100KO9TCpEPvsEFCj6cARCsj7ZDehmrQaO55AeAhX1P80W0UqS6ZhnK5KCIje2BkAxt19OGmA0UCyPJM11K2ysqFhaDxLZsj8eHnIutGar8Val3BM4MV3SpiDlqXi2DuorXb+N4K X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Bijan Tabatabai damos->migrate_dests provides a list of nodes the migrate_{hot,cold} actions should migrate to, as well as the weights which specify the ratio pages should be migrated to each destination node. This patch interleaves pages in the migrate_{hot,cold} actions according to the information provided in damos->migrate_dests if it is used. The interleaving algorithm used is similar to the one used in weighted_interleave_nid(). If damos->migration_dests is not provided, the actions migrate pages to the node specified in damos->target_nid as before. Co-developed-by: Ravi Shankar Jonnalagadda Signed-off-by: Ravi Shankar Jonnalagadda Signed-off-by: Bijan Tabatabai --- mm/damon/vaddr.c | 108 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 92 insertions(+), 16 deletions(-) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index b244ac056416..47d5f33f89c8 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -611,11 +611,69 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx) return max_nr_accesses; } +struct damos_va_migrate_private { + struct list_head *migration_lists; + struct damos_migrate_dests *dests; +}; + +/* + * Place the given folio in the migration_list corresponding to where the folio + * should be migrated. + * + * The algorithm used here is similar to weighted_interleave_nid() + */ +static void damos_va_migrate_dests_add(struct folio *folio, + struct vm_area_struct *vma, unsigned long addr, + struct damos_migrate_dests *dests, + struct list_head *migration_lists) +{ + pgoff_t ilx; + int order; + unsigned int target; + unsigned int weight_total = 0; + int i; + + /* + * If dests is empty, there is only one migration list corresponding + * to s->target_nid. + */ + if (!dests->nr_dests) { + i = 0; + goto isolate; + } + + order = folio_order(folio); + ilx = vma->vm_pgoff >> order; + ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); + + for (i = 0; i < dests->nr_dests; i++) + weight_total += dests->weight_arr[i]; + + /* If the total weights are somehow 0, don't migrate at all */ + if (!weight_total) + return; + + target = ilx % weight_total; + for (i = 0; i < dests->nr_dests; i++) { + if (target < dests->weight_arr[i]) + break; + target -= dests->weight_arr[i]; + } + +isolate: + if (!folio_isolate_lru(folio)) + return; + + list_add(&folio->lru, &migration_lists[i]); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE static int damos_va_migrate_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long next, struct mm_walk *walk) { - struct list_head *migration_list = walk->private; + struct damos_va_migrate_private *priv = walk->private; + struct list_head *migration_lists = priv->migration_lists; + struct damos_migrate_dests *dests = priv->dests; struct folio *folio; spinlock_t *ptl; pmd_t pmde; @@ -633,12 +691,9 @@ static int damos_va_migrate_pmd_entry(pmd_t *pmd, unsigned long addr, if (!folio) goto unlock; - if (!folio_isolate_lru(folio)) - goto put_folio; - - list_add(&folio->lru, migration_list); + damos_va_migrate_dests_add(folio, walk->vma, addr, dests, + migration_lists); -put_folio: folio_put(folio); unlock: spin_unlock(ptl); @@ -651,7 +706,9 @@ static int damos_va_migrate_pmd_entry(pmd_t *pmd, unsigned long addr, static int damos_va_migrate_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - struct list_head *migration_list = walk->private; + struct damos_va_migrate_private *priv = walk->private; + struct list_head *migration_lists = priv->migration_lists; + struct damos_migrate_dests *dests = priv->dests; struct folio *folio; pte_t ptent; @@ -663,12 +720,9 @@ static int damos_va_migrate_pte_entry(pte_t *pte, unsigned long addr, if (!folio) return 0; - if (!folio_isolate_lru(folio)) - goto out; + damos_va_migrate_dests_add(folio, walk->vma, addr, dests, + migration_lists); - list_add(&folio->lru, migration_list); - -out: folio_put(folio); return 0; } @@ -721,26 +775,48 @@ static unsigned long damos_va_migrate(struct damon_target *target, unsigned long *sz_filter_passed) { LIST_HEAD(folio_list); + struct damos_va_migrate_private priv; struct mm_struct *mm; + int nr_dests; + int nid; + bool use_target_nid; unsigned long applied = 0; + struct damos_migrate_dests *dests = &s->migrate_dests; struct mm_walk_ops walk_ops = { .pmd_entry = damos_va_migrate_pmd_entry, .pte_entry = damos_va_migrate_pte_entry, .walk_lock = PGWALK_RDLOCK, }; + use_target_nid = dests->nr_dests == 0; + nr_dests = use_target_nid ? 1 : dests->nr_dests; + priv.dests = dests; + priv.migration_lists = kmalloc_array(nr_dests, + sizeof(*priv.migration_lists), GFP_KERNEL); + if (!priv.migration_lists) + return 0; + + for (int i = 0; i < nr_dests; i++) + INIT_LIST_HEAD(&priv.migration_lists[i]); + + mm = damon_get_mm(target); if (!mm) - return 0; + goto free_lists; mmap_read_lock(mm); - walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &folio_list); + walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv); mmap_read_unlock(mm); mmput(mm); - applied = damon_migrate_pages(&folio_list, s->target_nid); - cond_resched(); + for (int i = 0; i < nr_dests; i++) { + nid = use_target_nid ? s->target_nid : dests->node_id_arr[i]; + applied += damon_migrate_pages(&priv.migration_lists[i], nid); + cond_resched(); + } +free_lists: + kfree(priv.migration_lists); return applied * PAGE_SIZE; } -- 2.43.0