From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C29DC54ED1 for ; Tue, 27 May 2025 18:50:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FCC46B009D; Tue, 27 May 2025 14:50:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AD3C6B00A0; Tue, 27 May 2025 14:50:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C2FD6B00A2; Tue, 27 May 2025 14:50:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2D7CF6B009D for ; Tue, 27 May 2025 14:50:26 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C9C121613F5 for ; Tue, 27 May 2025 18:50:25 +0000 (UTC) X-FDA: 83489578410.27.F4B8936 Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf03.hostedemail.com (Postfix) with ESMTP id 090D120011 for ; Tue, 27 May 2025 18:50:23 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="nh//koH/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf03.hostedemail.com: domain of sj@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748371824; a=rsa-sha256; cv=none; b=jpH7Dn4gBLVOjtFzF4qcs+Q9F96aSwANzv3FW1eppbGlRhr+p/RejTLLiZOBo4vD/APEv2 H4KinP4hzKsKybzgH49uhIE0o+8CXnTF/C3Z/gzVGes12LNWP+62vHxdanOcCzUUyGgcHG RFmbsdSpjo1C8TNLzy75NtBp2SQwVio= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="nh//koH/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf03.hostedemail.com: domain of sj@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748371824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uRf6i7oLjf6O7QdQad+scnbDk2HeNSYHr9fuE1Fipo8=; b=L6/HpwLtYL/xZ5e8NjmQ+DXwqdvJ7Auh97aX4qHk2OP3H5uNOOtZIpkA1tEj3VrLYeR+mU 79QsuPIvq3Dx9U7tnOb/YPjvcRDm8ux6ctJ2oFSda1MwSH401Zcczbh4T+zwYGC2BwTDcw Vkb8rRmK20ljliyg24cHXXN4y3S45Zc= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 176496135E; Tue, 27 May 2025 18:50:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92D47C4CEE9; Tue, 27 May 2025 18:50:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748371822; bh=6arYoFIhtasrIwdyo9c3OArXFmv3IflHhEXcgqbsST4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nh//koH/ba3SjdfBFtVrFMJwyd7JBy+pCyzJQ9VGsAIFjCAeP8Ezw5ZCw5dukdW8i yiqLOww0NWNom+jP0HdCk6Z3ulLcjU2RAItZsQShxg05tM7QxBowPlN88AiTAXzEVS 8ZE8Wd8Hnu3LnSOdk3DEVkYfE5gomZA5OQOuOAGexADIZPjeSVlu2neEkB1ekJb7d7 uEFRPoDefHgXAdXeTaoLpveKPAvj37K9JLlAMbDAjD7/TuLYJvlYxGwHo7v7vZcjte zlRH545AGA6GXUSrsapmyGb929Oikz5Bk9zarB385sqYnmVWGtMF8mMLrV1uFs9wsc eSOS8CdVaitEA== From: SeongJae Park To: Bharata B Rao Cc: SeongJae Park , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jonathan.Cameron@huawei.com, dave.hansen@intel.com, gourry@gourry.net, hannes@cmpxchg.org, mgorman@techsingularity.net, mingo@redhat.com, peterz@infradead.org, raghavendra.kt@amd.com, riel@surriel.com, rientjes@google.com, weixugc@google.com, willy@infradead.org, ying.huang@linux.alibaba.com, ziy@nvidia.com, dave@stgolabs.net, nifan.cxl@gmail.com, joshua.hahnjy@gmail.com, xuezhengchu@huawei.com, yiannis@zptcorp.com, akpm@linux-foundation.org, david@redhat.com Subject: Re: [RFC PATCH v0 0/2] Batch migration for NUMA balancing Date: Tue, 27 May 2025 11:50:19 -0700 Message-Id: <20250527185019.12457-1-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 090D120011 X-Stat-Signature: o4k9azhbpuihq3bk6fatsmjckojs46fy X-Rspam-User: X-HE-Tag: 1748371823-121482 X-HE-Meta: U2FsdGVkX1/gPpGR5P3+7Fzhg9y6dUjwxFSoG0HqTh9TplB2jx2zg6IVmamXMamUusjiPAnymIAJYjeseeDpoHjJCSBfOOiW/VIMmt/MtbtCNwbyuXSkUwjYbeWNAfmESXSTvWa0TEpmpshHTj+ygE0AENRhOKlKy04JRdzvyRAzYre+mjFM7a8wxiFNx+XZ00/Z1YU1Slv7aZI9ck379IwCK9e/E9A6p+irHYiRa36Km3l6PGGzEJjvqLGkK8Yysh1O31TcJ02wnYXPFUTelQOHOH6uOSxBb2iPJ7N/KPvp6EhG0oG5MVJcWRR5y+BQtOCvKK7f4CuActvhLDLgM9IBcVJN8E0z7qyjJSX3oBVyMgc1/0lqF08JYzNPWPz0kKRIuvbqN+K9RVjTGl4ga7ce3AxYFkzw8/JNABNpOauAWqHqpn5WX0EzSYUxYitBtkZxGYkPZ+d+UfZjI0bHxg3NkyECy9VzEFj34cAXC5y9wXImyt0yY/xgTeA/EU8swpN0uukKbJwxEMCxchcyDj92HVWt+HjOTaxMufOUCG/NGOacNYqPPahgDP9u5YTEsLb9f9UQHu9aQZEksTdoto+Widysh+rGFiuUu0JOfRcDhH2gzHOUUvuaQibSIm3JPu8Bxvg4Ae0qvrrN/gxh/wg+QlxNvA84daAnluPVGxu7E6dyU2DujGm62WeJGuOget8XhCp1aock+9kMLoYjLrE63gTPm8eq0cPZMe7FPL5xNc66ERsoKTtXmETTf7CWkk1VUyA3PyeCTKVL2RwKPw9KMeJ7imcN6KZp1bzoOnb854cu5U8Bpx0+sMKpDQbRBg2RdXNgSh849I68Dh8ybzKlaSHkoKRtS5ZWu4LlfBHRs+UHDugsYLMDD0f3w8aU6yTPdRg51G2D5ReLP0g3w5IxwfdWBKElcnpi6aPxXQuU310GYzCB7wVnBPOLn4pyTz3DqemUYppBWEJH7f+ sYzr6dQ/ JI+oGm45rtwcvQyysrar1xvI1zxpjxRHgPxYw4s6l/lWKall/CheABwFHaoDuTDPwvsmqcYK7PTEKmCX8ye4esOPLBVpkYjViq+gJCOen5q/NapgA6oOKC0QR5Bng4HVrgkmykhnEMuXpI2gxL188syjvuuXF2lMLh6qo37Kz/kEnOtXh80MO7lrKbKr/y2BNfmgEvyj5TCVKuNzS1hHcILNq+T+dDKG3Sgff4Ktn3p4tsStZle6liQm/I3HT52R4H4jSJKfAScwVT4o3B3ZvmCqrpLpEjR3olQtu4Sas45SHfsDDdHHtgf0Bqw4RVujmEUlIg+3KYya12eA0irufI2f61dStroYwzHAP+Ci56x6Kk6EGRDRmKaP0gcaUBOrsVDFg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 26 May 2025 10:50:02 +0530 Bharata B Rao wrote: > Hi SJ, > > On 22-May-25 12:15 AM, SeongJae Park wrote: > > Hi Bharata, > > > > On Wed, 21 May 2025 13:32:36 +0530 Bharata B Rao wrote: > > > >> Hi, > >> > >> This is an attempt to convert the NUMA balancing to do batched > >> migration instead of migrating one folio at a time. The basic > >> idea is to collect (from hint fault handler) the folios to be > >> migrated in a list and batch-migrate them from task_work context. > >> More details about the specifics are present in patch 2/2. > >> > >> During LSFMM[1] and subsequent discussions in MM alignment calls[2], > >> it was suggested that separate migration threads to handle migration > >> or promotion request may be desirable. Existing NUMA balancing, hot > >> page promotion and other future promotion techniques could off-load > >> migration part to these threads. Or if we manage to have a single > >> source of hotness truth like kpromoted[3], then that too can hand > >> over migration requests to the migration threads. I am envisaging > >> that different hotness sources like kmmscand[4], MGLRU[5], IBS[6] > >> and CXL HMU would push hot page info to kpromoted, which would > >> then isolate and push the folios to be promoted to the migrator > >> thread. > > > > I think (or, hope) it would also be not very worthless or rude to mention other > > existing and ongoing works that have potentials to serve for similar purpose or > > collaborate in future, here. > > > > DAMON is designed for a sort of multi-source access information handling. In > > LSFMM, I proposed[1] damon_report_access() interface for making it easier to be > > extended for more types of access information. Currenlty damon_report_access() > > is under early development. I think this has a potential to serve something > > similar to your single source goal. > > > > Also in LSFMM, I proposed damos_add_folio() for a case that callers want to > > utilize DAMON worker thread (kdamond) as an asynchronous memory > > management operations execution thread while using its other features such as > > [auto-tuned] quotas. I think this has a potential to serve something similar > > to your migration threads. I haven't started damos_add_folio() development > > yet, though. > > > > I remember we discussed about DAMON on mailing list and in LSFMM a bit, on your > > session. IIRC, you were also looking for a time to see if there is a chance to > > use DAMON in some way. Due to the technical issue, we were unable to discuss > > on the two new proposals on my LSFMM session, and it has been a bit while since > > our last discussion. So if you don't mind, I'd like to ask if you have some > > opinions or comments about these. > > > > [1] https://lwn.net/Articles/1016525/ > > Since this patchset was just about making the migration batched and > async for NUMAB, I didn't mention DAMON as an alternative here. I was thinking a clarification like this could be useful for readers though, since you were mentioning the future work together. Thank you for clarifying. > > One of the concerns I always had about DAMON when it is considered as > replacement for existing hot page migration is its current inability to > gather and maintain hot page info at per-folio granularity. I think this is a very valid concern. But I don't think DAMON should be a _replacement_. Rather, I'm looking for a chance to make existing approaches help each other. For example, I recommend running DAMON-based memory tiering[1] together with the LRU-based demotion. I think there is no reason to discourage using it together with NUMAB-2 based promotion, if the folio granularity is a real issue. That is, still NUMAB-2 will do synchronous promotion, but DAMON will do it asynchronously, so the amount of synchronous promotions and its overhead will reduce. I didn't encourage using NUMB-2 based promotion together with DAMON-based memory tiering[1] not because I show a problem at such co-usage, but just because I found no clear benefit of that from my test setup. In theory, I think running those together makes sense. That said, we're also making efforts for overcoming the folio-granularity issue on DAMON side, too. We implemented page-level filters that motivated by SK hynix' test results, and developed monitoring intervals auto-tuning for overall monitoring results accuracy. We proposed damon_report_access() and damos_add_folios() as yet another opportunitis to better deal with the issue. I was curious about your opinion to damon_report_access() and damos_add_folios() for the reason. I understand that could be out of the scope of this patch series, though. > How much > that eventually matters to the workloads has to be really seen. Cannot agree more. Nonetheless, as mentioned abovely, my test setup[1] didn't show the problem. That said, I'm not really convinced with my test setup, and I don't think the test setup is good for verifying the problem. Hence I'm trying to make a better test setup for this. I'll share more of the new setup if I make some progress. I will also be more than happy to learn about other's test setup if they have a good one or suggestions. [1] https://lore.kernel.org/20250420194030.75838-1-sj@kernel.org Thanks, SJ [...]