From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EB0FC3ABAA for ; Fri, 2 May 2025 15:49:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 877D26B0092; Fri, 2 May 2025 11:49:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 826326B0098; Fri, 2 May 2025 11:49:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F3086B0099; Fri, 2 May 2025 11:49:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4CD366B0092 for ; Fri, 2 May 2025 11:49:54 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 427F8121DA3 for ; Fri, 2 May 2025 15:49:54 +0000 (UTC) X-FDA: 83398403508.26.C927D8F Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf04.hostedemail.com (Postfix) with ESMTP id 6D6CC40003 for ; Fri, 2 May 2025 15:49:52 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ttOezoYp; spf=pass (imf04.hostedemail.com: domain of sj@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746200992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AlAZayccQ5Dozn1K62cET4fxx8GpSDNQeFrGhJKkaa8=; b=vGa7PJn9+BM3izARJ7vzPgw4O8fBjQlNj4L3jqJPLUBuIv+0AoIhMtt0mwtPXR2gMjqunR fWxnQ/ZUL/kJKsKLsSVumJ9dCh6vp+unzwVbyFfygLCCzNmEmAqeTDD2elSXfF0NyTicZD wYz8eq4BvxDTheJROV4UxjjTaOPR/LI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ttOezoYp; spf=pass (imf04.hostedemail.com: domain of sj@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746200992; a=rsa-sha256; cv=none; b=hiL8aA7zf98UUltbHiFcCbS+eI8tEPj9zIlhyWQA3KlqLv8+lMlCH3Axa0zUlu8Lkn4k8x HcR3s7g5aUyRRyph6cR1hwi4DlE0hiYbih53P02MGgXiGz2IMBkvf2aDLjbQF1YN3wjI+w RdWbUbMM06EL8v5isO93LeXM7jll3OM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id B2CD1441D8; Fri, 2 May 2025 15:49:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D26B6C4CEE4; Fri, 2 May 2025 15:49:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1746200991; bh=nugAQl8X0Xn9dvegWvqV8CZq9+e3Xlp7ZHKgXTkmWsU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ttOezoYpok+M3jGb5KMaW4nGBcY6w9V1odxe2yjfyWX6lGSJqh6PHecDmhccDj8da mBu3w4TK+Idjia3I7rTiffLbPqHbw0NfrxIzxF6wYGVVOUGAPrJ18ZPknNF8nMBEN1 PEtCAESVT+27kXrGzK8f564beQlx22GSODwQqcln1Bs7AVYpakQUwGzUrGTmRwdq0P UAeoAmOeVzH3q2U3CTVRK3g4NWYz0W1CDqBWO+VKiI0OxlBtq/ycQzHqZ16ab4ral/ n+h8msiCMn71MjdzwUf4c3Dnu/7nF+LrTIgWUa+qE3xG1dDZS80BQoPQB1T7Q3Qk58 oyO2IawvzQIKA== From: SeongJae Park To: Yunjeong Mun Cc: SeongJae Park , Jonathan Corbet , damon@lists.linux.dev, kernel-team@meta.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , kernel_team@skhynix.com Subject: Re: [PATCH 0/7] mm/damon: auto-tune DAMOS for NUMA setups including tiered memory Date: Fri, 2 May 2025 08:49:49 -0700 Message-Id: <20250502154949.50992-1-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250502073854.1689-1-yunjeong.mun@sk.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: wmcbsu7zenoiindteecuwabr8aix4udt X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6D6CC40003 X-Rspam-User: X-HE-Tag: 1746200992-155327 X-HE-Meta: U2FsdGVkX1+bqMDekhpZpG5bQoaixaBf642KQzOkkJOSZL+fUyp9zFiay0tdALyFGk59G6wz1vsVNpYqvshqLGCp8pavzvXP0McMR3egiXrcud+dtDAKgl+0Bep9wk1xGy8drgV2ABO4lDRPheFo1R459BUEv59LBqTliFuU2ttMfnCTg5sl81l7MHmvpZ0ApeCOjYws0SAfDOAOAHshUthQWxzl4bRd1ZbMZrofQ7D2RkKGm0xJPZzEFwrOhob+jpIgiORWGPf9YeRFJ6hRXN+NSXRdWMN6025JUTdizq/dmWD/HRldAQsGJduVlUmUpWFFvlDbaysULveniWE0bz2ITb0/1CZcMRqMiHec4zif4oFVyWqpOsv35Mu6gDOpJsn+KDkCx/B+Mk44Pd4/pypFcDpwYGM430OI6QP+ApJoLlRbuj8a2z2KpMZzU3j36Yx/+1Ew3bTuc4SaiEelbduQDIjDuxdVhiGc/JvMJ0+OIoS9wENwVKJqxgHtbFQ5Xo4mFvB9qOPGgS8BTsaAZymLVR0oFewWA3O0ulpL/zgPWhuclJCJRCEyQcCoGrPlc7oLOkdVFlLkRaXbdFb4WPYeqCrk4KU6MhlEjjA9W6NKF/nQtVW4N/fQSUnKUyAgLXrr99MEoYcutQoP+twKHjT6pMy212C7UgTCB2cTRScVHcbA4m2VMg0cd0h9JqtqCahOni9Dd8IZquzFU1aCT2bYKcHF10V10bSdELogBYqARSSiG+aOyyCxjFyI+c0jOFUaePBXsX5z8rFyc/BydkhDqJI3N143mP4Gh3LFzLiRNYXEOJeD57EgGYK7O24YgGUSFSsOfBZsHEw2lQmRZFAxqTw7aDPHOHH1JPvnI0FJsvHMIKwR4Bbul1UBHErI1Hz977640Uj8kc85or1iT8bhx3qvsqrV11my1O/G25WxnfJEF6QWo/3TYCkur1xDKYwnF/DFDtSK+GsPXhr dZJOWlOa BoPDOhaY5pOQNx4WLthIPhJumUbQL4yBpoj2EUiLXmJxOF01XsAAOgUf+Iq3C+ajlOJ2kvS3LxVUHYqGD/6G1z1kEuUef1Sx8GvhK1XuV4hhwA156qHO29mGs63LiLAT54Hp+57YDUeQE09G5zLCB5KyYls3pE1rBgiSDKcdnPZG6KjkNJ7twulB1soVwjodaptBXG+fytPEAJbAAtN+CxJkYPvSYIrYloKJAfmaexZhOsLERMPtZ+IJheDP/yShD3FPkqp3NhoekDwwe//7mc+iha8gu4/qah7skfB9LnyfjzrycN4nsKD4lLx4eDjRvE99I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Yunjeong, On Fri, 2 May 2025 16:38:48 +0900 Yunjeong Mun wrote: > Hi SeongJae, thanks for your helpful auto-tuning patchset, which optimizes > the ease of used of DAMON on tiered memory systems. I have tested demotion > mechanism with a microbenchmark and would like to share the result. Thank you for sharing your test result! [...] > Hardware. > - Node 0: 512GB DRAM > - Node 1: 0GB (memoryless) > - Node 2: 96GB CXL memory > > Kernel > - RFC patchset on top of v6.14-rc7 > https://lore.kernel.org/damon/20250320053937.57734-1-sj@kernel.org/ > > Workload > - Microbenchmark creates hot and cold regions based on the specified parameters. > $ ./hot_cold 1g 100g > It repetitively performs memset on a 1GB hot region, but only performs memset > once on a 100GB cold region. > > DAMON setup > - My intention is to demote most of all regions of cold memory from node 0 to > node 2. So, damo start with below yaml configuration: > ... > # damo v2.7.2 from https://git.kernel.org/pub/scm/linux/kernel/git/sj/damo.git/ > schemes: > - action: migrate_cold > target_nid: 2 > ... > apply_interval_us: 0 > quotas: > time_ms: 0 s > sz_bytes: 0 GiB > reset_interval_ms: 6 s > goals: > - metric: node_mem_free_bp > target_value: 99% > nid: 0 > current_value: 1 > effective_sz_bytes: 0 B > ... Sharing DAMON parameters you used can be helpful, thank you! Can you further share full parameters? I'm especially interested in how the parameters for monitoring targets and migrate_cold scheme's target access pattern, and if there are other DAMON contexts or DAMOS schemes running together. > > Results > I've run the hot_cold benchmark for approximately 2 days, and have monitored > the memory usage of each node as follows: > > $ numastat -c -p hot_cold > Per-node process memory usage (in MBs) > PID Node 0 Node 1 Node 2 Node 3 Total > --------------- ------ ------ ------ ------ ------ > 2689746 (watch) 2 0 0 1 3 > 2690067 (hot_col 100122 0 3303 0 103426 > 3770656 (watch) 0 0 0 1 1 > 3770657 (sh) 2 0 0 0 2 > --------------- ------ ------ ------ ------ ------ > Total 100127 0 3303 1 103432 > > I expected that most of cold data from node 0 would be demoted to node 2, but it isn't. > In this situation, DAMON's variables are displayed as follows: > > [2067202.863431] totalram 131938449 free 84504526 used 47433923 numerator 84504526 > [2067202.863446] goal->current_value: 6404 > [2067202.863452] score: 6468 > [2067202.863455] quota->esz: 1844674407370955 > > `score` 6468 means the goal hasn't been achieved yet, and the `quota->esz`, > which specifies the aggressiveness of the demotion action, has reached > ULONG_MAX. However, the demotion has not occured. Yes, as you intrpret, seems the auto-tuning is working as designed, but migration is not successfully happened. I'm curious if migration is tried but failed. DAMOS stats[1] may let us know that. Can you check and share those? > > [..snip..] > > I think there may be some errors or misunderstanding in my experiment. > I would be grateful for any insights or feedback you might have regarding these > results. I don't have clear idea at the moment, sorry. It would be helpful if you could share things I asked above. Also, it seems you suspect the auto-tuning as one of root causes. I'm curious if you tried some different tests (e.g., same one without auto-tuning) and it gave you some theories. If so, could you please share those? [1] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#statistics Thanks, SJ [...]