From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39AEACD11C2 for ; Fri, 5 Apr 2024 16:56:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E1466B008C; Fri, 5 Apr 2024 12:56:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9911A6B0092; Fri, 5 Apr 2024 12:56:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80AB96B0093; Fri, 5 Apr 2024 12:56:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5CF3C6B008C for ; Fri, 5 Apr 2024 12:56:32 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1B22F40415 for ; Fri, 5 Apr 2024 16:56:32 +0000 (UTC) X-FDA: 81976081824.02.83F46B0 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2094.outbound.protection.outlook.com [40.107.94.94]) by imf21.hostedemail.com (Postfix) with ESMTP id 886FB1C0025 for ; Fri, 5 Apr 2024 16:56:29 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=ypdILxWl; dmarc=pass (policy=none) header.from=memverge.com; spf=pass (imf21.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.94.94 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712336189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zErZbMb3jTyXqs57OMGmAAQj4J/j08jQ09xuRyuPM8g=; b=ag+GCj7GXK9IxTspn3bJl+iRhJGb27y48y1OBHQ0wQzvTYlXdmN1W3ZbduvsCIYWxEbkGf v1MnJIlHirTuWPGqVdlf00xMM7panOX38sO0pBsEJU6POgKK88bxoFSmTPMHgkhOIH24yK yHvbGZamjuphuwejq8tvnvF2cO88oCQ= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=ypdILxWl; dmarc=pass (policy=none) header.from=memverge.com; spf=pass (imf21.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.94.94 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1712336189; a=rsa-sha256; cv=pass; b=wtXbMvQ41Xi73J+nszD7DBBIUVnh8qSvFgwiqGx44NaYbFgxfGtdNNcVv4mmNwnWWTfNZq QJqe9vnx4eIefDabb7NENGVAVhuE0lzvhCG1jaB8IOrY7vb6QRC7XBSgZ5/s6SUFNjJ4/M QQpjqlDRM2/G+MD94fw+p2MH8S3gtqg= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EmzNqexo9cvcyGSFCG+R3Wsqpr5EwBC/FMP49rJNLrbqZP010o1WHaMKDTZYUjGAeNE0v1eYTDhCvWb/U8+vrX0T3DAAxFSs5ZeIHMIqYbtxfPZtgV1Lwm4TYiaFC/xW2olI/aEWUyrTTb70UlEuxVkAiSXH/f2/6lwMf68abI+R7Rc8zjhUFLngsLl1aFfftvIF+som/DD0d18jqpzWpWdYLGcCvh17Bj0VnXmdwjyvuX0Di1WSNN9veEANUzRVC8H/VDLplYZODWiPGZQrcNUyYIaLlbHzck9ta6Ss+y+b49ZJ31ufHIE40YEebQAIVs+2BU36QcQ7zWuT4IuEvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zErZbMb3jTyXqs57OMGmAAQj4J/j08jQ09xuRyuPM8g=; b=PAjeSCUb4Al6M+Kajf1DUAvPkqhVKqBwLjHKMKnqh+pxCSDXmcerO6392tN1hSmS3Vvnoh/2cprONrknr0RDx8+M6mVkU0+RPYxn1CB4ndRVTryelOA5hXUX3HJpxn15IDhAkSxJnUtEE1Qmst8HIRExs3paG9HzwZW7KZ4wDy+cQcbD3ut5GFX2hBUwDPFhLJupyWLWzc46fLdx7CYWOwdCW6Ch0t7twuNTVFOssdXRXD4CpPau/q27OqjeyVrfEAwgyHb0Y10fibprOdMvyAHNp5lu94FzQmrLem7Z/W7jQ8Z2P2pFNzl54osftipdpcEzo0sNjrscfSCeVaWbqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zErZbMb3jTyXqs57OMGmAAQj4J/j08jQ09xuRyuPM8g=; b=ypdILxWl4DSuOCj8IxeP2ldUvJo40EZB3LgtNqFPc1zyNUoOeIuLNzNr6rmIM8Uy5Qc6iaWLtCHlgbdtHRVVtlPcHQ1Ncc0qM4YYjMIMD1Odp7kQRw9l2XvPLTwXZrxfCfxxPvpsnQMiXgmKMxxyoCG3Ta/oVrMC8Nnn/unLbZg= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by SJ0PR17MB4582.namprd17.prod.outlook.com (2603:10b6:a03:35b::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.53; Fri, 5 Apr 2024 16:56:24 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::6657:814f:5df0:bb5b]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::6657:814f:5df0:bb5b%5]) with mapi id 15.20.7409.049; Fri, 5 Apr 2024 16:56:23 +0000 Date: Fri, 5 Apr 2024 12:56:14 -0400 From: Gregory Price To: Honggyu Kim Cc: sj@kernel.org, damon@lists.linux.dev, linux-mm@kvack.org, akpm@linux-foundation.org, apopple@nvidia.com, baolin.wang@linux.alibaba.com, dave.jiang@intel.com, hyeongtak.ji@sk.com, kernel_team@skhynix.com, linmiaohe@huawei.com, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, mathieu.desnoyers@efficios.com, mhiramat@kernel.org, rakie.kim@sk.com, rostedt@goodmis.org, surenb@google.com, yangx.jy@fujitsu.com, ying.huang@intel.com, ziy@nvidia.com, 42.hyeyoo@gmail.com, art.jeongseob@gmail.com Subject: Re: [RFC PATCH v3 0/7] DAMON based tiered memory management for CXL memory Message-ID: References: <20240405060858.2818-1-honggyu.kim@sk.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240405060858.2818-1-honggyu.kim@sk.com> X-ClientProxiedBy: SJ0P220CA0002.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:41b::10) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|SJ0PR17MB4582:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W50tsk1jRsL3CJGg0tCzaelddYpwBSdiQgiN3Cr//a/NVnM8S71mUuhBh7qv+6vipPsqWjp/O1WSL2/wmzB+ks4+2lhkg/Jqe/X6J7ZUtuqq+E8mQRDt6dYZ8WbJ1k5nT9Z6AYoObf+du6XZLXGV+pTNjfNOe+LsBgQu81IhD9vKjEQ7kRZo37gVbiupaBKlUvvc+cGRdCb30h5B+3te1pOVC7o7PUXOzsLomBFfP+gpuc06VxEm9leaIjCtlcdoeP5vqcCbOavnPsD2P3mqXzfUWaS5DSULsaIpCt74QNX6YzqzRxaPcsvq7kqT4Q+IoHpmiFbi21qitUz5pIGPMNEG4tLmJXH0uCCHTaQv/870mrBJ6UhmGCcUDfziSMmSYuro/DCRilSnfzgMd109AkNd9wdxgpRgpg/ZiWXt+7uiU1QGZ+hjHZQqTtNBYi+NJajzCRqs6h1RfiE2fZm0xJ3geiuFdACZJg2nmqwXQjpaEiFTzx532yAYjNcZvN/BW795Zox1whuvRfig670i8YaQepbEJDfC83n7OfVxGypKcoDZbdSCd1kYwlRdgJp57DA+qbFGSm1Lt8/2Mj+1bknb3AySi0/Luu60/sQ8CDuYtl/oNw9A+plbBby1pHeQnrBNjjTXhbzV/njpR2rDH7DYXvFkDJgxNPE7ZunmsAE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(7416005)(1800799015)(366007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?u3uA89lnbl1XjLpDljLpSx+EFbKrPa8e92AEIJA2eg+zB1ny2L3/gV8Qg9/v?= =?us-ascii?Q?wcmqhc2yi8DaWC9fvgDNxd6PkaR35SRxXdRAXuhy7VIUvuHrlQpX8JbtLbQa?= =?us-ascii?Q?WHL3K5gPSp3fkLUOXcj9IqqtvqkTJn0kgHWwxkX6DujT1C9/bLdd1alvLOIx?= =?us-ascii?Q?r9XUgWAsuCDxRG5O1ouee0zHz4YskyzcAlVRJFnq889ExUr3Nx7/C5Yg0SB6?= =?us-ascii?Q?wf84rLaaERa2af+b3FncGL1THgD+WBEUmn4Gx1wJ8SzSnP8/3uxIp+N5J+sV?= =?us-ascii?Q?f53Y/elCJNU4OEjg475PCsHZMMyNvVH/FDeHIlExXIruIGvUl61d30/v9y06?= =?us-ascii?Q?qcGDZCdnfLgduhHkhYbUD9CXkKaJFAI3QH2Yo77d6o/nrtLewQjlwsGJHFPj?= =?us-ascii?Q?czyzQKp62Keyn2kC8VtpSpTr7zRP2UbqgKH2lUcSHIldB/1PuhJUVjKc44I/?= =?us-ascii?Q?3nHPLwADQ7neG6Iism7HhhmJWHUnc5ci3myN2muWaPmPhn1ZpR9TR3GRNtRv?= =?us-ascii?Q?Rb9KKSqJd3HuzJye/R9tKHaGNGD+/gi+BkEKplMFpofkkB2ie5mmEUBzpFCc?= =?us-ascii?Q?K+0amEMcg/LE1ZnRrQA44sV4+MI8Ph9Ym7SKJ3AL+ZKnhVjOSe2woxBxITwo?= =?us-ascii?Q?fGLuyt/Z0y8hUPlRYQTQPo4ADWw9WphDEHg5VjbRkVMf5rtNehQnIDk3R6bB?= =?us-ascii?Q?ItRHxtdSyPTxZiRXuyu5k2G/m3cQS33+vnBcn54x9deeFP5VEjuG+mNJ9sxb?= =?us-ascii?Q?z0klIgIZTW0DEkjyXZO/QkKQLgjnOcvRMsAydG/0giVa+sJt9qgF7/rouNYu?= =?us-ascii?Q?ZA6DT3TAlYhZtd+0sNj+BFa6tqI7EXw6PDXiJDOtvaI7vjjd3hSA9Y7rTXjm?= =?us-ascii?Q?NC+8V1HedrbItuqTtFBdeXGK6vV5Mp8VKDXIMcGG+ccxNvsod/4WOkofgwMf?= =?us-ascii?Q?gBVJVVwpoJxzhVbldlHEtgfV7tFwZmNkeo1hBD3eXA7/NWTpaIbSCaE+zJMu?= =?us-ascii?Q?nLgzgQb2faSsniuXNVL5kcG5XDv8P7Yo29/LnL0196JU4GlhcxVV8LK9b6wK?= =?us-ascii?Q?3cQyzctryt1GNf7eXEzCyD+CyuI8FsfPF53Bn90yVtxD3AWA/HMSE58Bb7lx?= =?us-ascii?Q?Eshtol6LFJhI6CFV2Vxm1up/exZwxomRTsVomC/FUvtS7Gr5MUpH2kVwF56S?= =?us-ascii?Q?gGtS3ixv6VMKOgKASgKRFmLRd/PiLBRqH4/TZ575dlK3sAhtE6v8Of4/suet?= =?us-ascii?Q?1jGWsvOJiAW38wKkl3+FcWx/l63Y3+NI/cp7xndyjllelCztAEMcOkOknwVP?= =?us-ascii?Q?voAYGxwMXHashudX9asKuBZi4M/EMCY2pFWzwwFxi69orUuddx5u3E1PNVSz?= =?us-ascii?Q?WpUXJm7fm5pzQ4XbOPVShvxBUO3Cia1KCI90gUBbqfyqDxiE7WHvjFvVtM9i?= =?us-ascii?Q?46BNzP98PNJhlaYCQbgVaX7/22knhGPmEji6j53fPFfGjBPEkasMzwxq+Fgy?= =?us-ascii?Q?YbB8jFV3AFGjfjZSdHOY9+Yk2Fr7ptxOATJKeuXEtYOuqAb+og/UQSK9dN9+?= =?us-ascii?Q?TMHlreZk+aLwsOi75iSPYj41c/zlKgGgqG0Qd1jdUw59ADGBXADNqB5G7/qo?= =?us-ascii?Q?Jw=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6fbea0ab-734c-4fc0-54bc-08dc55915356 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2024 16:56:23.8254 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UeqfEUQg0/fVg0jo0o8bRoViY2vDLz8lEsCxvE92XDGxwp65zmDjTteUyTSaqlRis72353TqzOgym5ra8DFfwh3wIqbsbDKXIkbG+mTsOOM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4582 X-Rspamd-Queue-Id: 886FB1C0025 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 3ru6o4y5p7ciuj8tmfh81jabm3udazxa X-HE-Tag: 1712336189-622006 X-HE-Meta: U2FsdGVkX1+++IN23uTSa8zGc1zGj8loO5zuGHl1+JvEkataZI52tiA9v4vAZTwqOybWOBvuw0zpRhGh/gxpTWqqfurkwjYIgzK0bLDHO3KmjDXJ/Kdr7zh43KKznWerLLOuTpB0RVrNG4iy8W6I2QEhItlL2JC6XHuEwAhY+Sn9rddlxtNbL5OXRoQGqrZ4MnduE0Rtwhz0c5U2Dgr8052YKQpTaEJzhpfhUQHuNsUTuJQr/9lanBo+dJ1Yg3t483/VcKygTmAmu81P18vFt1tU3xE8o03JNJJFdfhvCYZzLAHbR9zoht8gV7Q8PGNxdX5CX0znJqN8aSPbUM1XNA5TYg6qijAZC4n/zII6KAKQ3ailpjrPwW+tu7i/u7Pl+C9jwA1nICVskToT0ItyW9YUBcBaxlysWzYOqSTAIGVCjgODBsIRZfNdKtWMcJcRAxgL1L72qiwuzi3LanXQwhBA3lJf2kWyb+fPYyfJa3PLhedJ4ZcIuTi/Uzaq4IJ050eAsFk4HmG9LXHxgGMkQzwMcWHpcncTHlHQTc/AT3C8Lp+9WUOvgCNOWxpB1FsQlW5ofDpCjYyVnXj3szcARpj/saWbonmAFFY3wYAJ6KVHfCCESoDgYkxHYUEkYxfeDOiRXe9LbZeo0ICxtSLMM2JybyWbvxEKjVasz8QlNqipKqLEq5w2hnuMtsGbrKXypqYCCcMFWGjyDOQnjVup0WLTbbyFON+YSZTPmLj6e8xMQj0K0XYjDAu5VuYPjqA8ne1joORE4Kk3To+lXrgjUG37VvFSa/82aVI8Ugap4j958rhuc+S5i90ULDFYKQSuFSuox54EWgbNbPum7GccW83E/7hqCq8loVbHHKgIDE9ELwtn3RYwM9PvceO3vdn5myQAQ0q8zw4bxpsLJcLEi3izoxkM/Ir16RirkUdkf0Wf9VjtUyEygNiN9amnkbGu/v4M6pPDiWCpJLGGM5p H2hgYhsc SZeax38FkI0tOsHI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.008661, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 05, 2024 at 03:08:49PM +0900, Honggyu Kim wrote: > There was an RFC IDEA "DAMOS-based Tiered-Memory Management" previously > posted at [1]. > > 1. YCSB zipfian distribution read only workload > memory pressure with cold memory on node0 with 512GB of local DRAM. > =============+================================================+========= > | cold memory occupied by mmap and memset | > | 0G 440G 450G 460G 470G 480G 490G 500G | > =============+================================================+========= > Execution time normalized to DRAM-only values | GEOMEAN > -------------+------------------------------------------------+--------- > DRAM-only | 1.00 - - - - - - - | 1.00 > CXL-only | 1.22 - - - - - - - | 1.22 > default | - 1.12 1.13 1.14 1.16 1.19 1.21 1.21 | 1.17 > DAMON tiered | - 1.04 1.03 1.04 1.06 1.05 1.05 1.05 | 1.05 > =============+================================================+========= > CXL usage of redis-server in GB | AVERAGE > -------------+------------------------------------------------+--------- > DRAM-only | 0.0 - - - - - - - | 0.0 > CXL-only | 52.6 - - - - - - - | 52.6 > default | - 20.4 27.0 33.1 39.5 45.6 50.5 50.3 | 38.1 > DAMON tiered | - 0.1 0.3 0.8 0.6 0.7 1.3 0.9 | 0.7 > =============+================================================+========= > > Each test result is based on the exeuction environment as follows. > > DRAM-only : redis-server uses only local DRAM memory. > CXL-only : redis-server uses only CXL memory. > default : default memory policy(MPOL_DEFAULT). > numa balancing disabled. > DAMON tiered: DAMON enabled with DAMOS_MIGRATE_COLD for DRAM nodes and > DAMOS_MIGRATE_HOT for CXL nodes. > > The above result shows the "default" execution time goes up as the size > of cold memory is increased from 440G to 500G because the more cold > memory used, the more CXL memory is used for the target redis workload > and this makes the execution time increase. > > However, "DAMON tiered" result shows less slowdown because the > DAMOS_MIGRATE_COLD action at DRAM node proactively demotes pre-allocated > cold memory to CXL node and this free space at DRAM increases more > chance to allocate hot or warm pages of redis-server to fast DRAM node. > Moreover, DAMOS_MIGRATE_HOT action at CXL node also promotes hot pages > of redis-server to DRAM node actively. > > As a result, it makes more memory of redis-server stay in DRAM node > compared to "default" memory policy and this makes the performance > improvement. > > The following result of latest distribution workload shows similar data. > > 2. YCSB latest distribution read only workload > memory pressure with cold memory on node0 with 512GB of local DRAM. > =============+================================================+========= > | cold memory occupied by mmap and memset | > | 0G 440G 450G 460G 470G 480G 490G 500G | > =============+================================================+========= > Execution time normalized to DRAM-only values | GEOMEAN > -------------+------------------------------------------------+--------- > DRAM-only | 1.00 - - - - - - - | 1.00 > CXL-only | 1.18 - - - - - - - | 1.18 > default | - 1.18 1.19 1.18 1.18 1.17 1.19 1.18 | 1.18 > DAMON tiered | - 1.04 1.04 1.04 1.05 1.04 1.05 1.05 | 1.04 > =============+================================================+========= > CXL usage of redis-server in GB | AVERAGE > -------------+------------------------------------------------+--------- > DRAM-only | 0.0 - - - - - - - | 0.0 > CXL-only | 52.6 - - - - - - - | 52.6 > default | - 20.5 27.1 33.2 39.5 45.5 50.4 50.5 | 38.1 > DAMON tiered | - 0.2 0.4 0.7 1.6 1.2 1.1 3.4 | 1.2 > =============+================================================+========= > > In summary of both results, our evaluation shows that "DAMON tiered" > memory management reduces the performance slowdown compared to the > "default" memory policy from 17~18% to 4~5% when the system runs with > high memory pressure on its fast tier DRAM nodes. > > Having these DAMOS_MIGRATE_HOT and DAMOS_MIGRATE_COLD actions can make > tiered memory systems run more efficiently under high memory pressures. > Hi, It's hard to determine from your results whether the performance mitigation is being caused primarily by MIGRATE_COLD freeing up space for new allocations, or from some combination of HOT/COLD actions occurring during execution but after the database has already been warmed up. Do you have test results which enable only DAMOS_MIGRATE_COLD actions but not DAMOS_MIGRATE_HOT actions? (and vice versa) The question I have is exactly how often is MIGRATE_HOT actually being utilized, and how much data is being moved. Testing MIGRATE_COLD only would at least give a rough approximation of that. Additionally, do you have any data on workloads that exceed the capacity of the DRAM tier? Here you say you have 512GB of local DRAM, but only test a workload that caps out at 500G. Have you run a test of, say, 550GB to see the effect of DAMON HOT/COLD migration actions when DRAM capacity is exceeded? Can you also provide the DRAM-only results for each test? Presumably, as workload size increases from 440G to 500G, the system probably starts using some amount of swap/zswap/whatever. It would be good to know how this system compares to swap small amounts of overflow. ~Gregory