From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C169C54E67 for ; Thu, 28 Mar 2024 05:37:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC2786B0088; Thu, 28 Mar 2024 01:37:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C739A6B008C; Thu, 28 Mar 2024 01:37:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B61A06B0092; Thu, 28 Mar 2024 01:37:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9D62A6B0088 for ; Thu, 28 Mar 2024 01:37:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 650C0140F21 for ; Thu, 28 Mar 2024 05:37:42 +0000 (UTC) X-FDA: 81945340764.07.4B23664 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) by imf02.hostedemail.com (Postfix) with ESMTP id F091E80007 for ; Thu, 28 Mar 2024 05:37:39 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ngQC02l1; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711604260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XiAmk//C1+jMPk+ZLi5lzMjZZ6zTKsxIFk4uq0FhAu8=; b=Daq2nhGEa6aPCAxLX80o+EhHdUx7oBdvi+ROpo+K6x2RR+ouBAzfufGwyjv22x0XqIJB8s aKPvSKYwPxFiAOj7nT6YNcjq2gixpCWJN7JQxUo+Ss2IH3mOwMIexYmCMnt6jqdWUFMsMk pU6+vH3BvzNSeib2h/XykmAy1Y1A6js= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ngQC02l1; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711604260; a=rsa-sha256; cv=none; b=K78jeSs6ka1igYKaXi9L6gDvxT8XijEbrCfkwcx3FQFpi02KpDoOsWZMINiwgq6GAGdFfh wVtHt13IFgiDd+GBKs+jaiekAKZgL7pS9pgDjN8/EE5XSdM6K5K+Tr7D9DPKMgUl5mgbDo p9/pe9l5cjt6IfEwzJ8qJju9Vq1yAVQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711604260; x=1743140260; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=I1zV8l2wxGw1MD2JHTbYzY6hhxw7196Gef6nl+ubcWg=; b=ngQC02l1/HP1+ipq3fwjccUwlwWVLGLFoogeAIAwkUOrVrR/Dy7UqHCg FYOCjV6MqjTd7z62Yuk+bGCMWYXF8fA8mp+OoZ2xQHSfNebeK7j7/E+Ny 6PkyEAPIiq2FVaaAwwuB+iBNo+JFJUKszgs+W1Axp4/f5YkImapQPIV9c lkc5gM+beIApjsAD+blSuCL3vEe4+boPkViP9t3I3vJ/JhKUMc4tjZunA qvmd3jP+Ag1o48I0rWG5NY2azd+ScRfHur9366541TgDBdzspmFArKWvP 7426a67YxHk7G869MgaVIPBVUOTe3RJZ0oTKPzkOfg4BzYxhxavTu9Au6 g==; X-CSE-ConnectionGUID: pPu24d5AQmaYZ5GkcdNPXA== X-CSE-MsgGUID: 8UjQTWA3Tp+kJjT0I+pGSA== X-IronPort-AV: E=McAfee;i="6600,9927,11026"; a="18122847" X-IronPort-AV: E=Sophos;i="6.07,160,1708416000"; d="scan'208";a="18122847" Received: from fmviesa001.fm.intel.com ([10.60.135.141]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2024 22:37:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,160,1708416000"; d="scan'208";a="47732570" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2024 22:37:36 -0700 From: "Huang, Ying" To: Bharata B Rao Cc: , , , , , , , , Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address space In-Reply-To: <20240327160237.2355-1-bharata@amd.com> (Bharata B. Rao's message of "Wed, 27 Mar 2024 21:32:35 +0530") References: <20240327160237.2355-1-bharata@amd.com> Date: Thu, 28 Mar 2024 13:35:42 +0800 Message-ID: <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: F091E80007 X-Stat-Signature: byofsckib1n9fmrtxqm89k3tsft9bz4k X-HE-Tag: 1711604259-217350 X-HE-Meta: U2FsdGVkX19KaqU7djKbc9eliwpZNwlFMxKHItDCZzbY2I2DZlVdhJyEoYyrcexow1Am66SahZM3s89FPiIOjvFy3FX9+Ddd+cs0d/Ccl2b2s8SHkLqwERsygcdPKQhrNtQLqoGYKDs3yA8ybjvQd1fCjFOECCLAVdSHZcp+3dGEraymHOEztIcuOztjbDpU1QtY/Fm0T0awdg8JXr1DDwDVqMGDAdcUIINdTiOA8wCpw9kPhF02XdTc77GEol3k94jT/rMx1r0iAu1D4CccUUXIPOxvP/hl7NG5e3cwTj3vdFjDaNgh0nhTXbJJ1OSmcBy/JUVooyx8imDhY/bmE+uSPwrnQH8HzSy0mU022WpTks4caVY7x3o+B80WA0SSp/kdtBhWdqVGAxF0wP8tbMI4QWWTkONw+sWs22kagKc0Jswiuv3izTWNR7I1JJgoIvmiG5rb7zx3hBNzv1UO+nVfn13E9wnDdaavE/Kk1RicyBiri0Hr3igaCs0Edpn94wyofAOmFfeeB8kvOR0fmKatyfdIQA1NmAahuQKP/Y718+gSjNOMqj7EEPqYm9Pu7xyTZgvYJvYWJqzVJHKzJJvcU4udPnQWBRgPtM+pkMpHvQ/Vgfdgel6zsIvYTPF3HBEz1pVCj0ntcDIAkSv7lQRIwGGeSA3YNS/DuvCbxrrc6lLUhS8vWLDOxxiEuAyiKCxoRvyIUD20f5DjJ1Hw5PJqxHHbXb2XLthDIW4Bw6UNo2wrgg0hMksA+1s5uilnbTSI+9fM5H2YMj78jIo7uDAWviZN3HPa85c5pH9NPP8Et9u2cwJsr6llLOU2mUk4CZglKvzJhStt9Wg/axZvmXD7n1TJm6SVfYl0PJyFHHCEmpga+Au+ukOxN0D6YR/sNHvj6bFf7NDYJ2E8p7DhgEGPDbh15M+n8LKvqfu0MkYEAadQcZfAHmFX53jEQshSaBjS7PPJweIlH+6FyhS f8fo3juV 9zAVbPwV9kZWGW7KNluGK5GCah7aeVPA74IrxXLDZ6aHiCghA2HApjV8ROV+zsRe6IMNn4j9K7bjD/0bU73NULRyjZEk+M3lBzKPqR+U+xa/7xEXVQZX9oKS4/kX4VWdR5XjgsjGNKtaUXT/LBVJC5R5mPICmSVm/Pbbt4epukh5guLE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Bharata B Rao writes: > In order to check how efficiently the existing NUMA balancing > based hot page promotion mechanism can detect hot regions and > promote pages for workloads with large memory footprints, I > wrote and tested a program that allocates huge amount of > memory but routinely touches only small parts of it. > > This microbenchmark provisions memory both on DRAM node and CXL node. > It then divides the entire allocated memory into chunks of smaller > size and randomly choses a chunk for generating memory accesses. > Each chunk is then accessed for a fixed number of iterations to > create the notion of hotness. Within each chunk, the individual > pages at 4K granularity are again accessed in random fashion. > > When a chunk is taken up for access in this manner, its pages > can either be residing on DRAM or CXL. In the latter case, the NUMA > balancing driven hot page promotion logic is expected to detect and > promote the hot pages that reside on CXL. > > The experiment was conducted on a 2P AMD Bergamo system that has > CXL as the 3rd node. > > $ numactl -H > available: 3 nodes (0-2) > node 0 cpus: 0-127,256-383 > node 0 size: 128054 MB > node 1 cpus: 128-255,384-511 > node 1 size: 128880 MB > node 2 cpus: > node 2 size: 129024 MB > node distances: > node 0 1 2 > 0: 10 32 60 > 1: 32 10 50 > 2: 255 255 10 > > It is seen that number of pages that get promoted is really low and > the reason for it happens to be that the NUMA hint fault latency turns > out to be much higher than the hot threshold most of the times. Here > are a few latency and threshold sample values captured from > should_numa_migrate_memory() routine when the benchmark was run: > > latency threshold (in ms) > 20620 1125 > 56185 1125 > 98710 1250 > 148871 1375 > 182891 1625 > 369415 1875 > 630745 2000 The access latency of your workload is 20s to 630s, which appears too long. Can you try to increase the range of threshold to deal with that? For example, echo 100000 > /sys/kernel/debug/sched/numa_balancing/hot_threshold_ms [snip] -- Best Regards, Huang, Ying