From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 329A8C4345F for ; Fri, 12 Apr 2024 07:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 647BD6B0083; Fri, 12 Apr 2024 03:30:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F6876B0087; Fri, 12 Apr 2024 03:30:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4984B6B0088; Fri, 12 Apr 2024 03:30:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2CD4F6B0083 for ; Fri, 12 Apr 2024 03:30:18 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BA39C160DC8 for ; Fri, 12 Apr 2024 07:30:17 +0000 (UTC) X-FDA: 82000056474.26.61281E9 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by imf03.hostedemail.com (Postfix) with ESMTP id C456420002 for ; Fri, 12 Apr 2024 07:30:14 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bdF3U0He; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.7 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712907015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A1hWJlsm6StbSFra3Yh61ZWoyECpOQ284nCmtxkx23Q=; b=qcdlRwdeS/8tHhng3frh/kam2IEapoBp6bIrTI3sG01ruMMgoLmoqgLxlpc7u3fwmNFbNU PSLUbg/pNRZPZnu62NRHWwR0lDPxlWtFNydjeIXL4kUD2wGuz4xcBM7Ee2Cib5HJ54/uut txS0KS6I8LLtbevWT3XM0qL1+hWVMMs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bdF3U0He; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.7 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712907015; a=rsa-sha256; cv=none; b=j8NRTdiR3Wc85IvHvNfPGZ0vSvXjrz1si3CTlm+kBIMGD0Lnl8xtjfOJ0tD+StBAZTUYdy WaV8iYfVN6tvGVOD6LTdQoikvdTp6/9ycQ+QjSa1isd7iXi/BM1PZoBI/LNzvtCU0ge0bg wruuO5igy90aSQ17DxIJqaXN7TIYImU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712907015; x=1744443015; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=5IO2NWCfpwWUyQ+ol8AQkFUqjjfRfj2qsguHSgUuUXs=; b=bdF3U0HeOdGd4JAc8urF+2yrJFFrxRlwCIhhUizTiWTaBrvjvR3Qmrwe 4uWtQ6E8QsDbQtUoh+7WcvAcgnHkx4jj7bfdwd0W/QkmtiitU5fHQozO7 XIpLU0FiEswPBPNs518Hfy29fGxem/J3EFVOYUrAzhdjf25YNl+lOVHJ4 RToMs7Qq+TzyaTz4jCYRSHjdsU1RFIdIbshV41FU+XIdubrVKDoOW2d6j Q9XuCO5TZ689EU6bKUhMYbbmafQxcMLxZtVBXki0FnGjG4XHd8WgY1WZL J6sV1F13w+A9W94ExbZ/sd0+agzjaSxVhWYNRSMvs6YsMRORxmJv58RhP Q==; X-CSE-ConnectionGUID: oFMsw0BbQAG36MF4+c7hSA== X-CSE-MsgGUID: fvVO1rcFQ3i9snsNyRYtWQ== X-IronPort-AV: E=McAfee;i="6600,9927,11041"; a="33748860" X-IronPort-AV: E=Sophos;i="6.07,195,1708416000"; d="scan'208";a="33748860" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Apr 2024 00:30:13 -0700 X-CSE-ConnectionGUID: 1qZw1JxBROSAf6QteJnCEw== X-CSE-MsgGUID: oSajzwv7R5GU1hSfdxDTNw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,195,1708416000"; d="scan'208";a="52123263" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Apr 2024 00:30:10 -0700 From: "Huang, Ying" To: Bharata B Rao Cc: , , , , , , , , Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address space In-Reply-To: <8692d514-d1c8-4fbf-84d7-1ad609480070@amd.com> (Bharata B. Rao's message of "Fri, 12 Apr 2024 09:30:45 +0530") References: <20240327160237.2355-1-bharata@amd.com> <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com> <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com> <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com> <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com> <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com> <9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com> <87il0yet4z.fsf@yhuang6-desk2.ccr.corp.intel.com> <8692d514-d1c8-4fbf-84d7-1ad609480070@amd.com> Date: Fri, 12 Apr 2024 15:28:16 +0800 Message-ID: <87cyqv59bj.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: C456420002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qgxk18p91d46j43a5d9d19xgd9j3ffdw X-HE-Tag: 1712907014-493902 X-HE-Meta: U2FsdGVkX1+OyjXtIGYS5OXHsDDd/cvD3pN3X8KgQ/p1e8G+PuXCfX72Tqu+HYcCqne8YTZhj9U08iVDpYk5smdMylQhxizEqkJpsDAWR2bSSbiyxQ9rooUykOMDJZwfFc8mT0N38czqdbvJzALc703Tcq2FzLXm30JqZSST9c6oEk0sgJ6+E4esnbdLtKiPfcZ9RtKy7DUpZTr3R8AZLuwQfItUyaYILN+DAI/jDC+Fl7fssJS3RzB9Xh9uo5ft9KiALDjxYJh/+rbQ1xjn+SG5WxvD/qJjwbGnMd/Yg+9u3Fqs1dM/Q1kcx0AnjlOrjYfvIejNageG8qYI9MR3EAe6G7CBodbZGSSIZxCK9ZMZBrO7wSFtRS+vNfdLwAIXNxj+TH+U4dzEUe4v3nyFca9fbhA6dhYSP+woYEWpTIxmQO32pbv9VlQbrwCyOQLoPPtnGJcE4de4knD/9lk8ebCAdF/6rlgWmmf275lyeGFG4aL2Tds4+/CpbRTs+0EuwSsx73cavP3LRH6rHxTQ1qgDgShyNVH186gVxk8c+4QjiGfHbDoAxRsc++6cr9YuBy0m0fZFKHGI6wF8npp/B6Ierdb4VURgd8fDBKjjseAOyzMFEz7Kw6hY/zZOLwomsJQ0kWEJhrFdeWVnOIl8hnL25kXNroYuk0UoImNA73eUvXCksyPb/xnxXLb8DXojxbLRNz22ErYfkOtWFgHmpME58qjXkJi4wfQM4dSmblla64nd+WsHC5Y6052BqibF7pfG6/3xhH/Snxy7aV6VQPqhQsOoWe6BnZcJl6KOZDnyc9R7YeDK9Sv6jXztGGbUyLMZf1p+F5IwHfvSYe62qxMcIyDE7DCiUhgWAKF8+olksM//jfvvFQz/oNqCkBJxTlhRE6GnTqhWJru7955BR4BAFvp1bIvXJP2mmkW3tF+RbrOS4gSBAailhjkkUGsyyMu7q0i8mAcPkgGmlI+ sRnsF6K3 T4qVfoOwAtrLmr4hSeZcwdfmUgb5F+APMHw51GcxsaIhezccTdnRYqV8aVHpFyuieOaOVreo87ch0+qDfDzMRojRPpMnowcU0wgrCxMyi/GJ5DRJ2KUYDxMmBorW85NikTA+lwbmoMaikGV/7rQdK/bf+H+wZosp8uSiDR/QAbITVbB8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Bharata B Rao writes: > On 03-Apr-24 2:10 PM, Huang, Ying wrote: >> Bharata B Rao writes: >> >>> On 02-Apr-24 7:33 AM, Huang, Ying wrote: >>>> Bharata B Rao writes: >>>> >>>>> On 29-Mar-24 6:44 AM, Huang, Ying wrote: >>>>>> Bharata B Rao writes: >>>>> >>>>>>> I don't think the pages are cold but rather the existing mechanism fails >>>>>>> to categorize them as hot. This is because the pages were scanned way >>>>>>> before the accesses start happening. When repeated accesses are made to >>>>>>> a chunk of memory that has been scanned a while back, none of those >>>>>>> accesses get classified as hot because the scan time is way behind >>>>>>> the current access time. That's the reason we are seeing the value >>>>>>> of latency ranging from 20s to 630s as shown above. >>>>>> >>>>>> If repeated accesses continue, the page will be identified as hot when >>>>>> it is scanned next time even if we don't expand the threshold range. If >>>>>> the repeated accesses only last very short time, it makes little sense >>>>>> to identify the pages as hot. Right? >>>>> >>>>> The total allocated memory here is 192G and the chunk size is 1G. Each >>>>> time one such 1G chunk is taken up randomly for generating memory accesses. >>>>> Within that 1G, 262144 random accesses are performed and 262144 such >>>>> accesses are repeated for 512 times. I thought that should be enough >>>>> to classify that chunk of memory as hot. >>>> >>>> IIUC, some pages are accessed in very short time (maybe within 1ms). >>>> This isn't repeated access in a long period. I think that pages >>>> accessed repeatedly in a long period are good candidates for promoting. >>>> But pages accessed frequently in only very short time aren't. >>> >>> Here are the numbers for the 192nd chunk: >>> >>> Each iteration of 262144 random accesses takes around ~10ms >>> 512 such iterations are taking ~5s >>> numa_scan_seq is 16 when this chunk is accessed. >>> And no page promotions were done from this chunk. All the >>> time should_numa_migrate_memory() found the NUMA hint fault >>> latency to be higher than threshold. >>> >>> Are these time periods considered too short for the pages >>> to be detected as hot and promoted? >> >> Yes. I think so. This is burst accessing, not repeated accessing. >> IIUC, NUMA balancing based promotion only works for repeated accessing >> for long time, for example, >100s. > > Hmm... When a page is accessed 512 times over a period of 5s and it is > still not detected as hot. This is understandable if fresh scanning couldn't > be done as the accesses were bursty and hence they couldn't be captured via > NUMA hint faults. But here the access captured via hint fault is being rejected > as not hot because the scanning was done a while back. But I do see the challenge > here since we depend on scanning time to obtain the frequency-of-access metric. Consider some pages that will be accessed once every 1 hour, should we consider it hot or not? Will your proposed method deal with that correctly? > BTW, for the above same scenario with numa_balancing_mode=1, the remote > accesses get detected and migration to source node is tried. It is a different > matter that eventually pages can't be migrated in this specific scenario as > the src node is already full. -- Best Regards, Huang, Ying