From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA95CCD1288 for ; Wed, 3 Apr 2024 08:50:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 820DE6B008A; Wed, 3 Apr 2024 04:50:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9A16B0092; Wed, 3 Apr 2024 04:50:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6235B6B0093; Wed, 3 Apr 2024 04:50:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 326986B008A for ; Wed, 3 Apr 2024 04:50:49 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EFE84A0DA2 for ; Wed, 3 Apr 2024 08:42:28 +0000 (UTC) X-FDA: 81967579176.08.850588E Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf17.hostedemail.com (Postfix) with ESMTP id 3685C4000C for ; Wed, 3 Apr 2024 08:42:25 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RSmqzXjb; spf=pass (imf17.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712133747; a=rsa-sha256; cv=none; b=e1AmX8T/zsKtsKuEAC2vRpmAPOoxBAF4NauMhQV6t4XgS6is6o4opmxYmtWc68t3O7NTGk 37mB4y9bQjAdRESn9nVOYn3ClTooMh+UvEqF8Ey2/Byzky07s5rEK5b4CetavfNFdoIIGr TuEiAvoXTjQL8IUlcFagF6Og6N0ClvM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RSmqzXjb; spf=pass (imf17.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712133747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t6GmiQ6YFOUpgGjwEihKAXDenv6JMkNouCZgpYQ0q28=; b=EUpS72WxeSKpMoppzlHVUTK068KnsmXtsGP1vXA/gl6NTIbbW7L27ziLkLyQOx3QGKlK39 lt3YEUd8s5eSUTFx+EW2yXR34+OD7zFqtivLJiz2wV8RcWcFjLfXq0fY1mTcVucC9Qtw4c NFHtNmoDSe5KYV1EwWzhB2wkeVfgW1k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712133746; x=1743669746; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=Yc/UxVgdcmJxp/86IHuzsAxrO66im/yKibWMN4SKocA=; b=RSmqzXjbpYHo6ynOZip6VXWMzvKaIJhN+LnxPH6NjVmytn6RFE1E8NPS VOomkc6lEd7sf7XMKA1trBENkPmtoSTKhWTXrM3TPDLRzAXq89jCWfEj0 b5SYNnOKBOC0Tlp0v9EErITf7s0d5+OeJoqN/HdUvH0fdwO51nXH4FuSt VSG4Iy5rra1SqCGlpv7THRvtMpxU0UpYxBAiyl8pTYPkFGTnBoJ2pACwF zg2Va0lYWjvGpkRivoq2FZw9iw366rcox6S0hsyZPUp6lXt6wtGRPj5m3 Xzd9AA3lM3cWB1+9iZ6u8BJNdkruC9geCBC6NLUPQ2XCztrrqV9lmGi7i Q==; X-CSE-ConnectionGUID: bHgA6oHCTXOV3UkQhHSZMA== X-CSE-MsgGUID: YWdwHHGRR+e6gwohkEDssw== X-IronPort-AV: E=McAfee;i="6600,9927,11032"; a="24847906" X-IronPort-AV: E=Sophos;i="6.07,177,1708416000"; d="scan'208";a="24847906" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2024 01:42:24 -0700 X-CSE-ConnectionGUID: FrRS57trSh+Ovwk9SHtEBA== X-CSE-MsgGUID: e7BMVgx9Q9ylI9EuWF6nMg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,177,1708416000"; d="scan'208";a="22993868" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2024 01:42:22 -0700 From: "Huang, Ying" To: Bharata B Rao Cc: , , , , , , , , Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address space In-Reply-To: <9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com> (Bharata B. Rao's message of "Tue, 2 Apr 2024 14:56:37 +0530") References: <20240327160237.2355-1-bharata@amd.com> <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com> <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com> <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com> <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com> <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com> <9ec3b04b-bde8-42ce-be1b-34f7d8e6762d@amd.com> Date: Wed, 03 Apr 2024 16:40:28 +0800 Message-ID: <87il0yet4z.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3685C4000C X-Stat-Signature: bb7mwppjpwcj8kmeca95knd7j1cyg31c X-Rspam-User: X-HE-Tag: 1712133745-810604 X-HE-Meta: U2FsdGVkX1+o050UsDtDNU6ghx4GS3u6UuJPvLTqWAd8Tm9VK4oAFgamo8i7ciPpC3Lap34VMC5JEj8zPq1adji101IjkYETbROyOaBkgaUC4qFHjHTLaEUnLd5BeBcEIY3UggjfmDgVlqzia24YG5EFVYziI9BUr8pU7pHcXOeUqp4P/0ZATS2YzN4warzb9VnzGgkYqHW+uLOozg8be21yD2eL+0VgnaQr9/VIqtUnZJ4VRYzm+g5dRjeZ4Vz1YQj/KtZt9pYsSxKOajwfVb5leQk72rdokxgFgRsGTwYPLIM3f7MaGok3EiPSFWlv4zpuiNLRdmeMczMWLqZyab5LnNOicRC3HRp+caZaq9yIxP07bJQbyCJER+nQ0gJkXNiSjgeQXXtXEZjw3TtL/cYsFbRYJNocril4DQAaPpAf7dCBIXc/OZt4CWD2cBFixCQwhbzLTvCWJqbh6cyqvUwWYqMSsoU9fqKGtAkit6y5eZkGBegoJ/M5zifEdg++ECIwGxdLKJqeSwxB6Av/+BIPSUJTkmuzg7eGrDrn4wTCFkmImZaGubIF0K4SUmyJepi+Na31SF9Rvlgg0NaGkGZi4iR5offu8ALmIq17ofZ2HhlMhsMw301vVtw2qGPkTBlB8FFQvg7l7ornaC6P9n1K9hr8po/Kyru9TSfhHNahk468hhyx4lxVZr8PfTnziPT7PX3E40dW3MUlS2pc/rZdp+YkkjUtfkJef4/KNij3NjmV0DuBTTErSseSK1KvA29dKwDGfAezr6mNFJnqtB214lHxKC3+xF8Fj+Dw+PUDwJysGD5rGllN9w/flPAy70B/bXLSMFT3r2yNwLMtKJ/46Ys7+N3W X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Bharata B Rao writes: > On 02-Apr-24 7:33 AM, Huang, Ying wrote: >> Bharata B Rao writes: >> >>> On 29-Mar-24 6:44 AM, Huang, Ying wrote: >>>> Bharata B Rao writes: >>> >>>>> I don't think the pages are cold but rather the existing mechanism fails >>>>> to categorize them as hot. This is because the pages were scanned way >>>>> before the accesses start happening. When repeated accesses are made to >>>>> a chunk of memory that has been scanned a while back, none of those >>>>> accesses get classified as hot because the scan time is way behind >>>>> the current access time. That's the reason we are seeing the value >>>>> of latency ranging from 20s to 630s as shown above. >>>> >>>> If repeated accesses continue, the page will be identified as hot when >>>> it is scanned next time even if we don't expand the threshold range. If >>>> the repeated accesses only last very short time, it makes little sense >>>> to identify the pages as hot. Right? >>> >>> The total allocated memory here is 192G and the chunk size is 1G. Each >>> time one such 1G chunk is taken up randomly for generating memory accesses. >>> Within that 1G, 262144 random accesses are performed and 262144 such >>> accesses are repeated for 512 times. I thought that should be enough >>> to classify that chunk of memory as hot. >> >> IIUC, some pages are accessed in very short time (maybe within 1ms). >> This isn't repeated access in a long period. I think that pages >> accessed repeatedly in a long period are good candidates for promoting. >> But pages accessed frequently in only very short time aren't. > > Here are the numbers for the 192nd chunk: > > Each iteration of 262144 random accesses takes around ~10ms > 512 such iterations are taking ~5s > numa_scan_seq is 16 when this chunk is accessed. > And no page promotions were done from this chunk. All the > time should_numa_migrate_memory() found the NUMA hint fault > latency to be higher than threshold. > > Are these time periods considered too short for the pages > to be detected as hot and promoted? Yes. I think so. This is burst accessing, not repeated accessing. IIUC, NUMA balancing based promotion only works for repeated accessing for long time, for example, >100s. >> >>> But as we see, often times >>> the scan time is lagging the access time by a large value. >>> >>> Let me instrument the code further to learn more insights (if possible) >>> about the scanning/fault time behaviors here. >>> >>> Leaving the fault count based threshold apart, do you think there is >>> value in updating the scan time for skipped pages/PTEs during every >>> scan so that the scan time remains current for all the pages? >> >> No, I don't think so. That makes hint page fault latency more >> inaccurate. > > For the case that I have shown, depending on a old value of scan > time doesn't work well when pages get accessed after a long time > since scanning. At least with the scheme I show in patch 2/2, > probability of detecting pages as hot increases. Yes. This may help your cases, but it will hurt other cases with incorrect hint page fault latency. To resolve your issue, we can increase the max value of the hot threshold automatically. We can work on that if you can find a real workload. -- Best Regards, Huang, Ying