From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9613ECD128A for ; Tue, 2 Apr 2024 02:05:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A8B86B0083; Mon, 1 Apr 2024 22:05:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 858716B0085; Mon, 1 Apr 2024 22:05:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71FA86B0088; Mon, 1 Apr 2024 22:05:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 549866B0083 for ; Mon, 1 Apr 2024 22:05:37 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CDABE40902 for ; Tue, 2 Apr 2024 02:05:35 +0000 (UTC) X-FDA: 81962950230.14.80A80A4 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) by imf28.hostedemail.com (Postfix) with ESMTP id 8B898C0011 for ; Tue, 2 Apr 2024 02:05:32 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KYKHD7b9; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.20 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712023533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DA0xUawUoVpglNNx1DE2UIoJRpcLMTMwRoxaoCaYWcI=; b=vATFZXgNtJJwiH1nd8utnj5Ujqk7S53EBeTWWtLQ+CLEUhU82qbGDPHe5UnU5SXePfVwuR Q1X9v2/zFMMNHPySWuexMnDX93bvKgJXCgFPyv6Y1hDwgD43XXYQnmY9g9HRVMxNUWMIHp p/O37d/L7lx2Trkeyfg+w4dnjmeb6Ys= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712023533; a=rsa-sha256; cv=none; b=wdoZ33F66oS7zYP5H2mKxdSveDury1N+Mx5kHZWc/GwXtLZrwE5QbE2ZU/JbpKMPW11jak ut+gOn5XBbNSFR1FYVbsB4rQ26KXrJ9uz7U9BHVexx+xqEOz74dxxF++dlhW/WVA4N2ysy 7pi9vhlBVqdydp6+L6ZbfDKMoPCW5tw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KYKHD7b9; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.20 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712023533; x=1743559533; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=Zsrq55HsUs8+9JmV9nnFtceZ2D2s/p9Q4BjxBGEMSEs=; b=KYKHD7b9nh+Bom1LRAKPWudeHjh+SxtRk6iZkilNfVoBbIuVrVMqxMWW IodvfXJpPjloiip6bUjiGWuY6mQY0/J+bfLzfntg5R3Gv8NnM2GMX+N5p 0RVf9z26W+NC7xnpn0GOk1N2vhDj09s8D3JoKfpliJo/Bu8fRVaYy6ekF 0qsybkwstNlNVtHfoLSAoP9mYOvgHM6G1XsN5i8Pv2JP5t5q45AnlMGnR fRioZlWQcZ2eD28QqZG4yPNomaAdFKUfdP4QMSA5dexi952UqAMbs+eUG s1fRwQjBKwr19qIs4Mo4tPmg5Pds0n/vDL9AUhg7esHsJNKSnkYBDHu6D Q==; X-CSE-ConnectionGUID: kzvSkUzsSIi9nIb/NzqAFw== X-CSE-MsgGUID: CYz30+IbRwOpWrEMaOGwYA== X-IronPort-AV: E=McAfee;i="6600,9927,11031"; a="7072602" X-IronPort-AV: E=Sophos;i="6.07,173,1708416000"; d="scan'208";a="7072602" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2024 19:05:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,173,1708416000"; d="scan'208";a="17864259" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa009-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2024 19:05:29 -0700 From: "Huang, Ying" To: Bharata B Rao Cc: , , , , , , , , Subject: Re: [RFC PATCH 0/2] Hot page promotion optimization for large address space In-Reply-To: <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com> (Bharata B. Rao's message of "Mon, 1 Apr 2024 17:50:57 +0530") References: <20240327160237.2355-1-bharata@amd.com> <87il16lxzl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edbulwom.fsf@yhuang6-desk2.ccr.corp.intel.com> <929b22ca-bb51-4307-855f-9b4ae0a102e3@amd.com> <875xx5lu05.fsf@yhuang6-desk2.ccr.corp.intel.com> <7e373c71-b2dc-4ae4-9746-c840f2a513a5@amd.com> Date: Tue, 02 Apr 2024 10:03:34 +0800 Message-ID: <87o7asfrm1.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 8B898C0011 X-Rspam-User: X-Stat-Signature: tq61ducu4h8oa44zgd3eaq7zqkkaf3wn X-Rspamd-Server: rspam03 X-HE-Tag: 1712023532-957629 X-HE-Meta: U2FsdGVkX19U+qX/H6AOJZUHX004W428heZgV38zbwzNZoyoW1zMjUclGgL/LA8Lu7HWq1CHDvd6lbJ6YslGqgrrcJK1HK8CqtIUDZgzuDAfAHK9Op73WxWCEMBOuAtZ9dY3+xDesvehhfnElNyF5ABIksgUXmEp+W0BtBQJx3710crUWf8oInc67vPgbqV9UEazs8m2jfkuUms+CRDyXSddcqOpI9Ivc5iBz07HFjDUp6EPmKFgnugv5hRGdmDtkKxqnSmtRTDKuO0yqoVKcgvNf+JttQjOKT3mY1H9c9ncTaoDxMqHmMo04VjXBNCNHWibqp8OmIEfa/nb635IxtsCey+CxFhh6bhErg3R23dyuIXGfTEp/oYpq+u4OB7xhUOfOPK84c/gMjee1cwHaVRGsU3e7lb6EbAfVncLiDBK1A5K9nTdMMz0iv0O3RgfvAIzDvm2L0pvrhuTdiUPYY7XAHFuzp2drmi8vYQvZrcNhLz17ws120zxPKz9lv/h22PAcvhJWa5xaTGnbm5EsbXWGYYrdWQHf3/MdIL45876SfWuMdzf8h3EF1k7L37EDyls+2dXymyl3T+WgstOOP34GsZoM4dpTX69bFWRXqKGxeYcSK631YalIfy3VXOFvBkxv+lUA3QYjSl2GuTzyqGmQJh1Mv4rFXT+rYml3uP+qUof70gLmgIWW1kLeEiKhnx7x/7ST8qQyn5sLzS3BBUn0jn5ckWZdZtB9LPuU39T+XCC9f9e34Qr1XX06ahLzEUtfCjZ+UgthLWANFB1g8fy67+1JTTMuCwAK8GCdT6vpsTe9uKmS1SijmPI7QMtXbGkWUZd9U6UHIU8dz28ai6I2bBgZoNBZ2Ofo9E4q4uycGxKiWLZwCBEl98F+6iQSUgHG0A4D9s8qaWGXJyhPuJW2GcXG2UJVNVE4aVUzuqxXUjxeVuzW0rjgbOp9zxpcJlNI8UtzbV0P93l50/ u0HJQvMa mxWtFVAQ771FIG0O4OPC5BxpOgiX/h1tP0e86pb2TdUYtHAQ1z/aXT+SM9r1g+EOvEz/RG7E2OBeGWrVN6M27xz5GU7hLpM1lQh49JhuczFPQjEfg6TpX1w2tdOaIXEKImHYrbTII5rUFFXKWsvuB4BXazHj8t0VmEt/xq/K5RshYeoE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Bharata B Rao writes: > On 29-Mar-24 6:44 AM, Huang, Ying wrote: >> Bharata B Rao writes: > >>> I don't think the pages are cold but rather the existing mechanism fails >>> to categorize them as hot. This is because the pages were scanned way >>> before the accesses start happening. When repeated accesses are made to >>> a chunk of memory that has been scanned a while back, none of those >>> accesses get classified as hot because the scan time is way behind >>> the current access time. That's the reason we are seeing the value >>> of latency ranging from 20s to 630s as shown above. >> >> If repeated accesses continue, the page will be identified as hot when >> it is scanned next time even if we don't expand the threshold range. If >> the repeated accesses only last very short time, it makes little sense >> to identify the pages as hot. Right? > > The total allocated memory here is 192G and the chunk size is 1G. Each > time one such 1G chunk is taken up randomly for generating memory accesses. > Within that 1G, 262144 random accesses are performed and 262144 such > accesses are repeated for 512 times. I thought that should be enough > to classify that chunk of memory as hot. IIUC, some pages are accessed in very short time (maybe within 1ms). This isn't repeated access in a long period. I think that pages accessed repeatedly in a long period are good candidates for promoting. But pages accessed frequently in only very short time aren't. > But as we see, often times > the scan time is lagging the access time by a large value. > > Let me instrument the code further to learn more insights (if possible) > about the scanning/fault time behaviors here. > > Leaving the fault count based threshold apart, do you think there is > value in updating the scan time for skipped pages/PTEs during every > scan so that the scan time remains current for all the pages? No, I don't think so. That makes hint page fault latency more inaccurate. >> >> The bits to record scan time or hint page fault is limited, so it's >> possible for it to overflow anyway. We scan scale time stamp if >> necessary (for example, from 1ms to 10ms). But it's hard to scale fault >> counter. And nobody can guarantee the frequency of hint page fault must >> be less 1/ms, if it's 10/ms, it can record even short interval. > > Yes, with the approach I have taken, the time factor is out of the > equation and the notion of hotness is purely a factor of the number of > faults (or accesses) Sorry, I don't get your idea here. I think that the fault count may be worse than time in quite some cases. -- Best Regards, Huang, Ying