From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EB6FC02181 for ; Fri, 24 Jan 2025 05:53:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D10C528003D; Fri, 24 Jan 2025 00:53:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9977280025; Fri, 24 Jan 2025 00:53:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B39FF28003D; Fri, 24 Jan 2025 00:53:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9499D280025 for ; Fri, 24 Jan 2025 00:53:14 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3FEE2B0E66 for ; Fri, 24 Jan 2025 05:53:14 +0000 (UTC) X-FDA: 83041277508.28.0419823 Received: from invmail3.skhynix.com (exvmail3.hynix.com [166.125.252.90]) by imf12.hostedemail.com (Postfix) with ESMTP id 9357540005 for ; Fri, 24 Jan 2025 05:53:11 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of hyeonggon.yoo@sk.com designates 166.125.252.90 as permitted sender) smtp.mailfrom=hyeonggon.yoo@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737697992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DqjLZb2G1ETjYfl826GRKEzgmuLnOkwwKXEoV7j61lU=; b=hwU+ExobdzGlD4Q+wi9cr09QAe1TNd5gEIs3wQf+arOyBECLnP9pBxC6c6K0R5iPuCm1i4 rii/5RaqWwXagLFqDVSbrezSiE37+e0reWPg5WA+m0EAoDPl8rlg2hl8Wv2q+p8Q88Xud+ wdrb6XnayHu9uZADFaf+74wRAI/K/0g= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of hyeonggon.yoo@sk.com designates 166.125.252.90 as permitted sender) smtp.mailfrom=hyeonggon.yoo@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737697992; a=rsa-sha256; cv=none; b=bG+OHpGbUhBDspmnf5yP27/jNiiQuYOyGV3bvrfKSE032vzc6kAuL0KTZR6KmkKvgTJ/C+ sK8VeOxEHmHcDxz2Fh7VcWUWnBRdH3nAnoJiDZlQGJalVIJoCAzdrohBAlLShir3YI1WSH Lz8+SNZOrh81hiDoPmbtjT5YroC4kxI= X-AuditID: a67dfc59-7a9ff700000194b3-2d-67932ac4f167 Message-ID: <0a9914f8-bec7-4e58-ab12-a87fe3876187@sk.com> Date: Fri, 24 Jan 2025 14:53:08 +0900 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: kernel_team@skhynix.com, 42.hyeyoo@gmail.com, "gourry@gourry.net" , "nehagholkar@meta.com" , "abhishekd@meta.com" , "ying.huang@linux.alibaba.com" , "nphamcs@gmail.com" , "hannes@cmpxchg.org" , "feng.tang@intel.com" , "kbusch@meta.com" , "Hasan.Maruf@amd.com" , "sj@kernel.org" , "david@redhat.com" , "willy@infradead.org" , "k.shutemov@gmail.com" , "mgorman@techsingularity.net" , "vbabka@suse.cz" , "hughd@google.com" , "rientjes@google.com" , "shy828301@gmail.com" , "liam.howlett@oracle.com" , "peterz@infradead.org" , "mingo@redhat.com" , "nadav.amit@gmail.com" , "shivankg@amd.com" , "ziy@nvidia.com" , "jhubbard@nvidia.com" , "AneeshKumar.KizhakeVeetil@arm.com" , "linux-kernel@vger.kernel.org" , "jon.grimm@amd.com" , "santosh.shukla@amd.com" , "Michael.Day@amd.com" , "riel@surriel.com" , "weixugc@google.com" , "leesuyeon0506@gmail.com" , honggyu.kim@sk.com, "leillc@google.com" , "kmanaouil.dev@gmail.com" , "rppt@kernel.org" , "dave.hansen@intel.com" , yuanchu@google.com Subject: Re: [LSF/MM/BPF TOPIC] Overhauling hot page detection and promotion based on PTE A bit scanning To: Raghavendra K T , "linux-mm@kvack.org" , "akpm@linux-foundation.org" , "lsf-pc@lists.linux-foundation.org" , "bharata@amd.com" References: <20250123105721.424117-1-raghavendra.kt@amd.com> Content-Language: en-US From: Hyeonggon Yoo In-Reply-To: <20250123105721.424117-1-raghavendra.kt@amd.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTcRTG+b93Z4vXpfW3qGhh1LrZhTiRSUTR/0uSRER+qZUv+tZcsqVp ENjNSLPmrXTa0syytjAnMe1i81JaaBdNaZJW5KKV5rTsZrFcEvjtx3Oe85znwxFolYudLsj6 A5JBr9WpOQWjGJhUuviBJjcu3JJJQ/bpcMj2dHJQXGnj4H2bk4by7M80DHveIBip/EXD2Xtv WfjZ08yD1b4ZThTe5MA9fIaBKvclHn65/yAoPPeVh9bOUgYGTA4Ed7vmw3CtiwZH2hseOm4X c9Br87FQ9+keDTXldRTUujpZaHeWUFA/UobA7D3Ng+PVeR6as5wUfMz7wUKX/RYL6ZerKGiq dFDQf0cPT2xfeChpGVtoHOmnoOjIGQSDXUM8jP4YO2G1xcP9nxUIih708usWkRPtfzhis9gQ aewfpEmtuYcnJfYkkpnewZPqCg0pu+uhiP36KY7Yh3N40lIwypDat6tJ7suriDx5nkKyjn3m yJC7m9kSEqOIiJV0crJkWBq5SxFfXv+USrROTTEdD0lDFlUGChCwuBL3drxn/3Npu5fxs1Jc jd2X8zk/M2IYHnrchsb1IPyosO+fJ0ScjV93F/AZSCHQYvVk3FZ2cmwgCFPEvfjC7e1+PVhM p3DOYy/tX1CJEbi7d/Qf0+I03N13kfL7OVGDa05JfjlAXIsbH15jxy2rcMatDDTOs7FjoJj2 Z2LRJ+CKPA8/XjoU11e8ZEwoyDyhn3nCCfOELPOErBLEXEfBsj45QSvrViwx7otP1cspS/bs T7CjsYe7cvj3jhr07dnWBiQKSD1JSVblxKlYbbIxNaEBYYFWByt9faY4lTJWm3pIMuzfaUjS ScYGNENg1NOUy78djFWJcdoD0j5JSpQM/6eUEDA9DSXecKjDDFHb5IKita9Njwb2Pt3oSZ+p mz9jedlcS4HVtbV13uTkLMonf98D6rDo3KRnzWtsXMdgKEse+sKd3i3rZ50PdLVHWgJb3D1N gS/kqK6I0Cm7lV5r7JyGTToq6oMm/1x+a0zT4WNy/ruFkbkbg6Ln5tVnxhw9vmGBc3udmjHG a5dpaINR+xelyn3xbAMAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTcRTG+b93R7PXpfliF2kU5KhZFHKELn4IfAmK6IvVh3LU27bUlZuZ BpZWGtm0aWVzXpqpteZIndTUzLxVJlmouFLKNJxg07y1sEzKJYHfHp7n+Z1zPhwGlzSSQYxa kyBoNYpYKSUiRMZNCZvbZDeVW/ry1kGOfgvkjPZSUFhpo2CkswmH8pxvOEyPDiLwVP7C4caz IRJ+fnpFQ4V9H6TnV1EwM+LGwTWdTUC16x4Nv1zzCPLzvtPwpreEgNai1ySMGxwIGpwbYbqu DwdH6iANPfWFFAzY/pDQ6H6GQ215IwZ1fb0kdDeZMWj2lCIwTeppcHy8Q8OrrCYMvt6aJcFp f0xCRlk1Bm2VDgzGnmrgrW2GBnP7AtDqGcOgIC0bwYRzioa52YUVFTYVPP9pQVDwYoCOkPPp 3fMUbyu2Ib51bALn60yfaN5sP8tfz+ih+RqLjC9tGMV4u/Uaxdunc2m+3ThH8HVD4fzNDw8Q /7Yric+6/I3ip1z9xIHAI6IdJ4RYdaKgDd0VLVKVN7/DzlSsTDJcCUhFxZJM5MNw7HaupHuS 8GoxG865ym5TXk2wG7ipjk606Ptxr/OH/3UC2GDuc7+RzkQiBmdrfLnO0qsLAcOsYE9xRfVR Xt+fzcC43I5J3AtI2B1c/8DcP42zgVz/8F3M26dYGVd7TfDaPuxOrvXlQ3KxEsZlPs5EizqY c4wX4gbka1pyhmnJJNMSxLQEMSPCivzVmsQ4hTo2TK6LUSVr1Eny46fj7Gjhq+6n/M6tRT96 IlsQyyDpMnFEaK5SQioSdclxLYhjcKm/+M+wQSkRn1Aknxe0p49pz8YKuha0iiGkgeK9UUK0 hFUqEoQYQTgjaP+nGOMTlIriSlIaPOtlFwqnLH5yy9oJZ3O7KlwZErZ89Xu/tEazc7xizbby 7LTfvRddu08GRsYbrW2sWX6RtPjt1e33P39YzxgO/Rjqts6PRoWQ8eyxvIjVs3s8j8RCltGM vUm/5D58tEqfE+3zJGDIfe6QZqRL/+WCVXGq60n9QevEiMQiJXQqxVYZrtUp/gIletfMUQMA AA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9357540005 X-Stat-Signature: hraxj5jypb8nky7oe1qioeeyfa9z1crk X-Rspam-User: X-HE-Tag: 1737697991-613712 X-HE-Meta: U2FsdGVkX1/jm9JdHzQjF021M/hA3hIUHzgSHeiCfXAlOWGUhiqCyAMkbbiD5VguMoypCJjtO5df24zK48ihaRwsi82nYb8TLidEe40nOMovoIxTM3qqRZa6wEsBYWtQUKuiEgBWkbZpRPg3lU7cuyBYPTrLPFhgwaZuYklvaAf2Nq0SCTk7OfZmS3Ya5dotgc3vNsZpyHJncPEpcX4mn1jt/MWcmDtZcXQK7TTJG0FWAt8Ke5uKC7GLNly8doM5If9efUe1ZPQ8x5+6ITKU1g3jfKHL3d787buyG0NjbzOIPnjaqMk5a+sO6Qux9hQ/aH7YA4FvZV2QqkERZBa9j5vxN3Aut04A8Ebj02WWfwTCCj4lDAEBhuGuSV05bsymv63Omq23+NOaw6xX1HugdS0XMu+OEdCh1Pq/OycnMbRL9MzNfbH9qp1quCHdxZ/KSMEYMR+8OvIK212xLfjxheK3hNaYJvZw+DQ/88AZ+wceSKhxoKP/QB/IUIK9WOULhYT2/obx/gBqy+rTtbx1Q0ISldyInmUxTGM/iCk1bxYC8t/K2Z0kN8SF4mvLpd0AupEeG32xYSIJM6qxaDLnGpCjI8MuAro/q0eVicKe45PsO7ve3uqYAtUJxj7Rxo/3ZUszPZGZvDKGLljAMBlmxWcpHmpCj+K7zp5i7T5ikNftVbvejUrtFZK0USQMB1J4hCJILeSxyS1rdtDp7ngjfRyPGmAEfOZ4NaQtRopmoe5X+5bHTDHi5cPBWYfh5f1Mefze57/MdFd24O18OP4Emfjbzj+rcuz4aehgbApr0ZpEHOwrYPgXw9McCHW8yNQWRCB3mAzOQ+Lug8BYjVCji7NoK6JLKt040pPXctbARu5hdQgTd62LgaDvrVzZEQoF6YaXNnpnwHmQFLPkPrpLcx/1jZiHBmCLcB1D0qY1llBqA2MXQzSZ4LXvFYCEY27DQwvXIs2iTR4pL0w48Q4 UXjnzSk1 yYb0VRgbl391fWNjHopRkGB2yv9nJuqXcluJGOlmuCclbNkBb4VdfsjMtlvD/Yldf8GoeK0RYOt9mHHKlYgA0CXoDyOlok4ydh5XG25Gq359//2j+3SVqMxH9P6FUg7zpqcA06ac0FowqN1Zxk1gF1exGQAPpYNk7igjuIBesdJ91IXg+O3nPuZewc4lrxpkGRjHRUagdbuKepZXs1K8Y+TJ/rzK1KgACCjwKaWvRf47FE31sdYUVUnz7oCchZ1GU7MxT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 1/23/2025 7:57 PM, Raghavendra K T wrote: > Bharata and I would like to propose the following topic for LSFMM. > > Topic: Overhauling hot page detection and promotion based on PTE A bit scanning. > > In the Linux kernel, hot page information can potentially be obtained from > multiple sources: > > a. PROT_NONE faults (NUMA balancing) > b. PTE Access bit (LRU scanning) > c. Hardware provided page hotness info (like AMD IBS) > > This information is further used to migrate (or promote) pages from slow memory > tier to top tier to increase performance. > > In the current hot page promotion mechanism, all the activities including the > process address space scanning, NUMA hint fault handling and page migration are > performed in the process context. i.e., scanning overhead is borne by the > applications. > > I had recently posted a patch [1] to improve this in the context of slow-tier > page promotion. Here, Scanning is done by a global kernel thread which routinely > scans all the processes' address spaces and checks for accesses by reading the > PTE A bit. The hot pages thus identified are maintained in list and subsequently> are promoted to a default top-tier node. Thus, the approach pushes overhead of > scanning, NUMA hint faults and migrations off from process context. > > The topic was presented in the MM alignment session hosted by David Rientjes [2]. > The topic also finds a mention in S J Park's LSFMM proposal [3]. > > Here is the list of potential discussion points: > 1. Other improvements and enhancements to PTE A bit scanning approach. Use of > multiple kernel threads, throttling improvements, promotion policies, per-process > opt-in via prctl, virtual vs physical address based scanning, tuning hot page > detection algorithm etc. Yuanchu's MGLRU periodic aging series [1] seems quite relevant here, you might want to look at it. adding Yuanchu to Cc. By the way, do you have any reason why you'd prefer opt-in prctl over per-memcg control? [1] https://lore.kernel.org/all/20221214225123.2770216-1-yuanchu@google.com/ > 2. Possibility of maintaining single source of truth for page hotness that would > maintain hot page information from multiple sources and let other sub-systems > use that info. > > 3. Discuss how hardware provided hotness info (like AMD IBS) can further aid > promotion. Bharata had posted an RFC [4] on this a while back. > > 4. Overlap with DAMON and potential reuse. > > Links: > > [1] https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/ > [2] https://lore.kernel.org/linux-mm/20241226012833.rmmbkws4wdhzdht6@ed.ac.uk/T/ > [3] https://lore.kernel.org/lkml/Z4XUoWlU-UgRik18@gourry-fedora-PF4VCD3F/T/ > [4] https://lore.kernel.org/lkml/20230208073533.715-2-bharata@amd.com/ > >