From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3F94C48BF6 for ; Mon, 4 Mar 2024 02:39:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 327726B009E; Sun, 3 Mar 2024 21:39:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 288F06B009F; Sun, 3 Mar 2024 21:39:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DBEE6B00A0; Sun, 3 Mar 2024 21:39:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EC6216B009E for ; Sun, 3 Mar 2024 21:39:49 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B197E1406A9 for ; Mon, 4 Mar 2024 02:39:49 +0000 (UTC) X-FDA: 81857801298.24.0F66B6D Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf04.hostedemail.com (Postfix) with ESMTP id 3B55240005 for ; Mon, 4 Mar 2024 02:39:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709519988; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UMPwjUrLS3I9dz2fEzg1h5GaMisteGMgmYHsibKgeAA=; b=G3EiMhoR9alo13IRMHiblPsaY9ayAQaKz07VpqR77Vbqj1Y9WdNBkSxa9FMcWdKLxoIPZr 0K4jVO6YIDouh28CHhP4t4SeG2cNj/9jCIgIxENyHXLCa7Tth+4j5zqNTge56XoQQxgdWQ W9cyA33MMRbAwAoprKN40yyRfqgJhlI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709519988; a=rsa-sha256; cv=none; b=Stv948qEJtgi2WxWDpQqdkkua9rx4kCNbEyzC+2LEcOI+XA/KcL+E3vV1bBhUwM/lLHJfz qXVf1Zz+rYfBh9fxUjv/dzt2vbvL8Wuwt97J2mMxK2J2CmDmW699iERoCTxie61eYav90K na2V5Oj7Jz7fi5xgd+/+jr1rrLcsoEQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-2a-65e5346b883a Date: Mon, 4 Mar 2024 11:39:34 +0900 From: Byungchul Park To: David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: Re: [RESEND PATCH v8 0/8] Reduce TLB flushes by 94% by improving folio migration Message-ID: <20240304023934.GA13332@system.software.com> References: <20240226030613.22366-1-byungchul@sk.com> <20240229092810.GC64252@system.software.com> <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrHIsWRmVeSWpSXmKPExsXC9ZZnoW6OydNUg4NbxC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG nb9PGAu+ild8vXOeqYFxiVAXIyeHhICJRPe1M4ww9qennSwgNouAisT/CQvB4mwC6hI3bvxk BrFFBDQkNrVtALK5OJgF3jJJTJnTzAaSEBaIlljWNYUdxOYVsJC41TKDDaRISGAGo8Tq5ivM EAlBiZMzn4BtYBbQkrjx7yVTFyMHkC0tsfwfB0iYU8BO4t+kc2CLRQWUJQ5sO84EMkdCYB27 xLQjN9ghLpWUOLjiBssERoFZSMbOQjJ2FsLYBYzMqxiFMvPKchMzc0z0MirzMiv0kvNzNzEC Y3FZ7Z/oHYyfLgQfYhTgYFTi4c3ofJIqxJpYVlyZe4hRgoNZSYS35hdQiDclsbIqtSg/vqg0 J7X4EKM0B4uSOK/Rt/IUIYH0xJLU7NTUgtQimCwTB6dUAyPnZ62GOPkPrPNPlnhGCKx+/Gtv 8bfXU+3/1jzNLNizUdydv6fU8OfNKY2hVRwxD3iWcjKnGl//cmUNY13xkTXP5L/ZTWXM6TMO 33bt6YO3HWdK3HvrqxT5e4P+OBvE7Pzh+3Jzsf+7mIk+wdk//ITns510M2kPbinek/liPeeM 5O3yyqXSVUosxRmJhlrMRcWJAMUelSDBAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA02RW0hTYQDH+c45OzsbGx6XsQ+Foiko81oofFBI1EMfGaIgBBW0Qx5yOadt KioFmmUoJfMymHOVZanpwu2MTEVkTGTtQcUskTAhcKQPteWFzMtsMyLffvwv/B/+DKkwU7GM Vl/OG/ScTkVLKWnu6frU4kw/n+EJKpFt0E6jdUeIRiuOhwBtDm6TyL/WTKGJJz4Rmhu10WjJ vi9CM6NvRGj41TiBPri7COR97CbQs6V1GrkEM4m87WaAdrbCYZ+1jTobjX89aKbwiPWLGHcJ FdjVp8bdY6sEFvobaSystYrxj+lpMX5v2aHwU18+Dox/onH3SpDAb7ezcZ15nsbrwrG8qCvS M4W8TlvJG9KzNdKixb1lULaprNpcnCFqwUtFE5AwkM2Ea/5GKsIUmwD3Tc9BhGk2ES4s/CYj HMMmQaHBEWYpQ7LfCdhuq6cjxhH2KuxpahdHWM4i+Pm+hY6EFKwFwIH6j+RfIxr6OpYPFkhW DRdCq0QTYMIcB3tDTESWsNkw1Dp9MHyUjYfuIS9hAnLrobb1UNv6v90FyH4Qo9VXlnBaXVaa sbioWq+tSrtRWiKA8Fk9d3dbhsHG3AUPYBmgkslN9mVeIeIqjdUlHgAZUhUjv7MdluSFXHUN byi9bqjQ8UYPiGMolVJ+8TKvUbA3uXK+mOfLeMM/l2AksbXgxey5lsmMLXNV51TAXRC1l0K9 RhqX83jcKVuSN2+kISEA+F0mpzM5q6CtJsXpHW2+Z42/xLWdJ775TROWTHvLu15DzteUxHFZ GROQBpduKedvD6RKnBVD6QV1IadRNqvuS/Y74aP8zj7Zz2saamrsxEZH0J7umlDZcydVlLGI O6kmDUbuDyUPXjKoAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 3B55240005 X-Rspam-User: X-Stat-Signature: 4k3ki5ueazeaazthdsgnkqkimbodehnn X-Rspamd-Server: rspam03 X-HE-Tag: 1709519983-576774 X-HE-Meta: U2FsdGVkX1/OMshvcJGRpPnm9JHCGDWBLYO1t5ri0oIUtmYynFPGCX7nYCV+jEw1jw5wh9Q3DTcJp/hXobUUTxrosujW2HEd3fyMAAbt+0hhiweXAo3rM365S0knVXDMOaHYiAQYff55QUNkphj3kpXbVnvTFHj5fxO/GtwKX/Ur8p6o0rmAVbhf7UE8/1zjtJClq6FH/FH6pmKqRuAk8jJQvYuf4oNGt5Da/SCc1ELgBDF5FNCUzew12+v1P30/ZOlNfUn82je/flNXhifOYSHdokU5kQ7JkLIIJxCuE3MgRMKPMvqlgj9y/Xik3OzJQU0MzsBN2Y1lbbSKlfYBjzbbm3O212f1FkCE4u3fSkhfvPR8MlIpzplwZXnZe7jn0DVYAfoaWx/0/kxmRfUbd5Qk3a7oNUTCJHA4O8gUQ5ncndF1JEUQbdKGGvWtt8wuDw2nwVzFpxXqI5c1Vz8pp1icry98XdDtW8NBIcuPunQnJaB1kfBSMkv86cuR7Ynno6280PaiYkMHn9uCb778xcKP4vQHXZ3j+f19hMXw+GM4BBttHswEp/ujr94H+npYofQfCbbk6/TJk2aluxu/oghZfh985qI/wd3ZemQCqoKLull62lETy9WkSB8o4cFcDdcSmUV8TlFlp9mWnH+W3n2lBgtXR9akjyc340mQUZ0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Feb 29, 2024 at 10:33:44AM +0100, David Hildenbrand wrote: > On 29.02.24 10:28, Byungchul Park wrote: > > On Mon, Feb 26, 2024 at 12:06:05PM +0900, Byungchul Park wrote: > > > Hi everyone, > > > > > > While I'm working with a tiered memory system e.g. CXL memory, I have > > > been facing migration overhead esp. TLB shootdown on promotion or > > > demotion between different tiers. Yeah.. most TLB shootdowns on > > > migration through hinting fault can be avoided thanks to Huang Ying's > > > work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE > > > is inaccessible"). See the following link: > > > > > > https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/ > > > > > > However, it's only for ones using hinting fault. I thought it'd be much > > > better if we have a general mechanism to reduce the number of TLB > > > flushes and TLB misses, that we can ultimately apply to any type of > > > migration, I tried it only for tiering for now tho. > > > > > > I'm suggesting a mechanism called MIGRC that stands for 'Migration Read > > > Copy', to reduce TLB flushes by keeping source and destination of folios > > > participated in the migrations until all TLB flushes required are done, > > > only if those folios are not mapped with write permission PTE entries. > > > > > > To achieve that: > > > > > > 1. For the folios that map only to non-writable TLB entries, prevent > > > TLB flush at migration by keeping both source and destination > > > folios, which will be handled later at a better time. > > > > > > 2. When any non-writable TLB entry changes to writable e.g. through > > > fault handler, give up migrc mechanism so as to perform TLB flush > > > required right away. > > > > > > I observed a big improvement of TLB flushes # and TLB misses # at the > > > following evaluation using XSBench like: > > > > > > 1. itlb flush was reduced by 93.9%. > > > 2. dtlb thread was reduced by 43.5%. > > > 3. stlb flush was reduced by 24.9%. > > > > Hi guys, > > Hi, > > > > > The TLB flush reduction is 25% ~ 94%, IMO, it's unbelievable. > > Can't we find at least one benchmark that shows an actual improvement on > some system? XSBench is more like a real workload that is used for performance analysis on high performance computing architectrues, not micro benchmark only for testing TLB things. XSBench : https://github.com/ANL-CESAR/XSBench Not to mention TLB numbers, the performance improvement is a little but clearly positive as you can see the result I shared. Byungchul > Staring at the number TLB flushes is nice, but if it does not affect actual > performance of at least one benchmark why do we even care? > > "12 files changed, 597 insertions(+), 59 deletions(-)" > > is not negligible and needs proper review. > > That review needs motivation. The current numbers do not seem to be > motivating enough :) > > -- > Cheers, > > David / dhildenb