From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D388C48BF6 for ; Fri, 1 Mar 2024 00:35:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB4796B00B0; Thu, 29 Feb 2024 19:35:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A64BF6B00B2; Thu, 29 Feb 2024 19:35:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92CC36B00B4; Thu, 29 Feb 2024 19:35:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 823286B00B0 for ; Thu, 29 Feb 2024 19:35:15 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 39322120FA5 for ; Fri, 1 Mar 2024 00:35:15 +0000 (UTC) X-FDA: 81846600990.01.4A1BFB9 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) by imf28.hostedemail.com (Postfix) with ESMTP id 8E7ABC0002 for ; Fri, 1 Mar 2024 00:35:12 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="A/N8WcNX"; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709253313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OetB1tcYOFnjN6t/6RngLGag3XA6vggjbInpmmsaKcE=; b=YLkmL5bz6yUA/7Z59brM2gDf+f6uJJljynUZ9dsETCwV0vIVBUqnWH++rHr+MfJgEaiEKi B93D13lDrEdOY8LT0bhJuPz4sJrYL0W30Djs5IHXgcY3gVlwbeRFfwe+AN/80FbRfa1Dt/ se5bZWlI3YiZxcPPWDvIbGce9KIZ5IQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="A/N8WcNX"; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709253313; a=rsa-sha256; cv=none; b=8J87A3yHXHpu4/kWjVBvgvjWILGoB8VG/4aDU/ukD2yVMXhcGlZkOA9ekbQfvk89GW260F jbRZ7tOWzyZYKsNFLGRegVk65gNVZj65fp3YrLGBzkEfCwyS0VKlX57PybYFt1rqudZJff XGfYMQgh2oTHJXQPfZZQm0Y3jP5TV5E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709253313; x=1740789313; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=ZD9CfVPw76C//U5gqjrmuX+/zXDZnhaO8eUOwWKoqXM=; b=A/N8WcNXvKCiQU7z3XuKoRbdvPm6nJA75mkra98y2ZO9U/8VZsXB4/XH v7GEI0XK/fzr52Ve53T1zf2p+oMEEpnMphr1L5dYDgpp8wcvK7/dJr/9k oVeJGbyOgdb1UZ9hyzJOjLSSovdmLMc1GMxcafjv3vprpKP54tp07+Aqi 2cIyMmVzbM1Gd5sApxxd3DCr1tBwaW7Kig0iyiidAApYBpN6tPOjF1XrK r1H2QonCNEVQFyPSf2TY4aUMqr1U9vw5kVAq/ch0IK/kxUM6B7trtPr8u Y7in4VoPN2CkE2BPLOe72yuQb9rx9VJS+mbyywVdr/rkLR7CyJyiWEdSA Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10999"; a="15184502" X-IronPort-AV: E=Sophos;i="6.06,194,1705392000"; d="scan'208";a="15184502" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Feb 2024 16:35:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,194,1705392000"; d="scan'208";a="12599031" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Feb 2024 16:35:07 -0800 From: "Huang, Ying" To: David Hildenbrand Cc: Byungchul Park , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: Re: [RESEND PATCH v8 0/8] Reduce TLB flushes by 94% by improving folio migration In-Reply-To: <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com> (David Hildenbrand's message of "Thu, 29 Feb 2024 10:33:44 +0100") References: <20240226030613.22366-1-byungchul@sk.com> <20240229092810.GC64252@system.software.com> <54053f0d-024b-4064-8d82-235cc71b61f8@redhat.com> Date: Fri, 01 Mar 2024 08:33:11 +0800 Message-ID: <87wmqmbxko.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 8E7ABC0002 X-Rspam-User: X-Stat-Signature: 8ysgkh9ipe57xi3aa8bdi5rmeb1ubcw9 X-Rspamd-Server: rspam01 X-HE-Tag: 1709253312-964725 X-HE-Meta: U2FsdGVkX1+/vcUCwl3QSWmUoiDdUSiqrFc+QHHeAEjr50IlNL8ieQ0gbZrtu/TSAGf2iN0y3fxkKycWXtbQgTDkAgdEt6aMklC8i9TiiJ6LVzL2bWkLwsOTEzdhdXF596k1v5yjjAvxXFGxNNjYLZuVznjWvn3fLTm2/TEEo+ocMPo30hPqkL7TbN5aq/vSsanEB7TrbV4tw5T3GfNVxZkYQz1ruhStw2uC/sXvl7tN+5KXyzGlDAvuC2Imf+FZKae/FQVg8bZ6GFhKeFwsLswKS4dNNzfA7Y8B40yzkZVO5aZYOAnaChpX2YnPj+ehNLd1T4aG5u3z2u0Y6g5XdYi8hsJ998h8nOrnfcMTIJ+cUP8zpFlQMWakkIlYVWWjOko+7RRgVpgCaCIvhKQskK80SM1atmKeK4gEU8eOaV8NyCH3qSIB64GtYPH5QEQBnWhjQmtASw9tDhJQyNEOUbaGFzwcmc5Skwr+Q5CKaWc2w5rIXjZe1hAM6A/AGj22QHPby9p3fq14nPBCeo0zs+RVfjUZqierwszjYLLXYxUbDh/Z9UzyqSra+CXXXnzZ27v4eJYeuQUVwqxu6WF9kqmWrbfaZ2UpQOQH3E2bwYhvbcn9KSPW32h9H0lp69xKXl9Hm9qvtxlIQ8ZpYtDD105sfDCqGaLuEY8gJzkZ8x9C7+anMod6naT34lZ7v8lo8F4U1uQVLWxGCOZCppqcl4IVpSlKTjn/boZzULLcsqMBYeE3bhKAKr9SvSqziMT0wOLXTLK67q35r0oA6O6NFIt1f8oyi/4WugC4uef9fTqRxq76hVVeuslI01dE47TVeNw979LmQuuogbvTVM2fAb7d/k+fTNFpwSF5ivZt2AReAY3NQqT42Dbozezv7UItbP6QN3QLIeKsbgzbTLILdbT/xCejhZ906ZZxpkIm2SZ2uFJdoMMY31mmyJrvf5J2pOMCPQSY9f3bG1UQB4t qMBZE2Oj XlclFeEj+zf4IfPSypHAxcy7S8xZJJehgTZsy2ZtoNpt5HF5rXfdC1/RoCAPHemZqy34IolBrQvzJ8j9iUCGTl6RkaU2UGccflDf5Jwkpjb5rKxh/p0czo5DdZJHPtClX2MdX6+oJpRzPo+JfvN15u7vdMHhRX93Z0nEyZ1aJf693heJttSA8L2+8RWT/NS4G4gIyQs4nO1/qsKwOtuD4hYhny6vI7xN3RchTjd6W4g917LK1NsImamWsvaXszZCIfESJjSwG9/F33JY5Xpu7UQlqMwZUYCaLEn4N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: David Hildenbrand writes: > On 29.02.24 10:28, Byungchul Park wrote: >> On Mon, Feb 26, 2024 at 12:06:05PM +0900, Byungchul Park wrote: >>> Hi everyone, >>> >>> While I'm working with a tiered memory system e.g. CXL memory, I have >>> been facing migration overhead esp. TLB shootdown on promotion or >>> demotion between different tiers. Yeah.. most TLB shootdowns on >>> migration through hinting fault can be avoided thanks to Huang Ying's >>> work, commit 4d4b6d66db ("mm,unmap: avoid flushing TLB in batch if PTE >>> is inaccessible"). See the following link: >>> >>> https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/ >>> >>> However, it's only for ones using hinting fault. I thought it'd be much >>> better if we have a general mechanism to reduce the number of TLB >>> flushes and TLB misses, that we can ultimately apply to any type of >>> migration, I tried it only for tiering for now tho. >>> >>> I'm suggesting a mechanism called MIGRC that stands for 'Migration Read >>> Copy', to reduce TLB flushes by keeping source and destination of folios >>> participated in the migrations until all TLB flushes required are done, >>> only if those folios are not mapped with write permission PTE entries. >>> >>> To achieve that: >>> >>> 1. For the folios that map only to non-writable TLB entries, prevent >>> TLB flush at migration by keeping both source and destination >>> folios, which will be handled later at a better time. >>> >>> 2. When any non-writable TLB entry changes to writable e.g. through >>> fault handler, give up migrc mechanism so as to perform TLB flush >>> required right away. >>> >>> I observed a big improvement of TLB flushes # and TLB misses # at the >>> following evaluation using XSBench like: >>> >>> 1. itlb flush was reduced by 93.9%. >>> 2. dtlb thread was reduced by 43.5%. >>> 3. stlb flush was reduced by 24.9%. >> Hi guys, > > Hi, > >> The TLB flush reduction is 25% ~ 94%, IMO, it's unbelievable. > > Can't we find at least one benchmark that shows an actual improvement > on some system? > > Staring at the number TLB flushes is nice, but if it does not affect > actual performance of at least one benchmark why do we even care? > > "12 files changed, 597 insertions(+), 59 deletions(-)" > > is not negligible and needs proper review. And, the TLB flush is reduced at cost of memory wastage. The old pages could have been freed. That may cause regression for some workloads. > That review needs motivation. The current numbers do not seem to be > motivating enough :) -- Best Regards, Huang, Ying