From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06728C25B77 for ; Wed, 22 May 2024 07:40:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69CB36B008A; Wed, 22 May 2024 03:40:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64BEF6B0092; Wed, 22 May 2024 03:40:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C5716B0093; Wed, 22 May 2024 03:40:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2AE0B6B008A for ; Wed, 22 May 2024 03:40:05 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A27191C0811 for ; Wed, 22 May 2024 07:40:04 +0000 (UTC) X-FDA: 82145233128.28.A02BE45 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by imf13.hostedemail.com (Postfix) with ESMTP id DC25420013 for ; Wed, 22 May 2024 07:40:01 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NbefEzPN; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf13.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.19 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716363602; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CK7zWuFZmvhXc0b7hYTYUls5I3hPhU4iRvQ5Hm/Nk18=; b=FqPqCawsNO+nj42lkZQX7ORcODWTewwek6qa2emI0WYRJavQMnsVlQHUOps27AKZRPpRhw rYkq9d18URgfvwJCKcIzAfvlmiPkGEqTicCRIXCdCC5TZ76wuOhb0u5r66TuaraoqVzaCw vdwQW+0HPcPXTTqT9qERHkq/QTH5u3A= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NbefEzPN; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf13.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.19 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716363602; a=rsa-sha256; cv=none; b=WbfzERVJTT0AR/3I+XQDegk23LQKw1Pcxp2NVVHZkIelX2D4u0VHmOtFB8Ce6Cn5XOzfWC eOCHTb/I2HlWCFxRsbybRAITO40ALTOrE4wBT2aceI2GsVDHIlG/o+wZXchd0Li3ad6dAU 4+zfMIdPNaA6sEjRey786EvVr0jdU14= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1716363602; x=1747899602; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=f3Hc/S5iPU1pxS7uFNla+UfZV6h4yjRsbFGxog9sYv4=; b=NbefEzPNKvwZGg4njTbv2WZ2sBg8E67/AOL8ogENgUjWMsMSp3dYhSNd wrMaISp/QTlLMey6/TFx/xRQeQkEKkdCzL649MxO1W8AYL0Y4a6qDh/AW W/9lNqpjVXFZ7p2qVuu/MU4Zn6O7v8Nk3D7V2wZuNjAXbsNXfdj8ts0s6 I5SwSqK1s4yF7fgm9PKtgDZ1igPMaPBtssXSavK2mnhlBWNZHdHxNwa0P znVdJtGcgwIUaEZCRgEqOYsdfgbuj2z4ObXU4rfiNskdrPC+F7DU9hcxA RTOjeZ0QTfLaurTddREXvHAbgAlIJuwg4OlIiUC3wi6WgXoSSxP+kpjSy w==; X-CSE-ConnectionGUID: T1v+dy5uT2uXZVHCFqaOOQ== X-CSE-MsgGUID: xvd3Gv1lRkyTeMZHXEr2IQ== X-IronPort-AV: E=McAfee;i="6600,9927,11079"; a="12443463" X-IronPort-AV: E=Sophos;i="6.08,179,1712646000"; d="scan'208";a="12443463" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2024 00:40:00 -0700 X-CSE-ConnectionGUID: 9A4E8CnSSlSCdnuTwJyWTQ== X-CSE-MsgGUID: OdClcumTQyadA7TS7a61Og== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,179,1712646000"; d="scan'208";a="56430458" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 May 2024 00:39:56 -0700 From: "Huang, Ying" To: Byungchul Park Cc: , , , , , , , , , , , , , , , Subject: Re: [PATCH v10 00/12] LUF(Lazy Unmap Flush) reducing tlb numbers over 90% In-Reply-To: <20240522021616.GA34580@system.software.com> (Byungchul Park's message of "Wed, 22 May 2024 11:16:16 +0900") References: <20240510065206.76078-1-byungchul@sk.com> <87eda8g6q2.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240513014428.GB38851@system.software.com> <20240522021616.GA34580@system.software.com> Date: Wed, 22 May 2024 15:38:04 +0800 Message-ID: <87h6eqb8kj.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: DC25420013 X-Stat-Signature: 7z8xieuat1dyg6buccxt4qgrysb1pbp4 X-HE-Tag: 1716363601-771423 X-HE-Meta: U2FsdGVkX1/tulC53rt9yzKMTlKlG5qphv74goXuQh1f7NnQNq8DR9p494+F7DgkcDvj0hZdOVlA9RPbanQLNbTrjLgFRWcykHmaxVNHy6xQydaXjttE7rErWCVk/tcDFLtsFXjh6eKVBbz8iv2XhzjiVe3eh8spUgSmfC5cRIin5aiP+uj/XdCqy4Mdc1RKiSTbOOu26Cq1g7g7tUuWZW5klYjxw4xq/4212lXMFZpALwQ18PNzLl5Sy1OsPNoEI2YmVbDKGOZa8qIR1uUFTA30meXtXkaqgl6KPwtJnzKXc65D6c4CLK/gBwUcRVZ1/PsK/vinnsU1EqfIQF20f0aw8NCGg8IsFCD16S+pTF4olc1jCQLv+46r93RssbW3joVimzeZ2QrZre4SSyOcUoNmUIYOnNkAtF38K3yrqW8vFwjOcnMjBwKw/mAfKaWMJhIkeh76A6Ly0Lbo+5zTWQbN1tDltkpRDzkin9pi3/DDdi1jDscXnTDlTBH5skT71deb6EnEE8uzfDEW2q70Yy3Al1yHbPsGDGZZJnp3uBkSNiBxKU4KPN1eOUD8yP37NmQEGqs5+HJu8PBz/LA9cblRD6Gjrkg9X4+LJD1PfOoOV6LsqX5VXHZCNcYGfqlfoajzGuqcASlOKyIml4K52A1yAh09CU4JQ82j+mrbYHByC2zri53pAf/KQW69hVHQIkxCOIHZkcPPzD3afzCD2TVszbXkaklQEzTaGFEDdU+Os/4N350bEzhmqLVehEtqXz05bWOarfo4Cz7frsQ5SFgUdiaHLmGUII5N8PhNvP8vkUZ5ECDuY/Fwv3+Yv4AtJIVsovgIntHGlzaQljVscmXQ8A9K2C159ZRakuKHLUI5iCzh8VlTu+9XbqKzYCHNlCtbz6tV9ozu1Q+LtDSQXjSCB6Z39xlhzl07W8OpTK8oHUxc2cZBOw71QjgZW+AmfdLezkWhUGnvKkk+CH0 nT4vPf1H P4oAbG3lGNlJ3RsHQOaueFcihLcciDaCKNshCO+u+/2gfstrGu8jbGqhe/eochVhs6kJQ2L4OTlgaACWFEnYlSyp2IP+OMInKwQOPEzQjEv3r5NxgGGr9XDdAD70Jk343UnEjGdE40HKlDtwYfQrFYuZitmVEnayE2ws9wXozLiX7hx+vBinkRhJK+1Y7sAj0VGFj7MAa+5uJE65o4uj91G44aqIi2YjOTh9PQvdx2fm34AKnTgBuoJF1FK45CMxBocciNU+RH0BMgm2KsO+E7853xwYgmxvDrZ8NwCfIwBMZhN1orXqnoX4a+CIwYDuircejQZ+4S6qhBp0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, Byungchul, Byungchul Park writes: > On Mon, May 13, 2024 at 10:44:29AM +0900, Byungchul Park wrote: >> On Sat, May 11, 2024 at 03:15:01PM +0800, Huang, Ying wrote: >> > Byungchul Park writes: >> > >> > > Hi everyone, >> > > >> > > While I'm working with a tiered memory system e.g. CXL memory, I have >> > > been facing migration overhead esp. tlb shootdown on promotion or >> > > demotion between different tiers. Yeah.. most tlb shootdowns on >> > > migration through hinting fault can be avoided thanks to Huang Ying's >> > > work, commit 4d4b6d66db ("mm,unmap: avoid flushing tlb in batch if PTE >> > > is inaccessible"). See the following link for more information: >> > > >> > > https://lore.kernel.org/lkml/20231115025755.GA29979@system.software.com/ >> > >> > And, I still have interest of the performance impact of commit >> > 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the email above, >> > you said that the performance of v6.5-rc5 + 7e12beb8ca2a reverted has >> > better performance than v6.5-rc5. Can you provide more details? For >> > example, the number of TLB flushing IPI for two kernels? >> >> Okay. I will test and share the result with what you asked me now once >> I get available for the test. > > I should admit that the test using qemu is so unstable. While using > qemu for the test, kernel with 7e12beb8ca2a applied gave better results > sometimes and worse ones sometimes. I should've used a bare metal from > the beginning. Sorry for making you confused with the unstable result. > > Since I thought you asked me for the test with the same environment in > the link above, I used qemu to reproduce the similar result but changed > the number of threads for the test from 16 to 14 to get rid of noise > that might be introduced by other than the intended test just in case. > > As expected, the stats are better with your work: > > ------------------------------------------ > v6.6-rc5 with 7e12beb8ca2a commit reverted > ------------------------------------------ > > 1) from output of XSBench > > Threads: 14 > Runtime: 1127.043 seconds > Lookups: 1,700,000,000 > Lookups/s: 1,508,371 > > 2) from /proc/vmstat > > numa_hit 15580171 > numa_miss 1034233 > numa_foreign 1034233 > numa_interleave 773 > numa_local 7927442 > numa_other 8686962 > numa_pte_updates 24068923 > numa_hint_faults 24061125 > numa_hint_faults_local 0 > numa_pages_migrated 7426480 > pgmigrate_success 15407375 > pgmigrate_fail 1849 > compact_migrate_scanned 4445414 > compact_daemon_migrate_scanned 4445414 > pgdemote_kswapd 7651061 > pgdemote_direct 0 > nr_tlb_remote_flush 8080092 > nr_tlb_remote_flush_received 109915713 > nr_tlb_local_flush_all 53800 > nr_tlb_local_flush_one 770466 > > 3) from /proc/interrupts > > TLB: 8022927 7840769 123588 7837008 7835967 7839837 > 7838332 7839886 7837610 7837221 7834524 407260 > 7430090 7835696 7839081 7712568 TLB shootdowns > > 4) from 'perf stat -a' > > 222371217 itlb.itlb_flush > 919832520 tlb_flush.dtlb_thread > 372223809 tlb_flush.stlb_any > 120210808042 dTLB-load-misses > 979352769 dTLB-store-misses > 3650767665 iTLB-load-misses > > ----------------------------------------- > v6.6-rc5 with 7e12beb8ca2a commit applied > ----------------------------------------- > > 1) from output of XSBench > > Threads: 14 > Runtime: 1105.521 seconds > Lookups: 1,700,000,000 > Lookups/s: 1,537,737 > > 2) from /proc/vmstat > > numa_hit 24148399 > numa_miss 797483 > numa_foreign 797483 > numa_interleave 772 > numa_local 12214575 > numa_other 12731307 > numa_pte_updates 24250278 > numa_hint_faults 24199756 > numa_hint_faults_local 0 > numa_pages_migrated 11476195 > pgmigrate_success 23634639 > pgmigrate_fail 1391 > compact_migrate_scanned 3760803 > compact_daemon_migrate_scanned 3760803 > pgdemote_kswapd 11932217 > pgdemote_direct 0 > nr_tlb_remote_flush 2151945 > nr_tlb_remote_flush_received 29672808 > nr_tlb_local_flush_all 124006 > nr_tlb_local_flush_one 741165 > > 3) from /proc/interrupts > > TLB: 2130784 2120142 2117571 844962 2071766 114675 > 2117258 2119596 2116816 1205446 2119176 2119209 > 2116792 2118763 2118773 2117762 TLB shootdowns > > 4) from 'perf stat -a' > > 60851902 itlb.itlb_flush > 334068491 tlb_flush.dtlb_thread > 223732916 tlb_flush.stlb_any > 120207083382 dTLB-load-misses > 446823059 dTLB-store-misses > 1926669373 iTLB-load-misses > Thanks a lot for test results! >From your test results, the TLB shootdown IPI can be reduced effectively with commit 7e12beb8ca2a. So that the benchmark score improved a little. And, your changes will reduce the TLB shootdown IPI further, right? Do you have the number? -- Best Regards, Huang, Ying