From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 703DBC021B2 for ; Sat, 22 Feb 2025 12:29:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D01806B007B; Sat, 22 Feb 2025 07:29:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0D06B0083; Sat, 22 Feb 2025 07:29:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9F586B0085; Sat, 22 Feb 2025 07:29:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9BFB16B007B for ; Sat, 22 Feb 2025 07:29:39 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 24EC31A1492 for ; Sat, 22 Feb 2025 12:29:39 +0000 (UTC) X-FDA: 83147511678.08.322A5DC Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf23.hostedemail.com (Postfix) with ESMTP id 8896B140012 for ; Sat, 22 Feb 2025 12:29:37 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lpLxVcDZ; spf=pass (imf23.hostedemail.com: domain of mingo@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=mingo@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740227377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PsEVVfrL1juTKN73ZwdK5NmpYjN2EVHd2jbKqXTEAHg=; b=o5Ous96iiBS8G4iDWNdYABS1yPveRD10QuU7JoJvG3CpmuPttSE0+GtfKpqk4TI1HT2D/C 7jZM2KIc5VUyuhrrX1udTKOJ76WA0XS9Y52cbrcxyX9LKYTUF7j/l7FNG/na630pAuewIo U0YBFFtcfo7N525W+w22b2Yu4jimbRI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lpLxVcDZ; spf=pass (imf23.hostedemail.com: domain of mingo@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=mingo@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740227377; a=rsa-sha256; cv=none; b=MaJt+VZO6okjh8139VIoZiC4xX9JZOVt7Q26vbGwgCUlZgtkj6dqzdvp2q0R/MdWGG5ZcK S9FghAPDJHBU6+G+sSqsIgT9uvqJco4+T2J37wp2aBITESlbOnHjTzoca5smvc+N98F6bz y5ro1zS2mHg2uUOUJGR+pcWkLisU3CI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 6513B61148; Sat, 22 Feb 2025 12:29:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6ECBC4CED1; Sat, 22 Feb 2025 12:29:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1740227376; bh=SSGSmoHatyl6HQLxw8fOkh4lN56WlcFQLHj1JxpvagQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=lpLxVcDZtSEnc2/4o39uLntPR3zWzkfv7KPT6J9r3JTW6sMirnbLY9YErrscDTsvO DN3LRsX67sK165Sv73Y+EZpsWd0lw/GDgZJLkQh5v5htZLCOJ+7ZCEroexeXPH6nJg BVZRwwaUF2fv8fe8pcIvlwMzdrmPLV6Y9EA9qyX+Kx0qFG0lLePhOqsP1w3VsnY/ce I/tlSOkPc4cneQPTc5hSJdef/SK2ra0n9puV25X+c1MfeloPatxqt1vCg55Oae8WB0 HBd0VftTlqM//MsUx80cQUjaHewOBL/bKmj7q5LspuJx1qs+IVUU+ZBIXWytHkhXaU zxO1OHtLq/LXw== Date: Sat, 22 Feb 2025 13:29:25 +0100 From: Ingo Molnar To: Manali Shukla Cc: Rik van Riel , x86@kernel.org, linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com Subject: Re: [PATCH v7 00/12] AMD broadcast TLB invalidation Message-ID: References: <20250123042447.2259648-1-riel@surriel.com> <93792758-bc88-4d84-bdea-f366988c2d53@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <93792758-bc88-4d84-bdea-f366988c2d53@amd.com> X-Rspam-User: X-Rspamd-Queue-Id: 8896B140012 X-Stat-Signature: efyhfjfznxazxjiqesqsryxzuu3ej76w X-Rspamd-Server: rspam03 X-HE-Tag: 1740227377-907722 X-HE-Meta: U2FsdGVkX18moTHoFs4Bwx5ZIfpN2K2cbgRACq3sNW18Q9+hpYEIKQcNw/bi1pZ/xd/unANjdZyEQIw05de41c7zuqgRbhkBIgM+b8ieifDAgWXOq/3LjP5K8Dt/MAJtTGwfnmoaEQfPkWEWjntE/IuOzjdQWRmGBGYGEzUWzkEwKgs0CSqSrg3/IyeNA3cIaIzL1B4pr0YilDRAC0Q0LylIpxUNHNFUT6dsTbrRizv+6fYu/kUdRv+5t/5+KzdpPvrdDsZfAgev52pw0dPjzZMydX+HOh+Lug6nDBxgfM56t10FB6MOuiWEHxWwtro6qewVnUbL4NN04jRCA96UHEEAacBGlg/PljqiFeF9lgo5OLw/DZTFCvp2BdHJFLBaGobXdQdPnyNgK378G2EfC0qSrmtxT298ZMkBW/e5/29vfv+Stpu+5nQ/ecI/Xs/aYD1KzOBsebEmJsm2yqZ4VoWIwtuK1J1zf2dWr3LosiD25bnI+pinB2esuus+gNxZkmW70eHElaVfHbwoQXPvCC5SLugkODX7L5brMyQgbzArWHAz5ub2KhWygCdYfhRmexlKWhwvltHj1ZU7LUbi+XUh1rM37HLtQZhY1fM+YWd8uGIFYTf6GAyn18iJxzMpM33pn+G1OibCMnx3l7Jzi+Z7Af5Jwn/sMP8cQFg0BMEXGywkEfxj18yPVhWajUG8rp5U6dcq/fiVK3/+BYcP4z65dFvDnpqJTwh0+7zbyX/3ejYcabsKO63mZr8BM7lqvA5JdPf6DvMv1IrADEVnH+xZMNu7d7zdxTtmWq7MiY2UU2TzRWqCc8a4H3g/u5sRvBI5w9CeO5SA71SKPG8TcaO+9M4PtbEgdrZV2mzAB4X3j7j+7xItnLmSlnAnaRPCJV4rfx+/VYFrIQ3ltTZ04e424flwLXb7z6paHjRzDyyvTuoIrp5GFITH60s1AyR7AxoS9tWHvl4NHi4gytW 5nA2fvyY y7d8QoOMWadR90GWECMStWZNUybnaQgm7pT3mCejjI/+OTe+t3D3WYzyHWv395g+lxyypuUAXCAo+lFVRoEx/oePTuNRLYwl4g4JwB3WXeQOy+An7pmBqqwbJodHh0MrvpmQWDw95bHs+LKSA2AQNlKOp13RhXt47cyLjy402g6ajnsfxEM47KpZsQO4H1hyZh7RzIGzVUNyuNrmNwWsIkC0olVSMDT2yTMDvQJqgGH3DpA15GnpqSdCBFitx0a4b9ge529tnhQXaKWob1UKszB5xiW6d5fl90b0iDUq8jwGIq1PWdB75PCDNCt0hSisUoM7JocjrT7YbRowR9FYRwHen9yVRBoYLaPdM26r088ptIEtctKZaAjGkbyCnsIRiB+5Pp4xfQ9Vx8pxret028RlkoHPpgmeBUuaPWonGNk5NwJw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: * Manali Shukla wrote: > On 1/23/2025 9:53 AM, Rik van Riel wrote: > > Add support for broadcast TLB invalidation using AMD's INVLPGB instruction. > > > > This allows the kernel to invalidate TLB entries on remote CPUs without > > needing to send IPIs, without having to wait for remote CPUs to handle > > those interrupts, and with less interruption to what was running on > > those CPUs. > > > > Because x86 PCID space is limited, and there are some very large > > systems out there, broadcast TLB invalidation is only used for > > processes that are active on 3 or more CPUs, with the threshold > > being gradually increased the more the PCID space gets exhausted. > > > > Combined with the removal of unnecessary lru_add_drain calls > > (see https://lkml.org/lkml/2024/12/19/1388) this results in a > > nice performance boost for the will-it-scale tlb_flush2_threads > > test on an AMD Milan system with 36 cores: > > > > - vanilla kernel: 527k loops/second > > - lru_add_drain removal: 731k loops/second > > - only INVLPGB: 527k loops/second > > - lru_add_drain + INVLPGB: 1157k loops/second > > > > Profiling with only the INVLPGB changes showed while > > TLB invalidation went down from 40% of the total CPU > > time to only around 4% of CPU time, the contention > > simply moved to the LRU lock. > > > > Fixing both at the same time about doubles the > > number of iterations per second from this case. > > > > Some numbers closer to real world performance > > can be found at Phoronix, thanks to Michael: > > > > https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits > > > > My current plan is to implement support for Intel's RAR > > (Remote Action Request) TLB flushing in a follow-up series, > > after this thing has been merged into -tip. Making things > > any larger would just be unwieldy for reviewers. > > > > v7: > > - a few small code cleanups (Nadav) > > - fix spurious VM_WARN_ON_ONCE in mm_global_asid > > - code simplifications & better barriers (Peter & Dave) > > v6: > > - fix info->end check in flush_tlb_kernel_range (Michael) > > - disable broadcast TLB flushing on 32 bit x86 > > v5: > > - use byte assembly for compatibility with older toolchains (Borislav, Michael) > > - ensure a panic on an invalid number of extra pages (Dave, Tom) > > - add cant_migrate() assertion to tlbsync (Jann) > > - a bunch more cleanups (Nadav) > > - key TCE enabling off X86_FEATURE_TCE (Andrew) > > - fix a race between reclaim and ASID transition (Jann) > > v4: > > - Use only bitmaps to track free global ASIDs (Nadav) > > - Improved AMD initialization (Borislav & Tom) > > - Various naming and documentation improvements (Peter, Nadav, Tom, Dave) > > - Fixes for subtle race conditions (Jann) > > v3: > > - Remove paravirt tlb_remove_table call (thank you Qi Zheng) > > - More suggested cleanups and changelog fixes by Peter and Nadav > > v2: > > - Apply suggestions by Peter and Borislav (thank you!) > > - Fix bug in arch_tlbbatch_flush, where we need to do both > > the TLBSYNC, and flush the CPUs that are in the cpumask. > > - Some updates to comments and changelogs based on questions. > > > > > > I have collected performance data using the will-it-scale > tlb_flush2_threads benchmark on my AMD Milan, Genoa, and Turin systems. > > As expected, I don't see any discrepancies in the data. > (Performance Testing is done based on 6.13.0-rc7). > > ------------------------------------------------------------------------------------------------------------------------------------------------ > | ./tlb_flush2_threads -s 5 -t 128 | Milan 1P (NPS1) | Milan 1P (NPS2) | Genoa 1P (NPS1) | Genoa 1P (NPS2) | Turin 2P (NPS1) | Turin 2P (NPS2) | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | Vanila | 357647 | 419631 | 319885 | 311069 | 380559 | 379286 | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | LRU drain removal | 784734 | 796056 | 540862 | 530472 | 549168 | 482683 | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | INVLPGB | 581069 | 950848 | 501033 | 553987 | 528660 | 536535 | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | LRU drain removal + INVLPGB | 1094941 | 1086826 | 980293 | 979005 | 1228823 | 1238440 | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | LRU drain vs. Vanila | 54.42% | 47.29% | 40.86% | 41.36% | 30.70% | 21.42% | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | INVLPGB vs. Vanila | 38.45% | 55.87% | 55.87% | 43.85% | 28.01% | 29.31% | > ------------------------------------------------------------------------------------------------------------------------------------------------ > | (LRU drain + INVLPGB) vs. Vanila | 67.34% | 61.39% | 67.37% | 68.23% | 69.03% | 69.37% | > ------------------------------------------------------------------------------------------------------------------------------------------------ > > Feel free to add: > Tested-by: Manali Shukla Great data! Could we please add all the scalability testing results to patch #9 or so, so that it's preserved in the kernel Git history and provides a background as to why we want this feature? Thanks, Ingo