From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ED52C02182 for ; Thu, 23 Jan 2025 12:43:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B35C96B007B; Thu, 23 Jan 2025 07:43:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE5256B0082; Thu, 23 Jan 2025 07:43:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ACB46B0083; Thu, 23 Jan 2025 07:43:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7D9876B007B for ; Thu, 23 Jan 2025 07:43:35 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B977A161058 for ; Thu, 23 Jan 2025 12:43:33 +0000 (UTC) X-FDA: 83038682706.27.3EBABD9 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf16.hostedemail.com (Postfix) with ESMTP id EC2DB180003 for ; Thu, 23 Jan 2025 12:43:31 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737636212; a=rsa-sha256; cv=none; b=eaW2OnSIJxnoPYkRd8ayS98nJ/XaU1Pv6KBoboSG1/BkYVNKNJuqD/kHBrBHackVJCVfAI ydZiZnNpogUl5/tzPSP8nkO4rkwCLCGtBiDs0y9Aujwd1Npfk3XMIi55iY1Xyq+pn61ocG CArbEUQ67G+ts82zUfEuocYWU9aUqUI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf16.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737636212; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e/Jqxk85JjejtfpBPsnKlYVmotJws6uqQaxJ7bF48Uw=; b=yqc919UP0K4Ptv7xYbsOhCz2pYN3E4rE0hbnN7ioy95OVX/DwdvIetJbhCiMblv7eTso+J TKeJ/EQLHf8BHYLcWEYVUN1pWRM6wPW4OfKjqIjp+/r9Nkd6Vj09l+p8IIYR4qUD1yzoul ezbqqKiX0TvAq1ohqXbS9C9fMLAQV5Q= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tawXe-000000003KH-16og; Thu, 23 Jan 2025 07:42:38 -0500 Message-ID: <5bdc9986fb755d84e31d4550a7d2a8ec9e7b0fa3.camel@surriel.com> Subject: Re: [PATCH v6 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes From: Rik van Riel To: Peter Zijlstra Cc: x86@kernel.org, linux-kernel@vger.kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Mark Rutland , Will Deacon Date: Thu, 23 Jan 2025 07:42:38 -0500 In-Reply-To: <20250123090720.GI3808@noisy.programming.kicks-ass.net> References: <20250120024104.1924753-1-riel@surriel.com> <20250120024104.1924753-10-riel@surriel.com> <20250122083835.GE7145@noisy.programming.kicks-ass.net> <5820b18ef0ba48c33a62553fcc444c47f963b907.camel@surriel.com> <20250123090720.GI3808@noisy.programming.kicks-ass.net> Autocrypt: addr=riel@surriel.com; prefer-encrypt=mutual; keydata=mQENBFIt3aUBCADCK0LicyCYyMa0E1lodCDUBf6G+6C5UXKG1jEYwQu49cc/gUBTTk33A eo2hjn4JinVaPF3zfZprnKMEGGv4dHvEOCPWiNhlz5RtqH3SKJllq2dpeMS9RqbMvDA36rlJIIo47 Z/nl6IA8MDhSqyqdnTY8z7LnQHqq16jAqwo7Ll9qALXz4yG1ZdSCmo80VPetBZZPw7WMjo+1hByv/ lvdFnLfiQ52tayuuC1r9x2qZ/SYWd2M4p/f5CLmvG9UcnkbYFsKWz8bwOBWKg1PQcaYHLx06sHGdY dIDaeVvkIfMFwAprSo5EFU+aes2VB2ZjugOTbkkW2aPSWTRsBhPHhV6dABEBAAG0HlJpayB2YW4gU mllbCA8cmllbEByZWRoYXQuY29tPokBHwQwAQIACQUCW5LcVgIdIAAKCRDOed6ShMTeg05SB/986o gEgdq4byrtaBQKFg5LWfd8e+h+QzLOg/T8mSS3dJzFXe5JBOfvYg7Bj47xXi9I5sM+I9Lu9+1XVb/ r2rGJrU1DwA09TnmyFtK76bgMF0sBEh1ECILYNQTEIemzNFwOWLZZlEhZFRJsZyX+mtEp/WQIygHV WjwuP69VJw+fPQvLOGn4j8W9QXuvhha7u1QJ7mYx4dLGHrZlHdwDsqpvWsW+3rsIqs1BBe5/Itz9o 6y9gLNtQzwmSDioV8KhF85VmYInslhv5tUtMEppfdTLyX4SUKh8ftNIVmH9mXyRCZclSoa6IMd635 Jq1Pj2/Lp64tOzSvN5Y9zaiCc5FucXtB9SaWsgdmFuIFJpZWwgPHJpZWxAc3VycmllbC5jb20+iQE +BBMBAgAoBQJSLd2lAhsjBQkSzAMABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDOed6ShMTe g4PpB/0ZivKYFt0LaB22ssWUrBoeNWCP1NY/lkq2QbPhR3agLB7ZXI97PF2z/5QD9Fuy/FD/jddPx KRTvFCtHcEzTOcFjBmf52uqgt3U40H9GM++0IM0yHusd9EzlaWsbp09vsAV2DwdqS69x9RPbvE/Ne fO5subhocH76okcF/aQiQ+oj2j6LJZGBJBVigOHg+4zyzdDgKM+jp0bvDI51KQ4XfxV593OhvkS3z 3FPx0CE7l62WhWrieHyBblqvkTYgJ6dq4bsYpqxxGJOkQ47WpEUx6onH+rImWmPJbSYGhwBzTo0Mm G1Nb1qGPG+mTrSmJjDRxrwf1zjmYqQreWVSFEt26tBpSaWsgdmFuIFJpZWwgPHJpZWxAZmIuY29tP okBPgQTAQIAKAUCW5LbiAIbIwUJEswDAAYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQznneko TE3oOUEQgAsrGxjTC1bGtZyuvyQPcXclap11Ogib6rQywGYu6/Mnkbd6hbyY3wpdyQii/cas2S44N cQj8HkGv91JLVE24/Wt0gITPCH3rLVJJDGQxprHTVDs1t1RAbsbp0XTksZPCNWDGYIBo2aHDwErhI omYQ0Xluo1WBtH/UmHgirHvclsou1Ks9jyTxiPyUKRfae7GNOFiX99+ZlB27P3t8CjtSO831Ij0Ip QrfooZ21YVlUKw0Wy6Ll8EyefyrEYSh8KTm8dQj4O7xxvdg865TLeLpho5PwDRF+/mR3qi8CdGbkE c4pYZQO8UDXUN4S+pe0aTeTqlYw8rRHWF9TnvtpcNzZw== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (3.54.1-1.fc41) MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: EC2DB180003 X-Stat-Signature: winqgopkxtu8ytmcboek8j6abd56pghr X-HE-Tag: 1737636211-922094 X-HE-Meta: U2FsdGVkX1+twbBnNrIEdQv7mDx9WISl2iyWch0A0lqiJ0XcwKOOagvC9EuKdd6lyS1v05Q4fs0v9h9T8rIhYGyGQvdAIUfNT1n9D3VbE09O47/gEpo0LHtoWugdDbrP7Ng5lq9TRKW1IRah6PY2xOfCMKgKKxa5rX625/H2qaOJmh3HBFy97mZ/S+99rTWWKTxF5NCV869C0xCC27BZt+xQoI4QUyszXN0KkyS4BEOAYEsH/16qb6T3fcW7oKuvo23D1cbK2uxepV48dzr/EtHRxRYmQNVDKsKtS0dypC5FMNwe60SzXwoVs4Yq+HZL29krc5YYrBpFCm7iSckt+IjlgyZIs1lxyvV3D3pxq4/v8tuGFsgzxyudMaATOY5OgoCS8lHZFlXWjCzCT8LDh+SHAQi8jqD46yldUZOaQ1ejSjxicOxAjBQaoZwlYPe+DC9MMmJRM0z0v/VLmILMBddaJ5N8S3UbOIpDMwrYuaIY3jI37jRDyFpkmc+90E/9Iioq3AU8bsx6am/LCeR/QXc2zjFyQsCTYyZkFOhqHNrW/caRfra0VWdSrZagb5QwFT9lgKGbBsWiYElNhMklRWbGLRw5riIcYa4NlqyRpp4qD08CLShhqUU+/K332edDwEHJk/eOLp/3VMhOlvNR4wtPxD42PQMy8xmib0SpO+4+OIsXs1Mjt7Lz4/DmG5VJY1kxUqF86mwtvPlG4uGjISHif2DzY41ziqfN+j0OTmFt8nCxaD673yR40AyC5TeStgnuEVHzvTWzrQmc/05bZxdYjsQYbkMlUykScK6YenPlhdXV9HFpVCrvu9zNNLjOF/hX0lyy/lpWsLdDa8yZFY732G0DDr/AHYKLI6c714FBsi6M/bKAXR2pzz6qAgOzlPuajXeU2xawDSRyheJ3QDkHtk6e3IdEuyRTXyQ/Xf5jVDlbkrIA144Gl9SVbKdqpf1h1TOqpFcgEK8EcXs cZ5Z5RDn ku1I2nhkS3BtrfrzYV+M6OoX4YtEVahJs8hP6EgPMk9DgffZ3NpOrVhDkOcoUPPS0mfpvR8xw5ImE6R97p9HedwFcj519a/GsWEgSeBO8L7ApDOwYHvVJ6tKFqcSWIBoniV/SnFYlm9t4ykgd8NEzYI0v8nUANXZE4G3jbfnAnFsMf6YZyuBcs4wBj9G7roBeSnD/qrhRVKViHb2M3XRWxUaFFQvBiHovfliswfYmpf6/deNp+t7XeraYLZ2zQjohdeW2P0eXaGTxP3EiNT25EbvT3IuILh9jiyjnLUlGheVxWuZFB0/zMEPrsKDkDKfQ5fgf3B5NfzbUyzm2dNIf6vrIrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 2025-01-23 at 10:07 +0100, Peter Zijlstra wrote: > On Wed, Jan 22, 2025 at 08:13:03PM -0500, Rik van Riel wrote: > > On Wed, 2025-01-22 at 09:38 +0100, Peter Zijlstra wrote: > > >=20 > > > Looking at this more... I'm left wondering, did 'we' look at any > > > other > > > architecture code at all?=20 > > >=20 > > > For example, look at arch/arm64/mm/context.c and see how their > > > reset > > > works. Notably, they are not at all limited to reclaiming free'd > > > ASIDs, > > > but will very aggressively take back all ASIDs except for the > > > current > > > running ones. > > >=20 > > I did look at the ARM64 code, and while their reset > > is much nicer, it looks like that comes at a cost on > > each process at context switch time. > >=20 > > In new_context(), there is a call to check_update_reserved_asid(), > > which will iterate over all CPUs to check whether this > > process's ASID is part of the reserved list that got > > carried over during the rollover. > >=20 > > I don't know if that would scale well enough to work > > on systems with thousands of CPUs. >=20 > So assuming something like 1k CPUs and !PTI, we only have like 4 > PCIDs > per CPU on average, and rollover could be frequent. >=20 > While an ARM64 with 1k CPUs and !PTI would have an average of 64 > ASIDs > per CPU, and rollover would be far less frequent. Not necessarily. On ARM64, every short lived task will get a global ASID, while on x86_64 only longer lived processes that are simultaneously active on multiple CPUs get a global ASID. The situation could be fairly bad for both, which is why I would like to solve the O(n^2) issues with the rollover code before adding that in to our x86_64 side :) I fully agree we should probably move in that direction, but I would like to make the worst case in the rollover-reuse cheaper. >=20 > That is to say, their larger ASID space (16 bits, vs our 12) > definitely > helps. But at some point yeah, this will become a problem. >=20 > Notably, I think think a 2 socket Epyc Turin with 192C is one of the > larger off-the-shelf systems atm, that gets you 768 CPUs and that is > already uncomfortably tight with our PCID space. >=20 >=20 >=20 --=20 All Rights Reversed.