From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28B93C02182 for ; Thu, 23 Jan 2025 09:07:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8029E6B007B; Thu, 23 Jan 2025 04:07:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B1C36B0082; Thu, 23 Jan 2025 04:07:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A0B66B0083; Thu, 23 Jan 2025 04:07:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4C77A6B007B for ; Thu, 23 Jan 2025 04:07:32 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E3FC280D4A for ; Thu, 23 Jan 2025 09:07:31 +0000 (UTC) X-FDA: 83038138302.12.80A4929 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id B6DAF1C0013 for ; Thu, 23 Jan 2025 09:07:29 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DsAWb6Fr; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737623250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sEevy1J0VAXnRgodm5rMxK1EoTYI9dE+a6pB13YJxTM=; b=72csIO1po+zbRnZiMWJMc/S6EQtIPDdGHujZWQ8jH8tjTLDs4RUuw9xTUxtWdR08f6kVxr mS+9b1fHs8sv9Gr8Bls48OgHzJbkutLK6zvWqJ30pBen55/pBV8MlhZyFd8REzhITlJEBy u4ea0P9f3eWRxazu3UVRL0caN1hE+Pw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737623250; a=rsa-sha256; cv=none; b=3faBFa5r6JhoBuNvh/gezmbigBMG7y8Bbck7NuimU5PZ1CgRLC1K3Lp5dqiN1vuERhbTq0 yT6X294rguOsqUE3gQyPFRZcYBEdUlvkid9krEVPIsz74wG4hfcwUlrpSTXbcuV92SdmZq ZV52+lI5spxyJppyEYdOaIi0VR+amUk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DsAWb6Fr; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=sEevy1J0VAXnRgodm5rMxK1EoTYI9dE+a6pB13YJxTM=; b=DsAWb6FrDXbbqVYyAWAcHgFrcU ygpHgd8phf/4B7SYcOWMgYYi4ehEe2jcgbfz3SNLkzAXiVeAOQd9i6WnTPxOuNcSlKj7uXaP8h0d9 oYB72qy75k/ep7iCS2LKuwqi8yGiZRuwe3ohN3sQESgnOoLbOl8Y6FTjkNYRQqp4s/WbNROn7P3zv s+nLGo6Yhj30utsPY9gRMhQ8Lgp6zRkH0jaqtM9tCRhZ1mrEoaVm/RX22f+EMisWvxF03DdNYuVjt fh3gcO/bB7G3PPDsVZ6ECsu4LwrBtzs6xgKf+fFQSKFLnvWGoHL7hkROyZMYi5evuOmorNu3QsE8G USf5+cFA==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tatBJ-00000008lRx-2Gy6; Thu, 23 Jan 2025 09:07:21 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 117E83006E6; Thu, 23 Jan 2025 10:07:21 +0100 (CET) Date: Thu, 23 Jan 2025 10:07:20 +0100 From: Peter Zijlstra To: Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Mark Rutland , Will Deacon Subject: Re: [PATCH v6 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes Message-ID: <20250123090720.GI3808@noisy.programming.kicks-ass.net> References: <20250120024104.1924753-1-riel@surriel.com> <20250120024104.1924753-10-riel@surriel.com> <20250122083835.GE7145@noisy.programming.kicks-ass.net> <5820b18ef0ba48c33a62553fcc444c47f963b907.camel@surriel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5820b18ef0ba48c33a62553fcc444c47f963b907.camel@surriel.com> X-Stat-Signature: offfty9ins336xraxke47asi4yzjacci X-Rspam-User: X-Rspamd-Queue-Id: B6DAF1C0013 X-Rspamd-Server: rspam03 X-HE-Tag: 1737623249-675182 X-HE-Meta: U2FsdGVkX1/HGeY887pqbeZlQ7KxDHzXf/FCst1isNOpkvwwN50U3s40AjiBK7/xVK90lBC+ET5VDDTYF5j3vhC2HmqEMP+ukjqnwqx1fLUdpyCgkQbI/hOF1MUL+AHWo44VPbbmW0wpPvM/9mn7nVi1SdbJ+z++GhrprFaQnwbinUrl2cXwZq5X/rp00vMsDmc7WvdoCrFAGtnMBQjgQNPjCKETdOE260bCWQj/9SHQMkStQBRRFQHc5O9gNi2KFoQtG1Pp+rApiiKdNA1I8MxRXE+MNIlk4KBEtuw/6NE1wWb6kXPqrkV3LHz4mQPNolZuRMmeqLjafqLfhVtHKbuLRIV71vnYky/3O3c6gFcv8ZLSffNEv0En5/8ARsPdPF9FImaF6aJMBqz+aYREdgNUptxaAOCXjJtti8Pg67BQXQ8xRfzafbqmm0NWazeeeRokYZ02EjEPESJcGaG8fLD/vVmd1uLihpLS85pokKb/nqIGhtP+kxGXxwvfsnZwEvGIMqvjU1zv1/yFSVmI8gz9IOjBsdwHIJYTjaciGOnPKFUOMBKhWR3RCfReao+XjYvNDopShnSwC/8bQSgxWB2NLu4mGrocqnp0EyNvRFxG/CsY1OoyoEhSRkloDG/6dLI5yub0z1LDtpueq9KIfbzc5+NONKvUZ7yLw277PvVI2HF2jSyAHoEzyEOsraqgVWQ2tuhEOCB0ogzUEmRTV780VW7gMvpVBHWb1ayZ+lFkTNC0+XCvm9xi27AuMLUnyDTUOQv7UKCyWFYe+lG7PiGwJtOlybp4lt5f0rxW6FYoyYl9Fl7/RLOdsJJeFnrOz3O3M92KHU7FWEdRWfTwydKczJDHEylv1Hd3hxeINSAGmmZv4SsHbutcYzREteB40ePGV10PiG5S+yij3/rgkXr6iuNuhw9/nBtu4IrfUmOO+h3B+DXLp7MQVvkGw/kA8xc1psVB72zqVxnwC64 g2CVTNdy 0mxHkZpgMuze5CyCMVzukL/908FPzkkVSstqPVG+zH+tWGUsTb0r+3qgJ2FRMqwT7cits2FC5xxOCxZB0LBZys0TodhiL/HwhJBYCEtw9FLkXDa4lwui6hO1cHvQoZA8tN+BSrkzBb+3lm+WF0fRRkkv/agFjtjVngWcgSgFLJu0skqnWQgplEXceSvb7qpC0ynfooq9/PL/hnrlfSSTptRNJd378mX6l2zqe+IfWk6+y2IcpMPak2DV0CTRmyAEpJdgpF8eVzLPRNSM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 22, 2025 at 08:13:03PM -0500, Rik van Riel wrote: > On Wed, 2025-01-22 at 09:38 +0100, Peter Zijlstra wrote: > > > > Looking at this more... I'm left wondering, did 'we' look at any > > other > > architecture code at all? > > > > For example, look at arch/arm64/mm/context.c and see how their reset > > works. Notably, they are not at all limited to reclaiming free'd > > ASIDs, > > but will very aggressively take back all ASIDs except for the current > > running ones. > > > I did look at the ARM64 code, and while their reset > is much nicer, it looks like that comes at a cost on > each process at context switch time. > > In new_context(), there is a call to check_update_reserved_asid(), > which will iterate over all CPUs to check whether this > process's ASID is part of the reserved list that got > carried over during the rollover. > > I don't know if that would scale well enough to work > on systems with thousands of CPUs. So assuming something like 1k CPUs and !PTI, we only have like 4 PCIDs per CPU on average, and rollover could be frequent. While an ARM64 with 1k CPUs and !PTI would have an average of 64 ASIDs per CPU, and rollover would be far less frequent. That is to say, their larger ASID space (16 bits, vs our 12) definitely helps. But at some point yeah, this will become a problem. Notably, I think think a 2 socket Epyc Turin with 192C is one of the larger off-the-shelf systems atm, that gets you 768 CPUs and that is already uncomfortably tight with our PCID space.