From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7291C0218C for ; Tue, 21 Jan 2025 10:33:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68961280003; Tue, 21 Jan 2025 05:33:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 63936280001; Tue, 21 Jan 2025 05:33:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5005B280003; Tue, 21 Jan 2025 05:33:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 321D1280001 for ; Tue, 21 Jan 2025 05:33:44 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CD824C0913 for ; Tue, 21 Jan 2025 10:33:43 +0000 (UTC) X-FDA: 83031097926.26.CEC3A6F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id E7616100002 for ; Tue, 21 Jan 2025 10:33:41 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AUGjuIvH; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737455622; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l6xknyGTGUdofrv2Cu24XlD1g2pDfkAAtlF645ZGeXI=; b=MmAUct3cQl8+IQMGS42QlYB6uSj2BcD0Fr7Qq1vy6Z0EBLiLVO7eSP2/XoIvba4d6/7zB5 cOfoEySjo9KEAw2cGhZWZbyojAhNMPK7e48BKDrbeGevudYKdUerzM9wl0rywlOnn7TbId 4QSWb0lnMjJnNj4y5B5puOnRACw763I= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AUGjuIvH; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737455622; a=rsa-sha256; cv=none; b=qFf+NEAkT8de3v0Oz9vVChF4zt9YvOmizIB2zXqsyFXYAhIybBf63Lr+cIIe29sbv+hiis hU1vQNXUdLTThvaq3mfEpRPWGyiae/G4BeCiWv2yaAY54I7XxWrBOAn8zflBHKlEzxT43Y 8p+cZM/kljwlTIDLIcjImCyaUL/taYU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=l6xknyGTGUdofrv2Cu24XlD1g2pDfkAAtlF645ZGeXI=; b=AUGjuIvHORwflO5V09JuzM+kSB nyhEcj/hhQLu6Ufv0A9fOXqufcIuEosXyjW+UeArVBJx97nab2GAb6di9o3+GMcdj/aGLTaLgQ1Tq toY7MF9hK9xIMPEUwz7dLXMi+42/b+f2Da9JhBdd0e8ePARN/RxFgwVJ7RmT2e2MReQDjP5IxF12Z BSWJ0cYE7m9hEEWyQtgyO7ivgIhqfMqPaw/C00MFncwbW1NXysZTtrSlxuDNyMNPDzple/koDRq+O ESA01K2Ibtx8rhoFhbWLIIqSmu/M9YeWU83/d7K7zyLl59/ldUfHnQttRC1dN/1dRJe8U1RF/aY4M gBKF8O9Q==; Received: from 77-249-17-89.cable.dynamic.v4.ziggo.nl ([77.249.17.89] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1taBZe-00000007LoL-17TN; Tue, 21 Jan 2025 10:33:34 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 529F3300619; Tue, 21 Jan 2025 11:33:33 +0100 (CET) Date: Tue, 21 Jan 2025 11:33:33 +0100 From: Peter Zijlstra To: Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com Subject: Re: [PATCH v6 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes Message-ID: <20250121103333.GA7145@noisy.programming.kicks-ass.net> References: <20250120024104.1924753-1-riel@surriel.com> <20250120024104.1924753-10-riel@surriel.com> <20250121095507.GB5388@noisy.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250121095507.GB5388@noisy.programming.kicks-ass.net> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E7616100002 X-Stat-Signature: xjbmrnrsyhqy4tgxawmsdeymc956mmts X-Rspam-User: X-HE-Tag: 1737455621-155208 X-HE-Meta: U2FsdGVkX1/bPoz8M3Ah8iIN0wpzJtp7Ui/gEcTAbWbIewPWtT4U79UIREPn9Ez67me+87ZNKiRBLj0QHFFZfLuZBCPDNPTF4+jsnD2MESyFzT6ynelzJIE+oWUt9Z8Gw7mNHjl3AoTC8ZQ6x7o8QqBnO4vV3rXSX0eM/Hyos0fI77ojqinhfMBrzAV8Tkd38NCCG+IgXlso/coT/MSSwC5FiwltPE4z9/sTs1Ds+2RZ+SJUX4w4BUQIHH6sSBWkOsFElmnZ6wHFwHxAJDUVvp/EYkORVy/UBIqgNoTT7OLVIja3fr98nKXBzkXDxuWOxSMhBz0ZPiAxsf57VlTnuI95Pl8xcy1VOUY+7rekP6UiOYRVHpdhhVt19ROMrPccBswe3bJ2X6uDJpTLiazjpjttmRnz/7E2JUnXV6URoF5pcMmJcW2leOuWgA7T429dpCVJcTXZ8+kV8tnO+E2/6DZNlQj82fbyWxb1Zm090tDEHhsst7aBYjf1R5j3B9XiAXKmrTF24dDzIuDEIa4v9iNmJ+SjLYVY8EV8sPAGmSzpOY8GooKftPxYNoZoR29BDdOUcROr8kryWT5A+yk70iC7Kwc/M0tMi4W5OoqpKLD0iAftYeaT/JmMUYBNHh7GgN1EPPvGoo50RWPRp2dfGOJAKw0H69z7+olaIgu+x4PIWtlUEE4Q7yS6M2Sh2IsAdKUMm4/F3/JBAL9XhjvVLVZk/bKrHM2o7cIww7EP0/Fih0uIDyzK49YawyVsJPvS2J6mKqZWjvQCFW32C5nNDYGhoDevlcLzC4IZRsW+kRnFKO7G9q+rileuBD3AUmwifc5H8QdjAiZ2pN/+g/itlF6/r9b7x5fuSaiNzOpL+GbWV9gaZv6kFEvr60UpX0FM/5YHpSaCf/cdP1ETZs9CDRsII5zP7syDZbIqdxaHvy7qXSKLyUNuh1OkKdplos1L8jbTOX9s3Wj4SzeamXb thfAhO+K aRbGqb4S03mpLLTXZPAiU/MRUuIyaGfEntsUduxErGrSVSS/misVQK5b2xMmLbIKZMXJdUc/y2HdeiNBPran1XFKnraRw++9PoTVI7fGnYHynlSWVzAo189UdFjgTdJ3848fg+k5fZtr6N/QEXj3FXWU8Z33lbifvJXLfbMX1PqG30dpFtNrP64tnizZ61epPVssrCULQaE2oXXqkT80xOLNGgfXmWS7xPLE2QGALR+cOSQ6pcMNfDeZ2rKLFxBZDszCOO+kyc/gOMao= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jan 21, 2025 at 10:55:07AM +0100, Peter Zijlstra wrote: > On Sun, Jan 19, 2025 at 09:40:17PM -0500, Rik van Riel wrote: > > +/* > > + * Figure out whether to assign a global ASID to a process. > > + * We vary the threshold by how empty or full global ASID space is. > > + * 1/4 full: >= 4 active threads > > + * 1/2 full: >= 8 active threads > > + * 3/4 full: >= 16 active threads > > + * 7/8 full: >= 32 active threads > > + * etc > > + * > > + * This way we should never exhaust the global ASID space, even on very > > + * large systems, and the processes with the largest number of active > > + * threads should be able to use broadcast TLB invalidation. > > + */ > > +#define HALFFULL_THRESHOLD 8 > > +static bool meets_global_asid_threshold(struct mm_struct *mm) > > +{ > > + int avail = global_asid_available; > > + int threshold = HALFFULL_THRESHOLD; > > + > > + if (!avail) > > + return false; > > + > > + if (avail > MAX_ASID_AVAILABLE * 3 / 4) { > > + threshold = HALFFULL_THRESHOLD / 4; > > + } else if (avail > MAX_ASID_AVAILABLE / 2) { > > + threshold = HALFFULL_THRESHOLD / 2; > > + } else if (avail < MAX_ASID_AVAILABLE / 3) { > > + do { > > + avail *= 2; > > + threshold *= 2; > > + } while ((avail + threshold) < MAX_ASID_AVAILABLE / 2); > > + } > > + > > + return mm_active_cpus_exceeds(mm, threshold); > > +} > > I'm still very much disliking this. Why do we need this? Yes, running > out of ASID space is a pain, but this increasing threshold also makes > things behave weird. > > Suppose our most used processes starts slow, and ends up not getting an > ASID because too much irrelevant crap gets started before it spawns > enough threads and then no longer qualifies. > > Can't we just start with a very simple constant test and poke at things > if/when its found to not work? Something like so perhaps? --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -268,7 +268,7 @@ static inline u16 mm_global_asid(struct if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) return 0; - asid = READ_ONCE(mm->context.global_asid); + asid = smp_load_acquire(&mm->context.global_asid); /* mm->context.global_asid is either 0, or a global ASID */ VM_WARN_ON_ONCE(is_dyn_asid(asid)); --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -308,13 +308,18 @@ static void reset_global_asid_space(void static u16 get_global_asid(void) { lockdep_assert_held(&global_asid_lock); + bool done_reset = false; do { u16 start = last_global_asid; u16 asid = find_next_zero_bit(global_asid_used, MAX_ASID_AVAILABLE, start); - if (asid >= MAX_ASID_AVAILABLE) { + if (asid > MAX_ASID_AVAILABLE) { + if (done_reset) + return asid; + reset_global_asid_space(); + done_reset = true; continue; } @@ -392,6 +398,12 @@ static bool mm_active_cpus_exceeds(struc */ static void use_global_asid(struct mm_struct *mm) { + u16 asid; + + /* This process is already using broadcast TLB invalidation. */ + if (mm->context.global_asid) + return; + guard(raw_spinlock_irqsave)(&global_asid_lock); /* This process is already using broadcast TLB invalidation. */ @@ -402,58 +414,25 @@ static void use_global_asid(struct mm_st if (!global_asid_available) return; + asid = get_global_asid(); + if (asid > MAX_ASID_AVAILABLE) + return; + /* - * The transition from IPI TLB flushing, with a dynamic ASID, - * and broadcast TLB flushing, using a global ASID, uses memory - * ordering for synchronization. - * - * While the process has threads still using a dynamic ASID, - * TLB invalidation IPIs continue to get sent. - * - * This code sets asid_transition first, before assigning the - * global ASID. - * - * The TLB flush code will only verify the ASID transition - * after it has seen the new global ASID for the process. + * Notably flush_tlb_mm_range() -> broadcast_tlb_flush() -> + * finish_asid_transition() needs to observe asid_transition == true + * once it observes global_asid. */ - WRITE_ONCE(mm->context.asid_transition, true); - WRITE_ONCE(mm->context.global_asid, get_global_asid()); + mm->context.asid_transition = true; + smp_store_release(&mm->context.global_asid, asid); } -/* - * Figure out whether to assign a global ASID to a process. - * We vary the threshold by how empty or full global ASID space is. - * 1/4 full: >= 4 active threads - * 1/2 full: >= 8 active threads - * 3/4 full: >= 16 active threads - * 7/8 full: >= 32 active threads - * etc - * - * This way we should never exhaust the global ASID space, even on very - * large systems, and the processes with the largest number of active - * threads should be able to use broadcast TLB invalidation. - */ -#define HALFFULL_THRESHOLD 8 static bool meets_global_asid_threshold(struct mm_struct *mm) { - int avail = global_asid_available; - int threshold = HALFFULL_THRESHOLD; - - if (!avail) + if (!global_asid_available) return false; - if (avail > MAX_ASID_AVAILABLE * 3 / 4) { - threshold = HALFFULL_THRESHOLD / 4; - } else if (avail > MAX_ASID_AVAILABLE / 2) { - threshold = HALFFULL_THRESHOLD / 2; - } else if (avail < MAX_ASID_AVAILABLE / 3) { - do { - avail *= 2; - threshold *= 2; - } while ((avail + threshold) < MAX_ASID_AVAILABLE / 2); - } - - return mm_active_cpus_exceeds(mm, threshold); + return mm_active_cpus_exceeds(mm, 4); } static void consider_global_asid(struct mm_struct *mm)