From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D4B1E77188 for ; Mon, 30 Dec 2024 19:24:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57C196B00A9; Mon, 30 Dec 2024 14:24:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 503996B00AA; Mon, 30 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 305F56B00AB; Mon, 30 Dec 2024 14:24:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0508C6B00A9 for ; Mon, 30 Dec 2024 14:24:32 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BBBA41C7400 for ; Mon, 30 Dec 2024 19:24:32 +0000 (UTC) X-FDA: 82952599632.01.3FEEEEE Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf14.hostedemail.com (Postfix) with ESMTP id 81C2A100012 for ; Mon, 30 Dec 2024 19:23:38 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mVgKiI1H; spf=pass (imf14.hostedemail.com: domain of nadav.amit@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735586635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MvZhetW4yctH43dLLBYpm6faFlepdUMw/rl8YLjOU9c=; b=UnlkCAzEZw/jPO+DKT3RLC07POWrhEDbfsFz4Geabfk2JWVqiiW9bR39SxEeEiUftA7IzK bmgMjWPmIZ3uowqhmB0LT5BgXUyuV6h7Yh55v3sS1nYpUHmgz0CsBvo1Cnr9kNs8KOfe6q SHVn3jQN60ktqstVN1pp1dEkvwfrnJM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mVgKiI1H; spf=pass (imf14.hostedemail.com: domain of nadav.amit@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735586635; a=rsa-sha256; cv=none; b=x1yNJ7F/QTWHl4izv3+K/Tyu2RTEqMZeBlqH0kXJTbWMoyPFqGquKN6KWnB6TcKNs8nEjs vnqfsx3a7P3QxfJs/NYTw2HdMFhuFnpQdM1PLu6JNeg3PCGcfcVblzf9pkrX54x8JkahkQ +1dyBxQRCP/oTpVhKTZql7ztRIPv1Vc= Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-aaf6b1a5f2bso120004966b.1 for ; Mon, 30 Dec 2024 11:24:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735586669; x=1736191469; darn=kvack.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=MvZhetW4yctH43dLLBYpm6faFlepdUMw/rl8YLjOU9c=; b=mVgKiI1HwI789oyeBT7obvawWEyarGXHEjQqNhBRRHqQ0MdEkyHz+aklMSC6++gQZv Ifai1XO9G8FwLZolI+ijexkaySjP/7FRZCLc2SfxtSKaq294sq32RpQtVOR8HEifC2aD oxM40/7B7bUxYyuifGnDMspMSk7YuP2jxfflhQANKRo6c7phDUBp9gT7zoefcTxFDOr7 OaCnJVVmVbrps9GC/6kW0MEOCz9ajoNSpiPzsKLma9/yG8DIcCBwTgv8EADh1huUzoEb tO72bvxPXzAZJWI3QHcnRZEdWIvqPHr/gnbYzJgF61epLQKkVW9uThQ976xJeMSbsnEb eDkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735586669; x=1736191469; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MvZhetW4yctH43dLLBYpm6faFlepdUMw/rl8YLjOU9c=; b=l5mHr9/icy30Jytc/0g/CJbc9OzPc/e/4vpA8f3kVZNWmhkK3X+EAlVwzrN3XdSgkJ TaORuA/kBtqn6sebGvVePcTIop4/p2sucsNKNTBPzXKvAWdCoX7MNYr9URBS5jTbcgMa ccijgucS/CP4zZ0rPPzB8xCni+c9qsYP0LkWWIrjicnFf2Tc4pdXDa61QobOoAv3Pbym gAyaRZw0N6o0Vog3gJXRcnvfkxHiME3lQ0/QgDoDLVSnF+0ZN+64nZzKSZZPfTh7e/hu Yblf9fa/m6jHlIO9KnWf5UhRzYlcwEJHlYH5/lbk+fS1CgMNzZDtL+lqVk+ZO7KfIJzL pQhA== X-Forwarded-Encrypted: i=1; AJvYcCVN25SUHZQlNLh8T3K7Z7zWo9CN2CZzmWO7Bt4vrsIAyQucVgp67gaQy+uCKl5iIeyI76ZkeNlQ4Q==@kvack.org X-Gm-Message-State: AOJu0YwtB8ZrZJlUh4mJhqdnqeU7rcKYb4fU9tdcT1QE9+9EGbT3unVu 80+RcgA6kkZHoM0iAn3YrfpDewslROMblybsx/MXVVgFiUuFv/XN X-Gm-Gg: ASbGncu6ILymI2biE/QCLEwbEyHnnjnYBTxkPJipkBK8/znCwyUj05zzsmCbUQLbtps Eh9xEx4US9v6UexHSPNkCW2OiNLLmxIa03h53YZsc3ltRlrRmj3U1tjJsIFBkL9J03pNYO1fR5K 0mreYO2jseZn0F2FaXBwKV2O4aJo+D/KCz1Co4nRQpEC9Pp2RGRcO234p8PEPJNkIBB61xgmGPK U00c39Q8rDa/QrjaVRoL7bJzhHfO0xRarvFmEVudC775b0p0dFAjgnuK89fIN1ZxvPfxOyHow== X-Google-Smtp-Source: AGHT+IGRWuNlvPcQdEx1bm6cGGFNa6MWn7Lw5KH6sGaNgmNSN8jx426e1ViadlzPJdWL7CIkY48bOQ== X-Received: by 2002:a17:906:dc8f:b0:aab:a1a8:92be with SMTP id a640c23a62f3a-aac080feebdmr3538958466b.5.1735586668551; Mon, 30 Dec 2024 11:24:28 -0800 (PST) Received: from smtpclient.apple ([132.69.243.32]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aac0e895194sm1495899366b.70.2024.12.30.11.24.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Dec 2024 11:24:27 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes From: Nadav Amit In-Reply-To: <20241230175550.4046587-10-riel@surriel.com> Date: Mon, 30 Dec 2024 21:24:13 +0200 Cc: the arch/x86 maintainers , Linux Kernel Mailing List , kernel-team@meta.com, Dave Hansen , luto@kernel.org, peterz@infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Andrew Morton , zhengqi.arch@bytedance.com, "open list:MEMORY MANAGEMENT" Content-Transfer-Encoding: quoted-printable Message-Id: References: <20241230175550.4046587-1-riel@surriel.com> <20241230175550.4046587-10-riel@surriel.com> To: Rik van Riel X-Mailer: Apple Mail (2.3826.300.87.4.3) X-Rspamd-Queue-Id: 81C2A100012 X-Rspamd-Server: rspam12 X-Stat-Signature: aj5k37b5dagz991x34zcu7kamwbcekrz X-Rspam-User: X-HE-Tag: 1735586618-784851 X-HE-Meta: U2FsdGVkX1+qD0SO6F6oIkBChnkPSNywilOFvjG0O6lXlHNcYp5VA9oHHYqvaj69rW0JIXfqX7WG+iGN4JACaBf8L/G79ZKOVB+yUhqHa4tlQn99QtWS19S2lgpkQmnfLxIZN/TzUM3MSGGplXhtjuZBT8DEGunuKSqeZSV/WOcQvXSy3JUU/y8+UKEXomj+FEMTi1VpR/iC3f1ayOJrRB5R6pkZUMjuxBYrfEyT1Pha5ZPzM9u/V0w+pRJW9CGoBmrKR4FYOmzFxrKtqVohhbnyvRRVJ9RRw+lorCLsf+GmIAflKCmNawwxMtbWEmmMiO1vENMLpuMtb546PTJl5xxm2EKbmOP4/wIx3UusIMG1kz70cT9iJWIOOmJOXYFDeZJdYb9TWomEXa7An91UZDJ9o9U6Go1iESp0oHrzdZo1WACzNfrTCo0eZ6xaVQVPi37ioeoMrQoCN+IAUW7y8V0AotbO/Qsw4SabEKRd3Fhypk/3n3nOOgLgYJPrWULqhHkYztoJ84FuBn2AdbnRGpdpq4wL8MzWMgVgduVc0iK/s9TaOx38fdQv7lA9IDQzhBMg+ZPsTvoZrxYlc0kOjtIxb8/Qbu2S6BXjURaFCAKu825NM380PshiBB1qSIky2xWaYQmasZmm0q1erYdv2rr+ox0OgmSM7O9+RLqKPP0oSnseooJFQGsmn3t7hnvz/oV5thtPVyUuZMun/6a1bOdXOuusBHs2G0kaNbcuX+zHabeR72Kuez3lRZH1QmJQOmo+gSKfl+rQg3cJICqVd5tLprtUmXGeFPd8kP9ZWW0R6y1XQG1TD68cse2O1EWkIYlKudjl12T0siFz8n3cUMdnpGbaVjkLR4EhoLQjRI1VO8PbpAEooKEiBsagpP8dbJK3JR2YAyMu9Sv67p4vf245XWC+oP1tIBjtAyRXmsv/hXt5FDoWBJndhn6KpKFTmd5fPn+V4W09J4Q50m+ tT1AinsI lHH/kY+mmK5gOnheULcVwL5IU/EGB1JExiR3H5WmuBlts+rSsXEnXok8Cbz2v15e3koHEM+fetspBiC4wR68Y+afPpCUrDoZ7+O1wjsJMxLzvtAzvV4o5in31EUf6JrETaw0b5MBpFUEi044d2kInGICkD98WSC2xFQScJ9Tm9DxVuWDJdaEJjeyzYI4BZaE0s7jFbex2t7BSYHP9Af77mIRnAWZnvZJIVyP1AOLS+KrPzsKJDPXfAPAd2ol+7R1kLYxWOiHsIrvZu8/N71B+Jq/ACzKVu0V/BKIRij8dEyYeZrBiPVS1zXQxF2O3Dp2AMiQ/KMG9DZjj2lSv6kGAIpa8JKN9XSV8F5D/0iwn0bs+FTJT0Gwd1+Fzh3r3Z0YIL8FqYD7g61VSMyOlNCaJ2mG5Sp+8oydtEwoVIFRq7A5HA1MwAZEu8yqLDoW7GaBmozW0RvTrTSjt8gx0WYc4XDpb+jscTwmwHHmG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > On 30 Dec 2024, at 19:53, Rik van Riel wrote: >=20 > Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD = EPYC 3 > and newer CPUs. >=20 > In order to not exhaust PCID space, and keep TLB flushes local for = single > threaded processes, we only hand out broadcast ASIDs to processes = active on > 3 or more CPUs, and gradually increase the threshold as broadcast ASID = space > is depleted. >=20 > Signed-off-by: Rik van Riel > --- >=20 [snip] > --- a/arch/x86/include/asm/mmu_context.h > +++ b/arch/x86/include/asm/mmu_context.h > @@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct = mm_struct *mm) > #define enter_lazy_tlb enter_lazy_tlb > extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct = *tsk); >=20 > +extern void destroy_context_free_broadcast_asid(struct mm_struct = *mm); > + > /* > * Init a new mm. Used on mm copies, like at fork() > * and on mm's that are brand-new, like at execve(). > @@ -161,6 +163,13 @@ static inline int init_new_context(struct = task_struct *tsk, > mm->context.execute_only_pkey =3D -1; > } > #endif > + > +#ifdef CONFIG_CPU_SUP_AMD > + INIT_LIST_HEAD(&mm->context.broadcast_asid_list); > + mm->context.broadcast_asid =3D 0; > + mm->context.asid_transition =3D false; > +#endif > + > mm_reset_untag_mask(mm); > init_new_context_ldt(mm); > return 0; > @@ -170,6 +179,9 @@ static inline int init_new_context(struct = task_struct *tsk, > static inline void destroy_context(struct mm_struct *mm) > { > destroy_context_ldt(mm); > +#ifdef CONFIG_CPU_SUP_AMD > + destroy_context_free_broadcast_asid(mm); > +#endif This ifdef=E2=80=99ry is not great. I think it=E2=80=99s better to have = entire functions in ifdef than put ifdef=E2=80=99s within the code.=20 > } >=20 > extern void switch_mm(struct mm_struct *prev, struct mm_struct *next, > diff --git a/arch/x86/include/asm/tlbflush.h = b/arch/x86/include/asm/tlbflush.h > index 20074f17fbcd..5e9956af98d1 100644 > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long = mask) > */ > #define TLB_NR_DYN_ASIDS 6 >=20 > +#ifdef CONFIG_CPU_SUP_AMD > +#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS > +#define is_broadcast_asid(asid) (asid) >=3D TLB_NR_DYN_ASIDS > +#define in_asid_transition(info) (info->mm && = info->mm->context.asid_transition) > +#define mm_broadcast_asid(mm) (mm->context.broadcast_asid) > +#else > +#define is_dyn_asid(asid) true > +#define is_broadcast_asid(asid) false > +#define in_asid_transition(info) false > +#define mm_broadcast_asid(mm) 0 I don=E2=80=99t see a reason why those should be #define instead of = inline functions. Arguably, those are better due to type-checking, etc. For instance = is_dyn_asid() is missing brackets to be safe. > + > +inline bool needs_broadcast_asid_reload(struct mm_struct *next, u16 = prev_asid) > +{ > + return false; > +} > +#endif > + > struct tlb_context { > u64 ctx_id; > u64 tlb_gen; > diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c > index 64f1679c37e1..eb83391385ce 100644 > --- a/arch/x86/mm/tlb.c > +++ b/arch/x86/mm/tlb.c > @@ -74,13 +74,15 @@ > * use different names for each of them: > * > * ASID - [0, TLB_NR_DYN_ASIDS-1] > - * the canonical identifier for an mm > + * the canonical identifier for an mm, dynamically allocated = on each CPU > + * [TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1] > + * the canonical, global identifier for an mm, identical = across all CPUs > * > - * kPCID - [1, TLB_NR_DYN_ASIDS] > + * kPCID - [1, MAX_ASID_AVAILABLE] > * the value we write into the PCID part of CR3; corresponds = to the > * ASID+1, because PCID 0 is special. > * > - * uPCID - [2048 + 1, 2048 + TLB_NR_DYN_ASIDS] > + * uPCID - [2048 + 1, 2048 + MAX_ASID_AVAILABLE] > * for KPTI each mm has two address spaces and thus needs two > * PCID values, but we can still do with a single ASID = denomination > * for each mm. Corresponds to kPCID + 2048. > @@ -225,6 +227,18 @@ static void choose_new_asid(struct mm_struct = *next, u64 next_tlb_gen, > return; > } >=20 > + /* > + * TLB consistency for this ASID is maintained with INVLPGB; > + * TLB flushes happen even while the process isn't running. > + */ > +#ifdef CONFIG_CPU_SUP_AMD I=E2=80=99m pretty sure IS_ENABLED() can be used here. > + if (static_cpu_has(X86_FEATURE_INVLPGB) && = mm_broadcast_asid(next)) { > + *new_asid =3D mm_broadcast_asid(next); Isn=E2=80=99t there a risk of a race changing broadcast_asid between the = two reads? Maybe use READ_ONCE() also since the value is modified asynchronously?=20= > + *need_flush =3D false; > + return; > + } > +#endif > + > if (this_cpu_read(cpu_tlbstate.invalidate_other)) > clear_asid_other(); >=20 > @@ -251,6 +265,245 @@ static void choose_new_asid(struct mm_struct = *next, u64 next_tlb_gen, > *need_flush =3D true; > } >=20 > +#ifdef CONFIG_CPU_SUP_AMD > +/* > + * Logic for AMD INVLPGB support. > + */ > +static DEFINE_RAW_SPINLOCK(broadcast_asid_lock); > +static u16 last_broadcast_asid =3D TLB_NR_DYN_ASIDS; > +static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) =3D { = 0 }; > +static LIST_HEAD(broadcast_asid_list); > +static int broadcast_asid_available =3D MAX_ASID_AVAILABLE - = TLB_NR_DYN_ASIDS - 1; Presumably some of these data structures are shared, and some are = accessed frequently together. Wouldn=E2=80=99t it make more sense to put them = inside a struct(s) and make it cacheline aligned? > + > +static void reset_broadcast_asid_space(void) > +{ > + mm_context_t *context; > + > + lockdep_assert_held(&broadcast_asid_lock); > + > + /* > + * Flush once when we wrap around the ASID space, so we won't = need > + * to flush every time we allocate an ASID for boradcast = flushing. > + */ > + invlpgb_flush_all_nonglobals(); > + tlbsync(); > + > + /* > + * Leave the currently used broadcast ASIDs set in the bitmap, = since > + * those cannot be reused before the next wraparound and flush.. > + */ > + bitmap_clear(broadcast_asid_used, 0, MAX_ASID_AVAILABLE); > + list_for_each_entry(context, &broadcast_asid_list, = broadcast_asid_list) > + __set_bit(context->broadcast_asid, broadcast_asid_used); > + > + last_broadcast_asid =3D TLB_NR_DYN_ASIDS; > +} > + > +static u16 get_broadcast_asid(void) > +{ > + lockdep_assert_held(&broadcast_asid_lock); > + > + do { > + u16 start =3D last_broadcast_asid; > + u16 asid =3D find_next_zero_bit(broadcast_asid_used, = MAX_ASID_AVAILABLE, start); > + > + if (asid >=3D MAX_ASID_AVAILABLE) { > + reset_broadcast_asid_space(); > + continue; > + } > + > + /* Try claiming this broadcast ASID. */ > + if (!test_and_set_bit(asid, broadcast_asid_used)) { IIUC, broadcast_asid_used is always protected with broadcast_asid_lock. So why test_and_set_bit ? > + last_broadcast_asid =3D asid; > + return asid; > + } > + } while (1); > +} > + > +/* > + * Returns true if the mm is transitioning from a CPU-local ASID to a = broadcast > + * (INVLPGB) ASID, or the other way around. > + */ > +static bool needs_broadcast_asid_reload(struct mm_struct *next, u16 = prev_asid) > +{ > + u16 broadcast_asid =3D mm_broadcast_asid(next); > + > + if (broadcast_asid && prev_asid !=3D broadcast_asid) > + return true; > + > + if (!broadcast_asid && is_broadcast_asid(prev_asid)) > + return true; > + > + return false; > +} > + > +void destroy_context_free_broadcast_asid(struct mm_struct *mm) > +{ > + if (!mm->context.broadcast_asid) mm_broadcast_asid()? > + return; > + > + guard(raw_spinlock_irqsave)(&broadcast_asid_lock); > + mm->context.broadcast_asid =3D 0; > + list_del(&mm->context.broadcast_asid_list); > + broadcast_asid_available++; > +} > + > +static bool mm_active_cpus_exceeds(struct mm_struct *mm, int = threshold) > +{ > + int count =3D 0; > + int cpu; > + > + if (cpumask_weight(mm_cpumask(mm)) <=3D threshold) > + return false; > + > + for_each_cpu(cpu, mm_cpumask(mm)) { > + /* Skip the CPUs that aren't really running this = process. */ > + if (per_cpu(cpu_tlbstate.loaded_mm, cpu) !=3D mm) > + continue; > + > + if (per_cpu(cpu_tlbstate_shared.is_lazy, cpu)) > + continue; > + > + if (++count > threshold) > + return true; > + } > + return false; > +} > + > +/* > + * Assign a broadcast ASID to the current process, protecting against > + * races between multiple threads in the process. > + */ > +static void use_broadcast_asid(struct mm_struct *mm) > +{ > + guard(raw_spinlock_irqsave)(&broadcast_asid_lock); > + > + /* This process is already using broadcast TLB invalidation. */ > + if (mm->context.broadcast_asid) > + return; > + > + mm->context.broadcast_asid =3D get_broadcast_asid(); This is read without the lock, so do you want WRITE_ONCE() here?=20 > + mm->context.asid_transition =3D true; And what about asid_transition? Presumably also need WRITE_ONCE(). But = more importantly than this theoretical compiler optimization, is there some = assumed ordering with setting broadcast_asid? > + list_add(&mm->context.broadcast_asid_list, = &broadcast_asid_list); > + broadcast_asid_available--; > +} > + > +/* > + * Figure out whether to assign a broadcast (global) ASID to a = process. > + * We vary the threshold by how empty or full broadcast ASID space = is. > + * 1/4 full: >=3D 4 active threads > + * 1/2 full: >=3D 8 active threads > + * 3/4 full: >=3D 16 active threads > + * 7/8 full: >=3D 32 active threads > + * etc > + * > + * This way we should never exhaust the broadcast ASID space, even on = very > + * large systems, and the processes with the largest number of active > + * threads should be able to use broadcast TLB invalidation. > + */ > +#define HALFFULL_THRESHOLD 8 > +static bool meets_broadcast_asid_threshold(struct mm_struct *mm) > +{ > + int avail =3D broadcast_asid_available; > + int threshold =3D HALFFULL_THRESHOLD; > + > + if (!avail) > + return false; > + > + if (avail > MAX_ASID_AVAILABLE * 3 / 4) { > + threshold =3D HALFFULL_THRESHOLD / 4; > + } else if (avail > MAX_ASID_AVAILABLE / 2) { > + threshold =3D HALFFULL_THRESHOLD / 2; > + } else if (avail < MAX_ASID_AVAILABLE / 3) { > + do { > + avail *=3D 2; > + threshold *=3D 2; > + } while ((avail + threshold) < MAX_ASID_AVAILABLE / 2); > + } > + > + return mm_active_cpus_exceeds(mm, threshold); > +} > + > +static void count_tlb_flush(struct mm_struct *mm) > +{ > + if (!static_cpu_has(X86_FEATURE_INVLPGB)) > + return; > + > + /* Check every once in a while. */ > + if ((current->pid & 0x1f) !=3D (jiffies & 0x1f)) > + return; > + > + if (meets_broadcast_asid_threshold(mm)) > + use_broadcast_asid(mm); > +} I don=E2=80=99t think count_tlb_flush() is a name that reflects what = this function does. > + > +static void finish_asid_transition(struct flush_tlb_info *info) > +{ > + struct mm_struct *mm =3D info->mm; > + int bc_asid =3D mm_broadcast_asid(mm); > + int cpu; > + > + if (!mm->context.asid_transition) is_asid_transition()? > + return; > + > + for_each_cpu(cpu, mm_cpumask(mm)) { > + if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) !=3D = mm) > + continue; > + > + /* > + * If at least one CPU is not using the broadcast ASID = yet, > + * send a TLB flush IPI. The IPI should cause stragglers > + * to transition soon. > + */ > + if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) !=3D = bc_asid) { > + flush_tlb_multi(mm_cpumask(info->mm), info); > + return; > + } > + } > + > + /* All the CPUs running this process are using the broadcast = ASID. */ > + mm->context.asid_transition =3D 0; > +} > + > +static void broadcast_tlb_flush(struct flush_tlb_info *info) > +{ > + bool pmd =3D info->stride_shift =3D=3D PMD_SHIFT; > + unsigned long maxnr =3D invlpgb_count_max; > + unsigned long asid =3D info->mm->context.broadcast_asid; > + unsigned long addr =3D info->start; > + unsigned long nr; > + > + /* Flushing multiple pages at once is not supported with 1GB = pages. */ > + if (info->stride_shift > PMD_SHIFT) > + maxnr =3D 1; > + > + if (info->end =3D=3D TLB_FLUSH_ALL) { > + invlpgb_flush_single_pcid(kern_pcid(asid)); > + /* Do any CPUs supporting INVLPGB need PTI? */ > + if (static_cpu_has(X86_FEATURE_PTI)) > + invlpgb_flush_single_pcid(user_pcid(asid)); > + } else do { I couldn=E2=80=99t find any use of =E2=80=9Celse do=E2=80=9D in the = kernel. Might it be confusing? > + /* > + * Calculate how many pages can be flushed at once; if = the > + * remainder of the range is less than one page, flush = one. > + */ > + nr =3D min(maxnr, (info->end - addr) >> = info->stride_shift); > + nr =3D max(nr, 1); > + > + invlpgb_flush_user_nr(kern_pcid(asid), addr, nr, pmd); > + /* Do any CPUs supporting INVLPGB need PTI? */ > + if (static_cpu_has(X86_FEATURE_PTI)) > + invlpgb_flush_user_nr(user_pcid(asid), addr, nr, = pmd); > + addr +=3D nr << info->stride_shift; > + } while (addr < info->end); > + > + finish_asid_transition(info); > + > + /* Wait for the INVLPGBs kicked off above to finish. */ > + tlbsync(); > +} > +#endif /* CONFIG_CPU_SUP_AMD */ > + > /* > * Given an ASID, flush the corresponding user ASID. We can delay = this > * until the next time we switch to it. > @@ -556,8 +809,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > */ > if (prev =3D=3D next) { > /* Not actually switching mm's */ > - = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=3D > - next->context.ctx_id); > + if (is_dyn_asid(prev_asid)) > + = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=3D > + next->context.ctx_id); Why not to add the condition into the VM_WARN_ON and avoid the nesting? >=20 > /* > * If this races with another thread that enables lam, = 'new_lam' > @@ -573,6 +827,23 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > !cpumask_test_cpu(cpu, = mm_cpumask(next)))) > cpumask_set_cpu(cpu, mm_cpumask(next)); >=20 > + /* > + * Check if the current mm is transitioning to a new = ASID. > + */ > + if (needs_broadcast_asid_reload(next, prev_asid)) { > + next_tlb_gen =3D = atomic64_read(&next->context.tlb_gen); > + > + choose_new_asid(next, next_tlb_gen, &new_asid, = &need_flush); > + goto reload_tlb; > + } > + > + /* > + * Broadcast TLB invalidation keeps this PCID up to date > + * all the time. > + */ > + if (is_broadcast_asid(prev_asid)) > + return; > + > /* > * If the CPU is not in lazy TLB mode, we are just = switching > * from one thread in a process to another thread in the = same > @@ -626,8 +897,10 @@ void switch_mm_irqs_off(struct mm_struct *unused, = struct mm_struct *next, > barrier(); > } >=20 > +reload_tlb: > new_lam =3D mm_lam_cr3_mask(next); > if (need_flush) { > + VM_BUG_ON(is_broadcast_asid(new_asid)); > this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, = next->context.ctx_id); > this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, = next_tlb_gen); > load_new_mm_cr3(next->pgd, new_asid, new_lam, true); > @@ -746,7 +1019,7 @@ static void flush_tlb_func(void *info) > const struct flush_tlb_info *f =3D info; > struct mm_struct *loaded_mm =3D = this_cpu_read(cpu_tlbstate.loaded_mm); > u32 loaded_mm_asid =3D = this_cpu_read(cpu_tlbstate.loaded_mm_asid); > - u64 local_tlb_gen =3D = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); > + u64 local_tlb_gen; > bool local =3D smp_processor_id() =3D=3D f->initiating_cpu; > unsigned long nr_invalidate =3D 0; > u64 mm_tlb_gen; > @@ -769,6 +1042,16 @@ static void flush_tlb_func(void *info) > if (unlikely(loaded_mm =3D=3D &init_mm)) > return; >=20 > + /* Reload the ASID if transitioning into or out of a broadcast = ASID */ > + if (needs_broadcast_asid_reload(loaded_mm, loaded_mm_asid)) { > + switch_mm_irqs_off(NULL, loaded_mm, NULL); > + loaded_mm_asid =3D = this_cpu_read(cpu_tlbstate.loaded_mm_asid); > + } > + > + /* Broadcast ASIDs are always kept up to date with INVLPGB. */ > + if (is_broadcast_asid(loaded_mm_asid)) > + return; > + > = VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) !=3D > loaded_mm->context.ctx_id); >=20 > @@ -786,6 +1069,8 @@ static void flush_tlb_func(void *info) > return; > } >=20 > + local_tlb_gen =3D = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); > + > if (unlikely(f->new_tlb_gen !=3D TLB_GENERATION_INVALID && > f->new_tlb_gen <=3D local_tlb_gen)) { > /* > @@ -953,7 +1238,7 @@ STATIC_NOPV void native_flush_tlb_multi(const = struct cpumask *cpumask, > * up on the new contents of what used to be page tables, while > * doing a speculative memory access. > */ > - if (info->freed_tables) > + if (info->freed_tables || in_asid_transition(info)) > on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, = true); > else > on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func, > @@ -1026,14 +1311,18 @@ void flush_tlb_mm_range(struct mm_struct *mm, = unsigned long start, > bool freed_tables) > { > struct flush_tlb_info *info; > + unsigned long threshold =3D tlb_single_page_flush_ceiling; > u64 new_tlb_gen; > int cpu; >=20 > + if (static_cpu_has(X86_FEATURE_INVLPGB)) > + threshold *=3D invlpgb_count_max; I know it=E2=80=99s not really impacting performance, but it is hard for = me to see such calculations happening unnecessarily every time...