From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7881AD78788 for ; Thu, 21 Nov 2024 16:26:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8953B6B007B; Thu, 21 Nov 2024 11:26:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81D8A6B0082; Thu, 21 Nov 2024 11:26:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6970F6B0085; Thu, 21 Nov 2024 11:26:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4B20A6B007B for ; Thu, 21 Nov 2024 11:26:37 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C1864120F11 for ; Thu, 21 Nov 2024 16:26:36 +0000 (UTC) X-FDA: 82810628712.21.6063E8E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id D2FC140019 for ; Thu, 21 Nov 2024 16:25:27 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VOP1mjd0; dmarc=none; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732206209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tuw6u61XVKgJzrI+RtEbzlt5dJzkuA15z61icpJOg2Y=; b=fAeojL99S8r2UQwv/UB8I0b1nxJmZ3J4HKUhsB0kfJzUEgzGUQ5UyUfqxXLJjXxzpEzk/R du2XD90GXMabJsy3fm+Z61uBWcKcO+BeD2hfrHOaqeG3+oMwuSZNc0hf8+/GpO6EHWV/yj agum7/YEq52TJa9hS0vc931Ue6NCf2o= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=VOP1mjd0; dmarc=none; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732206209; a=rsa-sha256; cv=none; b=dIbtJkuANW3URxgeIIFYHWUR4jOh9bNzUVFDHSy07CJMaKH/sN0QHnwyYKSXOwFq07dlYe bUC82lIxfAgRu3xUH40zFu1exVjSxHQwTBlUF0SkCRiNDeRDHUnIIaZ5pLXVSgUaFTW1nv oy/wZCgLOh1fcn6QWo3g7+/R4cBgtqs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Tuw6u61XVKgJzrI+RtEbzlt5dJzkuA15z61icpJOg2Y=; b=VOP1mjd0n9h7RsYZs0Q5ZkeARh 64Dxb1fCHluTUer5pycoxBMwlPq3sVsjV874CwaAx38wYIMDpIvkHeFWd9HsDmgvK/KbxK9OeviIn Jp4y7YwfdrXzLLVcCdIjTGtm5OdS1gIHpdDLqXEuJAsZPcAhvcJH3oI0v5Yls3X3XFEqCZGPK2q4p N3PTXsNIhlA1jtjZ9veHX/fqUeszwhWX+ijSVInuGupwCioCAnk+RkAOi1j3rZP02aiTgtdWSeKin FjQsDlflSJbwUF/xPs6pzoeDLcwufwdGAJnkcj+mXt2Ojs9QKOAVrDikRzL0hg4Zi2Yu23wOsdOgA u2DiSygA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux)) id 1tEA0Q-00000006ZOa-3hhy; Thu, 21 Nov 2024 16:26:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 9C68430068B; Thu, 21 Nov 2024 17:26:09 +0100 (CET) Date: Thu, 21 Nov 2024 17:26:09 +0100 From: Peter Zijlstra To: Valentin Schneider Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org, Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , Thomas =?iso-8859-1?Q?Wei=DFschuh?= , Juri Lelli , Marcelo Tosatti , Yair Podemsky , Daniel Wagner , Petr Tesarik Subject: Re: [RFC PATCH v3 13/15] context_tracking,x86: Add infrastructure to defer kernel TLBI Message-ID: <20241121162609.GM24774@noisy.programming.kicks-ass.net> References: <20241119153502.41361-1-vschneid@redhat.com> <20241119153502.41361-14-vschneid@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20241119153502.41361-14-vschneid@redhat.com> X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D2FC140019 X-Stat-Signature: 6tdo3ocyhasxfqrssziikmk78ht8y1bu X-Rspam-User: X-HE-Tag: 1732206327-485594 X-HE-Meta: U2FsdGVkX1/v4EhxiVfpdfak22Nct+Kp1I08wsyDX+MCE0Rqm8xhO9RV8wiL1Wpn6GDKAEIPsTBFbgnEDeDdp5IHbqBIpTqunoG66SUML+abAQddsZTcgS9sbA55PVTkHX/zCRnfZkxaLL9EwpIR2caJpPzJBWz7yMgAI4zbUBbhHmB5d47uSqOclpDHJy6eT3mVFq/UP9+Viq6RBtiIZwilTXdyzuOnaqag1DtKux890dpqOE1Qh7BDYBxONW0TkxeqkXoD8c8NrzPlbD2oYguSGCtXL+xOyTqhJaNhoUjthftwupqzL0ZlpwMZYjBMSAK9WdOf13okeCJxqXKlEuZdtouSVcBvl/dS+y5sGJOYpI5+e1r9r92krh21qhK/aP0nLv8JqxIRjvqXHi3w003icgpOomkGPMMdbFdoHpRKkgwpK1yYHaCI1lDbtr5n9WxDdhGljQr6xIDI4UDphGAZA2oh3RpAschyGiLMNnTRVGhzDaRs4icxyHnbo0iGw2WdlekJnuPRM6nvVfjfoOw+nzWi/zYnd1wUX0iDC5pxBOu2fD2L6dVIzh0AacBGyyL906G00ZAEB18Z09tBF4auv2xHcNXcgkcsHGj5iXuOFpspah8hw8zPPocvx1+bvmyA7ANuPDvKMlpoWJG6LzmWvlzo6DgmxW0dcGofctcrdc0MQ4gHfsfn95d3tuhdlNkV4/G+xzXI5E3Sv15b3PYwdNZ2sTpw9H2LsuxKYy8qopm9MWUNuaJ618KhEbA+GJm+5MAR1d73HdTfEzDnyxBrZ0gXMkrK9nb5ey4l5I38VWeUM42jynZ1aZgAdDeLrZEPdfSXoLUQDfLFS30E6KvwqpvdXzXCuG5Aw10MAkecYh8fDFjNU9Ne8PjwwURz1/n15+NEeSsKif0qui2HoCQKbf65CcM573cWHToZe8ERCYq8k9pa6l2n1O89ULniZ7knL9DO43Beja6LCc2 LhAyCPWR 6KeQaA0n+GCq73GaZ1AhswIShspJnBd8Wyhnt3lp7bg+CfuW8/uouDfd5XNnCrrBW9qjDq+xI8g71wXuTg0RS4udsKG825Msp9K6Nw47yFF6qeaAAbnB5NbdH6U5JglfuwUDQWeorqhP9YlBx/GHqsBoe8KrJKlkTh+pF2cZEZMCQgcRbumVYhcvuUSpQZ9jEEhYvnJZ5IAf57+zMwaqWD28pl28X0MVKUMJo5N3ISo+EteLfNRbwtnxydW5Vkv6xoCHezsdhQ7llj0qhI9p8tC/7Jd2jG2uWv5eRaITgY7grD4k8eRFMEqYK1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Nov 19, 2024 at 04:35:00PM +0100, Valentin Schneider wrote: > @@ -418,9 +419,20 @@ static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) > #endif > #endif /* !MODULE */ > > +#define __NATIVE_TLB_FLUSH_GLOBAL(suffix, cr4) \ > + native_write_cr4##suffix(cr4 ^ X86_CR4_PGE); \ > + native_write_cr4##suffix(cr4) > +#define NATIVE_TLB_FLUSH_GLOBAL(cr4) __NATIVE_TLB_FLUSH_GLOBAL(, cr4) > +#define NATIVE_TLB_FLUSH_GLOBAL_NOINSTR(cr4) __NATIVE_TLB_FLUSH_GLOBAL(_noinstr, cr4) > + > static inline void __native_tlb_flush_global(unsigned long cr4) > { > - native_write_cr4(cr4 ^ X86_CR4_PGE); > - native_write_cr4(cr4); > + NATIVE_TLB_FLUSH_GLOBAL(cr4); > } > + > +static inline void __native_tlb_flush_global_noinstr(unsigned long cr4) > +{ > + NATIVE_TLB_FLUSH_GLOBAL_NOINSTR(cr4); > +} How about something like this instead? I've only compile tested the tlb.c bit, but it should get __flush_tlb_global() to be noinstr I think, including the Xen bit (unless I missed something but then objtool should complain). --- diff --git a/arch/x86/include/asm/invpcid.h b/arch/x86/include/asm/invpcid.h index 734482afbf81..ff26136fcd9c 100644 --- a/arch/x86/include/asm/invpcid.h +++ b/arch/x86/include/asm/invpcid.h @@ -2,7 +2,7 @@ #ifndef _ASM_X86_INVPCID #define _ASM_X86_INVPCID -static inline void __invpcid(unsigned long pcid, unsigned long addr, +static __always_inline void __invpcid(unsigned long pcid, unsigned long addr, unsigned long type) { struct { u64 d[2]; } desc = { { pcid, addr } }; @@ -13,7 +13,7 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, * mappings, we don't want the compiler to reorder any subsequent * memory accesses before the TLB flush. */ - asm volatile("invpcid %[desc], %[type]" + asm_inline volatile("invpcid %[desc], %[type]" :: [desc] "m" (desc), [type] "r" (type) : "memory"); } @@ -23,26 +23,25 @@ static inline void __invpcid(unsigned long pcid, unsigned long addr, #define INVPCID_TYPE_ALL_NON_GLOBAL 3 /* Flush all mappings for a given pcid and addr, not including globals. */ -static inline void invpcid_flush_one(unsigned long pcid, - unsigned long addr) +static __always_inline void invpcid_flush_one(unsigned long pcid, unsigned long addr) { __invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR); } /* Flush all mappings for a given PCID, not including globals. */ -static inline void invpcid_flush_single_context(unsigned long pcid) +static __always_inline void invpcid_flush_single_context(unsigned long pcid) { __invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT); } /* Flush all mappings, including globals, for all PCIDs. */ -static inline void invpcid_flush_all(void) +static __always_inline void invpcid_flush_all(void) { __invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL); } /* Flush all mappings for all PCIDs except globals. */ -static inline void invpcid_flush_all_nonglobals(void) +static __always_inline void invpcid_flush_all_nonglobals(void) { __invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL); } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index d4eb9e1d61b8..b3daee3d4667 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -75,7 +75,7 @@ static inline void __flush_tlb_local(void) PVOP_VCALL0(mmu.flush_tlb_user); } -static inline void __flush_tlb_global(void) +static __always_inline void __flush_tlb_global(void) { PVOP_VCALL0(mmu.flush_tlb_kernel); } diff --git a/arch/x86/include/asm/xen/hypercall.h b/arch/x86/include/asm/xen/hypercall.h index a2dd24947eb8..b4c635b20538 100644 --- a/arch/x86/include/asm/xen/hypercall.h +++ b/arch/x86/include/asm/xen/hypercall.h @@ -357,8 +357,8 @@ MULTI_mmu_update(struct multicall_entry *mcl, struct mmu_update *req, trace_xen_mc_entry(mcl, 4); } -static inline void -MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count, +static __always_inline void +__MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count, int *success_count, domid_t domid) { mcl->op = __HYPERVISOR_mmuext_op; @@ -366,6 +366,13 @@ MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count, mcl->args[1] = count; mcl->args[2] = (unsigned long)success_count; mcl->args[3] = domid; +} + +static inline void +MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count, + int *success_count, domid_t domid) +{ + __MULTI_mmuext_op(mcl, op, count, success_count, domid); trace_xen_mc_entry(mcl, 4); } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index b0d5a644fc84..0cfc00a34b7e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1168,9 +1168,10 @@ void flush_tlb_one_user(unsigned long addr) /* * Flush everything */ -STATIC_NOPV void native_flush_tlb_global(void) +STATIC_NOPV noinstr void native_flush_tlb_global(void) { unsigned long flags; + unsigned long cr4; if (static_cpu_has(X86_FEATURE_INVPCID)) { /* @@ -1189,9 +1190,15 @@ STATIC_NOPV void native_flush_tlb_global(void) * be called from deep inside debugging code.) */ raw_local_irq_save(flags); - - __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4)); - + cr4 = this_cpu_read(cpu_tlbstate.cr4); + asm volatile("mov %0,%%cr4": : "r" (cr4 ^ X86_CR4_PGE) : "memory"); + asm volatile("mov %0,%%cr4": : "r" (cr4) : "memory"); + /* + * In lieu of not having the pinning crap, hard fail if CR4 doesn't + * match the expected value. This ensures that anybody doing dodgy gets + * the fallthrough check. + */ + BUG_ON(cr4 != this_cpu_read(cpu_tlbstate.cr4)); raw_local_irq_restore(flags); } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index 55a4996d0c04..4eb265eb867a 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -1231,22 +1231,22 @@ static noinstr void xen_write_cr2(unsigned long cr2) this_cpu_read(xen_vcpu)->arch.cr2 = cr2; } -static noinline void xen_flush_tlb(void) +static noinline noinstr void xen_flush_tlb(void) { struct mmuext_op *op; struct multicall_space mcs; - preempt_disable(); + preempt_disable_notrace(); mcs = xen_mc_entry(sizeof(*op)); op = mcs.args; op->cmd = MMUEXT_TLB_FLUSH_LOCAL; - MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF); + __MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF); - xen_mc_issue(XEN_LAZY_MMU); + __xen_mc_issue(XEN_LAZY_MMU); - preempt_enable(); + preempt_enable_notrace(); } static void xen_flush_tlb_one_user(unsigned long addr) diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h index e1b782e823e6..31eddca45c27 100644 --- a/arch/x86/xen/xen-ops.h +++ b/arch/x86/xen/xen-ops.h @@ -235,15 +235,19 @@ static inline struct multicall_space xen_mc_entry(size_t args) void xen_mc_flush(void); /* Issue a multicall if we're not in a lazy mode */ -static inline void xen_mc_issue(unsigned mode) +static __always_inline void __xen_mc_issue(unsigned mode) { - trace_xen_mc_issue(mode); - if ((xen_get_lazy_mode() & mode) == 0) xen_mc_flush(); /* restore flags saved in xen_mc_batch */ - local_irq_restore(this_cpu_read(xen_mc_irq_flags)); + raw_local_irq_restore(this_cpu_read(xen_mc_irq_flags)); +} + +static inline void xen_mc_issue(unsigned mode) +{ + trace_xen_mc_issue(mode); + __xen_mc_issue(mode); } /* Set up a callback to be called when the current batch is flushed */