From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 730A2C021B8 for ; Tue, 4 Mar 2025 11:01:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2BAC6B0085; Tue, 4 Mar 2025 06:01:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB17C6B0088; Tue, 4 Mar 2025 06:01:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C535E6B0089; Tue, 4 Mar 2025 06:01:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 958486B0085 for ; Tue, 4 Mar 2025 06:01:13 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1ABF4C1B25 for ; Tue, 4 Mar 2025 11:01:13 +0000 (UTC) X-FDA: 83183576826.19.13F2EBB Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) by imf25.hostedemail.com (Postfix) with ESMTP id 9321BA0018 for ; Tue, 4 Mar 2025 11:01:10 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=agbvZhRk; spf=pass (imf25.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741086071; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F5mv4iED1rr2rJ7SCsELcpcVz62iO8EDzCtAjovfhLA=; b=V8C00THE7fMkb+F7FXNGsY7Ndu8lLbytMI8BSvPIDEDSc8e/cpTwEicDapWZU6/njQBF2/ 48UbDkYubwVDLcqpRR1Qzohpdk5Ab6W/DnwLVaOGko7OfEBU0MxAQ50VHsXgS3gx4YcCri SG/mZsWy2pg66GAhpTOZDIpk63HtXfU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=agbvZhRk; spf=pass (imf25.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741086071; a=rsa-sha256; cv=none; b=ay/NgKdOea3d61NerB1bm9pAzQfDkBZOTPVfbx2et0rlH8YA/2YjCSUGeK+nir26N+uQxm TnKNPnMnOsXedukjLaxjL8/PADSdL7DbpS5rX1zg4onMBLe9X3fdRvPLT+K3GoKNwUyt30 RC8O5PaOXpjHDm9ksvz5USQh8n9jBlM= Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 91B1240E0215; Tue, 4 Mar 2025 11:01:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 8Rx0Nk4klulw; Tue, 4 Mar 2025 11:01:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1741086061; bh=F5mv4iED1rr2rJ7SCsELcpcVz62iO8EDzCtAjovfhLA=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=agbvZhRk1Su3jPL8rGiKwTEAO/7GD0UwfXSxP+ZcC31uvsFfTFaB7SXirBevkxMQr R4xMVLf0DlGVZkAP47+QPhw9OzIIsq6U/oDUC8w5x2whKV8n2+qJCLZEtNHLgWwoLV 57xKybmBkauzo2CCVuL0WSO4nhlUEK0k6ce5sVU5xRecvFzEr/vtkHcNi5iz8LnU0B XCeR51ZSD0caO63FVwP3HE7gVfY4u0vWQtHRU/2qL5ipfID9yeMzmSPlz5sExdVKzt 9oUdlCovyemquEeWkRb3zBX3uzt/vzCTM3FFievF6NuBaxxjYX95iMaKDHpOcaw3+E rxn+cxcZo3GrYtTZa9l6zYuZZQDBWxoTiBlnCOH5IC5KAP6MtZPTL/b+NDXcMndFDo b08HSMH+1CpCl5XE2oeXEAizNYc2clrpQ/7Ni2VrqiD0LNodeH+wBcJeSZtaK/WQxy 6KBhMDU3KFTFrGNVuIjqIoZ/ajjH1pvK8gcyCPhuT5+WvZWTqGu/vQzTS/E0PA/5KB IHD0G2Mg/d9eOgnXxuzQIFwwBZdnnbQsjtF2nPqezEiT59KeBWI4ifrtCFcbEkZi9u YusuMG788xZ3SQdPFzPCXlMkcxrTvc7eQUxHctRAK5OHLetzDDGo+2FoLY56KYcc5L KdsGNaUvJF8/dI2Xzk3K3/h0= Received: from zn.tnic (pd95303ce.dip0.t-ipconnect.de [217.83.3.206]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 0E14A40E0214; Tue, 4 Mar 2025 11:00:42 +0000 (UTC) Date: Tue, 4 Mar 2025 12:00:32 +0100 From: Borislav Petkov To: Dave Hansen Cc: Rik van Riel , x86@kernel.org, linux-kernel@vger.kernel.org, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org Subject: Re: [PATCH v14 03/13] x86/mm: add INVLPGB support code Message-ID: <20250304110032.GEZ8bdUOg2WLUrhMcm@fat_crate.local> References: <20250226030129.530345-1-riel@surriel.com> <20250226030129.530345-4-riel@surriel.com> <20250228194734.GGZ8IS1iFVpPzmEyYl@fat_crate.local> <30c721e0-338d-4172-989c-5226d584bcbc@intel.com> <34b80474-a309-493b-81e9-3a7d4de8a369@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <34b80474-a309-493b-81e9-3a7d4de8a369@intel.com> X-Rspam-User: X-Stat-Signature: htzczk3zzpwxgdt177xaz4oxhmt9dh6p X-Rspamd-Queue-Id: 9321BA0018 X-Rspamd-Server: rspam07 X-HE-Tag: 1741086070-875170 X-HE-Meta: U2FsdGVkX1/G9sFKuxe+W7YaC6u3V7KcEVk/T/0VbPJO9ryGxADGCd08J3AR+SnM8aEdHpxKvqYJU90lEntFKEC+VcVoFQzZefsikTt4A3Mrf65n5yj7ciJo2g8/15ptk6DCjE7I+W612sgX6ktjsv6CbOIkZjKMkdEKeH9T3QiyhNnFz5g8bfTlhuY/jWZU37PoYivd9iVEFWqx2sbaQLR0tXvGzodzWUjw3h2DFhzvWq7c/nBZxguQg3TzUxbhaCZ9mUmnw6PAVkIseeWZ0SGmNuO4Yi1bzWVD77bvq7+4lJwVg2rTqBLlSu+6dBo/8USSQfoofwXgWuam1CgYOEoyuUEs7KWh9DEaf0LcA4NVUzcpTB3+ydIsIMcgD4ox2MtlvFAHpumyEfkyjeoYVIkUfifpekf829xs5/4hFPxsr54BeFXueowKA+1cb8OPT2I5ti1r83SpA+qYa4LL7z/QopVfk1jG6U5alu4m6G9cZc87hYrdQQormgGrI07QOL5ax3H9pk5237n8xAJ5g+rzeB+SstSBKVYd4kZDbjh4bd5ec6hweTSSHMWVMZgD8ZdbOx7WlixenASccN5uImwgAvwnmicjHC2nu2nKkLCCJv3/peEEUuvRu81J0uJbUWhfihjMeMkrRAQB6hDYwckL9q1uZ4hDG3ceaoGUumHaDuDnZCkU+VwaGMorE1oOnYsDU2ixwjUUXiKZ551VDfw/akSK97p8VEs3L0zPijSlWuRl2YN15KdiZ5Nvrm77N9qlc5Vu74jvBBPxwUd4KdMZmUdILjqMYe5oQXCEhIyBOtS/12j1Bbpcuu8kT5TjkPAxreUCUIJgwDtgMscqb15i2uCjHjZzyX3KtOwtO2zyO7oIwAnH6sfjENNo7wm/0YAPC1Bit6Eu93o0po5Dq/RBGGLrWGp3FoRuhgP3pDE3wC/L+8W2oZhY0li2GobbdYG7Xr88lfEb+eJof02 h3knMobf HU33c+QCWXm7P9+/BiwuEhQ42b66Vis/faE6ojQv3vr+ck8nCSl+3t1zgWxG2OxfjV73IemfZ70EpXdZ65nBvogAfDoks9oOaVWkCGI/FQqPydJXP4CVudR87U/uPvg73nWcI8YvTMRFMO26/T0RmwAAYnzCShwp54mPzRR4knCCXEvkzqFBOzMpfupzwSfWxzOm1PsAG962eWKHhco69HX86l3bU9Hc/FAzAYIbwiYjZnkNBnPSvHfEG8A/w0WnvSL1lm1l3UKJfxinUjFIMtuAUOXM22la1IGo2MIE77jjRVKan2zjFz1dwPrn20soBLNo8I+bpup6hdMfWKUDCfIK3ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Mar 03, 2025 at 11:23:58AM -0800, Dave Hansen wrote: > Here's a plain diff if you just want to squish it in. > diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h > index 5375145eb9596..3bd617c204346 100644 > --- a/arch/x86/include/asm/tlb.h > +++ b/arch/x86/include/asm/tlb.h > @@ -28,6 +28,11 @@ static inline void invlpg(unsigned long addr) > asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); > } > > +enum invlpgb_stride { Right, this is an address stride, as the text calls it. > + NO_STRIDE = 0, > + PTE_STRIDE = 0, Ok, so those are confusing. No stride is PTE stride so let's just zap NO_STRIDE. > + PMD_STRIDE = 1 > +}; > > /* > * INVLPGB does broadcast TLB invalidation across all the CPUs in the system. ... > static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, > unsigned long addr, > u16 nr, > bool pmd_stride) You're relying on the fact that true == PMD_STRIDE and false to PTE_STRIDE but let's make it Right(tm), see below. Rest looks ok. IOW, I'm merging this into patch 3: diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 5375145eb959..6718835c3b0c 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -28,6 +28,10 @@ static inline void invlpg(unsigned long addr) asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); } +enum addr_stride { + PTE_STRIDE = 0, + PMD_STRIDE = 1 +}; /* * INVLPGB does broadcast TLB invalidation across all the CPUs in the system. @@ -48,10 +52,10 @@ static inline void invlpg(unsigned long addr) */ static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr, u16 nr_pages, - bool pmd_stride, u8 flags) + enum addr_stride stride, u8 flags) { u32 edx = (pcid << 16) | asid; - u32 ecx = (pmd_stride << 31) | (nr_pages - 1); + u32 ecx = (stride << 31) | (nr_pages - 1); u64 rax = addr | flags; /* The low bits in rax are for flags. Verify addr is clean. */ @@ -78,33 +82,38 @@ static inline void __tlbsync(void) /* * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination * of the three. For example: - * - INVLPGB_VA | INVLPGB_INCLUDE_GLOBAL: invalidate all TLB entries at the address - * - INVLPGB_PCID: invalidate all TLB entries matching the PCID + * - FLAG_VA | FLAG_INCLUDE_GLOBAL: invalidate all TLB entries at the address + * - FLAG_PCID: invalidate all TLB entries matching the PCID * - * The first can be used to invalidate (kernel) mappings at a particular + * The first is used to invalidate (kernel) mappings at a particular * address across all processes. * * The latter invalidates all TLB entries matching a PCID. */ -#define INVLPGB_VA BIT(0) -#define INVLPGB_PCID BIT(1) -#define INVLPGB_ASID BIT(2) -#define INVLPGB_INCLUDE_GLOBAL BIT(3) -#define INVLPGB_FINAL_ONLY BIT(4) -#define INVLPGB_INCLUDE_NESTED BIT(5) +#define INVLPGB_FLAG_VA BIT(0) +#define INVLPGB_FLAG_PCID BIT(1) +#define INVLPGB_FLAG_ASID BIT(2) +#define INVLPGB_FLAG_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FLAG_FINAL_ONLY BIT(4) +#define INVLPGB_FLAG_INCLUDE_NESTED BIT(5) + +/* The implied mode when all bits are clear: */ +#define INVLPGB_MODE_ALL_NONGLOBALS 0UL static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, unsigned long addr, - u16 nr, - bool pmd_stride) + u16 nr, bool stride) { - __invlpgb(0, pcid, addr, nr, pmd_stride, INVLPGB_PCID | INVLPGB_VA); + enum addr_stride str = stride ? PMD_STRIDE : PTE_STRIDE; + u8 flags = INVLPGB_FLAG_PCID | INVLPGB_FLAG_VA; + + __invlpgb(0, pcid, addr, nr, str, flags); } /* Flush all mappings for a given PCID, not including globals. */ static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) { - __invlpgb(0, pcid, 0, 1, 0, INVLPGB_PCID); + __invlpgb(0, pcid, 0, 1, PTE_STRIDE, INVLPGB_FLAG_PCID); } /* Flush all mappings, including globals, for all PCIDs. */ @@ -117,21 +126,21 @@ static inline void invlpgb_flush_all(void) * as it is cheaper. */ guard(preempt)(); - __invlpgb(0, 0, 0, 1, 0, INVLPGB_INCLUDE_GLOBAL); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); __tlbsync(); } /* Flush addr, including globals, for all PCIDs. */ static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) { - __invlpgb(0, 0, addr, nr, 0, INVLPGB_INCLUDE_GLOBAL); + __invlpgb(0, 0, addr, nr, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); } /* Flush all mappings for all PCIDs except globals. */ static inline void invlpgb_flush_all_nonglobals(void) { guard(preempt)(); - __invlpgb(0, 0, 0, 1, 0, 0); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_MODE_ALL_NONGLOBALS); __tlbsync(); } #endif /* _ASM_X86_TLB_H */ -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette