From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18227C021B8 for ; Tue, 4 Mar 2025 12:05:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 922CC280003; Tue, 4 Mar 2025 07:05:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AC01280001; Tue, 4 Mar 2025 07:05:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7738E280003; Tue, 4 Mar 2025 07:05:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 512EA280001 for ; Tue, 4 Mar 2025 07:05:24 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F24CF557B9 for ; Tue, 4 Mar 2025 12:05:23 +0000 (UTC) X-FDA: 83183738526.23.762A451 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) by imf23.hostedemail.com (Postfix) with ESMTP id C7C6F140023 for ; Tue, 4 Mar 2025 12:05:21 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=YkV2XnKS; spf=pass (imf23.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741089922; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vSIowNyCqsQ3yQ5qd5aOdrOx5Br2Nf7cQMl4V4s4NFI=; b=CpPQg6osJLVslOgzTreNI42GFdtE0XgowcPovDunttXHlGiBFDkf8GybaQxNRorLBZAjAG WMUg++htMgzO+sMxxEJkiv1oPW5V1buzxbHU6Zwn9ORKs/MKhY58KRRGYXoCLNBOx2gXl4 krW/mrS4ifrRURY3PkVZV1VSJoMVvyo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741089922; a=rsa-sha256; cv=none; b=Z6C+D18YA1YOJ7/xSm0HDy40Fw0U03Pb6ty39yl+yLk4vL5E3wy7bfTFGWbJEZatnOUGQg g8hXL6LjQD/Sr0rITX59wmGDZhlpdG4kffEVqOhOs88b9UVFOMf9yG4W9CXiKUj0TUjY06 ep1n8ifN4gUO0mnYR8HrTqJvW6pE3hM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=YkV2XnKS; spf=pass (imf23.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 399D240E0214; Tue, 4 Mar 2025 12:05:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5rp5czbxZx6k; Tue, 4 Mar 2025 12:05:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1741089913; bh=vSIowNyCqsQ3yQ5qd5aOdrOx5Br2Nf7cQMl4V4s4NFI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YkV2XnKSHaOiTk0OF/6kS1haFPLjq46gR2peMMrngiX5KptleErgGxN7RdJDPwS/r fwhb3x/TgfIs4Ltw0iYNEcI6u6dMrGREsPGBm8yU6WAz2yTr/Nbv5Ce93dChg8Bu9E zr1rZ9EMJUm7f0GMSL8jLDW9NugrajFtvE1nsabCTDVlY0YMDPFwoazzh9+CUjccTv 1I3UsSET6YQAirEMobmLphjdu7feNel210HJo3PCmt8iLSu7N7aUd7o16kLVFX6vPW 0RdLGpmI8P1mF/tpPWor+7UGOnqcCUXGSgQd9ll1DGuZYK5vZLzQi5VcF+xTEZ2Vof Ue2bOkZSoR+STn136u5TbpR+HJWpVhEQojVRoXnwkMN1rvB9u1gk6KaVhPnGTICqnx 6trVSM/EbI2l1nenGrzj5ILDE3KQp25YRYO12mXGFsp0UhjtrGDala/iO27dBS+nO9 cqsMnPeWS28m6g0iWzGE2UCFhluYB+sPu9zzVbgHDWr775MKGOhEhvyU/ug/W+Q2Aq jlsNgAIBpIEpWY1QGuj27dKFA3qjnfjo5k54a3CTfymqPxpj2ucS6zJZD+v55mPLZ/ HssgeyVzhT37H7dvbkndyYyVDup2GevoqLgPGigHiqzWSY4nPzIwvsd7K2r2I0RItg iN2DAPM99F4Co7lFRstS1XcY= Received: from zn.tnic (pd95303ce.dip0.t-ipconnect.de [217.83.3.206]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 3DF8240E020E; Tue, 4 Mar 2025 12:04:56 +0000 (UTC) Date: Tue, 4 Mar 2025 13:04:49 +0100 From: Borislav Petkov To: Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org Subject: [PATCH] x86/mm: Always set the ASID valid bit for the INVLPGB instruction Message-ID: <20250304120449.GHZ8bsYYyEBOKQIxBm@fat_crate.local> References: <20250226030129.530345-1-riel@surriel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20250226030129.530345-1-riel@surriel.com> X-Stat-Signature: bmk6kyz7pgk4j3rm6e68he319uzxnbgo X-Rspamd-Queue-Id: C7C6F140023 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741089921-940500 X-HE-Meta: U2FsdGVkX1+zWfrk4ofMbCIfKhhkggunGxiCB+LlOgCYFlT2GRyYSqTce+esxRblMSVjgO8z4uxBFp2d4Oupy+5rZ9Dh1E1EzTDnrJQ4gLoiRfyiJqFPRfA4buIxT/w3NMrEPbYs2OFzlST7XDZI1ChT5tp6ypjrnbz9KcdwI0fjgBXfNkJZWA2gmbJfk2noj7gynlx5kQaJdf++mXk6nuyCLi8tMkjA1S78eOfXXHIZxVUD4m7NCPU7pyRQ/4L2HEBhGnWHHnRshMu2Du+YqxY2ywUobLx33nEZJmvxyP0ARnbZ8ZKh47aE0m5XTyJy9Nvs+JLx6eRDDzs0kttVgYVTrXQQk4sOhsRYNhhwoEG8bGMVpLx3jfdScnzF9K8O52eYs4HQEUj1RPr1qt8Em8LpQjFYfOR1EmybaU0hY+rZ8+vTOnoPheqBXdpL5MAg9BjT3YljB+0hzneNHve/R4omI9KIUdcEFT6qEoLrys/AA11NFIxEiQ/sP8qshbxlg+61amecPBeu5oq1dRrwu5fqLIeGgv7OEqqYOcWxhGnV/cCyRbrP1wrbf7yuHB3oq06uy+KGRvh9bIQUM5Zz9TrM7zNiLk2naG51pyRbZwZZhSQdkh0emHDOkGviAZgRZeiTiIn17HVj9UTD/uvSKOT8FPfCWf8EmHyrwQHLjJhPPkmv3QHEfVlT09PeaDGHxT1QYyC2wWZOCf5FlB7hIBWx7FUWi1oEciygQpavACOXFJRwpoVwf/WAoAOM0+d1mUFYbZ9T6FbRljw9Ay7q9piCcxboTP5GU1ImYBU1TZ6HoLfaVUBFuTfQILEGr2CO8MJrCpqCdmaL3LeidOTWWPdDD43J6Hr5vqnFOTVsxMLnhB11OTdKuaQlMSZ7ayVHhUcduReqCuFz/xmYhLo954hfMFo80nAYWHa1IHvdLoZtZTssbUj2YL/dk04Gt+j4n0m4H0560VOSKnLnZxs vDVXreBv u3EjDaXOongKT8GtJGuxNsZUParemehRXMkSyDus3sJwSqpZko5zeFg43iRzb0W+DiLFl9YEJVVr2iGjFs3qPUBwrvI090zkCM8O/aDTZ5/nVFRDyEMP/9O5CyEACPunYvuJ41UjxYeflGHkd2wh5Z+sAbQ11IcIjMy2CAsfqQGDs5Dag/oCiT/5IKm6dQ1XykfNGZd3zUbAGXPqgU8hpCG6KS87ow8Tvk9C3oii537hRofHPXUZAlUddgbZcSXUv6mGisBVDMQEKf0gC4lPEkaSgb07OrGtvC32zLCIKcn6VV0SKAoE4CYgqDQVreGc94UAJV5HRw7BIqREHjntL5L2VeImTo4cY//61gppqJ/Tq70k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 25, 2025 at 10:00:35PM -0500, Rik van Riel wrote: > Add support for broadcast TLB invalidation using AMD's INVLPGB instruction. One more patch ontop from Tom. Now lemme test this a bit again... From: Tom Lendacky Date: Tue, 4 Mar 2025 12:59:56 +0100 Subject: [PATCH] x86/mm: Always set the ASID valid bit for the INVLPGB instruction When executing the INVLPGB instruction on a bare-metal host or hypervisor, if the ASID valid bit is not set, the instruction will flush the TLB entries that match the specified criteria for any ASID, not just the those of the host. If virtual machines are running on the system, this may result in inadvertent flushes of guest TLB entries. When executing the INVLPGB instruction in a guest and the INVLPGB instruction is not intercepted by the hypervisor, the hardware will replace the requested ASID with the guest ASID and set the ASID valid bit before doing the broadcast invalidation. Thus a guest is only able to flush its own TLB entries. So to limit the host TLB flushing reach, always set the ASID valid bit using an ASID value of 0 (which represents the host/hypervisor). This will will result in the desired effect in both host and guest. Signed-off-by: Tom Lendacky Signed-off-by: Borislav Petkov (AMD) --- arch/x86/include/asm/tlb.h | 58 +++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index e8561a846754..56fe331fb797 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -33,6 +33,27 @@ enum addr_stride { PMD_STRIDE = 1 }; +/* + * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination + * of the three. For example: + * - FLAG_VA | FLAG_INCLUDE_GLOBAL: invalidate all TLB entries at the address + * - FLAG_PCID: invalidate all TLB entries matching the PCID + * + * The first is used to invalidate (kernel) mappings at a particular + * address across all processes. + * + * The latter invalidates all TLB entries matching a PCID. + */ +#define INVLPGB_FLAG_VA BIT(0) +#define INVLPGB_FLAG_PCID BIT(1) +#define INVLPGB_FLAG_ASID BIT(2) +#define INVLPGB_FLAG_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FLAG_FINAL_ONLY BIT(4) +#define INVLPGB_FLAG_INCLUDE_NESTED BIT(5) + +/* The implied mode when all bits are clear: */ +#define INVLPGB_MODE_ALL_NONGLOBALS 0UL + #ifdef CONFIG_BROADCAST_TLB_FLUSH /* * INVLPGB does broadcast TLB invalidation across all the CPUs in the system. @@ -40,14 +61,20 @@ enum addr_stride { * The INVLPGB instruction is weakly ordered, and a batch of invalidations can * be done in a parallel fashion. * - * The instruction takes the number of extra pages to invalidate, beyond - * the first page, while __invlpgb gets the more human readable number of - * pages to invalidate. + * The instruction takes the number of extra pages to invalidate, beyond the + * first page, while __invlpgb gets the more human readable number of pages to + * invalidate. * * The bits in rax[0:2] determine respectively which components of the address * (VA, PCID, ASID) get compared when flushing. If neither bits are set, *any* * address in the specified range matches. * + * Since it is desired to only flush TLB entries for the ASID that is executing + * the instruction (a host/hypervisor or a guest), the ASID valid bit should + * always be set. On a host/hypervisor, the hardware will use the ASID value + * specified in EDX[15:0] (which should be 0). On a guest, the hardware will + * use the actual ASID value of the guest. + * * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from * this CPU have completed. */ @@ -55,9 +82,9 @@ static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr, u16 nr_pages, enum addr_stride stride, u8 flags) { - u32 edx = (pcid << 16) | asid; + u64 rax = addr | flags | INVLPGB_FLAG_ASID; u32 ecx = (stride << 31) | (nr_pages - 1); - u64 rax = addr | flags; + u32 edx = (pcid << 16) | asid; /* The low bits in rax are for flags. Verify addr is clean. */ VM_WARN_ON_ONCE(addr & ~PAGE_MASK); @@ -87,27 +114,6 @@ static inline void __invlpgb(unsigned long asid, unsigned long pcid, static inline void __tlbsync(void) { } #endif -/* - * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination - * of the three. For example: - * - FLAG_VA | FLAG_INCLUDE_GLOBAL: invalidate all TLB entries at the address - * - FLAG_PCID: invalidate all TLB entries matching the PCID - * - * The first is used to invalidate (kernel) mappings at a particular - * address across all processes. - * - * The latter invalidates all TLB entries matching a PCID. - */ -#define INVLPGB_FLAG_VA BIT(0) -#define INVLPGB_FLAG_PCID BIT(1) -#define INVLPGB_FLAG_ASID BIT(2) -#define INVLPGB_FLAG_INCLUDE_GLOBAL BIT(3) -#define INVLPGB_FLAG_FINAL_ONLY BIT(4) -#define INVLPGB_FLAG_INCLUDE_NESTED BIT(5) - -/* The implied mode when all bits are clear: */ -#define INVLPGB_MODE_ALL_NONGLOBALS 0UL - static inline void __invlpgb_flush_user_nr_nosync(unsigned long pcid, unsigned long addr, u16 nr, bool stride) -- 2.43.0 -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette