From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88D87C021B8 for ; Sat, 1 Mar 2025 12:20:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BFA76B007B; Sat, 1 Mar 2025 07:20:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 647F36B0082; Sat, 1 Mar 2025 07:20:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E8576B0083; Sat, 1 Mar 2025 07:20:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2CB946B007B for ; Sat, 1 Mar 2025 07:20:56 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D10B18206A for ; Sat, 1 Mar 2025 12:20:55 +0000 (UTC) X-FDA: 83172891270.16.F026B22 Received: from mail.alien8.de (mail.alien8.de [65.109.113.108]) by imf09.hostedemail.com (Postfix) with ESMTP id C5367140005 for ; Sat, 1 Mar 2025 12:20:53 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=SEa8ZM6Z; spf=pass (imf09.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740831654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2HKI4C6iYTI8d0b77jQiRO2WBjPRZTq44Ez/ai+zb8M=; b=R0HEwEQJP/iXMV2R9dLibgeFNej8qmMgEHz0Qh6IsNyPBSbLySdu0gAG092owdJ/b8290y VGuTv7fiplCMCcSDgLzfUZ79/J9IRCYadVpXVksAAMVLS9jNNOlBjTXpMEJFC27z4pvDXN vA234Fg1XrzbuMN+vAKTYgVWyfw9sIA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=alien8.de header.s=alien8 header.b=SEa8ZM6Z; spf=pass (imf09.hostedemail.com: domain of bp@alien8.de designates 65.109.113.108 as permitted sender) smtp.mailfrom=bp@alien8.de; dmarc=pass (policy=none) header.from=alien8.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740831654; a=rsa-sha256; cv=none; b=I6+Kqej/0QaNp6oJ606DXabhjvUwuPHMjo/foPZVLK3Gj41ZttQKv2UxwSFhL6IXigR3kP K5MbSP+wkK1bbY1tGvli71mxHRIFmzsu0G9XyqgiRX4KqiW5+d8ISm4pdHsA8btAUqfivv vgffBJPcVSVe8Nnjhaou+fot7b7K+g4= Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTP id 306A240E0173; Sat, 1 Mar 2025 12:20:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at mail.alien8.de Received: from mail.alien8.de ([127.0.0.1]) by localhost (mail.alien8.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id aAMgY6Qui3vK; Sat, 1 Mar 2025 12:20:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alien8.de; s=alien8; t=1740831646; bh=2HKI4C6iYTI8d0b77jQiRO2WBjPRZTq44Ez/ai+zb8M=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=SEa8ZM6ZDzgwLG1Sy6X1VKsTnDNPNiDmWVUcx54keQW6LFpqTrkOaJmOSC0ZjxABP UumLuYDhnxiGSEMB+06g60r5RKI3RJmzefY6DhBxAMAZAgpEr2Dw5lF0bEzn3eNM5K jTAeWntzw8EJHllnk/lOgzLhkG1cTXIM3T7UyDZLqKwVgYZ36p44S/RDc39IJtxd8l d4xGpXUbk53A9BDaBCeTWXoRBohC4Oyc1jrh5O4dLogucJCMxwjmg+XmBVleqOgDns FuUwAEp3hq14jQXuZE1cZEUauWkXWAMPCAtBM4VoPVz3m2vRz3w6Jh/hSPRghc3oKF OVWof9edTegcqpknaIDt2YWD0AczI6+FXqoqXy5lxO0nVuXrTL0b++oE6pnADETqxb IlAdrQmT2JQdiZbvfxKN9a9vLRGUTKomYnt/8jCEVMHcfGDUUYrFs223aX/LpSgeBC vwxxCjLYKwz8of/uAASFk0CphkDYN88JvygkjII1bVWLYd4CjVVuAZ4zmeWTMmV68i A8FITSN0yo0NjMNlv9BnFeEiyY+7KPrLPbl0DJ44lC1MoyGR0Nyr0tcgUnooKuLHCu 2gxdAR1iWpeGJXMk/g0HPt1dw0fMOo72wGlm+BE8vVu7FaBgOWllAa8b5Q+baef/PY nT/mGJz88bIstsHanta9E2Qg= Received: from zn.tnic (pd95303ce.dip0.t-ipconnect.de [217.83.3.206]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by mail.alien8.de (SuperMail on ZX Spectrum 128k) with ESMTPSA id 37F6E40E015F; Sat, 1 Mar 2025 12:20:28 +0000 (UTC) Date: Sat, 1 Mar 2025 13:20:22 +0100 From: Borislav Petkov To: Dave Hansen , Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jackmanb@google.com, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali.Shukla@amd.com, mingo@kernel.org Subject: Re: [PATCH v14 05/13] x86/mm: use INVLPGB in flush_tlb_all Message-ID: <20250301122022.GBZ8L7hlwP5cUffJA2@fat_crate.local> References: <20250226030129.530345-1-riel@surriel.com> <20250226030129.530345-6-riel@surriel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Stat-Signature: oimr69xqatcrmd58jj4a9ej1qk5ibutt X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C5367140005 X-Rspam-User: X-HE-Tag: 1740831653-40639 X-HE-Meta: U2FsdGVkX19o+evAUoyxH7yDF9YAXU3sMiTILxoLZPwrIaVrr7BJJpD8KP8aHorXPlAiZelmcc7lnF/n7OYdN92UyJOhMx6EPhVYgG7YKFNN5p8kgk+hmC0oSRoGQdbeiT43QVkG6CAnmz6o0ImrMUN3s4UBk6Pm+Tm4DR5sADGzxHS5Dfsl6swYbmwOOgIa8bnLL3fbQ0mOWEUEnHUBcWFftktpnRBvDOLh5pIbxveNgJHIKAMfcDVgEmu6IpLuaYIIkd8OHOgE92x2peiLy6+YsUgeYMdo5ixrSLV62mG8dbYhHRcwdCiPL3pM3mnGUwJSGtIRGgTlTOkVMlN3FPR/tawPGVLoQQyjPThK7LqBwqdXwvNE43smqth9l/xEBmqlJ83/nH/1Ulc+jBxbUHseay8UE2Jf4hxFLVg/qhEYcrt4BfUAQqwZ8MKjhAjukm1+jINv1Locc2J6DDi3s2Ue99Vg9fY2iQJn1rOKEcHEKKDoWuhzPCoiHIeOv2/feyOGrtHgraIbogWrYugE5+3pqa9VTREweO7shrjJ5CZ7XvG5McFFJyhTnxWRGxCpMeipnZoQ5T24O0C7+KuC78gQXK9iirkYqsvtEiaqgyRMn6GgaqWoj9n+r/+eYqBaMzud2mdG5Y9u5T1+Pvtv/NzwR+webVQN5qMCWVgMcynolhcbHXKReDOy8/gvLqKIRAxmH7I0nkfC0eqpwzxxkGW0d55s940m7JhY+OWybzcFOL3ZB724tVdZ6w4/8rH6TjkclNWnJiQUGtYpwXApCtp+mq2dt/WkMXXWt2KfPVjhMLwJYRe4wcsXjtNHPBgG2I3tGgJ5uNK0i4V59sq5CVELo/nhpj+yi79KxmvlxlyHmoHYPtt7JRnfLP3XgQcTiSIJYtpLoY2vXnAecQjoG07J+NjsOObn932do2uJD7aDUIo/LhrWXihN/A2x2Uk4XPtCqJhYub1PM13iUxs onTQrqtu LJfTodxrQAdf4+1e+lOqScIQd9JKqEOtPw8HYLnAUhiStwyyhYO6/dKZ56a0dkCRKqV+pHS3NHHvT0bU9SPu1rO5nFxkIkAtccB9r09Yeef0RvhUPpoeIP22FsiPwb6dJXw5Vn8ohv8ZlXSeu+bVa2Z+KBpnrIHKJll+YBPFtOrI8++rcp/DVNiQKIDiEMPscCpOrEYTHjK/DtRCrZLfF0QVbY+IjlxpzfebQ2783H1nB2mPKjq9K7hR6GWbNkBoVzAfYUzmN7/XWli7HwfAUFimCW//mKxC3yMfZ8rHiDBKTtGdeHgFYlnxVcRJzpqH5quoHKU41ah+F5P0Bt6DeUHjlDsHKljuX77e0FFWdO9cXRPXktj35HEqmPbPOzMyti1w1ZxKdRlDxxW8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 28, 2025 at 11:18:04AM -0800, Dave Hansen wrote: > We haven't talked at all about the locking rules for > invlpgb_flush_all(). It was used once in this series without any > explicit preempt twiddling. I assume that was because it was used in a > path where preempt is disabled. > > If it does need a universal rule about preempt, can we please add an: > > lockdep_assert_preemption_disabled() > > along with a comment about why it needs preempt disabled? So, after talking on IRC last night, below is what I think we should do ontop. More specifically: - I've pushed the preemption guard inside the functions which do INVLPGB+TLBSYNC so that callers do not have to think about it. - invlpgb_kernel_range_flush() I still don't like and we have to rely there on cant_migrate() in __tlbsync() - I'd like for all of them to be nicely packed but don't have an idea yet how to do that cleanly... - document what means for bits rax[0:2] being clear when issuing INVLPGB That ok? Anything I've missed? If not, I'll integrate this into the patches. Thx. diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 45d9c7687d61..0d90ceeb472b 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -39,6 +39,10 @@ static inline void invlpg(unsigned long addr) * the first page, while __invlpgb gets the more human readable number of * pages to invalidate. * + * The bits in rax[0:2] determine respectively which components of the address + * (VA, PCID, ASID) get compared when flushing. If neither bits are set, *any* + * address in the specified range matches. + * * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from * this CPU have completed. */ @@ -60,10 +64,10 @@ static inline void __invlpgb(unsigned long asid, unsigned long pcid, static inline void __tlbsync(void) { /* - * tlbsync waits for invlpgb instructions originating on the - * same CPU to have completed. Print a warning if we could have - * migrated, and might not be waiting on all the invlpgbs issued - * during this TLB invalidation sequence. + * TLBSYNC waits for INVLPGB instructions originating on the same CPU + * to have completed. Print a warning if the task has been migrated, + * and might not be waiting on all the INVLPGBs issued during this TLB + * invalidation sequence. */ cant_migrate(); @@ -106,6 +110,13 @@ static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) /* Flush all mappings, including globals, for all PCIDs. */ static inline void invlpgb_flush_all(void) { + /* + * TLBSYNC at the end needs to make sure all flushes done on the + * current CPU have been executed system-wide. Therefore, make + * sure nothing gets migrated in-between but disable preemption + * as it is cheaper. + */ + guard(preempt)(); __invlpgb(0, 0, 0, 1, 0, INVLPGB_INCLUDE_GLOBAL); __tlbsync(); } @@ -119,10 +130,7 @@ static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) /* Flush all mappings for all PCIDs except globals. */ static inline void invlpgb_flush_all_nonglobals(void) { - /* - * @addr=0 means both rax[1] (valid PCID) and rax[2] (valid ASID) are clear - * so flush *any* PCID and ASID. - */ + guard(preempt)(); __invlpgb(0, 0, 0, 1, 0, 0); __tlbsync(); } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index f49627e02311..8cd084bc3d98 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1075,19 +1075,11 @@ void flush_tlb_all(void) count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); /* First try (faster) hardware-assisted TLB invalidation. */ - if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { - /* - * TLBSYNC at the end needs to make sure all flushes done - * on the current CPU have been executed system-wide. - * Therefore, make sure nothing gets migrated - * in-between but disable preemption as it is cheaper. - */ - guard(preempt)(); + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(); - } else { + else /* Fall back to the IPI-based invalidation. */ on_each_cpu(do_flush_tlb_all, NULL, 1); - } } /* Flush an arbitrarily large range of memory with INVLPGB. */ -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette