From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 337C9C54754 for ; Tue, 20 May 2025 01:04:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A81026B009F; Mon, 19 May 2025 21:04:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E2656B00A3; Mon, 19 May 2025 21:04:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FAE56B009F; Mon, 19 May 2025 21:04:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 311666B00A1 for ; Mon, 19 May 2025 21:04:21 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D70091A199F for ; Tue, 20 May 2025 01:04:20 +0000 (UTC) X-FDA: 83461490280.17.8BCD1F9 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf09.hostedemail.com (Postfix) with ESMTP id 5285D140004 for ; Tue, 20 May 2025 01:04:19 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf09.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747703059; a=rsa-sha256; cv=none; b=zrVjhopwXz8Qo7/PwwPwkLXsuSIE0mvXmwej3sQ6IM9cPhNVcd57wpVLb3PgQJQI+vKc55 adMHFuLultjZov5LIm4i7SQQ8iTU8HVISv0xa8C1ZbDjTSwyaKPKYh6TKCfQlmu5cF1w1a feQzEOvITTn7p78MGwfF0q3hCWikAWg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf09.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747703059; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tsgbiggFhKY4PD1/TGvK6pgY8UM3OZ6s9ZWxsOlXQyg=; b=rii3/DAjBhW6sypPRlNRffEnaUazdXfdTSowWJGsXIPTq3XUbY39IzZdh0mNov+zqbDdrW aHiyHRNOQWT1dYaM1cHaavlM4+t5X6ocdgEMtb2OfcbDvoKR05hTmda4QAaL45lqbvXZEg QBYXsfMJNhieMyZcz4tPwgi4SC4Odcc= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1uHBOd-000000000aB-08xY; Mon, 19 May 2025 21:03:55 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, nadav.amit@gmail.com, Rik van Riel , Yu-cheng Yu , Rik van Riel Subject: [RFC v2 5/9] x86/mm: Change cpa_flush() to call flush_kernel_range() directly Date: Mon, 19 May 2025 21:02:30 -0400 Message-ID: <20250520010350.1740223-6-riel@surriel.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250520010350.1740223-1-riel@surriel.com> References: <20250520010350.1740223-1-riel@surriel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 78ywj187bpjfghjqcmotko16s39ozkto X-Rspam-User: X-Rspamd-Queue-Id: 5285D140004 X-Rspamd-Server: rspam06 X-HE-Tag: 1747703059-316941 X-HE-Meta: U2FsdGVkX19yj/f3bwkKJEHy/q5svJ95En0/0+stjTtjx1mlT9ZG1jb/Heoj3B0r04VfkBU+Cq5HzvPSV/O/FSVO6jDNbPdBxPtd47U12gVRVdeI7iWB0zuW/XAjQhAYGiePejnKYayS7zSY+rLaqm5lYqXLixbtElPqLlWti+BsDwc8btWff1m4+50jQNumJaZ2zqIsCiuva5v6zFWxocH5WMfZDP/FinGMrBz3vMnY5aMmQGw2X/DRKC3F2aByYxyx8JU/9XwlMGZnNUSARbCkylLNAmE7TbumxCoPo0JlP1th0KvAS8r+rLSQr2B4hVTs/pogqfDc2M1g+vM4wZ1fdq4x+qQDgWc9rDZbWyBJwC5M3ZE6XnFN1Ylvh27K4YdLY+YzP8MD8+gy1jAc9lYSpm1s+o3EUdYO6JmzTTkBcwYWsfNQPoGfboYXuNbD1i7vznwVPbRIQ9sM4x8hytSPnn6DevCoY/HOnDhGrQYkHiX06vQjNYYRlZR/BsH1bXcbMO5nLBhT8Ohz84U7qVfUHRGVOJ08rc6sBanJ847jPKZc0mlMHylFx1/9HomOoRRPm/J13+yJ3ZeOEP4Z0FdGI08NFYxSILoPKKrj2ILm+bYkRn8zX+Up2XBvwka1yM1SqGuAjKUdRCzwPO0t5QOauNG2SrVd1b1jBrrPiMI/IlhNcNTpmWoVNq8AWDIp7OUMVgiW9Ywlh//XxnINJlLuyzeZvk9nTcoRBcPoTjsrun0SWyq0zG8Lznh/UAiLRQslVVtw81KsFgxJ1bkRUghEpP7jg5lz3w2xdrdmAvew15Blk69pVVO2w4IJKCdRjH29gTBXl8F+VBBgCL7ZyrVjrIwKaBr1zW3UhV6V0jKp3TiQm4VXt/5PLm3mik30Dg5GnXvvWCd/nnfzdTFBUgIxuKAKjpMBFGxCvKEZGGbvyfz7+QMQsfiPcIL3h4BwGMm7HpkEorGNRzcyv+j 5hwHUBVm eC/BffWslH8E5y+V2vBW07WOaNLXYcMbqDkD70ZHsN3Ze5GKYLklhT3DXARNtXoy7ckDIvie5+myUfFhHaOnNNzBNl1b640q3JkEaDU2YaOogM2wwyQcAGcZyievogtdjd3mmFApqDUvHQryaOyI9nFYSr8dlDGBaKmltxWICjrzYRb6T3hhsBnLfQIqyvnxp1+JrVm7gpAsq5VB5BaDdb2PZ5gmpnniSWMak5jLpVxzRyCZEB1Y/M6amH2m7LZYXtJSo3vkbLROc4a9QQGhMhFMbIH36vn3+BHM0csXENAmwjqSVr8A20XLNjRtEUgoe+zsqS9OdmEPiIo0a/ukmCYGQ4/b4/xf0Il3+7+lIIorWUByurmlR74EHa3fIXtyZSa7TjChHVIgjGt72ILVZ+VFRGyKpBoQWsm1MtloRr49jKf7xE1rwI8zN5BrXSC3qKhLICUcx7u7alIqTWMStf1qGFQe5838ML6GY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel The function cpa_flush() calls __flush_tlb_one_kernel() and flush_tlb_all(). Replacing that with a call to flush_tlb_kernel_range() allows cpa_flush() to make use of INVLPGB or RAR without any additional changes. Initialize invlpgb_count_max to 1, since flush_tlb_kernel_range() can now be called before invlpgb_count_max has been initialized to the value read from CPUID. [riel: remove now unused __cpa_flush_tlb] Signed-off-by: Yu-cheng Yu Signed-off-by: Rik van Riel --- arch/x86/kernel/cpu/amd.c | 2 +- arch/x86/mm/pat/set_memory.c | 20 +++++++------------- 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 93da466dfe2c..b2ad8d13211a 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -31,7 +31,7 @@ #include "cpu.h" -u16 invlpgb_count_max __ro_after_init; +u16 invlpgb_count_max __ro_after_init = 1; static inline int rdmsrq_amd_safe(unsigned msr, u64 *p) { diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 30ab4aced761..2454f5249329 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -399,15 +399,6 @@ static void cpa_flush_all(unsigned long cache) on_each_cpu(__cpa_flush_all, (void *) cache, 1); } -static void __cpa_flush_tlb(void *data) -{ - struct cpa_data *cpa = data; - unsigned int i; - - for (i = 0; i < cpa->numpages; i++) - flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i))); -} - static int collapse_large_pages(unsigned long addr, struct list_head *pgtables); static void cpa_collapse_large_pages(struct cpa_data *cpa) @@ -444,6 +435,7 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) static void cpa_flush(struct cpa_data *cpa, int cache) { + unsigned long start, end; unsigned int i; BUG_ON(irqs_disabled() && !early_boot_irqs_disabled); @@ -453,10 +445,12 @@ static void cpa_flush(struct cpa_data *cpa, int cache) goto collapse_large_pages; } - if (cpa->force_flush_all || cpa->numpages > tlb_single_page_flush_ceiling) - flush_tlb_all(); - else - on_each_cpu(__cpa_flush_tlb, cpa, 1); + start = fix_addr(__cpa_addr(cpa, 0)); + end = fix_addr(__cpa_addr(cpa, cpa->numpages)); + if (cpa->force_flush_all) + end = TLB_FLUSH_ALL; + + flush_tlb_kernel_range(start, end); if (!cache) goto collapse_large_pages; -- 2.49.0