From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53BC8C3ABBF for ; Tue, 6 May 2025 00:38:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7058C6B0083; Mon, 5 May 2025 20:38:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 652C76B0096; Mon, 5 May 2025 20:38:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 393246B008C; Mon, 5 May 2025 20:38:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1984C6B0082 for ; Mon, 5 May 2025 20:38:28 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 19F9B1A0F20 for ; Tue, 6 May 2025 00:38:29 +0000 (UTC) X-FDA: 83410621938.29.0AFA537 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf10.hostedemail.com (Postfix) with ESMTP id 7EF4FC0006 for ; Tue, 6 May 2025 00:38:27 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746491907; a=rsa-sha256; cv=none; b=LEtreBOqjPCiLg11ofDs00mbW625Cd3fQ++1yqbAjPApT6eukXWi7BI2ao58DeIgaqHG+s 3JTKNX9aDTrX0VbmnIFf+jyYyC5SrHjsUMMhA3hvI7RVwGBfE+DqqGiXjIGc1B23yjQSWk SDIpu+XJ9zmBs74OW8+raTC/RQf+XUE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746491907; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OxNorbOjC9jGJqLvRnCW33QY0cbD963Af4qQkWB6acg=; b=JZOU46Tr1PwNFkoMmJnnEU87EbM3QzTVCxWbeOyySdkBh9/pwA3Qx3TsRIHtNbd2sX1KgO ocrLcyxXGu652E4nhrMYnO9k5KoZisCb3KPKajOXHmdJJc4jzjQb9CMiuFZo6SSKjF4ejT y2WwMaN5NULclims7JnYhhLtipsxu2Y= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1uC6K6-000000000IF-3ncU; Mon, 05 May 2025 20:38:14 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, Yu-cheng Yu , Rik van Riel Subject: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request Date: Mon, 5 May 2025 20:37:45 -0400 Message-ID: <20250506003811.92405-8-riel@surriel.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250506003811.92405-1-riel@surriel.com> References: <20250506003811.92405-1-riel@surriel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 7EF4FC0006 X-Stat-Signature: 39n1bh538eik5rgtymr1edf3nqcsy9iy X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1746491907-410682 X-HE-Meta: U2FsdGVkX1+NP2fku3zISZ0rkQUn93w4YrrV7cu6q11zQ4FIo6XZzOTxi7V0nEHN0N2veUaD5ga/pizUYsK5gQXWTKiPZiMT9km/lGJjgBJSsghLYw9WLmGb0GwyJD3CfcoZ2PAKZewxoaccKD367RxaFZdusBDsOef5lXkAu2GPxw1n1FmbFUa9h/WU02CUsYEvW5R4QsAjvWSCU9ot41azKAUuqP1MipXeHFpuP5nqB77JuLKNN9IGWbGMWAQzdAaSBntMRGapRLBhgTtil1sHYARyJY5Pi1HfFbf6B4vTxuxX94s9ZgSW9YL2kH/QVw9A4u/AcibmzxC1cGVfwAVRbUZJg87rdCttGsYCmFS05175V6KULICwh5m7B1ljNw2kPkeppDZfcGZf4bVaVCpRy3JpK2l7l2aFNJ77esoEGZPxfWeDLW0NWrLEjN6CwJZwHU5IVdtATZieO7c0MuhaMJBruXCy9QnPX8k7oKiCDjdAbdWe3BR5G+HQUcdmgkOe0rZkTDdyGRhSjWCh5I/bTYgYznSQ9Wg01M18P6SCWNaFQrQidmIjbRQB8QfqqTmH4X7rADoanlve2fGHhS2EhpJQwaS+lf+tyl44MFgXDGgoTdNslK3wkFzT+ys4nOSqJGGw+n7wlDHdBc0A+tCkfq/ObnvFCwYbPg+s3leqFVIbPVD8jTcoY6SXrxC2g1c/jXr39BrgsODj8nN4UfmKn/7Onz1EAkJLZLNu6nD6d13vyY/12mxuPzZ6eEwJ4LqeoNTaVsirCc0RJ7rse/vTUscOUYmf4/iekSqCN8W0rjvh+Ne8nklDHdqX+bkCZEk18o8rQ2BIQSwX8kd01RwJwjXtw3+JxGfgMDhH8akUB+6iSnzodTFrSqiaaw5/saj2a8dXjuoAai90lW9WylUU7yOhR39wIVzo2pvZkk2DJerv7B3mQ5vPj8ZELftxMYgblhfaMNTZIqS0W4V e6lawQ/+ 8W8XejX5TvayE8A2nzATnTRwK+OvCTIPKEQVFjdsE+96qMgEOKyvDKMB7PBAdwcUSwBouiMzkxkpb+g1F9qEtBPm6iWb1J2iYZGqQrIhnlDEM6/YFqpit/dOTI6ShcWFTp7LerbBTSOc63qBOVOuwlgn+nvAtRWlHSv7BnKq2rSBYz0qYo1MUVbnnqk33PP4AqLldACLQqYbFuDM5urPg3OqyMu3qfaH7IpUGAz8rbAUvM8qkANe5SCBe6J4bjwEKgP/S3dv0ve7kKqPzrmuyZGUQ0RxyPY9FTAZX6OswUYtYW0WzKDjugJTdaBOD/qWtmK1OXZqvffbEfyvFmkKMfnhFRg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu-cheng Yu Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement. [ riel: - add pcid parameter to smp_call_rar_many so other mms can be flushed - ensure get_payload only allocates valid indices - make sure rar_cpu_init does not write reserved bits - fix overflow in range vs full flush decision ] Signed-off-by: Yu-cheng Yu Signed-off-by: Rik van Riel --- arch/x86/include/asm/rar.h | 69 +++++++++++ arch/x86/kernel/cpu/common.c | 4 + arch/x86/mm/Makefile | 1 + arch/x86/mm/rar.c | 226 +++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+) create mode 100644 arch/x86/include/asm/rar.h create mode 100644 arch/x86/mm/rar.c diff --git a/arch/x86/include/asm/rar.h b/arch/x86/include/asm/rar.h new file mode 100644 index 000000000000..b5ba856fcaa8 --- /dev/null +++ b/arch/x86/include/asm/rar.h @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_RAR_H +#define _ASM_X86_RAR_H + +/* + * RAR payload types + */ +#define RAR_TYPE_INVPG 0 +#define RAR_TYPE_INVPG_NO_CR3 1 +#define RAR_TYPE_INVPCID 2 +#define RAR_TYPE_INVEPT 3 +#define RAR_TYPE_INVVPID 4 +#define RAR_TYPE_WRMSR 5 + +/* + * Subtypes for RAR_TYPE_INVLPG + */ +#define RAR_INVPG_ADDR 0 /* address specific */ +#define RAR_INVPG_ALL 2 /* all, include global */ +#define RAR_INVPG_ALL_NO_GLOBAL 3 /* all, exclude global */ + +/* + * Subtypes for RAR_TYPE_INVPCID + */ +#define RAR_INVPCID_ADDR 0 /* address specific */ +#define RAR_INVPCID_PCID 1 /* all of PCID */ +#define RAR_INVPCID_ALL 2 /* all, include global */ +#define RAR_INVPCID_ALL_NO_GLOBAL 3 /* all, exclude global */ + +/* + * Page size for RAR_TYPE_INVLPG + */ +#define RAR_INVLPG_PAGE_SIZE_4K 0 +#define RAR_INVLPG_PAGE_SIZE_2M 1 +#define RAR_INVLPG_PAGE_SIZE_1G 2 + +/* + * Max number of pages per payload + */ +#define RAR_INVLPG_MAX_PAGES 63 + +typedef struct { + uint64_t for_sw : 8; + uint64_t type : 8; + uint64_t must_be_zero_1 : 16; + uint64_t subtype : 3; + uint64_t page_size: 2; + uint64_t num_pages : 6; + uint64_t must_be_zero_2 : 21; + + uint64_t must_be_zero_3; + + /* + * Starting address + */ + uint64_t initiator_cr3; + uint64_t linear_address; + + /* + * Padding + */ + uint64_t padding[4]; +} rar_payload_t; + +void rar_cpu_init(void); +void smp_call_rar_many(const struct cpumask *mask, u16 pcid, + unsigned long start, unsigned long end); + +#endif /* _ASM_X86_RAR_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 5666620e7153..75b43db0b129 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -71,6 +71,7 @@ #include #include #include +#include #include "cpu.h" @@ -2395,6 +2396,9 @@ void cpu_init(void) if (is_uv_system()) uv_cpu_init(); + if (cpu_feature_enabled(X86_FEATURE_RAR)) + rar_cpu_init(); + load_fixmap_gdt(cpu); } diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index cebe5812d78d..d49d16412569 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -54,6 +54,7 @@ obj-$(CONFIG_ACPI_NUMA) += srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_BROADCAST_TLB_FLUSH) += rar.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o diff --git a/arch/x86/mm/rar.c b/arch/x86/mm/rar.c new file mode 100644 index 000000000000..77a334f1e212 --- /dev/null +++ b/arch/x86/mm/rar.c @@ -0,0 +1,226 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * RAR Tlb shootdown + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +static DEFINE_PER_CPU(int, rar_lock); +static DEFINE_PER_CPU(struct cpumask, rar_cpu_mask); + +#define RAR_ACTION_OK 0x00 +#define RAR_ACTION_START 0x01 +#define RAR_ACTION_ACKED 0x02 +#define RAR_ACTION_FAIL 0x80 + +#define RAR_MAX_PAYLOADS 32UL + +static unsigned long rar_in_use = ~(RAR_MAX_PAYLOADS - 1); +static rar_payload_t rar_payload[RAR_MAX_PAYLOADS] __page_aligned_bss; +static DEFINE_PER_CPU_ALIGNED(u64[(RAR_MAX_PAYLOADS + 8) / 8], rar_action); + +static __always_inline void lock(int *lock) +{ + smp_cond_load_acquire(lock, !(VAL & 1)); + *lock |= 1; + + /* + * prevent CPU from reordering the above assignment + * to ->flags with any subsequent assignments to other + * fields of the specified call_single_data structure: + */ + smp_wmb(); +} + +static __always_inline void unlock(int *lock) +{ + WARN_ON(!(*lock & 1)); + + /* + * ensure we're all done before releasing data: + */ + smp_store_release(lock, 0); +} + +static unsigned long get_payload(void) +{ + while (1) { + unsigned long bit; + + /* + * Find a free bit and confirm it with + * test_and_set_bit() below. + */ + bit = ffz(READ_ONCE(rar_in_use)); + + if (bit >= RAR_MAX_PAYLOADS) + continue; + + if (!test_and_set_bit((long)bit, &rar_in_use)) + return bit; + } +} + +static void free_payload(unsigned long idx) +{ + clear_bit(idx, &rar_in_use); +} + +static void set_payload(unsigned long idx, u16 pcid, unsigned long start, + uint32_t pages) +{ + rar_payload_t *p = &rar_payload[idx]; + + p->must_be_zero_1 = 0; + p->must_be_zero_2 = 0; + p->must_be_zero_3 = 0; + p->page_size = RAR_INVLPG_PAGE_SIZE_4K; + p->type = RAR_TYPE_INVPCID; + p->num_pages = pages; + p->initiator_cr3 = pcid; + p->linear_address = start; + + if (pcid) { + /* RAR invalidation of the mapping of a specific process. */ + if (pages >= RAR_INVLPG_MAX_PAGES) + p->subtype = RAR_INVPCID_PCID; + else + p->subtype = RAR_INVPCID_ADDR; + } else { + /* + * Unfortunately RAR_INVPCID_ADDR excludes global translations. + * Always do a full flush for kernel invalidations. + */ + p->subtype = RAR_INVPCID_ALL; + } + + smp_wmb(); +} + +static void set_action_entry(unsigned long idx, int target_cpu) +{ + u8 *bitmap = (u8 *)per_cpu(rar_action, target_cpu); + + WRITE_ONCE(bitmap[idx], RAR_ACTION_START); +} + +static void wait_for_done(unsigned long idx, int target_cpu) +{ + u8 status; + u8 *bitmap = (u8 *)per_cpu(rar_action, target_cpu); + + status = READ_ONCE(bitmap[idx]); + + while ((status != RAR_ACTION_OK) && (status != RAR_ACTION_FAIL)) { + cpu_relax(); + status = READ_ONCE(bitmap[idx]); + } + + WARN_ON_ONCE(bitmap[idx] == RAR_ACTION_FAIL); +} + +void rar_cpu_init(void) +{ + u64 r; + u8 *bitmap; + int this_cpu = smp_processor_id(); + + per_cpu(rar_lock, this_cpu) = 0; + cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu)); + + rdmsrl(MSR_IA32_RAR_INFO, r); + pr_info_once("RAR: support %lld payloads\n", r >> 32); + + bitmap = (u8 *)per_cpu(rar_action, this_cpu); + memset(bitmap, 0, RAR_MAX_PAYLOADS); + wrmsrl(MSR_IA32_RAR_ACT_VEC, (u64)virt_to_phys(bitmap)); + wrmsrl(MSR_IA32_RAR_PAYLOAD_BASE, (u64)virt_to_phys(rar_payload)); + + r = RAR_CTRL_ENABLE | RAR_CTRL_IGNORE_IF; + // reserved bits!!! r |= (RAR_VECTOR & 0xff); + wrmsrl(MSR_IA32_RAR_CTRL, r); +} + +/* + * This is a modified version of smp_call_function_many() of kernel/smp.c, + * without a function pointer, because the RAR handler is the ucode. + */ +void smp_call_rar_many(const struct cpumask *mask, u16 pcid, + unsigned long start, unsigned long end) +{ + unsigned long pages = (end - start + PAGE_SIZE) / PAGE_SIZE; + int cpu, next_cpu, this_cpu = smp_processor_id(); + cpumask_t *dest_mask; + unsigned long idx; + + if (pages > RAR_INVLPG_MAX_PAGES || end == TLB_FLUSH_ALL) + pages = RAR_INVLPG_MAX_PAGES; + + /* + * Can deadlock when called with interrupts disabled. + * We allow cpu's that are not yet online though, as no one else can + * send smp call function interrupt to this cpu and as such deadlocks + * can't happen. + */ + WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled() + && !oops_in_progress && !early_boot_irqs_disabled); + + /* Try to fastpath. So, what's a CPU they want? Ignoring this one. */ + cpu = cpumask_first_and(mask, cpu_online_mask); + if (cpu == this_cpu) + cpu = cpumask_next_and(cpu, mask, cpu_online_mask); + + /* No online cpus? We're done. */ + if (cpu >= nr_cpu_ids) + return; + + /* Do we have another CPU which isn't us? */ + next_cpu = cpumask_next_and(cpu, mask, cpu_online_mask); + if (next_cpu == this_cpu) + next_cpu = cpumask_next_and(next_cpu, mask, cpu_online_mask); + + /* Fastpath: do that cpu by itself. */ + if (next_cpu >= nr_cpu_ids) { + lock(this_cpu_ptr(&rar_lock)); + idx = get_payload(); + set_payload(idx, pcid, start, pages); + set_action_entry(idx, cpu); + arch_send_rar_single_ipi(cpu); + wait_for_done(idx, cpu); + free_payload(idx); + unlock(this_cpu_ptr(&rar_lock)); + return; + } + + dest_mask = this_cpu_ptr(&rar_cpu_mask); + cpumask_and(dest_mask, mask, cpu_online_mask); + cpumask_clear_cpu(this_cpu, dest_mask); + + /* Some callers race with other cpus changing the passed mask */ + if (unlikely(!cpumask_weight(dest_mask))) + return; + + lock(this_cpu_ptr(&rar_lock)); + idx = get_payload(); + set_payload(idx, pcid, start, pages); + + for_each_cpu(cpu, dest_mask) + set_action_entry(idx, cpu); + + /* Send a message to all CPUs in the map */ + arch_send_rar_ipi_mask(dest_mask); + + for_each_cpu(cpu, dest_mask) + wait_for_done(idx, cpu); + + free_payload(idx); + unlock(this_cpu_ptr(&rar_lock)); +} +EXPORT_SYMBOL(smp_call_rar_many); -- 2.49.0