From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAE9BEE14C3 for ; Sat, 9 Sep 2023 20:17:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A5546B0125; Sat, 9 Sep 2023 16:17:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02B2F6B0126; Sat, 9 Sep 2023 16:17:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4B906B0127; Sat, 9 Sep 2023 16:17:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BA2FB6B0125 for ; Sat, 9 Sep 2023 16:17:46 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9C12B160602 for ; Sat, 9 Sep 2023 20:17:46 +0000 (UTC) X-FDA: 81218169732.28.79670B7 Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by imf18.hostedemail.com (Postfix) with ESMTP id 71D021C0008 for ; Sat, 9 Sep 2023 20:17:44 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=sholland.org header.s=fm2 header.b="K u0dd0R"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="J 6plQnu"; dmarc=pass (policy=none) header.from=sholland.org; spf=pass (imf18.hostedemail.com: domain of samuel@sholland.org designates 64.147.123.20 as permitted sender) smtp.mailfrom=samuel@sholland.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694290664; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vYgeei/BDcJ6bUB2DAGppW0ND32ofla8o2DCj1NxYPE=; b=yMElJdqaL4zjf52n4ZbFGNlmGHoOUL+o7hzefduWvwBSp1iyKXm2uFh97fklr60ODdqvOp m7JsithTYtygpZqZ/PxeI58ZaZ3vMcfwyzIAT7150ERb/iMRYb1Ki7aQee6TiapGp0/aI0 D+dLTB2LjiY9Aw3/d21kT34EjLroNyk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=sholland.org header.s=fm2 header.b="K u0dd0R"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="J 6plQnu"; dmarc=pass (policy=none) header.from=sholland.org; spf=pass (imf18.hostedemail.com: domain of samuel@sholland.org designates 64.147.123.20 as permitted sender) smtp.mailfrom=samuel@sholland.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694290664; a=rsa-sha256; cv=none; b=J7R2F3cKlwevyPaKpR0GHwYKhLZOtVDbyA9Prl6lf3T3m7aUxJzIRR0tbuZsNY6rhWtr6e NkrwLlSuUaL2jL7roFkZsCzOJrU0XVivpPSjOYJsHTulGA6XW3g8lHecgvOqv9fHL19pJd 4A7O+JAPSDw4WBo1KTmuDehao8QtjH8= Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id A23CD3200805; Sat, 9 Sep 2023 16:17:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Sat, 09 Sep 2023 16:17:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sholland.org; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; t=1694290662; x= 1694377062; bh=vYgeei/BDcJ6bUB2DAGppW0ND32ofla8o2DCj1NxYPE=; b=K u0dd0RZHMaDqMns7iRitHO88BySItmXc1V/+Vb/OkmR3KGcO1DvaRAtq3OLHETze naLsbxraep105inN1mW8i3aVWoN2CHwYKGEBnzeDVwtB/CY825a4Z6xplHeKufSB j5nFyWllPSvSBWdcnnya2PQa8PQnYYZV8XJ5rJi+x/EQ026aTvbwXVV8VdsVpeGh K6pNLRO3moFbl1eU9gKfEAOCfyD/8MKFiMMKp6OiMj6Z78ZchXerkP5VT2pQbY1v Kah1XXspL8hkSGENcggXYwqoYZTnzsPBZM5B7wK0UGfXDstVHgvLEPFUuxcvWJ4X I7OZP8RqM2VGeFapjE9CA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1694290662; x= 1694377062; bh=vYgeei/BDcJ6bUB2DAGppW0ND32ofla8o2DCj1NxYPE=; b=J 6plQnuSCnXtX5IBcgF5KYZGLQQG5MhLcW0wynftAZ0np5C/3SOw1d4GtGQhHZoeA RUCoGTMdxPBabmUAs8z9fzJypFl0W39j4LSRzE/kj+YtSJ+y2/NJxCWOMlC+OiBh E06QkMy8GJfKVsumpoaI812sFBrDYMDbN7JgsvEuVRLEJ8/imdo2h3gHSfEr1mtS U5OdXOVeblb2nv0zlpwDBRQC818RhI7+SAipU2efh9earF6Cmks7jZbChNmnwBYD Vd/3hzPGOd+tI1EvQdpIoT/1VmuvUeJxT7/xF3v23ep25CoStRL7VhCvoOIJo699 CEAt1LarUDmRPpTBESL8w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudehledgudegiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefurghm uhgvlhcujfholhhlrghnugcuoehsrghmuhgvlhesshhhohhllhgrnhgurdhorhhgqeenuc ggtffrrghtthgvrhhnpedukeetueduhedtleetvefguddvvdejhfefudelgfduveeggeeh gfdufeeitdevteenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfh hrohhmpehsrghmuhgvlhesshhhohhllhgrnhgurdhorhhg X-ME-Proxy: Feedback-ID: i0ad843c9:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 9 Sep 2023 16:17:41 -0400 (EDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Samuel Holland Subject: [PATCH 6/7] riscv: mm: Always flush a single MM context by ASID Date: Sat, 9 Sep 2023 15:16:34 -0500 Message-ID: <20230909201727.10909-7-samuel@sholland.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230909201727.10909-1-samuel@sholland.org> References: <20230909201727.10909-1-samuel@sholland.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 71D021C0008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: qem5paamak44up8myafa9ywfyoazswhz X-HE-Tag: 1694290664-145949 X-HE-Meta: U2FsdGVkX1+a7OffLhbdJzWjavRtDitPWQqwZhO2v3Xx4xEKAwfCFug9Z/310HDMOrhpmRKMunN3MDJ+1gHiYT4OQM74syt6ltMuTPooUwviRewyfC8HuEMS6hQrXDdHnc6gRZrRoZ4McoITzHgYTYMI0e+D6hVIhs2XBbXIJGxxS04cg8IQCEy6qyxvfDTXgCJOBrKGd0UdqwDrT0l+JV8ukxkfUN6tEAFUBBGziYbfk7UWgBemC1sA2Ky3DFHMcSD1eUa7SRdOeYNdf6/dToS9R1TJevxZ7kUPy7RN6/Z0zAesuzKthH6Z/Z4Mxt9AE/rNAgkT25xJwBZlIpcOn1oinFXh8LYx92ldcJhiEm/CieGU5AYxAb2jtsEeXOEDhtn90OMjWuSdPbL/djGhfIsbEtf4qhAXodDvVzukfUsJ45rEKTYYBzt/97z6GF0HhoBle55DzhPLeDBOYTKNXBXjWD4/evAsikh8MYF5j9Qx/GgQUO7Cfp6Hr6jQegIsNMUCmPOSzt5oBrTTywzktQmhI6orCetiCc4i173pLmV85jCS3/0HBvOjnuyMBDZaxM51kqvGwonOIUlx+YjKLu1toqVHAtajKs/z+G+MwljL5gk6BsC01fEH7EI46Z5Cr2FzsnvDF/kK4ByoYWUY2F/ZIElZWNlzj+8ZvIOMguo9yJCPuR2/kpWIMgakTmcEmCFDZZC+7aw8s8nj+YcPpc1EX7rKh/7YQsjttkWaHMcQRmJYdA+hopRyjNe0rBI73ui+SV68EZ/gSCdHiDnrFUsh/PIYFUfysqO1Y0pQ+jLdlN/olSTlgBDpwPR2263VnnIy1p6QK87eJPuykkVyJEdgU0SVuEXfzMYrXfcAU8V3t5WKYs5bxkApu9o1sbsK1gXMxghU2S38eH4KdoaPQr/tYePAXEZ0vjm5hIBHblNsLic6RH/EwYoEyN6TgYMI5k+PeDDyXf8/yti+O5d Z0mxMZIh jgs800MEzWkHmpUoxsM+KhxAcORJPi8L44wjKSSlNIuzy7wUTm7XOsa2lxrkMz7Ha9RtM13TICkYWvM1aWAvQXKpzG2F/tQXwS3PQqJg7guBGy1bt4/ThnQY6BGmPhklnbWA6xw4tFMjdvhiFJ0+DBBeDPJwguqc7E7CUZwf46Uzt+ENpVo0GH0pzv6t+9tlLDLL7J5iB2NpTe2cevx9kPvmL2FXbedsfhU4YXmkkZrQCqbPQN7zYyb3vp+J694j+02SfeyDWhVkQWMsRigIRd1+At7w4K+l3aZOdhTJJwmsBSqLHYqCFcYImLRhvc6rdzIEo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even if ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always most efficient to use the single-ASID code path. Signed-off-by: Samuel Holland --- arch/riscv/include/asm/mmu_context.h | 2 - arch/riscv/include/asm/tlbflush.h | 11 +++-- arch/riscv/mm/context.c | 3 +- arch/riscv/mm/tlbflush.c | 68 ++++++---------------------- 4 files changed, 24 insertions(+), 60 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index e55831edfc19..ba27cf68b170 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -54,13 +54,18 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, #define flush_tlb_all() local_flush_tlb_all() #define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) +static inline void flush_tlb_mm(struct mm_struct *mm) +{ + unsigned long asid = cntx2asid(atomic_long_read(&mm->context.id)); + + local_flush_tlb_all_asid(asid); +} + static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - local_flush_tlb_all(); + flush_tlb_mm(vma->vm_mm); } - -#define flush_tlb_mm(mm) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 54c3e70ccd81..56c2d40681a2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -6,15 +6,6 @@ #include #include -static inline void local_flush_tlb_range(unsigned long start, - unsigned long size, unsigned long stride) -{ - if (size <= stride) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); -} - static inline void local_flush_tlb_range_asid(unsigned long start, unsigned long size, unsigned long stride, unsigned long asid) { @@ -51,62 +42,33 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } -static void __ipi_flush_tlb_range(void *info) -{ - struct flush_tlb_range_data *d = info; - - local_flush_tlb_range(d->start, d->size, d->stride); -} - static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { + unsigned long asid = cntx2asid(atomic_long_read(&mm->context.id)); struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; - bool broadcast; if (cpumask_empty(cmask)) return; cpuid = get_cpu(); /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid = cntx2asid(atomic_long_read(&mm->context.id)); - - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { - local_flush_tlb_range_asid(start, size, stride, asid); - } - } else { - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = 0; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range, - &ftd, 1); - } else - sbi_remote_sfence_vma(cmask, start, size); - } else { - local_flush_tlb_range(start, size, stride); - } - } - + if (cpumask_any_but(cmask, cpuid) < nr_cpu_ids) { + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); + } else + local_flush_tlb_range_asid(start, size, stride, asid); put_cpu(); } -- 2.41.0