From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DB8AC4332F for ; Fri, 10 Nov 2023 14:08:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 735F0280026; Fri, 10 Nov 2023 09:08:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69611280022; Fri, 10 Nov 2023 09:08:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55F69280026; Fri, 10 Nov 2023 09:08:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 45A6D280022 for ; Fri, 10 Nov 2023 09:08:29 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 98669C0ED3 for ; Fri, 10 Nov 2023 14:08:28 +0000 (UTC) X-FDA: 81442224696.18.E4982F9 Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) by imf10.hostedemail.com (Postfix) with ESMTP id 88551C000D for ; Fri, 10 Nov 2023 14:08:26 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=g+IxDWSj; dmarc=none; spf=pass (imf10.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699625306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=oBo3Iq7Rw3dC7j71Ou2b9qV5k2WhbIOOeCwqM4BaPN+KCi3K3V+/TeXlUddm0DPumhLZ+h i9q4jsYrYDiqPHIB0Ul7eEJg4uQHQBO9dxMh2aONrsmtTsR6RFH1MJYTqoGNMZ2YMoOxE0 GyYFNFZYje9Nqt0Ehwg5csO4ZxV+Gjo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=g+IxDWSj; dmarc=none; spf=pass (imf10.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699625306; a=rsa-sha256; cv=none; b=JVYedP3Sp4z61h/1Oz1J2xktFzyN5eHYvDhgaIVHNTwWFRbNLuSHMyvA0qpZylP6B6Clxb V8e4bsMGxeq0T0fjzrebS5P8Soi8iSYSiwoJzRss/bLfF/OvkG6gG/Bu6U0cOQlBz7HhRy uXDQDEA13XgH/6mYXZk/YvqblnBwL2M= Received: by mail-lf1-f43.google.com with SMTP id 2adb3069b0e04-50a71aac023so653407e87.3 for ; Fri, 10 Nov 2023 06:08:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1699625304; x=1700230104; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=g+IxDWSjwiapiG9dWmnScQobJMlgkPv1sK3iljbBpXqkB6GSlu6roQtkVTDW7wv3zN p2EpjdToLeolR03wZ4pHxrQ/Be8VjjdWs/1hoc0H9hJCqhH2guBp/JxChotTWCu+C4rg VIkorkGBrahJ0mFaOecyrgMjg0ABwYj6kT6dfxn0i14O1j0beeAPOGWYPtNg6DPrrFC+ SAai8YPI0eOBjiCY5amKLM3Oj/XI0YIBLL+NNjup97Cu9jLEIFivFBF2dBMG1DNOnaUS azpAx40xDrnUcJJ5JFW6qOyLThiSNK8ErWhaj5pbj5qSXag4eNqzBSxuidhdJ/UXe7Lv 0Nyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699625304; x=1700230104; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2AMKJOgOBU+36iIxGMD9Trqw8IVdjZaTyhAA3F26Egw=; b=SYF/p0l6XjeLjp4770wwPJPFERdocicvKfFScUy8Kpl2YOv6dHj4EOJG66hQfZhtc2 oX4i+mTbT3UJd8Gbk+aMTF2K8PfA6MyinBl1BbRIhCa6xSZajMS1YiwNXkhS7Q+zdFu3 pJiiLormnxefXTLC9XoxPAn4lYh3agp/Gj4b1suVXte/QLuVyYeevDfkfOvK74zLHQYR Rm7J9S0lFppdKN6JHAuHB625sHsXY6xK5TzPUHUlscwNE1IW20sbVZoPq/gqyWXO88YQ F7/CxHGYa9KtSMud2K0BaxhNZMBFw7MgBEjggi6xhfnXgplFsaA7CJ8aarfAp+C0tWGm 9DDw== X-Gm-Message-State: AOJu0Yz/cRfXLej/zkMrqnpAJlmaGSsyRSQNG+O6EYkSr05ZzNexVuq2 zCdQ1m/Mkw6Vsk4Caz1qp5RlwA== X-Google-Smtp-Source: AGHT+IEGn96XLIR0ZFAsjYB2eJfL/m69l1crDYV4Ee9Qc9cnFj8KCnXUpx20enm/StvHsssXdR1xaw== X-Received: by 2002:ac2:5209:0:b0:503:1783:d5a9 with SMTP id a9-20020ac25209000000b005031783d5a9mr3895555lfl.3.1699625304483; Fri, 10 Nov 2023 06:08:24 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t18-20020a05600c451200b004076f522058sm5366438wmo.0.2023.11.10.06.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Nov 2023 06:08:24 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 1/2] mm: Introduce flush_cache_vmap_early() and its riscv implementation Date: Fri, 10 Nov 2023 15:07:20 +0100 Message-Id: <20231110140721.114235-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231110140721.114235-1-alexghiti@rivosinc.com> References: <20231110140721.114235-1-alexghiti@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 88551C000D X-Stat-Signature: 63fgjqfgqnax96xsycq8b9pe7ek6fj19 X-HE-Tag: 1699625306-901817 X-HE-Meta: U2FsdGVkX19uB3qpNIK2vDyOxAy0sK7Mb4kyeRSDQQTdpI0r2eCeiRC5BvxVBfMLbFlCcV0KV5U3/jqsADjapq6v4Wq9n1UIPecH1HWIQpooPF4TxvY+goPVgX4Ri3NXyq3grH+rkQHxPI6GU0J2j9cDZKcZ2aDe/ek5ghVUKWp9pgUtJWJb2FAJUi98dYHDb4Cl49pmB2d7sjqZNxUg04cWmHCTTQo3KiOJjVRGN+oXY+afyNaoyL3Z/GMIpijYohLOH93iMJTgPM32bRx/f+dkO4se46luvpVAGANE+aSUAso6CRKe+nzJ9tVA3HCRR9Msr9OJzVxh2+CeVZ2cRT3mSLa/yDI0NSqphc8+iIEH6QfUzNiWgsGr2A2ZFudT2Qf4vySz5FJZfa76pVnuqL5hea3yfXFAjPBRzep0o04WjiK1h53k0tGPI1OlgXZAAZG78ovlA9ebGCVmyXNeWHQ3PTS4fE9uZXrAWVHBw5CjBhTfOn3lRryPFrpw5W3BnLxNEG2i+QsZ5yMLg7uJQb3b1CuJv3bKhdpGfS2PZe0mMwp2FhPZKaSlrr1rXq30Ubo8kT4/fUsC2j+vddAQ/36rTLVZutJTa+nsxW4/FCMxfj7v6VRwWA6yTw+0UMeHdF5H73GKIdO6UtfVj46aDwpkDiNk+w9y34yFDnMfxwrg0rAEQuJtg7iEnoQvce3k0YZpwXKF90cymQIrdHU4OeKS/EsnTY33qaKidkGjZ77+tU9xK18Pd2k1wbYdKI8B9RRKIi3i9wEcca2IRVVZKESz0Txs+PBZLUhJ8KAB0ZybxXLzAd5zKwyPNxp5rceQb5l/kphXhBBRQ6wTK0kNVV4x2rob+fWmtJZC4D1D27auC8BjFkqqOwXKo6/XoEyLVrEL903guv6c7AHA5Lr8zUjFif/P9rAMpPmdsqqIxmCw45IOtAjT59i4VHZn9UwQnP4XUTPSkE7+fOueyOP 3l2b02X+ 5hhx43TtJI302R5G0rG8OK5qfe7z/95sQNA2ejweNRHSOj8+22j379GjFR89LfT7z+hktUWXV0csAk1NacCclsjgr0aueCtDwAwsdVzgKmQZacMecLor8TyufH6ZkRJaS1VyFN+xj+0oxnz9YzugqyEiXDr6EKPUgTONwS3gDqlj9iqDOFS/ZF8MhHsAQLANw2blaIuo9n666EFYLpW2p5ayMtH+mIBJjHFJcCenjVBAZ4brHw+TRv4tM6UHaE5Kr3BznltuXpLhTCPJ264jx+6n0IdIspgi8P8AU+BrluXRuEYTOynwbVs0Ecm6rRiLFKb99+cH8LlLCNAj0xX1RauhsDa5KYd6+X1nrBKeHJm1NQUZ/tlENvhIt4rtCS2yTN0WfKarslrjzQwc7LpeGPrfhijNCS9ZIKbSRAYpmB1LhykmigeouId0tfNq8ntOPu6nO4zOZ4Bpu2mYNLI6tsf50ZoxuQ1FQ8STvn/XlirboOHLCavYvNQj2GMgZmoIi5qCLS+PdlAv4P/pPQLXslGlIrU+2vcw6AWKD6lcVXdhsNswrAYi15EQG1x2zWoa7S9pdw9HyKt998vVb/34sAFhSb2SYpmZWpnaxUjXv15SuOY4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pcpu setup when using the page allocator sets up a new vmalloc mapping very early in the boot process, so early that it cannot use the flush_cache_vmap() function which may depend on structures not yet initialized (for example in riscv, we currently send an IPI to flush other cpus TLB). But on some architectures, we must call flush_cache_vmap(): for example, in riscv, some uarchs can cache invalid TLB entries so we need to flush the new established mapping to avoid taking an exception. So fix this by introducing a new function flush_cache_vmap_early() which is called right after setting the new page table entry and before accessing this new mapping. This new function implements a local flush tlb on riscv and is no-op for other architectures (same as today). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cacheflush.h | 3 ++- arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 5 +++++ include/asm-generic/cacheflush.h | 6 ++++++ mm/percpu.c | 8 +------- 5 files changed, 16 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 3cb53c4df27c..a129dac4521d 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,8 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_range(start, end) #endif #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..f0d6328076b6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -41,6 +41,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); void flush_tlb_kernel_range(unsigned long start, unsigned long end); +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -64,6 +65,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, local_flush_tlb_all(); } +#define local_flush_tlb_kernel_range(start, end) flush_tlb_kernel_range(start, end) #define flush_tlb_mm(mm) flush_tlb_all() #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..8aadc5f71c93 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,11 @@ static inline void local_flush_tlb_range_asid(unsigned long start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + local_flush_tlb_range_asid(start, end, PAGE_SIZE, FLUSH_TLB_NO_ASID); +} + static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cacheflush.h index 84ec53ccc450..7ee8a179d103 100644 --- a/include/asm-generic/cacheflush.h +++ b/include/asm-generic/cacheflush.h @@ -91,6 +91,12 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end) } #endif +#ifndef flush_cache_vmap_early +static inline void flush_cache_vmap_early(unsigned long start, unsigned long end) +{ +} +#endif + #ifndef flush_cache_vunmap static inline void flush_cache_vunmap(unsigned long start, unsigned long end) { diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..d287cebd58ca 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3306,13 +3306,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t if (rc < 0) panic("failed to map percpu area, err=%d\n", rc); - /* - * FIXME: Archs with virtual cache should flush local - * cache for the linear mapping here - something - * equivalent to flush_cache_vmap() on the local cpu. - * flush_cache_vmap() can't be used as most supporting - * data structures are not set up yet. - */ + flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size); /* copy static data */ memcpy((void *)unit_addr, __per_cpu_load, ai->static_size); -- 2.39.2