From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F6DC36010 for ; Fri, 4 Apr 2025 06:02:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5625D280004; Fri, 4 Apr 2025 02:02:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50F40280001; Fri, 4 Apr 2025 02:02:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D792280004; Fri, 4 Apr 2025 02:02:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1ED5E280001 for ; Fri, 4 Apr 2025 02:02:51 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0C1E71CE1B0 for ; Fri, 4 Apr 2025 06:02:52 +0000 (UTC) X-FDA: 83295317784.23.96397A3 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 0D03E2000B for ; Fri, 4 Apr 2025 06:02:49 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743746570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zNYtSL5xDVUQFOcxHTJM9IsX6pO/gXm5fBHgLRD7xyY=; b=QRf8R17rAjLoORyJx2x45a4wbvUGxaxl88e1wu4t1T2xXLnRG9SZFrpwhnf0UZi1VInwt9 k9Xj+l8JnR3xV58tiIbY+rXsPLmKZ/EhSmJ83QNtkt+XW7z1du3LbkhC/hL5dxv1xpsf/a ZWUKKpY7T4yJUI8z713xBvQVJfTKe6s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743746570; a=rsa-sha256; cv=none; b=zSeknTRC9s2jYpgAHBBXnON9qQ+ttH1n+Ms3wAk2QhJ62rWb0iPy/yBFSlKBXyF6iI67IL 1Y+inhUBH8vVSwm6I/68zupSOjpomyfutNVUl445yCDFWPdgpLsbl2VnLCKHfF21aos83C oaEqU8G9hUjORFbwakd1EaWBE2PqMSM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6DFFB1063; Thu, 3 Apr 2025 23:02:51 -0700 (PDT) Received: from [10.162.40.17] (a077893.blr.arm.com [10.162.40.17]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 42B2F3F694; Thu, 3 Apr 2025 23:02:43 -0700 (PDT) Message-ID: <00a0d9f1-d0a1-41fe-a0af-7e2174efc2e0@arm.com> Date: Fri, 4 Apr 2025 11:32:41 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 11/11] arm64/mm: Batch barriers when updating kernel mappings To: Ryan Roberts , Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Alexandre Ghiti , Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20250304150444.3788920-1-ryan.roberts@arm.com> <20250304150444.3788920-12-ryan.roberts@arm.com> Content-Language: en-US From: Anshuman Khandual In-Reply-To: <20250304150444.3788920-12-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 0D03E2000B X-Stat-Signature: iuon8gtx73msqd8hc5znttohee7gdiu5 X-HE-Tag: 1743746569-406352 X-HE-Meta: U2FsdGVkX1+WFxKmECCrBsgucHeiMK/nFPAK9cvpT8aLh4gttAfoJ/dbU56Ey496a2bAlYJf2rEFBeIceVrv78tnS9jaZl2Ut62hVPNOqfp9K43BgqIjf+UOnyqYOnmi5dNiglUrt2nUZAU81vyMSxRD9NebSON/ghVJkXIPwOeQ4G7K/krhCROXByxonClD5njx0jBoi5pzM01/4jr58foA22a/ROQWjbP1Ut6UsHluEnC6qc37Yi8CREKrDf8QpebCSQjsAl6L0gkgNiEHhY/d1vJid8B9I7nuzoAbuZ+Ju2D+Y0FLyHhNi7xytSwPWSDfMhxHFo9v5Kk+hyqBI3/kJqrdAL3Ru3wHzQd4wuQIDrSnr3JlZklgXNxKmEZgBFWSR6LcWRykgxwmn8IO/Vvv1Cb8LaGsRUo4oFU09G9kEdXZB11M3SbKy38cFuTQ8WQ7qn+VQLFIAuRyRKm46oHSWiAQMqM4iwEgSFLeLRKSF24w4ORPv4HR22mLfHHJRLw/ZlQ/arcqUDv+zlUTOpn69DOL7Wv0YKjWBv9G1hOqPY0v9Lfjhh5BxuwGk/sLa8gqzPTy9Nc2NiuVNcZpIwHfom+jWwekSTbgNo2ddZ33cBa2mGxb10a59bLr/qDnErbITTKGNWXcnKg4XcxgKPsHXnwN57ojFWqrpvZ7GVtog3hob4Fff9gSWUVy306+IeUNVyLaR82L26ttSW2nt7F/ggU2zhH7jMIl1W2vhKC6fGO0VdapX8JpM5Gl4MiqPX7FHlH3og5vviMvXMPPuVfk3zgfeqyZPCfBD+7nvKHe7MklXusPXXql8jKLUJc5o4NldpBSZsHeUklqGazEnQOTH18CMOWNzq72qVkMzqc9+gweWEcdiAUliizHdvPg5eq6ZA9ci/k+wY/TQaA9/VBcHUYc2HNr1udbPvrVqy8dKMLio8iMeKgpPSN9JTM0rHPo1JeYRYBeDpJBttX a4AwC1UI Elv7tltC57vyY7jdskjozxORZcc3ZxzPq0ywmG06taGXXBQXgAVLiYcxsAMmpQRu2KupttcAtw7615Dzg/nNkzQU/gz2/3gcuSZiOWN55eVtzupYHZK2Ye2qY6ukxEEjtiUX3+rn9DJcCeF/RwxyDYFWrRaci1lHaOQ3jVWbvKcgYKsPwo/u+LOn2pADVOXP3LyjGJNeGxyzqQOI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 3/4/25 20:34, Ryan Roberts wrote: > Because the kernel can't tolerate page faults for kernel mappings, when > setting a valid, kernel space pte (or pmd/pud/p4d/pgd), it emits a > dsb(ishst) to ensure that the store to the pgtable is observed by the > table walker immediately. Additionally it emits an isb() to ensure that > any already speculatively determined invalid mapping fault gets > canceled. > > We can improve the performance of vmalloc operations by batching these > barriers until the end of a set of entry updates. > arch_enter_lazy_mmu_mode() and arch_leave_lazy_mmu_mode() provide the > required hooks. > > vmalloc improves by up to 30% as a result. > > Two new TIF_ flags are created; TIF_LAZY_MMU tells us if the task is in > the lazy mode and can therefore defer any barriers until exit from the > lazy mode. TIF_LAZY_MMU_PENDING is used to remember if any pte operation > was performed while in the lazy mode that required barriers. Then when > leaving lazy mode, if that flag is set, we emit the barriers. > > Since arch_enter_lazy_mmu_mode() and arch_leave_lazy_mmu_mode() are used > for both user and kernel mappings, we need the second flag to avoid > emitting barriers unnecessarily if only user mappings were updated. Agreed and hence for that an additional TIF flag i.e TIF_LAZY_MMU_PENDING can be justified. > > Signed-off-by: Ryan Roberts > --- > arch/arm64/include/asm/pgtable.h | 73 ++++++++++++++++++++++------ > arch/arm64/include/asm/thread_info.h | 2 + > arch/arm64/kernel/process.c | 9 ++-- > 3 files changed, 64 insertions(+), 20 deletions(-) > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h > index 1898c3069c43..149df945c1ab 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -40,6 +40,55 @@ > #include > #include > > +static inline void emit_pte_barriers(void) > +{ > + /* > + * These barriers are emitted under certain conditions after a pte entry > + * was modified (see e.g. __set_pte_complete()). The dsb makes the store > + * visible to the table walker. The isb ensures that any previous > + * speculative "invalid translation" marker that is in the CPU's > + * pipeline gets cleared, so that any access to that address after > + * setting the pte to valid won't cause a spurious fault. If the thread > + * gets preempted after storing to the pgtable but before emitting these > + * barriers, __switch_to() emits a dsb which ensure the walker gets to > + * see the store. There is no guarrantee of an isb being issued though. typo ^^^^^^^^ > + * This is safe because it will still get issued (albeit on a > + * potentially different CPU) when the thread starts running again, > + * before any access to the address. > + */ > + dsb(ishst); > + isb(); > +} > + > +static inline void queue_pte_barriers(void) > +{ > + if (test_thread_flag(TIF_LAZY_MMU)) > + set_thread_flag(TIF_LAZY_MMU_PENDING); > + else > + emit_pte_barriers(); > +} > + > +#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE > +static inline void arch_enter_lazy_mmu_mode(void) > +{ > + VM_WARN_ON(in_interrupt()); > + VM_WARN_ON(test_thread_flag(TIF_LAZY_MMU)); > + > + set_thread_flag(TIF_LAZY_MMU); > +} > + > +static inline void arch_flush_lazy_mmu_mode(void) > +{ > + if (test_and_clear_thread_flag(TIF_LAZY_MMU_PENDING)) > + emit_pte_barriers(); > +} > + > +static inline void arch_leave_lazy_mmu_mode(void) > +{ > + arch_flush_lazy_mmu_mode(); > + clear_thread_flag(TIF_LAZY_MMU); > +} > + > #ifdef CONFIG_TRANSPARENT_HUGEPAGE > #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE > > @@ -323,10 +372,8 @@ static inline void __set_pte_complete(pte_t pte) > * Only if the new pte is valid and kernel, otherwise TLB maintenance > * has the necessary barriers. > */ > - if (pte_valid_not_user(pte)) { > - dsb(ishst); > - isb(); > - } > + if (pte_valid_not_user(pte)) > + queue_pte_barriers(); > } > > static inline void __set_pte(pte_t *ptep, pte_t pte) > @@ -778,10 +825,8 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) > > WRITE_ONCE(*pmdp, pmd); > > - if (pmd_valid(pmd)) { > - dsb(ishst); > - isb(); > - } > + if (pmd_valid(pmd)) > + queue_pte_barriers(); > } > > static inline void pmd_clear(pmd_t *pmdp) > @@ -845,10 +890,8 @@ static inline void set_pud(pud_t *pudp, pud_t pud) > > WRITE_ONCE(*pudp, pud); > > - if (pud_valid(pud)) { > - dsb(ishst); > - isb(); > - } > + if (pud_valid(pud)) > + queue_pte_barriers(); > } > > static inline void pud_clear(pud_t *pudp) > @@ -925,8 +968,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) > } > > WRITE_ONCE(*p4dp, p4d); > - dsb(ishst); > - isb(); > + queue_pte_barriers(); > } > > static inline void p4d_clear(p4d_t *p4dp) > @@ -1052,8 +1094,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) > } > > WRITE_ONCE(*pgdp, pgd); > - dsb(ishst); > - isb(); > + queue_pte_barriers(); > } > > static inline void pgd_clear(pgd_t *pgdp) > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h > index 1114c1c3300a..1fdd74b7b831 100644 > --- a/arch/arm64/include/asm/thread_info.h > +++ b/arch/arm64/include/asm/thread_info.h > @@ -82,6 +82,8 @@ void arch_setup_new_exec(void); > #define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */ > #define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */ > #define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */ > +#define TIF_LAZY_MMU 31 /* Task in lazy mmu mode */ > +#define TIF_LAZY_MMU_PENDING 32 /* Ops pending for lazy mmu mode exit */ > > #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c > index 42faebb7b712..45a55fe81788 100644 > --- a/arch/arm64/kernel/process.c > +++ b/arch/arm64/kernel/process.c > @@ -680,10 +680,11 @@ struct task_struct *__switch_to(struct task_struct *prev, > gcs_thread_switch(next); > > /* > - * Complete any pending TLB or cache maintenance on this CPU in case > - * the thread migrates to a different CPU. > - * This full barrier is also required by the membarrier system > - * call. > + * Complete any pending TLB or cache maintenance on this CPU in case the > + * thread migrates to a different CPU. This full barrier is also > + * required by the membarrier system call. Additionally it makes any > + * in-progress pgtable writes visible to the table walker; See > + * emit_pte_barriers(). > */ > dsb(ish); > Otherwise, LGTM. I will try and think through again if these deferred sync and flush can cause subtle problems else where.