From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2421DC0219B for ; Mon, 10 Feb 2025 08:03:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB0FE6B007B; Mon, 10 Feb 2025 03:03:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A606C6B0083; Mon, 10 Feb 2025 03:03:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94F276B0085; Mon, 10 Feb 2025 03:03:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 76F916B007B for ; Mon, 10 Feb 2025 03:03:29 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 040C9160684 for ; Mon, 10 Feb 2025 08:03:28 +0000 (UTC) X-FDA: 83103295338.25.52E56BF Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 3406314000D for ; Mon, 10 Feb 2025 08:03:27 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739174607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k0WdaW5rf9iE7C+e7Q6inh5VPyPLaI6FIrdlWLqp1iY=; b=kvZiTeqd4Oeh9bDtlcOQ4wpRnHKfse1rd+FRAkIChX9ke2j6WnzTXEtM42JmcciqfnKFDi uXhAiKnzfWU6DSJM7+eOFy6utUws8/Szbdfnlrnt4AVWXDeLl/sqYlf+kVbTxz5Ovy9Mms Pzwhyny3/s5w6EE9quPBocfutkcs0DU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739174607; a=rsa-sha256; cv=none; b=uQkX/MpCwYCGR0RZT5ej2W+5QNQF5Ie0B8Ge86gZdw72bkY16VaJJ4YfUW291f4VsxDFPi z5Ly5cZpXyEU3fcZubQ9g1rXyj60HcFAqP1h6nlw9RukOGPrRxfNGDO8aBU4CBYiCXbyc5 3nGhSFd80ZNUvB4EL5YcVmyny7MAjP4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3285F1BA8; Mon, 10 Feb 2025 00:03:48 -0800 (PST) Received: from [10.163.35.99] (unknown [10.163.35.99]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3B4D53F58B; Mon, 10 Feb 2025 00:03:20 -0800 (PST) Message-ID: <9bc5527e-16f4-45cc-aced-55b1ace6c143@arm.com> Date: Mon, 10 Feb 2025 13:33:21 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 16/16] arm64/mm: Defer barriers when updating kernel mappings To: Ryan Roberts , Catalin Marinas , Will Deacon , Muchun Song , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Mark Rutland , Ard Biesheuvel , Dev Jain , Alexandre Ghiti , Steve Capper , Kevin Brodsky Cc: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20250205151003.88959-1-ryan.roberts@arm.com> <20250205151003.88959-17-ryan.roberts@arm.com> Content-Language: en-US From: Anshuman Khandual In-Reply-To: <20250205151003.88959-17-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 3406314000D X-Stat-Signature: m3jjsmijyaaxcokwtsszm3iug1bhbxzn X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739174607-987058 X-HE-Meta: U2FsdGVkX1/MGLiBMPjdi/nuE+rcwYMt3Vf4DlSZLc8PkIOC5Qb1SHZSOVbD2RfTzHlB9v7bqIulSMuFKe1Ojx8Bemua0nlGR4vnBvgW9DlWif0urFewCGcEbES5qm/WKNb6XEpj7fPaKqQAlUBbK/5RjxeeluR8zKjU0/SV6VpTK7ykyFVYLrFPmf8OxAdLf0W1ZK8Ryj59NOw7NPgL+lxdJgdojYlNcYMY5ArudZe5YPrNi1XVrrcGZd33+wfQThgfTu5JmqFQU5KMaai3sIIFXwn0ObSQkQxzKO2rJ9R26ch94zRj7GxQHs37QmZDTaBWYyYSfVqjx77Es6t8rFGM78J8uiBRvFMi2NZ9v/ycohyGH0pwyA/QLgWEjSo9ZqDfaiK1LF6UGbxlMziYlweJvDMeF6lUa/iQ9wiV+vG+cohIBrUVeZj8kUqQZb2NbK3ZDhhNFbqqHBquLhnrHM/dDmcMTu4S2MeftU5Ijxote/SRUlrLUiqJaByeNa3eA14Qvl6xM9rta8bsZaC0eONu4Sgb2YHLrqP9oEbpzrfjyZ2MFjZM22jXt7bihnQarPLpWLQkGp+kqPyh52c0PLkBo81ziIKY0vGlmvui3zcsIA1BpGm7BKxiHSSkDYLHF+frtIYCn9ZSShNbH/5ERVf4wnX8xZ1VdbO6fEyS+79nMx+7hKYFG5hHwVuPZ5E+Q7jlIXzpuzmiO4FCf9b/Zt6f+3wKk2Ka3p5I7i26o0h4mdQS5RxOHN2Z9EhJswWEGbCxRLyc+VaDU41/DqMxEJBdo2p3A38eZns84HFSBNwyT1O7htfIYEtfHN95U/onuXgd97+fWKVbDQtX3JHrmkuw/E4QjoPiaxbfsRbPWt8tYCZPleICEeIKce41S84Z62tJEoG4dBqvIWgjixn9T+K5nE5+KSKXaUyXFZmB8sNpGx48SFLwXh7UOMHIQ5IwAowaSSpFmWyvjNAFDTj 0hvvZb9B CfjQkwIzECIyUrhDJ/rFQseVTzDd4TFMgXExAB+O/MRIYrHNjTh8Tucmnky5z7pdF+vbNqYZtBLzoAgUGvYOmRfsOm/aAqIGuc8Oa5eMvx/n8Fa/xmFl9svnPTseBHYYP/e+aZQ4oL3oCConS6g8uNWf5HREvnzRMpZmpK0LsNWbD7b6JhF2fp1k53mOnRW1GatX2oUArhLRrCNw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/5/25 20:39, Ryan Roberts wrote: > Because the kernel can't tolerate page faults for kernel mappings, when > setting a valid, kernel space pte (or pmd/pud/p4d/pgd), it emits a > dsb(ishst) to ensure that the store to the pgtable is observed by the > table walker immediately. Additionally it emits an isb() to ensure that > any already speculatively determined invalid mapping fault gets > canceled.> > We can improve the performance of vmalloc operations by batching these > barriers until the end of a set up entry updates. The newly added > arch_update_kernel_mappings_begin() / arch_update_kernel_mappings_end() > provide the required hooks. > > vmalloc improves by up to 30% as a result. > > Two new TIF_ flags are created; TIF_KMAP_UPDATE_ACTIVE tells us if we > are in the batch mode and can therefore defer any barriers until the end > of the batch. TIF_KMAP_UPDATE_PENDING tells us if barriers are queued to > be emited at the end of the batch. Why cannot this be achieved with a single TIF_KMAP_UPDATE_ACTIVE which is set in __begin(), cleared in __end() and saved across a __switch_to(). > > Signed-off-by: Ryan Roberts > --- > arch/arm64/include/asm/pgtable.h | 65 +++++++++++++++++++--------- > arch/arm64/include/asm/thread_info.h | 2 + > arch/arm64/kernel/process.c | 20 +++++++-- > 3 files changed, 63 insertions(+), 24 deletions(-) > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h > index ff358d983583..1ee9b9588502 100644 > --- a/arch/arm64/include/asm/pgtable.h > +++ b/arch/arm64/include/asm/pgtable.h > @@ -39,6 +39,41 @@ > #include > #include > #include > +#include > + > +static inline void emit_pte_barriers(void) > +{ > + dsb(ishst); > + isb(); > +} There are many sequence of these two barriers in this particular header, hence probably a good idea to factor this out into a common helper. > + > +static inline void queue_pte_barriers(void) > +{ > + if (test_thread_flag(TIF_KMAP_UPDATE_ACTIVE)) { > + if (!test_thread_flag(TIF_KMAP_UPDATE_PENDING)) > + set_thread_flag(TIF_KMAP_UPDATE_PENDING); > + } else > + emit_pte_barriers(); > +} > + > +#define arch_update_kernel_mappings_begin arch_update_kernel_mappings_begin > +static inline void arch_update_kernel_mappings_begin(unsigned long start, > + unsigned long end) > +{ > + set_thread_flag(TIF_KMAP_UPDATE_ACTIVE); > +} > + > +#define arch_update_kernel_mappings_end arch_update_kernel_mappings_end > +static inline void arch_update_kernel_mappings_end(unsigned long start, > + unsigned long end, > + pgtbl_mod_mask mask) > +{ > + if (test_thread_flag(TIF_KMAP_UPDATE_PENDING)) > + emit_pte_barriers(); > + > + clear_thread_flag(TIF_KMAP_UPDATE_PENDING); > + clear_thread_flag(TIF_KMAP_UPDATE_ACTIVE); > +} > > #ifdef CONFIG_TRANSPARENT_HUGEPAGE > #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE > @@ -323,10 +358,8 @@ static inline void __set_pte_complete(pte_t pte) > * Only if the new pte is valid and kernel, otherwise TLB maintenance > * or update_mmu_cache() have the necessary barriers. > */ > - if (pte_valid_not_user(pte)) { > - dsb(ishst); > - isb(); > - } > + if (pte_valid_not_user(pte)) > + queue_pte_barriers(); > } > > static inline void __set_pte(pte_t *ptep, pte_t pte) > @@ -791,10 +824,8 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) > > WRITE_ONCE(*pmdp, pmd); > > - if (pmd_valid_not_user(pmd)) { > - dsb(ishst); > - isb(); > - } > + if (pmd_valid_not_user(pmd)) > + queue_pte_barriers(); > } > > static inline void pmd_clear(pmd_t *pmdp) > @@ -869,10 +900,8 @@ static inline void set_pud(pud_t *pudp, pud_t pud) > > WRITE_ONCE(*pudp, pud); > > - if (pud_valid_not_user(pud)) { > - dsb(ishst); > - isb(); > - } > + if (pud_valid_not_user(pud)) > + queue_pte_barriers(); > } > > static inline void pud_clear(pud_t *pudp) > @@ -960,10 +989,8 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) > > WRITE_ONCE(*p4dp, p4d); > > - if (p4d_valid_not_user(p4d)) { > - dsb(ishst); > - isb(); > - } > + if (p4d_valid_not_user(p4d)) > + queue_pte_barriers(); > } > > static inline void p4d_clear(p4d_t *p4dp) > @@ -1098,10 +1125,8 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) > > WRITE_ONCE(*pgdp, pgd); > > - if (pgd_valid_not_user(pgd)) { > - dsb(ishst); > - isb(); > - } > + if (pgd_valid_not_user(pgd)) > + queue_pte_barriers(); > } > > static inline void pgd_clear(pgd_t *pgdp) > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h > index 1114c1c3300a..382d2121261e 100644 > --- a/arch/arm64/include/asm/thread_info.h > +++ b/arch/arm64/include/asm/thread_info.h > @@ -82,6 +82,8 @@ void arch_setup_new_exec(void); > #define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */ > #define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */ > #define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */ > +#define TIF_KMAP_UPDATE_ACTIVE 31 /* kernel map update in progress */ > +#define TIF_KMAP_UPDATE_PENDING 32 /* kernel map updated with deferred barriers */ > > #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c > index 42faebb7b712..1367ec6407d1 100644 > --- a/arch/arm64/kernel/process.c > +++ b/arch/arm64/kernel/process.c > @@ -680,10 +680,10 @@ struct task_struct *__switch_to(struct task_struct *prev, > gcs_thread_switch(next); > > /* > - * Complete any pending TLB or cache maintenance on this CPU in case > - * the thread migrates to a different CPU. > - * This full barrier is also required by the membarrier system > - * call. > + * Complete any pending TLB or cache maintenance on this CPU in case the > + * thread migrates to a different CPU. This full barrier is also > + * required by the membarrier system call. Additionally it is required > + * for TIF_KMAP_UPDATE_PENDING, see below. > */ > dsb(ish); > > @@ -696,6 +696,18 @@ struct task_struct *__switch_to(struct task_struct *prev, > /* avoid expensive SCTLR_EL1 accesses if no change */ > if (prev->thread.sctlr_user != next->thread.sctlr_user) > update_sctlr_el1(next->thread.sctlr_user); > + else if (unlikely(test_thread_flag(TIF_KMAP_UPDATE_PENDING))) { > + /* > + * In unlikely event that a kernel map update is on-going when > + * preemption occurs, we must emit_pte_barriers() if pending. > + * emit_pte_barriers() consists of "dsb(ishst); isb();". The dsb > + * is already handled above. The isb() is handled if > + * update_sctlr_el1() was called. So only need to emit isb() > + * here if it wasn't called. > + */ > + isb(); > + clear_thread_flag(TIF_KMAP_UPDATE_PENDING); > + } > > /* the actual thread switch */ > last = cpu_switch_to(prev, next);