From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D483CCD195 for ; Wed, 15 Oct 2025 08:28:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04A918E0013; Wed, 15 Oct 2025 04:28:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3D598E0002; Wed, 15 Oct 2025 04:28:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2C258E0013; Wed, 15 Oct 2025 04:28:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CE25D8E0002 for ; Wed, 15 Oct 2025 04:28:22 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 870475CBEF for ; Wed, 15 Oct 2025 08:28:22 +0000 (UTC) X-FDA: 83999671644.15.327DAF8 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id D2E3E1C000C for ; Wed, 15 Oct 2025 08:28:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1760516901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NQTwqTlBXr6retjra5GPHcjItFw8MKUqPe4SY+8hd5k=; b=lYa1LwFfYSvH7iynqREM0Njgh1hC9UqeeLNCVwNdYsogQEZdE+02ZAQuAn9W9C2V/gl6EF SE3j04nTnYNsBbbmhx5KK5tQjsYuzfNIcXxs/D7S3a58NRqIhb0/9/HILzur9cW/95K0az KTOBtsB4aW74PTmoglWqARtYI5/VXI0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1760516901; a=rsa-sha256; cv=none; b=flXX+BESrByWyB5Nik9KG5F1LnhKMYzHWQWMewwrOT5tz9pDsyi2dEPeo/ZABa/5XUi9NX fPe9dDR8Dj7xyFkPcUXnR3PPqOsZNuaqvknrkpgUCpfEDl7teteE4Wg4DmIwBLpPsjXUhR xXKb3vnIhcFeKAQyq8m8lqWmQMkKzDs= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44DD11A32; Wed, 15 Oct 2025 01:28:12 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3FFB93F66E; Wed, 15 Oct 2025 01:28:15 -0700 (PDT) From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 07/13] mm: enable lazy_mmu sections to nest Date: Wed, 15 Oct 2025 09:27:21 +0100 Message-ID: <20251015082727.2395128-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: D2E3E1C000C X-Rspamd-Server: rspam03 X-Stat-Signature: b9ke3aq9gkqpcidwxata147wpzy11ssi X-HE-Tag: 1760516900-837834 X-HE-Meta: U2FsdGVkX1/gUdzNWLbNJpWUSMKxv1V7qpif1hAv5qhniHAcbYWwRGCJY2z1v19CBLoAIK/oJhLYA7QJAz5EeW6NqCncGsZkSfW4yhiTNhom5JVkjTsJpc+oUrOuyy2UJsv/TMLXutYxurUiB5WC0q0eZVkWgYov8hJMfCIR9F6zQlu1tLgMNVbuzFmuoz4+CChLOB8a1O5UjRiJpB20jwnvRqeWQNzTL6xvNUgLs0bHzae9VJASo5CO03D2yTrn02I2I3tjygbuZ20vI3nbth9mqIqWLNWHE5BXhLdyWNvh8zi/VAlsN3Mj99Wetr5Px6JHpXofTdAR5L5s1s7sxHmVJd1wl0W6Y/wODubijSuuOxuavJpKR33k7nlUwjHb92aAWGEXGqEBRevkRSEPhVpT0PYzh4i2qFVmFub64SkGIUMAkdDE++E46j/uzJBPyG/y9rghpIsOZ+/x0LgXHiBznqH9Zt/xkqj7aWNNIm/8qzle3Zs9jatyM+ZBG+1te/6WTp2XMb+d0iXznZZMmVj7/4MkfN0Qze3qz812fxyhieqOzyj3d6Khy0H4hzrMfOnaCKFZx1Sc4AM0gLE02jEFV3YPx0R3Rme6VjxrLlRXmcvuVAQn3pAPsJvxNCVQQyao4JeqijGlCbXdfiDxrhB6pImbmD1umG/5zIpv9MQ2nlEebutMK+GV8HoaHmfn2xGd6VmTkeCRmTq+fZF0C2toO1Tl2JAihcPsMwRXAla966eOkCQn0F5lOQ8lys01txj3DsVc27vFo/ufai3TjOcTvz+dTE9lnTEc7tJWjlIf88j9vb14+nYnsNDMbyykZ4OdgSgi9ji3yHnFJOg7/w/0QlV6SLpTW+4R47pJ7ft+aKXwGb6FntyTY8MwD6sYIaS49TO/cbxy7gaakA3W32AljS3sbELKdtldwKSCjP3UkZL4KJj9QiZI0+dLpSjRbanz6pGytyA1rLxvKCe +qiBEg7S 5mLK3IgFer9xEq9Z8Lyx8wjw6dD1m6iBfspD7Zk9u4Y25dtPmTgvM+nEuJPCD6blnVrTrvcT/DfRL7bp0lM6omiTtHSVWWQELyEnJHbTY9H2pLHQK1qL5vCbFeG6WLu5chpziGykE6Wvo4+0wKGyPOSlXQQcBy6nTgqz9iIXSCArwhK7bOy8B+gwVB3dSL/zlAlc0x3Oiy6sYd3xJ1RhdlFyQuUEcMkREuwv8LWmuRSzq4DIgvqMRCI3MF3jI44Fk2H+emQ2YyS/EkL1rfhZFTtwllYGRu/tm8lDplsCPi7rgnNNLnGMmd2x+oWNX3ZThYGL4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Despite recent efforts to prevent lazy_mmu sections from nesting, it remains difficult to ensure that it never occurs - and in fact it does occur on arm64 in certain situations (CONFIG_DEBUG_PAGEALLOC). Commit 1ef3095b1405 ("arm64/mm: Permit lazy_mmu_mode to be nested") made nesting tolerable on arm64, but without truly supporting it: the inner call to leave() disables the batching optimisation before the outer section ends. This patch actually enables lazy_mmu sections to nest by tracking the nesting level in task_struct, in a similar fashion to e.g. pagefault_{enable,disable}(). This is fully handled by the generic lazy_mmu helpers that were recently introduced. lazy_mmu sections were not initially intended to nest, so we need to clarify the semantics w.r.t. the arch_*_lazy_mmu_mode() callbacks. This patch takes the following approach: * The outermost calls to lazy_mmu_mode_{enable,disable}() trigger calls to arch_{enter,leave}_lazy_mmu_mode() - this is unchanged. * Nested calls to lazy_mmu_mode_{enable,disable}() are not forwarded to the arch via arch_{enter,leave} - lazy MMU remains enabled so the assumption is that these callbacks are not relevant. However, existing code may rely on a call to disable() to flush any batched state, regardless of nesting. arch_flush_lazy_mmu_mode() is therefore called in that situation. A separate interface was recently introduced to temporarily pause the lazy MMU mode: lazy_mmu_mode_{pause,resume}(). pause() fully exits the mode *regardless of the nesting level*, and resume() restores the mode at the same nesting level. Whether the mode is actually enabled or not at any point is tracked by a separate "enabled" field in task_struct; this makes it possible to check invariants in the generic API, and to expose a new in_lazy_mmu_mode() helper to replace the various ways arch's currently track whether the mode is enabled (this will be done in later patches). In summary (count/enabled represent the values *after* the call): lazy_mmu_mode_enable() -> arch_enter() count=1 enabled=1 lazy_mmu_mode_enable() -> ΓΈ count=2 enabled=1 lazy_mmu_mode_pause() -> arch_leave() count=2 enabled=0 lazy_mmu_mode_resume() -> arch_enter() count=2 enabled=1 lazy_mmu_mode_disable() -> arch_flush() count=1 enabled=1 lazy_mmu_mode_disable() -> arch_leave() count=0 enabled=0 Note: in_lazy_mmu_mode() is added to to allow arch headers included by to use it. Signed-off-by: Kevin Brodsky --- Alexander Gordeev suggested that a future optimisation may need lazy_mmu_mode_{pause,resume}() to call distinct arch callbacks [1]. For now arch_{leave,enter}() are called directly, but introducing new arch callbacks should be straightforward. [1] https://lore.kernel.org/all/5a0818bb-75d4-47df-925c-0102f7d598f4-agordeev@linux.ibm.com/ --- arch/arm64/include/asm/pgtable.h | 12 ------ include/linux/mm_types_task.h | 5 +++ include/linux/pgtable.h | 69 ++++++++++++++++++++++++++++++-- include/linux/sched.h | 16 ++++++++ 4 files changed, 86 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e3cbb10288c4..f15ca4d62f09 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -82,18 +82,6 @@ static inline void queue_pte_barriers(void) static inline void arch_enter_lazy_mmu_mode(void) { - /* - * lazy_mmu_mode is not supposed to permit nesting. But in practice this - * does happen with CONFIG_DEBUG_PAGEALLOC, where a page allocation - * inside a lazy_mmu_mode section (such as zap_pte_range()) will change - * permissions on the linear map with apply_to_page_range(), which - * re-enters lazy_mmu_mode. So we tolerate nesting in our - * implementation. The first call to arch_leave_lazy_mmu_mode() will - * flush and clear the flag such that the remainder of the work in the - * outer nest behaves as if outside of lazy mmu mode. This is safe and - * keeps tracking simple. - */ - if (in_interrupt()) return; diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a82aa80c0ba4..2ff83b85fef0 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -88,4 +88,9 @@ struct tlbflush_unmap_batch { #endif }; +struct lazy_mmu_state { + u8 count; + bool enabled; +}; + #endif /* _LINUX_MM_TYPES_TASK_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 194b2c3e7576..269225a733de 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -228,28 +228,89 @@ static inline int pmd_dirty(pmd_t pmd) * of the lazy mode. So the implementation must assume preemption may be enabled * and cpu migration is possible; it must take steps to be robust against this. * (In practice, for user PTE updates, the appropriate page table lock(s) are - * held, but for kernel PTE updates, no lock is held). Nesting is not permitted - * and the mode cannot be used in interrupt context. + * held, but for kernel PTE updates, no lock is held). The mode cannot be used + * in interrupt context. + * + * The lazy MMU mode is enabled for a given block of code using: + * + * lazy_mmu_mode_enable(); + * + * lazy_mmu_mode_disable(); + * + * Nesting is permitted: may itself use an enable()/disable() pair. + * A nested call to enable() has no functional effect; however disable() causes + * any batched architectural state to be flushed regardless of nesting. After a + * call to disable(), the caller can therefore rely on all previous page table + * modifications to have taken effect, but the lazy MMU mode may still be + * enabled. + * + * In certain cases, it may be desirable to temporarily pause the lazy MMU mode. + * This can be done using: + * + * lazy_mmu_mode_pause(); + * + * lazy_mmu_mode_resume(); + * + * This sequence must only be used if the lazy MMU mode is already enabled. + * pause() ensures that the mode is exited regardless of the nesting level; + * resume() re-enters the mode at the same nesting level. must not modify + * the lazy MMU state (i.e. it must not call any of the lazy_mmu_mode_* + * helpers). + * + * in_lazy_mmu_mode() can be used to check whether the lazy MMU mode is + * currently enabled. */ #ifdef CONFIG_ARCH_LAZY_MMU static inline void lazy_mmu_mode_enable(void) { - arch_enter_lazy_mmu_mode(); + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; + + VM_BUG_ON(state->count == U8_MAX); + /* enable() must not be called while paused */ + VM_WARN_ON(state->count > 0 && !state->enabled); + + if (state->count == 0) { + arch_enter_lazy_mmu_mode(); + state->enabled = true; + } + ++state->count; } static inline void lazy_mmu_mode_disable(void) { - arch_leave_lazy_mmu_mode(); + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; + + VM_BUG_ON(state->count == 0); + VM_WARN_ON(!state->enabled); + + --state->count; + if (state->count == 0) { + state->enabled = false; + arch_leave_lazy_mmu_mode(); + } else { + /* Exiting a nested section */ + arch_flush_lazy_mmu_mode(); + } } static inline void lazy_mmu_mode_pause(void) { + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; + + VM_WARN_ON(state->count == 0 || !state->enabled); + + state->enabled = false; arch_leave_lazy_mmu_mode(); } static inline void lazy_mmu_mode_resume(void) { + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; + + VM_WARN_ON(state->count == 0 || state->enabled); + arch_enter_lazy_mmu_mode(); + state->enabled = true; } #else static inline void lazy_mmu_mode_enable(void) {} diff --git a/include/linux/sched.h b/include/linux/sched.h index cbb7340c5866..2862d8bf2160 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1441,6 +1441,10 @@ struct task_struct { struct page_frag task_frag; +#ifdef CONFIG_ARCH_LAZY_MMU + struct lazy_mmu_state lazy_mmu_state; +#endif + #ifdef CONFIG_TASK_DELAY_ACCT struct task_delay_info *delays; #endif @@ -1724,6 +1728,18 @@ static inline char task_state_to_char(struct task_struct *tsk) return task_index_to_char(task_state_index(tsk)); } +#ifdef CONFIG_ARCH_LAZY_MMU +static inline bool in_lazy_mmu_mode(void) +{ + return current->lazy_mmu_state.enabled; +} +#else +static inline bool in_lazy_mmu_mode(void) +{ + return false; +} +#endif + extern struct pid *cad_pid; /* -- 2.47.0