From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 017E7C369DC for ; Mon, 28 Apr 2025 12:05:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E1666B008C; Mon, 28 Apr 2025 08:05:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3944F6B00B4; Mon, 28 Apr 2025 08:05:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20EEE6B00B5; Mon, 28 Apr 2025 08:05:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 017106B008C for ; Mon, 28 Apr 2025 08:05:38 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 06A071216CE for ; Mon, 28 Apr 2025 12:05:40 +0000 (UTC) X-FDA: 83383323240.01.ABB6230 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf25.hostedemail.com (Postfix) with ESMTP id 69318A0002 for ; Mon, 28 Apr 2025 12:05:38 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745841938; a=rsa-sha256; cv=none; b=vAHVIojAT3B8SWjcgGUtIRuMocVwwsTBKtR4X31FFj2RX9NTVZiIscU/6/KW8yLDSQTA1z 20UEH+vaemDvWKKSlBXV9OcTOEGvtPlyC/HOIwujdNwaq3ubT3zmMErANTZlp/DKu1j8ot MMQN1jdDzTNTJHZMrl2/RZcLLMqokxk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf25.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745841938; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q8cZD9ZzY3v39fKexFRUN197TL2q7m4akqDWnsZ08ZA=; b=urEQAq0mGZcv7V/CRIWCtuBYFEDw25RSEauvQIoZ8uYYR68ESBNRDmSNgzI0b+OCp0fndK g5G229YvJeaNXP3CSDL1rjDLPpZnGs++vhqwQnKrkHeahhBQC9mukkZDiMPwHC/C+TZpeS IGQu5JbKxS+YF7IVVTVAOQY0jIuo0r0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4F13D22D7; Mon, 28 Apr 2025 05:05:31 -0700 (PDT) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7DED83F66E; Mon, 28 Apr 2025 05:05:29 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, catalin.marinas@arm.com, will@kernel.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, anshuman.khandual@arm.com, peterx@redhat.com, joey.gouly@arm.com, ioworker0@gmail.com, baohua@kernel.org, kevin.brodsky@arm.com, quic_zhenhuah@quicinc.com, christophe.leroy@csgroup.eu, yangyicong@hisilicon.com, linux-arm-kernel@lists.infradead.org, namit@vmware.com, hughd@google.com, yang@os.amperecomputing.com, ziy@nvidia.com, Dev Jain Subject: [PATCH 6/7] mm: Batch around can_change_pte_writable() Date: Mon, 28 Apr 2025 17:34:13 +0530 Message-Id: <20250428120414.12101-7-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250428120414.12101-1-dev.jain@arm.com> References: <20250428120414.12101-1-dev.jain@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 69318A0002 X-Stat-Signature: zuhicuep7y16xonfi4twxuprsqd8ox1p X-HE-Tag: 1745841938-774011 X-HE-Meta: U2FsdGVkX18OLLgVX7M2fW0RV8A+U8Ftsxk5D8D6MQ8ck6PsEVGbvsqsTzfjfk0oS61Z2Pva7WPqFhPxF7C9KiEfDZ0510O9wOCpuatsDbOyaZUHshsAM00+R9Nv0TKZVPUytJuglx3GNiVMAzgxVjZXg8m2SCw23/7892NAAxd4nA1BHgRtd7TIIRGOMffEhdKolY7o2X+ZiaDSTsE97hZOKM0yB29qk7JuQcoX6XzHL5sMe+Kf2wN4HyjdTkxN0gj755gSEex8XnUxmYmi89FnXkpK5vkbhANUvcAciPbeDoumHrhMZfBQnjwPTQlPYeVsKx/xClG8COfF6F7BW00l9Wxj559mwxzu1llyDWVg0ICbIyZa0o1GSw0g2loEXWaVfL5yn4CjRindXNe7N2XK208ggHqlMOsopMzMKvjJEJuhnRnKA20L8Y6ugdkKu1zmbs+zmw5/XmMhV6PVxe3vMbdOsCQzvvX+agl4rT13BJrlVuCNuc0qS/ZXwWpMw3MogL1RVZ9bl97n/EVDEAtH7YiTwp5NdE7fRRIbJOZOQy2HMRiJQTan7bfiPSq3J9OuPNxkOL2/g8x+PlPpIaw4C8NH8a5bR/frQqJGD9Zq456YO2XlV69PF4FLcLDzJsvRus+aQDw0RGciyFUVNKhmzwCN0+oBjANauAUupCprVDH2rZG4kOpGe/zF+1sKpfcYW8HvXCl6J5tM6OLA7AAmbHfBaRAs2Y4ubpXjeAoNod+v5PazMYHsKwRhOBTyvNB1pbzKk+NlGc5QjK9KxmHdJ5ybV5DqPd4DaRrCA1vGmGg/nAkrs0OQjYQkvP40jLbGLn/eyuMw+0J8UM9UreomA/fJGAIs0glj0xW+5heesAgeIj+apVzlCkpQIT8KNpSEn6OpvSQDul7vjQ81WF+q5Kmx9GyOeKQ2sqWxsuUqsvZHycvBua29/mwyuWEYAAfN4VqQdT66CI+49V2 CMv6Fpiy INelWtYq/+WdITcvpTf6CzsijTv5UrN+VaPpgGWxXBRxhvPR4x3XyXf//uUFBFZZpGF8DmyVViY9yvb0S5kkkqwCUGdwVAXDFLtCTFSRRYUfMghPP8kjEOM1ExsVyoXogJvnBjGVI8RXZGmadO7Ga/i0dK5MHaAP055WKPqZSKaChQnmDy94EbM+JA2VJvoi28v3HHmrMUNykn2Y= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for patch 7, we need to properly batch around can_change_pte_writable(). We batch around pte_needs_soft_dirty_wp() by the corresponding fpb flag, we batch around the page-anon exclusive check using folio_maybe_mapped_shared(); modify_prot_start_ptes() collects the dirty and access bits across the batch, therefore batching across pte_dirty(): this is correct since the dirty bit on the PTE really is just an indication that the folio got written to, so even if the PTE is not actually dirty (but one of the PTEs in the batch is), the wp-fault optimization can be made. Signed-off-by: Dev Jain --- include/linux/mm.h | 4 ++-- mm/gup.c | 2 +- mm/huge_memory.c | 4 ++-- mm/memory.c | 6 +++--- mm/mprotect.c | 9 ++++++--- 5 files changed, 14 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5eb0d77c4438..ffa02e15863f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2710,8 +2710,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen); #define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) -bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, - pte_t pte); +bool can_change_ptes_writable(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct folio *folio, unsigned int nr); extern long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags); diff --git a/mm/gup.c b/mm/gup.c index 84461d384ae2..6a605fc5f2cb 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -614,7 +614,7 @@ static inline bool can_follow_write_common(struct page *page, return false; /* - * See can_change_pte_writable(): we broke COW and could map the page + * See can_change_ptes_writable(): we broke COW and could map the page * writable if we have an exclusive anonymous page ... */ return page && PageAnon(page) && PageAnonExclusive(page); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 28c87e0e036f..e5496c0d9e7e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2032,12 +2032,12 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, return false; if (!(vma->vm_flags & VM_SHARED)) { - /* See can_change_pte_writable(). */ + /* See can_change_ptes_writable(). */ page = vm_normal_page_pmd(vma, addr, pmd); return page && PageAnon(page) && PageAnonExclusive(page); } - /* See can_change_pte_writable(). */ + /* See can_change_ptes_writable(). */ return pmd_dirty(pmd); } diff --git a/mm/memory.c b/mm/memory.c index b9e8443aaa86..b1fda3de8d27 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -750,7 +750,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, pte = pte_mkuffd_wp(pte); if ((vma->vm_flags & VM_WRITE) && - can_change_pte_writable(vma, address, pte)) { + can_change_ptes_writable(vma, address, pte, NULL, 1)) { if (folio_test_dirty(folio)) pte = pte_mkdirty(pte); pte = pte_mkwrite(pte, vma); @@ -5767,7 +5767,7 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru ptent = pte_modify(ptent, vma->vm_page_prot); writable = pte_write(ptent); if (!writable && pte_write_upgrade && - can_change_pte_writable(vma, addr, ptent)) + can_change_ptes_writable(vma, addr, ptent, NULL, 1)) writable = true; } @@ -5808,7 +5808,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) */ writable = pte_write(pte); if (!writable && pte_write_upgrade && - can_change_pte_writable(vma, vmf->address, pte)) + can_change_ptes_writable(vma, vmf->address, pte, NULL, 1)) writable = true; folio = vm_normal_folio(vma, vmf->address, pte); diff --git a/mm/mprotect.c b/mm/mprotect.c index 33eabc995584..362fd7e5457d 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -40,8 +40,8 @@ #include "internal.h" -bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, - pte_t pte) +bool can_change_ptes_writable(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct folio *folio, unsigned int nr) { struct page *page; @@ -67,6 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, * write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ + if (unlikely(nr != 1)) + return !folio_maybe_mapped_shared(folio); + page = vm_normal_page(vma, addr, pte); return page && PageAnon(page) && PageAnonExclusive(page); } @@ -222,7 +225,7 @@ static long change_pte_range(struct mmu_gather *tlb, */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pte_write(ptent) && - can_change_pte_writable(vma, addr, ptent)) + can_change_ptes_writable(vma, addr, ptent, folio, 1)) ptent = pte_mkwrite(ptent, vma); ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); -- 2.30.2