From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C684AC3ABA5 for ; Tue, 29 Apr 2025 05:24:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CA526B0027; Tue, 29 Apr 2025 01:24:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 656B76B0028; Tue, 29 Apr 2025 01:24:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F75F6B0029; Tue, 29 Apr 2025 01:24:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2D5D26B0027 for ; Tue, 29 Apr 2025 01:24:54 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8933FBD7A9 for ; Tue, 29 Apr 2025 05:24:54 +0000 (UTC) X-FDA: 83385942108.23.995E8F0 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id CC61C100003 for ; Tue, 29 Apr 2025 05:24:52 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745904293; a=rsa-sha256; cv=none; b=XdFz5TisUanQkLlFVuOMWNThrS5nh2iSRqdnHmtoXtUMx/PnTeQS1nWUcs8ftQzH08pJdK PIYEj4fffwTkBGQ2zM9p+9d9fBDCnWJFJspQg/yQC0r+d5Cz0S0ymJkwkXeO9yiD65AtEg HpgLk+zIx3OZx53tcd+qP/gWj9WsEs0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745904292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2qjZmg0eEJEeO+MiKEt7kQSfHjMNyNDs5BR4+Dl9DEQ=; b=OFokwVktUNWHqpYyhs5qe4JN7gKnXSLL43DJ2b2VLaW80A333AsMqsp8aPvqoM4LlffCXJ JPpNIaubaSRdS2reuSlQXdZvyby9RVnho/FyiQFje+Kbm4KFC2FblojI20njXwUYDqAhSR 9fyMT1DvcjBH5qSrWT8oiet4vGihTjk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 748811515; Mon, 28 Apr 2025 22:24:45 -0700 (PDT) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.253]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4330A3F5A1; Mon, 28 Apr 2025 22:24:41 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, catalin.marinas@arm.com, will@kernel.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, anshuman.khandual@arm.com, peterx@redhat.com, joey.gouly@arm.com, ioworker0@gmail.com, baohua@kernel.org, kevin.brodsky@arm.com, quic_zhenhuah@quicinc.com, christophe.leroy@csgroup.eu, yangyicong@hisilicon.com, linux-arm-kernel@lists.infradead.org, namit@vmware.com, hughd@google.com, yang@os.amperecomputing.com, ziy@nvidia.com, Dev Jain Subject: [PATCH v2 6/7] mm: Batch around can_change_pte_writable() Date: Tue, 29 Apr 2025 10:53:35 +0530 Message-Id: <20250429052336.18912-7-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250429052336.18912-1-dev.jain@arm.com> References: <20250429052336.18912-1-dev.jain@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: CC61C100003 X-Stat-Signature: 7cuk1ije6sh57fq5pup5jsamh9c1o5oi X-Rspam-User: X-HE-Tag: 1745904292-62227 X-HE-Meta: U2FsdGVkX1/9F+B2MrRP6xd/HtYfUnUXbXD+wcslGZ30pROEQTdd2lk1rbBpwHwr0Ol+m1hExTgeG2AFSqmo7xSk9KFMJCWi35nReyxFQQBzCQaux3IKqpMh61cw5SxVNtLaA/IlfjAqXt1qDhpXndBkNUqBw0wp7uBjqRfonlaIbS26d6XUgzf1p44LX0mysZGaV3uLDodJ2Z2iy3FKrVsmoDwIDjVKTcGSF++Ymnuebm2fnm4S5/uNKTO2pr5L9NPT+vv71QVkwxRkEcTaqEkPDzUM94186Ff2bi0Yb6tVqmxozoY8djRpwni2amKWO/dLCx9grkBlnCEi/tY60TuF8hSipJbqWbzduxvkPA/RBxpAmRb5nQl5gxXc9LqPg7D/LlEU8K+Anuv1AjlODfuHBTZwGcSYdxdLddSamaKkl+keU6XjrrZBG8gS1UVHvCPUz5fBbZBAMQ7mNdBZjbxtgKPLWw4Lh2Bwi5ygu6CLsI1QsZHVxt1Bf9IERDHB2lqL4/PGa8M3Bbx0E7Gby499m+5o9butz8MZF8PO9jJJpBlZJUY2ql7uplMxw6K/mXO/Y9UcNaycvNfZfaS7lRBG9jcWvJznjD0Vdttw2Kve4+F7HgEDqg/T2YAhV+cfuB8Z3P7V3zUWwVLrRyDNPImIJjyztDTFimaLZre+QwlY6d7ats5q9vu85FaYh69GC6pA8431z8oIPeQAPaLuWJ8efNnlZqK/vt6laLXzwgYuZgtL5Mz16hppmoN4IgzacunC5FNEeyoWOMMl/5fyVnR3HYBnqniPIWbmFPWPnF+IwehVqVlEUgJKI+OR+9kXYMx7czFAOo7WhRE82uo4HvG8YlLS+tztQuC9fOwOnnC2t+x5CgcO3E8pmXT4Y24ZknnFS9Xylpt7GsBf7BBGsGJbTrLKyAoQeCuBZg1c7oZKKF6vE8y5hvqpMBJsH65Tlc2JuAL8v/f6UeaA5Ag h/t800Iz bz6BIeLCaQfdZ/iiFmdnI2TSjs8DERTx3Tjf6oPnCeLxmPe8kR6q5ir6PEg6aLUzNg9QEnyxlIzEfhfWLKLfgqGhCUkyfd6e1T9CkZTEPlgw4erqLWsSdTeNRYmPgQbqx8l23ysEMrgB/ZK1eYl+kAbL5i5WwWB3prqYJyEs5sXaBKBlkwmTlIOXKuBMWv8NSade/IUchxnm2HJ0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for patch 7, we need to properly batch around can_change_pte_writable(). We batch around pte_needs_soft_dirty_wp() by the corresponding fpb flag, we batch around the page-anon exclusive check using folio_maybe_mapped_shared(); modify_prot_start_ptes() collects the dirty and access bits across the batch, therefore batching across pte_dirty(): this is correct since the dirty bit on the PTE really is just an indication that the folio got written to, so even if the PTE is not actually dirty (but one of the PTEs in the batch is), the wp-fault optimization can be made. Signed-off-by: Dev Jain --- include/linux/mm.h | 4 ++-- mm/gup.c | 2 +- mm/huge_memory.c | 4 ++-- mm/memory.c | 6 +++--- mm/mprotect.c | 11 ++++++----- 5 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 21dd110b6655..2f639f6d93f9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2487,8 +2487,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen); #define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \ MM_CP_UFFD_WP_RESOLVE) -bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, - pte_t pte); +bool can_change_ptes_writable(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct folio *folio, unsigned int nr); extern long change_protection(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long cp_flags); diff --git a/mm/gup.c b/mm/gup.c index f32168339390..c39f587842a0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -614,7 +614,7 @@ static inline bool can_follow_write_common(struct page *page, return false; /* - * See can_change_pte_writable(): we broke COW and could map the page + * See can_change_ptes_writable(): we broke COW and could map the page * writable if we have an exclusive anonymous page ... */ return page && PageAnon(page) && PageAnonExclusive(page); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2780a12b25f0..a58445fcedfc 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2032,12 +2032,12 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, return false; if (!(vma->vm_flags & VM_SHARED)) { - /* See can_change_pte_writable(). */ + /* See can_change_ptes_writable(). */ page = vm_normal_page_pmd(vma, addr, pmd); return page && PageAnon(page) && PageAnonExclusive(page); } - /* See can_change_pte_writable(). */ + /* See can_change_ptes_writable(). */ return pmd_dirty(pmd); } diff --git a/mm/memory.c b/mm/memory.c index 68c1d962d0ad..e7ebc6b70421 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -750,7 +750,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, pte = pte_mkuffd_wp(pte); if ((vma->vm_flags & VM_WRITE) && - can_change_pte_writable(vma, address, pte)) { + can_change_ptes_writable(vma, address, pte, NULL, 1)) { if (folio_test_dirty(folio)) pte = pte_mkdirty(pte); pte = pte_mkwrite(pte, vma); @@ -5796,7 +5796,7 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru ptent = pte_modify(ptent, vma->vm_page_prot); writable = pte_write(ptent); if (!writable && pte_write_upgrade && - can_change_pte_writable(vma, addr, ptent)) + can_change_ptes_writable(vma, addr, ptent, NULL, 1)) writable = true; } @@ -5837,7 +5837,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) */ writable = pte_write(pte); if (!writable && pte_write_upgrade && - can_change_pte_writable(vma, vmf->address, pte)) + can_change_ptes_writable(vma, vmf->address, pte, NULL, 1)) writable = true; folio = vm_normal_folio(vma, vmf->address, pte); diff --git a/mm/mprotect.c b/mm/mprotect.c index ec5d17af7650..baff009fc981 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -40,8 +40,8 @@ #include "internal.h" -bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, - pte_t pte) +bool can_change_ptes_writable(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct folio *folio, unsigned int nr) { struct page *page; @@ -67,8 +67,9 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr, * write-fault handler similarly would map them writable without * any additional checks while holding the PT lock. */ - page = vm_normal_page(vma, addr, pte); - return page && PageAnon(page) && PageAnonExclusive(page); + if (!folio) + folio = vm_normal_folio(vma, addr, pte); + return folio_test_anon(folio) && !folio_maybe_mapped_shared(folio); } VM_WARN_ON_ONCE(is_zero_pfn(pte_pfn(pte)) && pte_dirty(pte)); @@ -222,7 +223,7 @@ static long change_pte_range(struct mmu_gather *tlb, */ if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pte_write(ptent) && - can_change_pte_writable(vma, addr, ptent)) + can_change_ptes_writable(vma, addr, ptent, folio, 1)) ptent = pte_mkwrite(ptent, vma); ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent); -- 2.30.2