From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C29C4C369AB for ; Thu, 24 Apr 2025 10:52:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E5106B0099; Thu, 24 Apr 2025 06:52:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06E886B00B3; Thu, 24 Apr 2025 06:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E00546B00B4; Thu, 24 Apr 2025 06:52:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B9E1B6B0099 for ; Thu, 24 Apr 2025 06:52:36 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E7AAE1A1BA6 for ; Thu, 24 Apr 2025 10:52:37 +0000 (UTC) X-FDA: 83368623954.16.CE80C1B Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by imf12.hostedemail.com (Postfix) with ESMTP id 0536D40007 for ; Thu, 24 Apr 2025 10:52:35 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=igalia.com header.s=20170329 header.b=JWqr8Wh7; dmarc=pass (policy=none) header.from=igalia.com; spf=pass (imf12.hostedemail.com: domain of gavinguo@igalia.com designates 178.60.130.6 as permitted sender) smtp.mailfrom=gavinguo@igalia.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745491956; a=rsa-sha256; cv=none; b=QgosUgS649stYdMCK0ZD47xxDoxfyHf2lfetkNZyvfvbex2foWy9JWlzp37E+tF9uIMvE9 EGt+5m46eEcBCZb3mrPCE/FPyOqop7ffQgoVksPa4M4jHkRwkDFpIFL6TScsv+7uFAVCBU bKsouOdmdsFSjWqDZwwxlsWiD54bUgE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=igalia.com header.s=20170329 header.b=JWqr8Wh7; dmarc=pass (policy=none) header.from=igalia.com; spf=pass (imf12.hostedemail.com: domain of gavinguo@igalia.com designates 178.60.130.6 as permitted sender) smtp.mailfrom=gavinguo@igalia.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745491956; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=r/V7XqoJ6G3ABAqhFrYPZSJ+Oh7KOTFLRGs/+RlpRZE=; b=LyWG3Bu5TOIqCdaBVE9CxG6vXlMp4QhDUaRISTqBPhzT7JGk34Th3t9YCsQBz1v/5m2qwi 5BC5QkQCMkUTSGs+7g80bFqKITUiZx2JW3EU7Zy+4zFxTADqjxkedIpnRCmGXYIbBTtBz8 Gnca0hCYlWDO8OCoODkLOdumOAN9Lc0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:MIME-Version:Message-ID:Date:Subject: Cc:To:From:Sender:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=r/V7XqoJ6G3ABAqhFrYPZSJ+Oh7KOTFLRGs/+RlpRZE=; b=JWqr8Wh767TvMjoU9/4S0auRgE VxGbHmNwUUovjCw2+UJQRR9UTtPGN2vnnUR9kSH2W6y4odylvdPzssJxhJGKNPKSPlOOFU5Kz/DqF BBjhGMPBldKlEhy3BVOXHGRNCkzPMdWEcsUyAW1rIr9Yjlr9KHy5gjUGWcbdT7N73qYvkE2cqYtXt m6dgiPHPEKosmhtS4xdcA5lARxflSsPqVGfNWL2/YEFyKwRCF75yD5JRB0z39pyscVzof+U52TjdT rt8Smfh/M5y6G1atySzGsoIp6r04Y3JIWWy1N9KTa+KfAWKSWQoP87YQuTSwFmqRFZcOtioXqjPfF xPCwNGjA==; Received: from 114-44-196-209.dynamic-ip.hinet.net ([114.44.196.209] helo=gavin-HP-Z840-Workstation..) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim) id 1u7uBx-007n4l-P0; Thu, 24 Apr 2025 12:52:30 +0200 From: Gavin Guo To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: gshan@redhat.com, david@redhat.com, willy@infradead.org, ziy@nvidia.com, linmiaohe@huawei.com, hughd@google.com, revest@google.com, kernel-dev@igalia.com, linux-kernel@vger.kernel.org Subject: [PATCH] mm/huge_memory: Simplify the split_huge_pmd_locked function Date: Thu, 24 Apr 2025 18:50:12 +0800 Message-ID: <20250424105012.427390-1-gavinguo@igalia.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0536D40007 X-Stat-Signature: kiqrqo4qnhiucp36nszd6mxbhxjc1opm X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1745491955-96188 X-HE-Meta: U2FsdGVkX1+7b03yrnbT5jqcc56fpRAVkXhCauaGAs9guS2LId4Y9Ezh2Z7QGjg8In4AAGqhnE8dJsMejIXDwcxHxy3mqVV3OR39Sgyx6NryhRNIRcEDxWdlu6GnEiq5YAvb1ahfzSeANhnjvU+G+3e7Zb60eDcxGym0wt3MULMzUWs+UeQwcII8+K0CqWVaOmpuoZ7E9Ol91P7PXIKB8nVaOO1XHc5VqRsxIJK/UZgBfpHG+pTcuB/6+r18/cGd1TT3m5OLFJQQXqZ+j0GLwZxsO9tIN3kWPwWqSOyEZUZEC99nUDmAEOMa3Sgrt7HHyqAQoDaUm1ucmcRE/MG/M73xQWswq8flSWYFmtHyTVd/xr5dtO5vOh8PD5uaCr0ocPnTLJh6IVfM5M/myyovbZtd00lO5YJXk5Kkc2IzAClwmiLrrWU2CJnmSABAvy1VfwKyV/HfgDtlt5GDzmbpavzP6SJQmRb3B4pifCm3NnmFSVtwV7dA54wlN8McoEMlyisnzWGChfPytBP5Hy0MMzfqbt2IqDAoL2JiNTXT70wfOZzQSFa6xxZIxHHVYouCB4qYwCOrdQQ2VqJuBZ2jOfRw9Y14ijFr6L1v9dITz3r3VvRRKw47txW+73rVziJUYRPMQgG2prPVSMmxKHsBdkWxtKrBLOmSBoLAOs78/C0fZlxN71oVdqsQrRPDmnCiiBeJbDxce8/Qa/n4OzHIXVs0GRa00T3jVSCXA9KESRm8rYy9UV2Tt1HJJCXNmRhq6g70hApbfmSp/OSIgOu/OpHTFTGuKsbItKjjePTReP3RT03kKsD19zN/UKVvcVEUO0P2oY4Un4o8H8wJgOfii1IYtqEg39K1ARVULteV19xbZYYQdT483HTGrDXSEoqL7x5Yu4zhfCg26o/q6p8so7qQ3efT4z/Jvr6Q5Gnzo1eAZPLA7dr8ML6Z5/160V/zrOE1G2wTKj5tZaHjQp8 9Y/+dOvF PmRV7Js50aY9L1dOats2MnZtU9xCcBFuaIZ8ZDLB99972rXSiCM5ZyH4tXaDI2yONR/MLUSVfAU/VeEYv8n3B+7Z602HPtWo1sh9McUbM4DU7nKzCzODyiWYCl88c/zJ4h9PIe1B7sG2pXFrFDdFw4qTDJxOctXhM/poT2Q2LuL8jUbP+joWbl/tMMdqMfXg99MaD+6E7dWSRfj/VpiiZIezzL0/xzntdtnZ7NNvAahPzFgzD8KP9uEgiY8wX3GZk7UWAdLojDr9LlcZuOL2ZPLxyVGLWVEn5MOfId8BRMyA1wuk9Li93LIOa3P23whJ4ah5pr3iOwhSShMDSirMqwjIzjBE3fUbhQPJH2ccLXFhZgwM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The split_huge_pmd_locked function currently performs redundant checks for migration entries and folio validation that are already handled by the page_vma_mapped_walk mechanism in try_to_migrate_one. Specifically, page_vma_mapped_walk already ensures that: - The folio is properly mapped in the given VMA area - pmd_trans_huge, pmd_devmap, and migration entry validation are performed To leverage page_vma_mapped_walk's work, moving TTU_SPLIT_HUGE_PMD handling to the while loop checking, removing these duplicate checks from split_huge_pmd_locked and also removing the unnecessary folio argument since it's no longer needed for the validation. Suggested-by: David Hildenbrand Link: https://lore.kernel.org/all/98d1d195-7821-4627-b518-83103ade56c0@redhat.com/ Signed-off-by: Gavin Guo --- include/linux/huge_mm.h | 15 +++++++-------- mm/huge_memory.c | 36 ++++++++++-------------------------- mm/memory.c | 4 ++-- mm/mprotect.c | 2 +- mm/rmap.c | 25 +++++++++++++++---------- 5 files changed, 35 insertions(+), 47 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e893d546a49f..01a6d998d212 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -395,7 +395,7 @@ static inline int split_huge_page(struct page *page) void deferred_split_folio(struct folio *folio, bool partially_mapped); void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze, struct folio *folio); + unsigned long address, bool freeze); #define split_huge_pmd(__vma, __pmd, __address) \ do { \ @@ -403,12 +403,11 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, if (is_swap_pmd(*____pmd) || pmd_trans_huge(*____pmd) \ || pmd_devmap(*____pmd)) \ __split_huge_pmd(__vma, __pmd, __address, \ - false, NULL); \ + false); \ } while (0) - void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, - bool freeze, struct folio *folio); + bool freeze); void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); @@ -503,7 +502,7 @@ static inline bool thp_migration_supported(void) } void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, - pmd_t *pmd, bool freeze, struct folio *folio); + pmd_t *pmd, bool freeze); bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, struct folio *folio); @@ -578,12 +577,12 @@ static inline void deferred_split_folio(struct folio *folio, bool partially_mapp do { } while (0) static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze, struct folio *folio) {} + unsigned long address, bool freeze) {} static inline void split_huge_pmd_address(struct vm_area_struct *vma, - unsigned long address, bool freeze, struct folio *folio) {} + unsigned long address, bool freeze) {} static inline void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, pmd_t *pmd, - bool freeze, struct folio *folio) {} + bool freeze) {} static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 47d76d03ce30..f7721800f4da 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1786,7 +1786,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); - __split_huge_pmd(src_vma, src_pmd, addr, false, NULL); + __split_huge_pmd(src_vma, src_pmd, addr, false); return -EAGAIN; } add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); @@ -2008,7 +2008,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) folio_unlock(folio); spin_unlock(vmf->ptl); fallback: - __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); + __split_huge_pmd(vma, vmf->pmd, vmf->address, false); return VM_FAULT_FALLBACK; } @@ -3073,33 +3073,17 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, } void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, - pmd_t *pmd, bool freeze, struct folio *folio) + pmd_t *pmd, bool freeze) { - bool pmd_migration = is_pmd_migration_entry(*pmd); - - VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio)); VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); - VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); - VM_BUG_ON(freeze && !folio); - - /* - * When the caller requests to set up a migration entry, we - * require a folio to check the PMD against. Otherwise, there - * is a risk of replacing the wrong folio. - */ - if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || pmd_migration) { - /* - * Do not apply pmd_folio() to a migration entry; and folio lock - * guarantees that it must be of the wrong folio anyway. - */ - if (folio && (pmd_migration || folio != pmd_folio(*pmd))) - return; + if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || + is_pmd_migration_entry(*pmd)) { __split_huge_pmd_locked(vma, pmd, address, freeze); } } void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long address, bool freeze, struct folio *folio) + unsigned long address, bool freeze) { spinlock_t *ptl; struct mmu_notifier_range range; @@ -3109,20 +3093,20 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); ptl = pmd_lock(vma->vm_mm, pmd); - split_huge_pmd_locked(vma, range.start, pmd, freeze, folio); + split_huge_pmd_locked(vma, range.start, pmd, freeze); spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, - bool freeze, struct folio *folio) + bool freeze) { pmd_t *pmd = mm_find_pmd(vma->vm_mm, address); if (!pmd) return; - __split_huge_pmd(vma, pmd, address, freeze, folio); + __split_huge_pmd(vma, pmd, address, freeze); } static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned long address) @@ -3134,7 +3118,7 @@ static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned if (!IS_ALIGNED(address, HPAGE_PMD_SIZE) && range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PMD_SIZE), ALIGN(address, HPAGE_PMD_SIZE))) - split_huge_pmd_address(vma, address, false, NULL); + split_huge_pmd_address(vma, address, false); } void vma_adjust_trans_huge(struct vm_area_struct *vma, diff --git a/mm/memory.c b/mm/memory.c index ba3ea0a82f7f..4f85167baff9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1799,7 +1799,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, next = pmd_addr_end(addr, end); if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { if (next - addr != HPAGE_PMD_SIZE) - __split_huge_pmd(vma, pmd, addr, false, NULL); + __split_huge_pmd(vma, pmd, addr, false); else if (zap_huge_pmd(tlb, vma, pmd, addr)) { addr = next; continue; @@ -5892,7 +5892,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) split: /* COW or write-notify handled on pte level: split pmd. */ - __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); + __split_huge_pmd(vma, vmf->pmd, vmf->address, false); return VM_FAULT_FALLBACK; } diff --git a/mm/mprotect.c b/mm/mprotect.c index 62c1f7945741..88608d0dc2c2 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -379,7 +379,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || pgtable_split_needed(vma, cp_flags)) { - __split_huge_pmd(vma, pmd, addr, false, NULL); + __split_huge_pmd(vma, pmd, addr, false); /* * For file-backed, the pmd could have been * cleared; make sure pmd populated if diff --git a/mm/rmap.c b/mm/rmap.c index 67bb273dfb80..507dc13add61 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1944,7 +1944,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * restart so we can process the PTE-mapped THP. */ split_huge_pmd_locked(vma, pvmw.address, - pvmw.pmd, false, folio); + pvmw.pmd, false); flags &= ~TTU_SPLIT_HUGE_PMD; page_vma_mapped_walk_restart(&pvmw); continue; @@ -2291,13 +2291,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (flags & TTU_SYNC) pvmw.flags = PVMW_SYNC; - /* - * unmap_page() in mm/huge_memory.c is the only user of migration with - * TTU_SPLIT_HUGE_PMD and it wants to freeze. - */ - if (flags & TTU_SPLIT_HUGE_PMD) - split_huge_pmd_address(vma, address, true, folio); - /* * For THP, we have to assume the worse case ie pmd for invalidation. * For hugetlb, it could be much worse if we need to do pud @@ -2323,9 +2316,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); while (page_vma_mapped_walk(&pvmw)) { -#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION /* PMD-mapped THP migration entry */ if (!pvmw.pte) { + /* + * unmap_folio() in mm/huge_memory.c is the only user of + * migration with TTU_SPLIT_HUGE_PMD and it wants to + * freeze. + */ + if (flags & TTU_SPLIT_HUGE_PMD) { + split_huge_pmd_locked(vma, pvmw.address, + pvmw.pmd, true); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION subpage = folio_page(folio, pmd_pfn(*pvmw.pmd) - folio_pfn(folio)); VM_BUG_ON_FOLIO(folio_test_hugetlb(folio) || @@ -2337,8 +2342,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } continue; - } #endif + } /* Unexpected PMD-mapped THP? */ VM_BUG_ON_FOLIO(!pvmw.pte, folio); base-commit: a79be02bba5c31f967885c7f3bf3a756d77d11d9 prerequisite-patch-id: 9c9c975b11ad0f73acd863049b4f1732caa04e53 -- 2.43.0