From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A225CEFB7FA for ; Tue, 24 Feb 2026 05:12:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DB306B0093; Tue, 24 Feb 2026 00:12:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 099C06B0095; Tue, 24 Feb 2026 00:12:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB2EF6B0096; Tue, 24 Feb 2026 00:12:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DAA096B0093 for ; Tue, 24 Feb 2026 00:12:53 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A991AB900D for ; Tue, 24 Feb 2026 05:12:53 +0000 (UTC) X-FDA: 84478180626.10.AB91DCC Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id E9A5440008 for ; Tue, 24 Feb 2026 05:12:51 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1771909972; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UmwHNPDGtWbF+MXPvm57i1l+rUY6KArEvQFmb7JFGz8=; b=JhtRiJdlSBsbgKXDnrRVE/ygJcT3if4/e+LQfVf7rKoiS7wDHdUfY5KNTfV8MClkS/VNft cIfZydOMaMjDqG12GmwjnktLGNGMmlt5H01m/aYJtf5BPd4IuneTjlHZkg3MvqFNK09jLW y/9BkAJ+ssJ4AqZIE/69SyiFpMwg3Q8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1771909972; a=rsa-sha256; cv=none; b=lyOuMv2UH7BTRgMaqLGk3sSbudU1bKZ+NSzmvZZVYcvxXXX1eAxdAZ4Wao7pyjXZAlPtyK Va9QjzM34VQq+upblPPYJysobc8Me+ifNQlKjnTG9KoKiq8hrdL8EOD5a4V3Vp4X+0IHxe dN/1VGrh6WbSbeXoHNhMDj2hWHMpQSE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B9A7B497; Mon, 23 Feb 2026 21:12:44 -0800 (PST) Received: from a085714.blr.arm.com (a085714.arm.com [10.164.18.87]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 91ED33F7BD; Mon, 23 Feb 2026 21:12:46 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Cc: Anshuman Khandual , Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [RFC V1 05/16] arm64/mm: Convert READ_ONCE() as pmdp_get() while accessing PMD Date: Tue, 24 Feb 2026 10:41:42 +0530 Message-ID: <20260224051153.3150613-6-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260224051153.3150613-1-anshuman.khandual@arm.com> References: <20260224051153.3150613-1-anshuman.khandual@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: E9A5440008 X-Stat-Signature: gscypu4zh56zg7z9gyci4q8ccg7bu8s3 X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1771909971-548618 X-HE-Meta: U2FsdGVkX18y3NP66huIk43d2IlLD0AclUaAGOGNt4Z9AHafpfl/qv6Nksf1AUWCmdBIrYNDQ2aCyCeFg0jcfIggjm8gzQbT6WYc+EsGTPi1tWR/V1GY1mFhgWqKxawmssx6QtymN2I37ggouskix2MrycIJptJYKwKcN7KURwQYYs02Smt2dz8mLIAY07RGnn9FOPiTbG6v9dLYrlzFfjCrRNm6HkqxVLdUJwBPWmO0VMSCxKhL6KXOUDh6+XeCjMAqk4LJiF8388UsT8wLIQvZnoReuEbSZ4UL9PCN3gAurhg7dUBDhzSUx3nzXydLBi39Ase8PG69v8E8EXqNlqqx3L2BCCADkdJfERQtHKL0ZZ+NDbhvqp62aduHAOF2GFdwkuX9ujnmpv4oZEX7BFLKBr+cuQeUo3JNVVQcBh47906XzGLzT+KlaTQMj+qv7dtjGm6UyjU8qsMkAMbs5nKKyG4FzBzjAxBurrI+TrhUumcSroc+HvNOOzyJcR987eFwpcAgbDnVoCvQ+X6dOGLkA/XGH7ZyQSqHg/adJz5Px5y/BjN2ClWr0QdR311nKVDURykOtpsGYnY8elY+2X2x5sWcQsDl1BHysPcG9hoV27ichG3w1RbiAqRXvm5JdnpzMIyNimHbg4LWJspk5C0mF9/RJxHWy8K958qjtYxIP36Q5Rwp4T5pwD+993ySJhrZankp6e1IEcWNewbHvkR+huhJebDYOgCive4zC3cMAMXDs9pmxmgMNpFjdZsdmsZTn2iQQMwIkyCZt75KWiKo1ySwynaEY9k1VZdO+pZZgkCH7/iu2xdFBw38BRCPNAWZxTaLb5GXYSZCKplFD0rJltDCbq+sw67jnwm8MoWi/vcaa5evvKjS7HEMVxrQxh26stcEqahO5sHygyDRFsGAsYnvY23rdBfvTkibT0pQAARTAuN9vTMEmZC4ncW7UQd6BtJj+ln0Ut0mHZn u1VCkbvC P/SF2rYEUCAN3YsxBFBCgmOFChOA+b/cywlHDtiP0bDqeUKt1HQxLtPsUWZwDcZMNbmprnn8HG7HfvTs12YfWVDY6zdTqZWjmoLXkwq5opbJy89TWFBrSzianSM/nOI+kAWXvstvY38aWjXvGIi9yXycSTqcsOnJ5T5LEeE0BHqbDjGQdquG2i0Qy8Q57Fooondf7yyWlO0nxwvlHqlYxTCXevBB6xpODWhhSburAVC8ynQP5Xd/uGipaz/Q88DVOAaMnVQlKcVOHtYeeGG6LcLKF1DKMQM9udHSzHJSbHH0a5UOuOWHKPQjEXwP8WYillt8CPQTKm7PtmlJb8jd9mp3QFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert all READ_ONCE() based PMD accesses as pmdp_get() instead which will support both D64 and D128 translation regime going forward. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgtable.h | 12 +++-------- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 35 ++++++++++++++++++++++---------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 34 insertions(+), 27 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b3e58735c49b..4b5bc2c09bf2 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -852,7 +852,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) } /* Find an entry in the third-level page table. */ -#define pte_offset_phys(dir,addr) (pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t)) +#define pte_offset_phys(dir, addr) (pmd_page_paddr(pmdp_get(dir)) + \ + pte_index(addr) * sizeof(pte_t)) #define pte_set_fixmap(addr) ((pte_t *)set_fixmap_offset(FIX_PTE, addr)) #define pte_set_fixmap_offset(pmd, addr) pte_set_fixmap(pte_offset_phys(pmd, addr)) @@ -1328,14 +1329,7 @@ static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pmd_t *pmdp) -{ - /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */ - VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft()); - return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); -} +int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm, diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index be9dab2c7d6a..1389ba26ec74 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -177,7 +177,7 @@ static void show_pte(unsigned long addr) break; pmdp = pmd_offset(pudp, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); pr_cont(", pmd=%016llx", pmd_val(pmd)); if (pmd_none(pmd) || pmd_bad(pmd)) break; diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c5c5425791da..7a4bbcb39094 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr) static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr) { - pmd_t pmd = READ_ONCE(*pmdp); + pmd_t pmd = pmdp_get(pmdp); pte_t *ptep; if (pmd_none(pmd)) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index a42c05cf5640..6117aca2bac7 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, addr &= CONT_PMD_MASK; pmdp = pmd_offset(pudp, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) && pmd_none(pmd)) return NULL; diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index abeb81bf6ebd..709e8ad15603 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node) static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, bool early) { - if (pmd_none(READ_ONCE(*pmdp))) { + if (pmd_none(pmdp_get(pmdp))) { phys_addr_t pte_phys = early ? __pa_symbol(kasan_early_shadow_pte) : kasan_alloc_zeroed_page(node); @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, do { next = pmd_addr_end(addr, end); kasan_pte_populate(pmdp, addr, next, node, early); - } while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp))); + } while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp))); } static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index a6a00accf4f9..dea1b595f237 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -201,7 +201,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, int flags) { unsigned long next; - pmd_t pmd = READ_ONCE(*pmdp); + pmd_t pmd = pmdp_get(pmdp); pte_t *ptep; BUG_ON(pmd_sect(pmd)); @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, unsigned long next; do { - pmd_t old_pmd = READ_ONCE(*pmdp); + pmd_t old_pmd = pmdp_get(pmdp); next = pmd_addr_end(addr, end); @@ -271,7 +271,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), - READ_ONCE(pmd_val(*pmdp)))); + pmd_val(pmdp_get(pmdp)))); } else { int ret; @@ -281,7 +281,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, return ret; BUG_ON(pmd_val(old_pmd) != 0 && - pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); + pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp))); } phys += next - addr; } while (pmdp++, addr = next, addr != end); @@ -1475,7 +1475,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr, do { next = pmd_addr_end(addr, end); pmdp = pmd_offset(pudp, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); if (pmd_none(pmd)) continue; @@ -1623,7 +1623,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, do { next = pmd_addr_end(addr, end); pmdp = pmd_offset(pudp, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); if (pmd_none(pmd)) continue; @@ -1644,7 +1644,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, */ pmdp = pmd_offset(pudp, 0UL); for (i = 0; i < PTRS_PER_PMD; i++) { - if (!pmd_none(READ_ONCE(pmdp[i]))) + if (!pmd_none(pmdp_get(pmdp + i))) return; } @@ -1763,7 +1763,7 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, { vmemmap_verify((pte_t *)pmdp, node, addr, next); - return pmd_sect(READ_ONCE(*pmdp)); + return pmd_sect(pmdp_get(pmdp)); } int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, @@ -1810,7 +1810,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot) pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot)); /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)), + if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)), pmd_val(new_pmd))) return 0; @@ -1835,7 +1835,7 @@ int pud_clear_huge(pud_t *pudp) int pmd_clear_huge(pmd_t *pmdp) { - if (!pmd_sect(READ_ONCE(*pmdp))) + if (!pmd_sect(pmdp_get(pmdp))) return 0; pmd_clear(pmdp); return 1; @@ -1847,7 +1847,7 @@ static int __pmd_free_pte_page(pmd_t *pmdp, unsigned long addr, pte_t *table; pmd_t pmd; - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); if (!pmd_table(pmd)) { VM_WARN_ON(1); @@ -2245,4 +2245,17 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long i return 0; } + +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) +int pmdp_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pmd_t *pmdp) +{ + pmd_t pmdval = pmdp_get(pmdp); + + /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */ + VM_WARN_ON(pmd_table(pmdval) && !system_supports_haft()); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ + #endif diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 358d1dc9a576..ed1eec4c757d 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -408,7 +408,7 @@ bool kernel_page_present(struct page *page) return true; pmdp = pmd_offset(pudp, addr); - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get(pmdp); if (pmd_none(pmd)) return false; if (pmd_sect(pmd)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 18543b603c77..ddde0f2983b0 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -100,7 +100,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp, src_pmdp = pmd_offset(src_pudp, start); do { - pmd_t pmd = READ_ONCE(*src_pmdp); + pmd_t pmd = pmdp_get(src_pmdp); next = pmd_addr_end(addr, end); if (pmd_none(pmd)) -- 2.43.0