From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A94F6CD1284 for ; Tue, 2 Apr 2024 14:48:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB8336B0089; Tue, 2 Apr 2024 10:48:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C67746B008A; Tue, 2 Apr 2024 10:48:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE1536B0092; Tue, 2 Apr 2024 10:48:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8AE7A6B0089 for ; Tue, 2 Apr 2024 10:48:26 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3D9A3A0BB8 for ; Tue, 2 Apr 2024 14:48:26 +0000 (UTC) X-FDA: 81964872612.20.A19C1FE Received: from relay.hostedemail.com (unirelay06 [10.200.18.69]) by imf22.hostedemail.com (Postfix) with ESMTP id 5BBB9C0005 for ; Tue, 2 Apr 2024 14:48:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; arc=pass ("hostedemail.com:s=arc-20220608:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712069304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BNj19qfqQuYW+Z2n5jCtMAjv/baqAYqa+B/Vqj50/h8=; b=mfqshWP/rQ6rAXFSk13wTISSop47SXpYOTmI5HbG5/OadGjxCBzLnMC/A7KRuWXeH7OGxD OGCUL4OWx+UuRxXWl/1Q4bMAh1381uWZhNvFJvW1Q16REAqCeuEshC+/wREKaxgFPOkrpj JTlHG85t9Tlj0NF0R24u2qH6rSHEyFw= ARC-Authentication-Results: i=2; imf22.hostedemail.com; arc=pass ("hostedemail.com:s=arc-20220608:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1712069304; a=rsa-sha256; cv=pass; b=1xCbHtck4Ih5Dkcnau9kJ5tx9T2PvTM2Dytvt7ffoaIGyLVWxVcsoLfVn7ly5NQEQFtLAI +HlokXaUDnEy1kUu43ON7T2mtGPSk6Wuv96riotHaOEFywe9kKLQZtM7LD8N4La5eS8fuW CDRQiz0cwMhosJqF4/Cdnfr7X/vs5Ho= Received: from relay.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CF920A0DA5 for ; Tue, 2 Apr 2024 14:48:23 +0000 (UTC) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B9123A0CC0 for ; Tue, 2 Apr 2024 14:48:23 +0000 (UTC) X-FDA: 81964872486.26.FEACB8D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf18.hostedemail.com (Postfix) with ESMTP id A63931C0008 for ; Tue, 2 Apr 2024 14:48:21 +0000 (UTC) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712069302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BNj19qfqQuYW+Z2n5jCtMAjv/baqAYqa+B/Vqj50/h8=; b=OWIcZPUJTgYJAAvii+rRX2kYM2b+tHFJeKLpiTOVGdRWfyoX7nAU8XcxjEZnvxjq7rw55G wlK4Ro5uiCrOSE11svAE8KcjN8hj6Jr3qcvt5krZaATRS1eRdkCfz6b7My6oLqSBNL0OeL nMOYeL4CQoO7DuHRJ8vWhiaLDcqsmqI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf18.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712069302; a=rsa-sha256; cv=none; b=qZi2GbJitB+tuvSJBDMhpfw8tovxcs7SOIusHuFD/IVt4lMraBSopl2uL94C0PJIhMjDPW gPP9hyLAs8G2QS3NFyKzjnWzzLwRXBEXi6ZLVyJEXd3e5GFv+O0AeSREUQa00bHyZikcdL Dr2qZimYjDx5DYfLwG0XxDh8bll3B68= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 00D961007; Tue, 2 Apr 2024 07:48:52 -0700 (PDT) Received: from [10.1.38.163] (XHFQ2J9959.cambridge.arm.com [10.1.38.163]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7C1E13F64C; Tue, 2 Apr 2024 07:48:17 -0700 (PDT) Message-ID: Date: Tue, 2 Apr 2024 15:48:16 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 13/13] mm/gup: Handle hugetlb in the generic follow_page_mask code Content-Language: en-GB To: peterx@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen References: <20240327152332.950956-1-peterx@redhat.com> <20240327152332.950956-14-peterx@redhat.com> From: Ryan Roberts In-Reply-To: <20240327152332.950956-14-peterx@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-HE-Meta: U2FsdGVkX1+0p0+scmLqDyXLADTY4lCEGWLr4PBlADxJoxVkhQC5YKkal/b5SGZLep7uUv9D83J2KZ30vG8XIjuQVoZRj8NbtnuY7aF36vW8EdDMIbsidBkDxYIlhrmxrcydutjKGmZ+OWQS7+3vT8sfYq6Fipb6TSZean5/KtGzqzukhheg5ctDBEZzGz4hZfmq540hJUMH+Vn4l+BeYA3rfLuG5XUTpzNwZRbHQhaqAQSjxhcSDHXRfeu9oeSq6nuF9QrsncZ5/Fj2Qm4DNm7M+L4Cy/pcdyDXg9Hum9/b+8xNY4KXJFplKNfZU9z6e1JYEWfpo4JEcZ/e/L+lY42mopvH33BXNWYwoVg+gGynySdsnhufbpO3i/RtK6VDD6JHAHqB8sTpBvLltuAJFd6vJt7HBrDqcK6/rrb3QMxuSMOMIP9XfRALPVB7CGnRuiVPQTJXkz7kCJ+zw2+KIsTyaYuYYOJ7koEyN4OZgng1x1KWvnC8f5X+V1jZ8ojn2IhvPSnHMD2Zvj6XSw6LvTFnN2ItYOCprzPq4OEaDoYBlqY30dWXioiiDZHUAM0sS9UPbXBcAg3ecF+4jYlIGDCS1WMviGLYJ2DLKOb0Q6g9+M2dHpDw3dCm5IWylw0OMqyvBa9+Iknq1QeK736AA8lw9fFUE0SjhTTWmkviMcU81TBgCbSY77PqYSW0Sk6A8uakWrZsbJy9zsAp8XyPvQO8XkiFiKixLdCXssBzLXoB/9+vlXtcw5GDfV3hicUlt/QOWqeeut9it/OBejMkNpPij8QcPy+AvrdAexxN880y4e6CBcglWNU4jGrs5gh8nEEYZHJzf5wVwSBp1kt0jwmghZPyzb6Hzyt/PT5DvNrn6ZOJiqEXdk6v0sWvN4GiRqemgodo29SFW2+tuWbbz0uh+AptSA5nEsC/rtjdOJhGs7pCvZpcB9S/tGuFJiK4ISHRhjYtUReqxhC89sk 9LDtkHOZ uStTLj6ZWVMEwzPZOXlG7uBeHtCCLVRlLushn1d845FqNjd3CWvJFgKXVfHDUDKZ0PbmNZ3H8dEOgJ0eKskVE1Bevz1IFoi64GtecQwfTjw9gF8z5uFmiWeEFABjItF3wfW9akh4z0mN+60vAC1qaJpbP157fQjyWPte5c6kKoTdzVAfFPAF7s3CYFt95XbEs1GyNWIyyJvjsRSM2jy9ujqhmzErZ6GzRKyTMeckNBvwJXsKmCszFpo73eA== X-Rspamd-Queue-Id: 5BBB9C0005 X-Rspam-User: X-Stat-Signature: d9ftnnyknf6h7ww1uaj53g3zrmhtbork X-Rspamd-Server: rspam07 X-HE-Tag-Orig: 1712069301-792893 X-HE-Tag: 1712069304-267179 X-HE-Meta: U2FsdGVkX187fUqJDO0DCdDmDK2jN1ORi1yYynR1nFOTyH+94S3XKG2CfgONmkzudwpWTK5g1JlpvKrYOHdwkiE80fUG/ziLG705OLRT+NFkMhX51SUDRoL8ZfwaO188eNIxeZ9bCY6Gm9uL9/pg/xSpmTKRnU7MTexakuJ3zzm9oPpKPMtBl53A6mlJhugIx7ewdvSXcjtayz3XZMcy1DeJNnOhIa5iCdvIFFpTcRWehhCyUowelUlMjZZ6h7qaG4vGWJgHmehdvtGEXZNJhDEGYoRfE+uEz1Dj59joI/KpuHvwQwtsvGGXJR6iAHbeRWPtTwBNtyq8FKhcjH4SE97T+AD7q+V5phdn4RiKVxwuL2GSO8K9pmk5H7b2GbVuHJSiGMoaKprtv2aEV4cIHHfAfTLkLUGWjJqOPmW2wWp+F/NJ48aoFEaXd4Zs/ByHVpj4H/z8CQd0oV+1f5MIT3+yEosdjLduN44AimjY0hXEAeO4no4Idrwh534XVrl8ygwbgLh4hSn1HFFSopVBeDKJVgWSNdItjdq7bU032VIJy/q9XlKhNrK3Ywyn5KK0sBOMB18j4N+uSTfSuARMsDKB/SL0Uj8f9GOtloI0X6/tgoPBGfhaOiqpWZ0uwJ7PO03mcp6a7meUL2N8XD/0WSmoKFBaiW2c1BJM7dxTjG5BVBxgZVAoG1SY2K01BbQzEOH/YPajYi1FCqZQPippKwxT3BLrP1uYsQzYfdXI091ZTBcXgdxxhJN1L3j/BN3PLS32OOBOwt1lYn9OhStniPQqRO4AKWMiVlDX3xTWPOEUzqOjRJdnpQK93Z4bMTikfSmuOoImi/Hj8D2M1q4m1eke4mmzKmpFOHE+LbwEPkYB83Fx65VdRUD0yGhGKe6CdDZO/RimqFKssKIkQsyWtXzsN4nAc40+OP+XE0232NU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Peter, On 27/03/2024 15:23, peterx@redhat.com wrote: > From: Peter Xu > > Now follow_page() is ready to handle hugetlb pages in whatever form, and > over all architectures. Switch to the generic code path. > > Time to retire hugetlb_follow_page_mask(), following the previous > retirement of follow_hugetlb_page() in 4849807114b8. > > There may be a slight difference of how the loops run when processing slow > GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each > loop of __get_user_pages() will resolve one pgtable entry with the patch > applied, rather than relying on the size of hugetlb hstate, the latter may > cover multiple entries in one loop. > > A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over > a tight loop of slow gup after the path switched. That shouldn't be a > problem because slow-gup should not be a hot path for GUP in general: when > page is commonly present, fast-gup will already succeed, while when the > page is indeed missing and require a follow up page fault, the slow gup > degrade will probably buried in the fault paths anyway. It also explains > why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: > accelerate thp gup even for "pages != NULL"") lands, the latter not part of > a performance analysis but a side benefit. If the performance will be a > concern, we can consider handle CONT_PTE in follow_page(). > > Before that is justified to be necessary, keep everything clean and simple. > > Reviewed-by: Jason Gunthorpe > Signed-off-by: Peter Xu Afraid I'm seeing an oops when running gup_longterm test on arm64 with current mm-unstable. Git bisect blames this patch. The oops reproduces for me every time on 2 different machines: [ 9.340416] kernel BUG at mm/gup.c:778! [ 9.340746] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP [ 9.341199] Modules linked in: [ 9.341481] CPU: 1 PID: 1159 Comm: gup_longterm Not tainted 6.9.0-rc2-00210-g910ff1a347e4 #11 [ 9.342232] Hardware name: linux,dummy-virt (DT) [ 9.342647] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 9.343195] pc : follow_page_mask+0x4d4/0x880 [ 9.343580] lr : follow_page_mask+0x4d4/0x880 [ 9.344018] sp : ffff8000898b3aa0 [ 9.344345] x29: ffff8000898b3aa0 x28: fffffdffc53973e8 x27: 00003c0005d08000 [ 9.345028] x26: ffff00014e5cfd08 x25: ffffd3513a40c000 x24: fffffdffc5d08000 [ 9.345682] x23: ffffc1ffc0000000 x22: 0000000000080101 x21: ffff8000898b3ba8 [ 9.346337] x20: 0000fffff4200000 x19: ffff00014e52d508 x18: 0000000000000010 [ 9.347005] x17: 5f656e6f7a5f7369 x16: 2120262620296567 x15: 6170286461654865 [ 9.347713] x14: 6761502128454741 x13: 2929656761702865 x12: 6761705f65636976 [ 9.348371] x11: 65645f656e6f7a5f x10: ffffd3513b31d6e0 x9 : ffffd3513852f090 [ 9.349062] x8 : 00000000ffffefff x7 : ffffd3513b31d6e0 x6 : 0000000000000000 [ 9.349753] x5 : ffff00017ff98cc8 x4 : 0000000000000fff x3 : 0000000000000000 [ 9.350397] x2 : 0000000000000000 x1 : ffff000190e8b480 x0 : 0000000000000052 [ 9.351097] Call trace: [ 9.351312] follow_page_mask+0x4d4/0x880 [ 9.351700] __get_user_pages+0xf4/0x3e8 [ 9.352089] __gup_longterm_locked+0x204/0xa70 [ 9.352516] pin_user_pages+0x88/0xc0 [ 9.352873] gup_test_ioctl+0x860/0xc40 [ 9.353249] __arm64_sys_ioctl+0xb0/0x100 [ 9.353648] invoke_syscall+0x50/0x128 [ 9.354022] el0_svc_common.constprop.0+0x48/0xf8 [ 9.354488] do_el0_svc+0x28/0x40 [ 9.354822] el0_svc+0x34/0xe0 [ 9.355128] el0t_64_sync_handler+0x13c/0x158 [ 9.355489] el0t_64_sync+0x190/0x198 [ 9.355793] Code: aa1803e0 d000d8e1 91220021 97fff560 (d4210000) [ 9.356280] ---[ end trace 0000000000000000 ]--- [ 9.356651] note: gup_longterm[1159] exited with irqs disabled [ 9.357141] note: gup_longterm[1159] exited with preempt_count 2 [ 9.358033] ------------[ cut here ]------------ [ 9.358800] WARNING: CPU: 1 PID: 0 at kernel/context_tracking.c:128 ct_kernel_exit.constprop.0+0x108/0x120 [ 9.360157] Modules linked in: [ 9.360541] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 6.9.0-rc2-00210-g910ff1a347e4 #11 [ 9.361626] Hardware name: linux,dummy-virt (DT) [ 9.362087] pstate: 204003c5 (nzCv DAIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 9.362758] pc : ct_kernel_exit.constprop.0+0x108/0x120 [ 9.363306] lr : ct_idle_enter+0x10/0x20 [ 9.363845] sp : ffff8000801abdc0 [ 9.364222] x29: ffff8000801abdc0 x28: 0000000000000000 x27: 0000000000000000 [ 9.364961] x26: 0000000000000000 x25: ffff00014149d780 x24: 0000000000000000 [ 9.365557] x23: 0000000000000000 x22: ffffd3513b299d48 x21: ffffd3513a785730 [ 9.366239] x20: ffffd3513b299c28 x19: ffff00017ffa7da0 x18: 0000fffff5ffffff [ 9.366869] x17: 0000000000000000 x16: 1fffe0002a21a8c1 x15: 0000000000000000 [ 9.367524] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000002 [ 9.368207] x11: 0000000000000001 x10: 0000000000000ad0 x9 : ffffd35138589230 [ 9.369123] x8 : ffff00014149e2b0 x7 : 0000000000000000 x6 : 000000000f8c0fb2 [ 9.370403] x5 : 4000000000000002 x4 : ffff2cb045825000 x3 : ffff8000801abdc0 [ 9.371170] x2 : ffffd3513a782da0 x1 : 4000000000000000 x0 : ffffd3513a782da0 [ 9.372279] Call trace: [ 9.372519] ct_kernel_exit.constprop.0+0x108/0x120 [ 9.373216] ct_idle_enter+0x10/0x20 [ 9.373562] default_idle_call+0x3c/0x160 [ 9.374055] do_idle+0x21c/0x280 [ 9.374394] cpu_startup_entry+0x3c/0x50 [ 9.374797] secondary_start_kernel+0x140/0x168 [ 9.375220] __secondary_switched+0xb8/0xc0 [ 9.375875] ---[ end trace 0000000000000000 ]--- The oops trigger is at mm/gup.c:778: VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); This is the output of gup_longterm (last output is just before oops): # [INFO] detected hugetlb page size: 2048 KiB # [INFO] detected hugetlb page size: 32768 KiB # [INFO] detected hugetlb page size: 64 KiB # [INFO] detected hugetlb page size: 1048576 KiB TAP version 13 1..70 # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd ok 1 Should have worked # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with tmpfile ok 2 Should have failed # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with local tmpfile ok 3 Should have failed # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (2048 kB) ok 4 Should have worked # [RUN] R/W longterm GUP pin in MAP_SHARED file mapping ... with memfd hugetlb (32768 kB) So 2M passed ok, and its failing for 32M, which is cont-pmd. I'm guessing you're trying to iterate 2M into a cont-pmd folio and ending up with an unexpected tail page? I'm running with defconfig plus these: ./scripts/config --enable CONFIG_SQUASHFS_LZ4 ./scripts/config --enable CONFIG_SQUASHFS_LZO ./scripts/config --enable CONFIG_SQUASHFS_XZ ./scripts/config --enable CONFIG_SQUASHFS_ZSTD ./scripts/config --enable CONFIG_XFS_FS ./scripts/config --enable CONFIG_FTRACE ./scripts/config --enable CONFIG_FUNCTION_TRACER ./scripts/config --enable CONFIG_KPROBES ./scripts/config --enable CONFIG_HIST_TRIGGERS ./scripts/config --enable CONFIG_FTRACE_SYSCALLS ./scripts/config --enable CONFIG_DEBUG_VM ./scripts/config --enable CONFIG_DEBUG_VM_MAPLE_TREE ./scripts/config --enable CONFIG_DEBUG_VM_RB ./scripts/config --enable CONFIG_DEBUG_VM_PGFLAGS ./scripts/config --enable CONFIG_DEBUG_VM_PGTABLE ./scripts/config --enable CONFIG_PAGE_TABLE_CHECK ./scripts/config --enable CONFIG_USERFAULTFD ./scripts/config --enable CONFIG_TEST_VMALLOC ./scripts/config --enable CONFIG_GUP_TEST Thanks, Ryan > --- > include/linux/hugetlb.h | 7 ---- > mm/gup.c | 15 +++------ > mm/hugetlb.c | 71 ----------------------------------------- > 3 files changed, 5 insertions(+), 88 deletions(-) > > diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h > index 294c78b3549f..a546140f89cd 100644 > --- a/include/linux/hugetlb.h > +++ b/include/linux/hugetlb.h > @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( > { > } > > -static inline struct page *hugetlb_follow_page_mask( > - struct vm_area_struct *vma, unsigned long address, unsigned int flags, > - unsigned int *page_mask) > -{ > - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ > -} > - > static inline int copy_hugetlb_page_range(struct mm_struct *dst, > struct mm_struct *src, > struct vm_area_struct *dst_vma, > diff --git a/mm/gup.c b/mm/gup.c > index a02463c9420e..c803d0b0f358 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -1135,18 +1135,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, > { > pgd_t *pgd; > struct mm_struct *mm = vma->vm_mm; > + struct page *page; > > - ctx->page_mask = 0; > - > - /* > - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use > - * special hugetlb page table walking code. This eliminates the > - * need to check for hugetlb entries in the general walking code. > - */ > - if (is_vm_hugetlb_page(vma)) > - return hugetlb_follow_page_mask(vma, address, flags, > - &ctx->page_mask); > + vma_pgtable_walk_begin(vma); > > + ctx->page_mask = 0; > pgd = pgd_offset(mm, address); > > if (unlikely(is_hugepd(__hugepd(pgd_val(*pgd))))) > @@ -1157,6 +1150,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, > else > page = follow_p4d_mask(vma, address, pgd, flags, ctx); > > + vma_pgtable_walk_end(vma); > + > return page; > } > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index 65b9c9a48fd2..cc79891a3597 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -6870,77 +6870,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, > } > #endif /* CONFIG_USERFAULTFD */ > > -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, > - unsigned long address, unsigned int flags, > - unsigned int *page_mask) > -{ > - struct hstate *h = hstate_vma(vma); > - struct mm_struct *mm = vma->vm_mm; > - unsigned long haddr = address & huge_page_mask(h); > - struct page *page = NULL; > - spinlock_t *ptl; > - pte_t *pte, entry; > - int ret; > - > - hugetlb_vma_lock_read(vma); > - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); > - if (!pte) > - goto out_unlock; > - > - ptl = huge_pte_lock(h, mm, pte); > - entry = huge_ptep_get(pte); > - if (pte_present(entry)) { > - page = pte_page(entry); > - > - if (!huge_pte_write(entry)) { > - if (flags & FOLL_WRITE) { > - page = NULL; > - goto out; > - } > - > - if (gup_must_unshare(vma, flags, page)) { > - /* Tell the caller to do unsharing */ > - page = ERR_PTR(-EMLINK); > - goto out; > - } > - } > - > - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); > - > - /* > - * Note that page may be a sub-page, and with vmemmap > - * optimizations the page struct may be read only. > - * try_grab_page() will increase the ref count on the > - * head page, so this will be OK. > - * > - * try_grab_page() should always be able to get the page here, > - * because we hold the ptl lock and have verified pte_present(). > - */ > - ret = try_grab_page(page, flags); > - > - if (WARN_ON_ONCE(ret)) { > - page = ERR_PTR(ret); > - goto out; > - } > - > - *page_mask = (1U << huge_page_order(h)) - 1; > - } > -out: > - spin_unlock(ptl); > -out_unlock: > - hugetlb_vma_unlock_read(vma); > - > - /* > - * Fixup retval for dump requests: if pagecache doesn't exist, > - * don't try to allocate a new page but just skip it. > - */ > - if (!page && (flags & FOLL_DUMP) && > - !hugetlbfs_pagecache_present(h, vma, address)) > - page = ERR_PTR(-EFAULT); > - > - return page; > -} > - > long hugetlb_change_protection(struct vm_area_struct *vma, > unsigned long address, unsigned long end, > pgprot_t newprot, unsigned long cp_flags)