From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0DF4FA373D for ; Fri, 21 Oct 2022 16:37:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1599F8E001B; Fri, 21 Oct 2022 12:37:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3B948E0001; Fri, 21 Oct 2022 12:37:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEE408E001B; Fri, 21 Oct 2022 12:37:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B51F58E0001 for ; Fri, 21 Oct 2022 12:37:41 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8EEFB1405D4 for ; Fri, 21 Oct 2022 16:37:41 +0000 (UTC) X-FDA: 80045512722.12.3B88811 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 3415C140014 for ; Fri, 21 Oct 2022 16:37:41 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id o4-20020a258d84000000b006bcfc1aafbdso3722239ybl.14 for ; Fri, 21 Oct 2022 09:37:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iBv8Qeh3umlbRvLwHQeNwyurmrBSLw5TULsp1T6zjOQ=; b=suyR7+ETHnPADETLXijl2rxvK4S0l8lKXahB00jRy94KrLql58bgTIu6Xtq+wEAutw 3qSk5gGSMSLsMT2bwUVmLbiHjZ0JM//d+Va50N+iIzaNRXrtWFUP2ujYlH1+zkVlnTBY o4N+HujD+Uvd3YUSbtc6XaGAqm8geaCN4uRH8TMKVxMsk5SRi2j9odHj3AlMttSaThcn piV7sQs60kSX1tbZIg1XS5TZQJHNgM0uTI7X8y1AXFlegWGyn7RhVj6zPo9K9F6hUJ3t H8alvqf47S17+GW/5DSv1v7fEO6GtqbrtOg9qQM+8ny5ZkncQHoN4Y8WiC0iIHWzuQsy 2f6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iBv8Qeh3umlbRvLwHQeNwyurmrBSLw5TULsp1T6zjOQ=; b=nBmr6pVRTxWw4oezp2f639Rw5xzpjkG2mTuDPShi6TAwbiI3risGsMVPlhC2amjX/z yKrY9s/+/M61xX37gN8TR0u3IBLnH7UqAS/Z0bVTJEUNUlu5bEMP74WVo2yOwQIMsRK5 CkhIK1z09E0eK4eGZz/nlnVLfbxW0PjF+pDem5Z6K84yd9T1SMkLVPgOeGXDwSsyHBuF wozGYpbidO5a8zC0jgRd7OEp9wMiaAIfyu9hEi7QqRIhFtag8F6GEA17AJiiH/805ZK6 zfG/NugzGE1pRFdJNyi1Bqp/d3XrIxRjFYxIoBCLVoDOGuxI7ngwzEl+O2DRQdEU4gEJ 8AQQ== X-Gm-Message-State: ACrzQf3EaW6gHIOckBKG2qkHBYK+mWqR/KFDiHkRJ3jnzZ09XCiTtdYm 2XVpM7Vt52UaR3OfzAqXqo34nThdeCVoM4Ny X-Google-Smtp-Source: AMsMyM6dL+1MEn3usFlfe2w4rDmoALEsJetzdzp0dKvN3alW4LkUGv7Gm0+RZmYNLBhMIeQTdlGoqaYDlTAsktx8 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:84cc:0:b0:6c4:95:f1c1 with SMTP id x12-20020a2584cc000000b006c40095f1c1mr18790616ybm.131.1666370260537; Fri, 21 Oct 2022 09:37:40 -0700 (PDT) Date: Fri, 21 Oct 2022 16:36:42 +0000 In-Reply-To: <20221021163703.3218176-1-jthoughton@google.com> Mime-Version: 1.0 References: <20221021163703.3218176-1-jthoughton@google.com> X-Mailer: git-send-email 2.38.0.135.g90850a2211-goog Message-ID: <20221021163703.3218176-27-jthoughton@google.com> Subject: [RFC PATCH v2 26/47] hugetlb: make move_hugetlb_page_tables compatible with HGM From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=suyR7+ET; spf=pass (imf26.hostedemail.com: domain of 31MpSYwoKCNUAK8FL78KFE7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=31MpSYwoKCNUAK8FL78KFE7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666370261; a=rsa-sha256; cv=none; b=Mj2NSRmcYh4xvIaKOMrpb7AwxBLjuA66M/7RZJmVuOLYRrZ1sX6WYuyq+mhEZc2k9kVUF+ GYdcNWi9qnDuZqa1RaGbcFFdKqqEhTacJc6VS8K5dx7siIeU+NVciLhptK2ABL8TRIJMBP P7Q4rD32jnTBujN8SIwLrpYtPHsJmxA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666370261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iBv8Qeh3umlbRvLwHQeNwyurmrBSLw5TULsp1T6zjOQ=; b=OE8VPrqXVlDO+tudMgIZBUFqzjQfC+iNuyZPN62rDzuKIaDiAbvCVljtzSe8SYdKFPPHBx eA5blxZkz851a++IuG3pkLZiXehW5o4z3djnhi1y3LsjCcRBCkcfCGdM2IO5m39Z9uqiej z4hK51OlIsHD17iF8CDcO8UDEtS8BTg= X-Stat-Signature: jt8gs87dccmawzc9y1op1kcyfiizzek9 X-Rspamd-Queue-Id: 3415C140014 X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=suyR7+ET; spf=pass (imf26.hostedemail.com: domain of 31MpSYwoKCNUAK8FL78KFE7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=31MpSYwoKCNUAK8FL78KFE7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-HE-Tag: 1666370261-382419 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is very similar to the support that was added to copy_hugetlb_page_range. We simply do a high-granularity walk now, and most of the rest of the code stays the same. Signed-off-by: James Houghton --- mm/hugetlb.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7d692907cbf3..16b0d192445c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5174,16 +5174,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, return ret; } -static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte) +static void move_hugetlb_pte(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, struct hugetlb_pte *src_hpte, + struct hugetlb_pte *dst_hpte) { - struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; spinlock_t *src_ptl, *dst_ptl; pte_t pte; - dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); + dst_ptl = hugetlb_pte_lock(mm, dst_hpte); + src_ptl = hugetlb_pte_lockptr(mm, src_hpte); /* * We don't have to worry about the ordering of src and dst ptlocks @@ -5192,8 +5192,8 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, if (src_ptl != dst_ptl) spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); - pte = huge_ptep_get_and_clear(mm, old_addr, src_pte); - set_huge_pte_at(mm, new_addr, dst_pte, pte); + pte = huge_ptep_get_and_clear(mm, old_addr, src_hpte->ptep); + set_huge_pte_at(mm, new_addr, dst_hpte->ptep, pte); if (src_ptl != dst_ptl) spin_unlock(src_ptl); @@ -5214,6 +5214,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; bool shared_pmd = false; + struct hugetlb_pte src_hpte, dst_hpte; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, old_addr, old_end); @@ -5229,20 +5230,28 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, /* Prevent race with file truncation */ hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); - for (; old_addr < old_end; old_addr += sz, new_addr += sz) { + while (old_addr < old_end) { src_pte = huge_pte_offset(mm, old_addr, sz); if (!src_pte) { - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz; continue; } - if (huge_pte_none(huge_ptep_get(src_pte))) + + hugetlb_pte_populate(&src_hpte, src_pte, huge_page_shift(h), + hpage_size_to_level(sz)); + hugetlb_hgm_walk(mm, vma, &src_hpte, old_addr, + PAGE_SIZE, /*stop_at_none=*/true); + if (huge_pte_none(huge_ptep_get(src_hpte.ptep))) { + old_addr += hugetlb_pte_size(&src_hpte); + new_addr += hugetlb_pte_size(&src_hpte); continue; + } - if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { + if (huge_pmd_unshare(mm, vma, old_addr, src_hpte.ptep)) { shared_pmd = true; - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr = (old_addr | last_addr_mask) + sz; + new_addr = (new_addr | last_addr_mask) + sz; continue; } @@ -5250,7 +5259,15 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, if (!dst_pte) break; - move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte); + hugetlb_pte_populate(&dst_hpte, dst_pte, huge_page_shift(h), + hpage_size_to_level(sz)); + if (hugetlb_hgm_walk(mm, vma, &dst_hpte, new_addr, + hugetlb_pte_size(&src_hpte), + /*stop_at_none=*/false)) + break; + move_hugetlb_pte(vma, old_addr, new_addr, &src_hpte, &dst_hpte); + old_addr += hugetlb_pte_size(&src_hpte); + new_addr += hugetlb_pte_size(&src_hpte); } if (shared_pmd) -- 2.38.0.135.g90850a2211-goog