From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3989AC54E76 for ; Thu, 5 Jan 2023 10:19:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C29AC900003; Thu, 5 Jan 2023 05:19:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD95E900002; Thu, 5 Jan 2023 05:19:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94120900003; Thu, 5 Jan 2023 05:19:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8503A900002 for ; Thu, 5 Jan 2023 05:19:48 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 523C6AAE17 for ; Thu, 5 Jan 2023 10:19:48 +0000 (UTC) X-FDA: 80320349256.21.02C680A Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id AB23380017 for ; Thu, 5 Jan 2023 10:19:46 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=atlQfqiE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3QaS2YwoKCIgv5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QaS2YwoKCIgv5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672913986; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nQJKiUDoV/4j52zLo+Pfg4lOVIUQ9TTfXf4yaRNkk0o=; b=2hnpsX2cBTrD6NvteDK90/ULdmmQKSDnFX+tABFCOzYDxvWD3LlWRoK9MTNDYj4JP5e3gV 3WstWe43OUPtjbMb+aU9p4TFsDedU+BjdtHyl0nj1jdGeRdw9vAIG04Ff6Bv7P5OW7ugto QEgjQUoFkI+CR0igISFUeKoZyGHIaMs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=atlQfqiE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3QaS2YwoKCIgv5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3QaS2YwoKCIgv5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672913986; a=rsa-sha256; cv=none; b=z6FgdIFLJlplAIApseXv0KkIBW/It4QUOSUhBoja0nZYTAEnRkpoUKr8TyZIGgeDKPUH3N wqRbIOpXtxlLBDfB9Vjpsro6NQ6T67RA1AmCyilHodSl2XHVW2uxkDhq90u9dm1f4I0euK wBtmgz+H71XPNRboUBwCZ+hu1RjgGBw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-47ede4426e1so257857727b3.7 for ; Thu, 05 Jan 2023 02:19:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nQJKiUDoV/4j52zLo+Pfg4lOVIUQ9TTfXf4yaRNkk0o=; b=atlQfqiEgy1xYzhT4IyUVUGbPAhpghAx33YY9Ut2pzp0mD93Mh/Yq8hJbUQYMZUGyv 79g4SUp7Afeizjld4KBniyNCxadcDLxlzNbxhQuOC/KyWBzgN0ZCC7l0KieF/fpc4kkJ iUr6M0pdgU5RSZag7PDJbcOaLT6TYs/HCTL9W9Jb1R+becqgtzJQGEONP4TOXQWnH4F0 DYOLfXy51tLR/SmFY1q18P7ZJv+GrFzaVXxYkXHmZUnrB2+y7sCsAz/aoRq7ER2ueTN4 kZf3HAYzz5w5uKAKpFJxkGcQKV4Ix4sjQ5F5h/IZxCSAjyzQRxzHbXvVGGDZzdryGq3g sSOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nQJKiUDoV/4j52zLo+Pfg4lOVIUQ9TTfXf4yaRNkk0o=; b=WdvqspTiP1sPwSA6PSL0r0K08oLUQjkEfzJYlHcwAFl2csyrz5KeGb2OM+CKLgicNH n1DrShjb/xEefe/PXskqMGrZ7rO2RRq5gXeKT3VNljmC7LfBLAIsX3ww6j9oL9Dp0e5e WR/NDG7gvk7OCAKrTcMiOG8iZqt5CqtRbOcPxVe6x0ar+3jEw2+Y6/EIrIBBWTb1aCUv zsrcob5kCuVK4bOU0GqgB/O3Nvce17xRqlFF9aFM6cyUJ+zAAhKfCF6DYUciMlHJqam/ 0B5jnEAzAzLWPdDgDAMdDZ/4hVgUnp9MX1MPhogaC11KLXbcmG1l7oD2xc0pPjhe4VZ+ judA== X-Gm-Message-State: AFqh2kodYctjuIWACgni0zUTvkKvs5vaBR2JQLmlP9rMdcxEcjwyynH0 cMv3mU5xKQ/xuK4J0pNU1ahHi2arGCDuBqx5 X-Google-Smtp-Source: AMrXdXskgc9vPcHYbKiHVU6KI+i99SMBDDXKXKUoGyd1uzZY3LEXY9tDZtAeOm/gOkCBpHD2OdRdbv4oR1vsSaUo X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a81:cc8:0:b0:38d:c23a:c541 with SMTP id 191-20020a810cc8000000b0038dc23ac541mr53560ywm.109.1672913985945; Thu, 05 Jan 2023 02:19:45 -0800 (PST) Date: Thu, 5 Jan 2023 10:18:34 +0000 In-Reply-To: <20230105101844.1893104-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230105101844.1893104-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230105101844.1893104-37-jthoughton@google.com> Subject: [PATCH 36/46] hugetlb: remove huge_pte_lock and huge_pte_lockptr From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: AB23380017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: p46kso6t51ennifbdtyrfztudw145my8 X-HE-Tag: 1672913986-600316 X-HE-Meta: U2FsdGVkX1/jCOU+/4joOGJEHGG8d8aTVt2uRlJvJWKnui3EbjkK6/8KI/rn1GFFq7Jfaaga5vm79LULjasQQbz3zOzo6N98qfKHQo5RqtMRVkV3WnMpz3BVnn8J/ZsjTkruW6ExFqfL0xPbuHy6EHnMFG3lV+O3jMqN9aFPedLgs7huQ1RI8r/sKDl0j1NqyEgsU6q8m1SubbBhcwokr6D/7u3JvVY8qLipJi9YsW00U7PLVGnm65kGZLjmN4KIU/Zku0qOt0Xw+dGZIN88+2V9XOXVzjFXpIVGA3RUY/4krta+VgfvqLP4hSeNaa484cxDPiqTpwQ0sLJROJE9RCox8T9HnJ69qdj2WnAVSerwSmpu6YuVVcRWSM3x+L/nJL4QR4EIxYmmGJt/mxZFCfrVuOletyW4/eD4H0NJrRwly1XVv+tCx3devkLLdFy3jZSC5y3hnOKHqQCqIZqM//RfMRxtRcW4i4YuHHk7VGUrnZqmxfrrtJ8XBDNLuWLqAIgIvXj5C6ItqlEKA317jJ0qvWe/SCXNBUWIEDvKNof1VzNdJPLE0YuEoZA0laL2FPOUxFla/W24san94g0fVAPCWHm1RaFOTcoUsa7OytUumcah3oxVUJkD0gr6sp6BZD/Jbhvofn4XV2TiUZNHzZK1Vv00Xb4YO2CMZzSrXTynngjOLaDMa1ipSXxoAa6ItDt1ln5ApWydDlIqY2i3Mn10MYVMOPp8YJuwLfvO1rC0MYHr/s2Um3EtqPk7qnkoH7VVZ3nZXJohTzGPVuKbmFrBBtgrJDLYk2L/Lhp9LBFwngncbkBIrs7YDtUahUKoVUTzvYwM6msZZHJ5MoPt9WMM67W/5V/iBIesGILo5ITmHU+1hmfSB96iP5HM75Gbx/S7qSUa1dy3wBl0/3yEdwnUuEHHzLxMeMdC61kipBIyVyhaHuSTsd2yC6AWGZb2Z69d3ACW5omccR+D0GY dHT1O+fX c4u52fOTPxss3So3MlaDCKE/oD9z0pu3FYsVEY+FuZAt789wr2+SH5vmswPNWoMoNAkpzgT1gvcmcIlmQRfCJVQ4yHqUgr4tcl2DCL088Cg2W1ZX4BhjevQRKWgDBPPC1EoNBS+7pVR66k/PNsPEktHBLqVMNnsTnmwgrXUqJviVPXp755C4epJQLiQHqbovQhuIhQ2wQx+lJYpdi66nPxTgrqrmYTQxWFo8lmzxBHTmTFG3Y8qBzAmq4nNdCFHDGEV0Y6TrfA2MuXOqWir92/IHMYA7LX5WZOgbtg9W0+biV0bNHRVIijCQd0WV5qEkm4sJsUD4zeSHiTqwNWjGVMU4588YI2GdSWrygUwfuFBRCLc2CyBcqzlvqA9eqyyVzf7pjV/bmb0jBHmCioNO59URss4ZdvUyFKbAe6+ZmUPoI5jpJ2XiKILh1cEt6f+z47YJf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: They are replaced with hugetlb_pte_lock{,ptr}. All callers that haven't already been replaced don't get called when using HGM, so we handle them by populating hugetlb_ptes with the standard, hstate-sized huge PTEs. Signed-off-by: James Houghton --- arch/powerpc/mm/pgtable.c | 7 +++++-- include/linux/hugetlb.h | 42 +++++++++++++++------------------------ mm/hugetlb.c | 22 +++++++++++++------- 3 files changed, 36 insertions(+), 35 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 035a0df47af0..e20d6aa9a2a6 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -258,11 +258,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, #ifdef CONFIG_PPC_BOOK3S_64 struct hstate *h = hstate_vma(vma); + struct hugetlb_pte hpte; psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), - vma->vm_mm, ptep)); + /* HGM is not supported for powerpc yet. */ + hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h), + hpage_size_to_level(psize)); + assert_spin_locked(hpte.ptl); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e1baf939afb6..4d318bf2ced9 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1032,14 +1032,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(unsigned int shift, - struct mm_struct *mm, pte_t *pte) -{ - if (shift == PMD_SHIFT) - return pmd_lockptr(mm, (pmd_t *) pte); - return &mm->page_table_lock; -} - #ifndef hugepages_supported /* * Some platform decide whether they support huge pages at boot @@ -1248,12 +1240,6 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(unsigned int shift, - struct mm_struct *mm, pte_t *pte) -{ - return &mm->page_table_lock; -} - static inline void hugetlb_count_init(struct mm_struct *mm) { } @@ -1328,16 +1314,6 @@ int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma, } #endif -static inline spinlock_t *huge_pte_lock(struct hstate *h, - struct mm_struct *mm, pte_t *pte) -{ - spinlock_t *ptl; - - ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); - spin_lock(ptl); - return ptl; -} - static inline spinlock_t *hugetlb_pte_lockptr(struct hugetlb_pte *hpte) { @@ -1358,8 +1334,22 @@ void hugetlb_pte_populate(struct mm_struct *mm, struct hugetlb_pte *hpte, pte_t *ptep, unsigned int shift, enum hugetlb_level level) { - __hugetlb_pte_populate(hpte, ptep, shift, level, - huge_pte_lockptr(shift, mm, ptep)); + spinlock_t *ptl; + + /* + * For contiguous HugeTLB PTEs that can contain other HugeTLB PTEs + * on the same level, the same PTL for both must be used. + * + * For some architectures that implement hugetlb_walk_step, this + * version of hugetlb_pte_populate() may not be correct to use for + * high-granularity PTEs. Instead, call __hugetlb_pte_populate() + * directly. + */ + if (level == HUGETLB_LEVEL_PMD) + ptl = pmd_lockptr(mm, (pmd_t *) ptep); + else + ptl = &mm->page_table_lock; + __hugetlb_pte_populate(hpte, ptep, shift, level, ptl); } #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 388c46c7e77a..d71adc03138d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5303,9 +5303,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, put_page(hpage); /* Install the new huge page if src pte stable */ - dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(huge_page_shift(h), - src, src_pte); + dst_ptl = hugetlb_pte_lock(&dst_hpte); + src_ptl = hugetlb_pte_lockptr(&src_hpte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -7383,7 +7382,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long saddr; pte_t *spte = NULL; pte_t *pte; - spinlock_t *ptl; + struct hugetlb_pte hpte; + struct hstate *shstate; i_mmap_lock_read(mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { @@ -7404,7 +7404,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, if (!spte) goto out; - ptl = huge_pte_lock(hstate_vma(vma), mm, spte); + shstate = hstate_vma(svma); + + hugetlb_pte_populate(mm, &hpte, spte, huge_page_shift(shstate), + hpage_size_to_level(huge_page_size(shstate))); + spin_lock(hpte.ptl); if (pud_none(*pud)) { pud_populate(mm, pud, (pmd_t *)((unsigned long)spte & PAGE_MASK)); @@ -7412,7 +7416,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, } else { put_page(virt_to_page(spte)); } - spin_unlock(ptl); + spin_unlock(hpte.ptl); out: pte = (pte_t *)pmd_alloc(mm, pud, addr); i_mmap_unlock_read(mapping); @@ -8132,6 +8136,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) unsigned long address, start, end; spinlock_t *ptl; pte_t *ptep; + struct hugetlb_pte hpte; if (!(vma->vm_flags & VM_MAYSHARE)) return; @@ -8156,7 +8161,10 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) ptep = hugetlb_walk(vma, address, sz); if (!ptep) continue; - ptl = huge_pte_lock(h, mm, ptep); + + hugetlb_pte_populate(mm, &hpte, ptep, huge_page_shift(h), + hpage_size_to_level(sz)); + ptl = hugetlb_pte_lock(&hpte); huge_pmd_unshare(mm, vma, address, ptep); spin_unlock(ptl); } -- 2.39.0.314.g84b9a713c41-goog