From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F57AC433FE for ; Fri, 21 Oct 2022 16:37:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB9038E000B; Fri, 21 Oct 2022 12:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B41E28E0001; Fri, 21 Oct 2022 12:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8ACE98E000D; Fri, 21 Oct 2022 12:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7364E8E000B for ; Fri, 21 Oct 2022 12:37:26 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3887F1A0512 for ; Fri, 21 Oct 2022 16:37:26 +0000 (UTC) X-FDA: 80045512092.07.55D30B5 Received: from mail-vs1-f73.google.com (mail-vs1-f73.google.com [209.85.217.73]) by imf11.hostedemail.com (Postfix) with ESMTP id B5F6040004 for ; Fri, 21 Oct 2022 16:37:25 +0000 (UTC) Received: by mail-vs1-f73.google.com with SMTP id q185-20020a6743c2000000b003a6ee1e0c90so1048493vsa.17 for ; Fri, 21 Oct 2022 09:37:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uDuCPiOW4B8iVzI/6AmJPz8Wx+abw4sbfMgiEm8c4Xc=; b=qq0EliFW2HK5B98D2udo/aAD/3vjrh0KvnobMtG+PZI475PQxdoifbXuauS2SCXR7s NBAGHfMDTBh2R7M3djga9zPIyUlhx0bNxKeL4L+V01PzyGv4mm0erA2L0Y1ny5S6vxJy 4gip8m0AqQsH7lfmx5uVfNaKn30YrEf2kISS2JxneX57zOtYW6lbymXlNNHiSfHhJ/y1 yUdl8fYNmg7ptYMobwK3mFqxuJ7Y4BDoNlS3denm9rDalcC6tLL8UL3VoKuOEfO+zFSR UdMrIX+D3fSXo2nL8Ka8LYE/7mxAdhiJ2pdIhLJL9bKOH/gF6yAWL/qKQ3jBuanK9jCL 6s5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uDuCPiOW4B8iVzI/6AmJPz8Wx+abw4sbfMgiEm8c4Xc=; b=uGlLQ/MdHQj83H2/WKWuSNwyIud9xreeTBM8VszYZ8ywpSlHwXVY8GxNUOr9/ZGqh2 tI1AMcoMES1YqoszgirVVXTgSOq66/aMOCr742sfBuXRiu5RoLcyuUMUbvm/9kwaEqX1 UDvN2wE3AHGagF3exNDp2tJTQsQB50RJY/4s7bIIlYRPWBnjNhn40+eqY8WfndGG8Zwn hjkJNqhCRiZjmJgKgwoNrKKKym07Q25ZotWUHgZJ68gDwE278N0c2pCeLWUH+PkXB5L4 CZ5N4mGjrKtxDR3WhWjkVxYgTnkm3seSjDlBH5KZECVf74N1CPkgLUNtQgq29v+OKCQW 8grw== X-Gm-Message-State: ACrzQf29GhuqfteT3OmlF+6V2J69SjyVbLaRPdPAu4OzqltXqO9vy9xH HurJNYKJieHhjuVoXSedXcewq0jIK0nT3Kpx X-Google-Smtp-Source: AMsMyM5MOfTzLEOJgE3E6XYRUQL+J9vN27S5Ka8UI3ikwk+Wd0rafhqiIzJtQHdX6T/3LZGa8ANB+n7hmfewb2Cy X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:3172:b0:3a7:319c:ffef with SMTP id l18-20020a056102317200b003a7319cffefmr15180512vsm.80.1666370244975; Fri, 21 Oct 2022 09:37:24 -0700 (PDT) Date: Fri, 21 Oct 2022 16:36:25 +0000 In-Reply-To: <20221021163703.3218176-1-jthoughton@google.com> Mime-Version: 1.0 References: <20221021163703.3218176-1-jthoughton@google.com> X-Mailer: git-send-email 2.38.0.135.g90850a2211-goog Message-ID: <20221021163703.3218176-10-jthoughton@google.com> Subject: [RFC PATCH v2 09/47] hugetlb: make huge_pte_lockptr take an explicit shift argument. From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666370245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uDuCPiOW4B8iVzI/6AmJPz8Wx+abw4sbfMgiEm8c4Xc=; b=lE56P3TWBjpMbNJEfGfUuePmCzT7wF2G5+DO3ZGmOZrqfdLvRZigdw/mNQT7f6bhnGJewZ yzA/UrTqUHtOPlvUBC3jtiG6flizTFUEa0uimBh9FaKHyOLm8Kh/wQxFaP3nsXX9uSs7+Y Md+DFYbWYdVTqeTMFR/CqxQ85NDGbGQ= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qq0EliFW; spf=pass (imf11.hostedemail.com: domain of 3xMpSYwoKCMUu4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com designates 209.85.217.73 as permitted sender) smtp.mailfrom=3xMpSYwoKCMUu4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666370245; a=rsa-sha256; cv=none; b=gxAmdu151YWjpCACZ8yV3BpHkov9kaOannaaCGQag/1EWPzGqrA6UFnwM9Kcp98LjClVZV t8qiBYvTLRerJO6EuB6BCQccVXwOuMMGX5/uTJNSD3E4OanNP6wYWBHrALkoeHRIqTGptQ CAlECqLjME6lZUn/6Rt9JGq1aHW4Qg8= X-Stat-Signature: h995ain7yjin8xck7nzrykyy8h1hdqqc X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qq0EliFW; spf=pass (imf11.hostedemail.com: domain of 3xMpSYwoKCMUu4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com designates 209.85.217.73 as permitted sender) smtp.mailfrom=3xMpSYwoKCMUu4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B5F6040004 X-HE-Tag: 1666370245-313350 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE. Signed-off-by: James Houghton --- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 9 ++++----- mm/hugetlb.c | 7 ++++--- mm/migrate.c | 3 ++- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index cb2dcdb18f8e..035a0df47af0 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); + assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), + vma->vm_mm, ptep)); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 6e0c36b08a0c..db3ed6095b1c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -934,12 +934,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { - if (huge_page_size(h) == PMD_SIZE) + if (shift == PMD_SHIFT) return pmd_lockptr(mm, (pmd_t *) pte); - VM_BUG_ON(huge_page_size(h) == PAGE_SIZE); return &mm->page_table_lock; } @@ -1144,7 +1143,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { return &mm->page_table_lock; @@ -1206,7 +1205,7 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, { spinlock_t *ptl; - ptl = huge_pte_lockptr(h, mm, pte); + ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); spin_lock(ptl); return ptl; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a18143add956..ef7662bd0068 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4847,7 +4847,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); again: @@ -4925,7 +4925,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Install the new huge page if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), + src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -4979,7 +4980,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, pte_t pte; dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(h, mm, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); /* * We don't have to worry about the ordering of src and dst ptlocks diff --git a/mm/migrate.c b/mm/migrate.c index 1457cdbb7828..a0105fa6e3b2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -334,7 +334,8 @@ void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); + spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), + vma->vm_mm, pte); __migration_entry_wait_huge(pte, ptl); } -- 2.38.0.135.g90850a2211-goog