From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CB00C19F4F for ; Wed, 8 May 2024 14:35:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97F276B007B; Wed, 8 May 2024 10:35:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92E3B6B0093; Wed, 8 May 2024 10:35:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F55A6B0096; Wed, 8 May 2024 10:35:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6327A6B007B for ; Wed, 8 May 2024 10:35:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 052071A144C for ; Wed, 8 May 2024 14:35:46 +0000 (UTC) X-FDA: 82095477492.16.BDD9286 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf18.hostedemail.com (Postfix) with ESMTP id 186F21C001B for ; Wed, 8 May 2024 14:35:43 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RlutZaMK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715178944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mSdFQi7WEyAPecRMBfbjj3HF+3TVM1BEUIWbOf9ypA8=; b=b39fYLNtxsniEqFnJM8jRBgZGrt1by+ZOYys0TMZrRs91L5E+FYhsB9GJ9cw5fJHsbqgya WKWp2hEa/D8B+8GbWCti15OuMoFyasj2QCzeiXbsQlXCSVsDscOnsRrFhF2x0vXTHTUsFg 3TMVk8F/x4fpkJQgtopWJ2Hbl8058r0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RlutZaMK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715178944; a=rsa-sha256; cv=none; b=gPWqt/M3l5Z9+T0WKVllpe1hbzVbifsVsxDkawFR+Gwhtk42WXISJnITF0HZ6HLbb+Nf0L 8inkSNxVQuTGR28YJCYeVTiAqZvhWkyHvsJErW8RnAx/J+sf6Yg3IY1L8BVSeGvraboVsl isCMpuRh8PZdOkASv3gpO0/ODEczN8E= Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-a59a0168c75so1175485966b.1 for ; Wed, 08 May 2024 07:35:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715178942; x=1715783742; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=mSdFQi7WEyAPecRMBfbjj3HF+3TVM1BEUIWbOf9ypA8=; b=RlutZaMKuY5okKnoRsH+PTlC02sOpU4clRxvCaroE7RaTp/nm8ul+IUq58qnmlhC6b FiC26oeyDQh4bYv0/SUReGM7sgKO0p+pBYNoDkDFby5B67ugqYipEgUZYBXblOTPkW6E Xk9R1f6AMs3tvM7mM1CUNLawbtsw/AeHyfMkNqywQOYkovvhqIj227reRMi8OiHVfGjt FQVOrR7yt8AsWclbL8rtmCwmgB1cJVt/4kCWwwbxyFiwDAtR8+NfWGY/VDxkkZVSaKOg CWM0xASfC1N+/VZIDb+w6LkUtjHrFbF1DpuEjfqeauzKR0Q0y8kSH+oKfSxHHLTDrLST Pyqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715178942; x=1715783742; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mSdFQi7WEyAPecRMBfbjj3HF+3TVM1BEUIWbOf9ypA8=; b=d5ceGdJ7fbD24kLevYFQyT4ZSz3OBhWwUId/SQVCEb0IWQTSv6io/5auZNe1X8Hv8I WLXDXXwjtv9YLTE1Tt41KPMDAeN9ue9jhClwdRfpNtXTsff9cITJg7pP0zJ6orKGUvXL DffdDSz0FKfTI6rT74GBeRPKXNH8wU3BEOih2spEHY8r9wt/aZD6qV/Sho7BElyT0AA7 s8v0n8h+0Rk7EydLZSrXwoNEk29NZH7cLnACCTGodyjiTueVxsYjdibOvA/PqxAgA8v4 htpmZvisStbWX5Az7FSFVjNa4D+WuBkpSdUPmDsfShX6sXzwSaGgPbhgev8mGYVexmUy a9Jw== X-Forwarded-Encrypted: i=1; AJvYcCUxRY3c2ce5EraP9GsXSA5YJQm+D7sf7Mx1b7VZTu/HOxHbD/icLtKooFZt1KeBcGJ6Glu1zZyHI8e13MsJXFP35Ag= X-Gm-Message-State: AOJu0YwD3oJQ8yYckz7C5mWA+iONH5WeirCZRXb1CaTv9QuY9xGYwduu rug3joPAk/Zxfv5O2GyzpjxEIMFQKX42U52q4nh3/2vWKpouBc3zryQcNmbxl03ZxFX/HeMG6oZ TNh3RIKl/8C5vkTsuE9IcfIyaqYs= X-Google-Smtp-Source: AGHT+IFM3fxW+ad9Mrsz43fhcWZSiYP7UCfoc7C2RZAuglscuVMehmOhWV4iB9t7DmAW9r/6Sn7RL7PPzMw3wIWsBGc= X-Received: by 2002:a50:c34f:0:b0:572:943a:cdf2 with SMTP id 4fb4d7f45d1cf-5731d9b659bmr1881465a12.1.1715178942095; Wed, 08 May 2024 07:35:42 -0700 (PDT) MIME-Version: 1.0 References: <20240501042700.83974-1-ioworker0@gmail.com> <20240501042700.83974-3-ioworker0@gmail.com> <0077A412-0AF1-4022-8F49-EE77AE601ECB@nvidia.com> In-Reply-To: From: Lance Yang Date: Wed, 8 May 2024 22:35:30 +0800 Message-ID: Subject: Re: [PATCH v4 2/3] mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop To: Zi Yan Cc: akpm@linux-foundation.org, willy@infradead.org, sj@kernel.org, maskray@google.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Baolin Wang Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 186F21C001B X-Stat-Signature: ujq5ypmk919ama6gfcnym9kgs6x9ejrg X-HE-Tag: 1715178943-556025 X-HE-Meta: U2FsdGVkX1+ECezb71ER1c2yaKIj6sZPCLN9+2HnTcldKlGNlYD95RDWY7QmIsaUBgsSP/AXn0qaluvVJe4DdBi+q4e763DjVfQa6qupKJ4Dn4g1+GhMw0jfvfJ6399JYMpPXssUiLJOCpaZKoR7ckZwvx0wzxgYRYs3UaQVTBNvzvT4FmS+SHKh64Pe3bg1mgGnWlm8xsOK6uvEUK2U7T4BzOTehORal1wfihBJB88YVhtSwb28e1LhSsNoeAHtP5mnVrEjZvoXBQdE8L1CPJVptDD3xbIDF1vbU2M/Cgw4vQK29rwP9yQSqSNCbpzlUemNQzagiB+yU+0hnaEb2Jl9PoFzUDG0Lmg0aaY7wtLOZvJ5gINVjXKSlricrwd90pXjnwlxWr4y/02TgowQDjfU+8ASzlo/S0F35d6imgP1ru2qeWHCmZYBhiFPRe1f+5BRpyTwdu24YX7ZrNURSQA/Z24zt27GW1sKYBMpfXr3p2gGAjSO9HaRRQ3Vk4T8f2G3Vt+2Bjtk6duUzPSDXnpKTSYj8H1VpEYd6o1svj36GQn7wy7d2N7K6FHQ8dOYE/9w6VoRC12jvxbCoRrXCeMjieejdgmo0nTVzpMdfOPBkW916FnQ0Ri+n7kjrGFhMawM7xBRx27qhxiSjRz588vGj/qrVVL5OpufiOVP6g9/PJau1o1VBF1xfpIgq/iTKw6D49iApuvMXrWyKDVDtVcHlfPiAoVWX4svAPqqGRUpdFFv5IOL22OFulvNUyF7R0yzp4uUa/bHMJ8B/tuWzTFUU8uG0sRpjEfqP1fWWCrTPoO8ooCBAJnXosB8xWZHpak3sLsjCzHuatgKgz1e7oquFSMqmV+6Y5GFSNaZF7Dr6lIIbosUHwwNWsO8W2g1ob+nuOm6vQdVY8/EHivbc0/DeCnBvhxPDxfJcSpsqP379HEnSfiFi1PgscwllvEvLS9vioV04pZ3MXIVXFW mRUbaW0l TCk6wsdbpsyvy1TQ1KSPw8suXfN/Epq/thnY9AwL+pjd+Kd7Mup8adPHdQdUhC7tuBMh0VEqDlRVsdwb1JfWh5YTW6K4uwzvBEOpSOWnuE/IRBLQcv1nj8jPWwikwYxw2/vsPCzSwPap6socV3QJ2RJwqAMfHEBj5FAveqi+1Cx1+XtWJU9IIYgt9erDcMf/OwslruB6uiTuSsUWx7WzEjv/vqLsyiADpOZev27QGmOThXMHl/1/rLEWNX6R99N+zwwh9COqqOHzoQRT10fELzCtD+BBBihvqtqWd8y3Br40xPdNsk9rg2u/c5bF8vceYEzby8rb2Qa+gZkK/DOBPA0KHFjx++OcseHVbKmPhJpy6sX33oioSHCmkiZ+X2F21foYl/H9nf5Yancem+qQ3r+9UIIlccUh2Ri1IIYdwWn7Pb/QUxGAiSH1JksxHlNd0PwwP/LCkX1i93+pkcoNXssswzWpGrfXXYkrr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, May 8, 2024 at 10:07=E2=80=AFPM Zi Yan wrote: > > On 8 May 2024, at 1:43, Lance Yang wrote: > > > On Tue, May 7, 2024 at 11:26=E2=80=AFPM Zi Yan wrote: > >> > >> On 1 May 2024, at 0:26, Lance Yang wrote: > >> > >>> In preparation for supporting try_to_unmap_one() to unmap PMD-mapped > >>> folios, start the pagewalk first, then call split_huge_pmd_address() > >>> to split the folio. > >>> > >>> Suggested-by: David Hildenbrand > >>> Signed-off-by: Lance Yang > >>> --- > >>> include/linux/huge_mm.h | 20 ++++++++++++++++++++ > >>> mm/huge_memory.c | 42 +++++++++++++++++++++------------------= -- > >>> mm/rmap.c | 24 +++++++++++++++++------ > >>> 3 files changed, 60 insertions(+), 26 deletions(-) > >>> > >>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > >>> index c8d3ec116e29..38c4b5537715 100644 > >>> --- a/include/linux/huge_mm.h > >>> +++ b/include/linux/huge_mm.h > >>> @@ -409,6 +409,20 @@ static inline bool thp_migration_supported(void) > >>> return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); > >>> } > >>> > >>> +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long= address, > >>> + pmd_t *pmd, bool freeze, struct folio *folio= ); > >>> + > >>> +static inline void align_huge_pmd_range(struct vm_area_struct *vma, > >>> + unsigned long *start, > >>> + unsigned long *end) > >>> +{ > >>> + *start =3D ALIGN(*start, HPAGE_PMD_SIZE); > >>> + *end =3D ALIGN_DOWN(*end, HPAGE_PMD_SIZE); > >>> + > >>> + VM_WARN_ON_ONCE(vma->vm_start > *start); > >>> + VM_WARN_ON_ONCE(vma->vm_end < *end); > >>> +} > >>> + > >>> #else /* CONFIG_TRANSPARENT_HUGEPAGE */ > >>> > >>> static inline bool folio_test_pmd_mappable(struct folio *folio) > >>> @@ -471,6 +485,12 @@ static inline void __split_huge_pmd(struct vm_ar= ea_struct *vma, pmd_t *pmd, > >>> unsigned long address, bool freeze, struct folio *folio= ) {} > >>> static inline void split_huge_pmd_address(struct vm_area_struct *vma= , > >>> unsigned long address, bool freeze, struct folio *folio= ) {} > >>> +static inline void split_huge_pmd_locked(struct vm_area_struct *vma, > >>> + unsigned long address, pmd_t *= pmd, > >>> + bool freeze, struct folio *fol= io) {} > >>> +static inline void align_huge_pmd_range(struct vm_area_struct *vma, > >>> + unsigned long *start, > >>> + unsigned long *end) {} > >>> > >>> #define split_huge_pud(__vma, __pmd, __address) \ > >>> do { } while (0) > >>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c > >>> index 8261b5669397..145505a1dd05 100644 > >>> --- a/mm/huge_memory.c > >>> +++ b/mm/huge_memory.c > >>> @@ -2584,6 +2584,27 @@ static void __split_huge_pmd_locked(struct vm_= area_struct *vma, pmd_t *pmd, > >>> pmd_populate(mm, pmd, pgtable); > >>> } > >>> > >>> +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long= address, > >>> + pmd_t *pmd, bool freeze, struct folio *folio= ) > >>> +{ > >>> + VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio)); > >>> + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); > >>> + VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); > >>> + VM_BUG_ON(freeze && !folio); > >>> + > >>> + /* > >>> + * When the caller requests to set up a migration entry, we > >>> + * require a folio to check the PMD against. Otherwise, there > >>> + * is a risk of replacing the wrong folio. > >>> + */ > >>> + if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || > >>> + is_pmd_migration_entry(*pmd)) { > >>> + if (folio && folio !=3D pmd_folio(*pmd)) > >>> + return; > >>> + __split_huge_pmd_locked(vma, pmd, address, freeze); > >>> + } > >>> +} > >>> + > >>> void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, > >>> unsigned long address, bool freeze, struct folio *folio= ) > >>> { > >>> @@ -2595,26 +2616,7 @@ void __split_huge_pmd(struct vm_area_struct *v= ma, pmd_t *pmd, > >>> (address & HPAGE_PMD_MASK) + HPAGE_PMD_= SIZE); > >>> mmu_notifier_invalidate_range_start(&range); > >>> ptl =3D pmd_lock(vma->vm_mm, pmd); > >>> - > >>> - /* > >>> - * If caller asks to setup a migration entry, we need a folio t= o check > >>> - * pmd against. Otherwise we can end up replacing wrong folio. > >>> - */ > >>> - VM_BUG_ON(freeze && !folio); > >>> - VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); > >>> - > >>> - if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || > >>> - is_pmd_migration_entry(*pmd)) { > >>> - /* > >>> - * It's safe to call pmd_page when folio is set because= it's > >>> - * guaranteed that pmd is present. > >>> - */ > >>> - if (folio && folio !=3D pmd_folio(*pmd)) > >>> - goto out; > >>> - __split_huge_pmd_locked(vma, pmd, range.start, freeze); > >>> - } > >>> - > >>> -out: > >>> + split_huge_pmd_locked(vma, range.start, pmd, freeze, folio); > >>> spin_unlock(ptl); > >>> mmu_notifier_invalidate_range_end(&range); > >>> } > >>> diff --git a/mm/rmap.c b/mm/rmap.c > >>> index 7e2575d669a9..432601154583 100644 > >>> --- a/mm/rmap.c > >>> +++ b/mm/rmap.c > >>> @@ -1636,9 +1636,6 @@ static bool try_to_unmap_one(struct folio *foli= o, struct vm_area_struct *vma, > >>> if (flags & TTU_SYNC) > >>> pvmw.flags =3D PVMW_SYNC; > >>> > >>> - if (flags & TTU_SPLIT_HUGE_PMD) > >>> - split_huge_pmd_address(vma, address, false, folio); > >>> - > >>> /* > >>> * For THP, we have to assume the worse case ie pmd for invalid= ation. > >>> * For hugetlb, it could be much worse if we need to do pud > >>> @@ -1650,6 +1647,8 @@ static bool try_to_unmap_one(struct folio *foli= o, struct vm_area_struct *vma, > >>> range.end =3D vma_address_end(&pvmw); > >>> mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm= , > >>> address, range.end); > >>> + if (flags & TTU_SPLIT_HUGE_PMD) > >>> + align_huge_pmd_range(vma, &range.start, &range.end); > >>> if (folio_test_hugetlb(folio)) { > >>> /* > >>> * If sharing is possible, start and end will be adjust= ed > >>> @@ -1664,9 +1663,6 @@ static bool try_to_unmap_one(struct folio *foli= o, struct vm_area_struct *vma, > >>> mmu_notifier_invalidate_range_start(&range); > >>> > >>> while (page_vma_mapped_walk(&pvmw)) { > >>> - /* Unexpected PMD-mapped THP? */ > >>> - VM_BUG_ON_FOLIO(!pvmw.pte, folio); > >>> - > >>> /* > >>> * If the folio is in an mlock()d vma, we must not swap= it out. > >>> */ > >>> @@ -1678,6 +1674,22 @@ static bool try_to_unmap_one(struct folio *fol= io, struct vm_area_struct *vma, > >>> goto walk_done_err; > >>> } > >>> > >>> + if (!pvmw.pte && (flags & TTU_SPLIT_HUGE_PMD)) { > >>> + /* > >>> + * We temporarily have to drop the PTL and star= t once > >>> + * again from that now-PTE-mapped page table. > >>> + */ > >>> + split_huge_pmd_locked(vma, range.start, pvmw.pm= d, false, > >>> + folio); > >> > >> Just in case you might miss here, since you will no longer align > >> range.start as Baolin mentioned in another email and you have a VM_WAR= N_ONCE > >> in split_huge_pmd_locked(), you will need to align the input address n= ow. > > > > Thanks for bringing that up! > > > > I do miss the alignment here when I decide to no longer align range.sta= rt > > in another email - thanks! > > > No problem. > > > Zi, could I move the alignment here? > > IIUC, we will not encounter a partially mapped THP here, and range.star= t > > and range.end should also not beyond the VMA limits. > > > > align_huge_pmd_range(vma, &range.start, &range.end= ); > > split_huge_pmd_locked(vma, range.start, pvmw.pmd, = false, > > folio); > > I think you can just do > > split_huge_pmd_locked(vma, ALIGN(range.start, HPAGE_PMD_SIZE), pvmw.pmd, = false, folio); > > since range will later be used by mmu_notifier_invalidate_range_end() and= changing > it might cause secondary TLB invalidation issues. Ok, makes sense to me - thanks! But we probably cannot use the HPAGE_PMD_SIZE here; it will cause broken compilation as seen in v3[1]. Perhaps we still need to add a new alignment function for the huge PMD? [1] https://lore.kernel.org/linux-mm/20240429202040.187453-1-sj@kernel.org/ Thanks again for the review! Best, Lance > > > > > Thanks, > > Lance > > > >> > >>> + pvmw.pmd =3D NULL; > >>> + spin_unlock(pvmw.ptl); > >>> + flags &=3D ~TTU_SPLIT_HUGE_PMD; > >>> + continue; > >>> + } > >>> + > >>> + /* Unexpected PMD-mapped THP? */ > >>> + VM_BUG_ON_FOLIO(!pvmw.pte, folio); > >>> + > >>> pfn =3D pte_pfn(ptep_get(pvmw.pte)); > >>> subpage =3D folio_page(folio, pfn - folio_pfn(folio)); > >>> address =3D pvmw.address; > >>> -- > >>> 2.33.1 > >> > >> > >> -- > >> Best Regards, > >> Yan, Zi > > > -- > Best Regards, > Yan, Zi