From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A7E9C10F1A for ; Tue, 7 May 2024 03:40:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0F356B0088; Mon, 6 May 2024 23:40:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9996C6B0089; Mon, 6 May 2024 23:40:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 811376B008A; Mon, 6 May 2024 23:40:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5ECB36B0088 for ; Mon, 6 May 2024 23:40:25 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E33A0805F5 for ; Tue, 7 May 2024 03:40:24 +0000 (UTC) X-FDA: 82090197168.07.440816C Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by imf14.hostedemail.com (Postfix) with ESMTP id ADC2A100008 for ; Tue, 7 May 2024 03:40:21 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=LJUJYWKO; spf=pass (imf14.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.133 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715053223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T1kb8aaj1YB35LBiWabfJHjOOXKZswjfPzIXa7wPy+Q=; b=xzjop4yMkw3kSd4NlYTQtGZ+/xVy/l+QgPIze/3rxw4CBRNK2do1G6RKT9AdipQGIvXhCC ZLI26ZMQ5lQrRiVZlWq3IxqQ2KDNnVQzfhLldBNgCHRx40irnJNGzpCcK5m6nKeNt6jjUh ABU45Id+XkZ0PvGlrKIXnaWqxLKnJLQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=LJUJYWKO; spf=pass (imf14.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.133 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715053223; a=rsa-sha256; cv=none; b=4orFmqxqRPQ3pOmW1ae73I1OCjVDvAZlx7AdqGj0DAXS3DUcf5BN+iodB9cmmdNzgVSRca uEGcSURxFIRp6W45QX6LQdR1djKTwuGU1T8S2/SVs9EkSn8LhDXBwiJ0cgzV9R3wE0n2lw uf2NS7TuzTCN0PaDS3mSyo4bcYDH4Z4= DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715053218; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=T1kb8aaj1YB35LBiWabfJHjOOXKZswjfPzIXa7wPy+Q=; b=LJUJYWKOkahz77xHULgvaVN+aWiJOeQOAGy0/F/ajIHV8l+0k714AmnBJvUSq81X+ZKGLmM4HgHGBBlrM20rzJdsalJpoPCQVZ2iyaml9EE38pjnQ5r9SZREN2cUsPSsGwSLEXlXCXE+LEAR29K/A4bop83n28zm3iKCEo6BMO8= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067110;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=21;SR=0;TI=SMTPD_---0W6-n0uS_1715053215; Received: from 30.97.56.67(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W6-n0uS_1715053215) by smtp.aliyun-inc.com; Tue, 07 May 2024 11:40:16 +0800 Message-ID: Date: Tue, 7 May 2024 11:40:15 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 2/3] mm/rmap: integrate PMD-mapped folio splitting into pagewalk loop To: Lance Yang , akpm@linux-foundation.org Cc: willy@infradead.org, sj@kernel.org, maskray@google.com, ziy@nvidia.com, ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, libang.li@antgroup.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240501042700.83974-1-ioworker0@gmail.com> <20240501042700.83974-3-ioworker0@gmail.com> From: Baolin Wang In-Reply-To: <20240501042700.83974-3-ioworker0@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: ADC2A100008 X-Rspamd-Server: rspam06 X-Stat-Signature: hc17gmjk6ax5r1hjc8fdq9wtz999qrmb X-HE-Tag: 1715053221-357211 X-HE-Meta: U2FsdGVkX19FG26LKsin9veNIawm2JxtJT4zbbFoDWZdK9f5GZr+jWiXpFmEOuw27PPpzpBiFZt1RCQJGoNb1d7wCFnJVxDHolfuhVN6J5GNOyzB/bq7J4FWtx6v+DQcGJV/2ftRiTVLhUSwWMIe8ykKlwol1uPFJgSQhw33ePuTCK2Tg6Xkq2UJ+lSx7CDse7TL+G/Cub2ZRdo1QwF/P2Ua9Ix80A+8y+UzjojmjQ54jMUfVhuSzACpkENNdNkLpBh+kl0uDhqeuwKFEd0POhVuT7urxsFN63Ft+n90TK6vDMdnacc5TkZ8S+onpxCZS1lSuFRT+69YAqcmezjPBWKEGDzi4meVXfkeNUfLgHTSbIpZSHp5JDxfOXFp7jRcB9y4oVkIswzbdQ+CFnfaPAG5qMveGxJ19YJItHfp1ZXE9Q8GpfZRQN5V8MmeJ7QW0PNT3YzBX4EPneMEpLYNUu1+GQjXWvVFubg/haCqxtHtqMQREgYL5M6uwiLoPlF0cXJQ/LrKz/0cEF99jpWRA1V1gPIeOza+XgBnYyA6afRIjh9y6vPvZAacrncKTsfJMPGEFuWuYMiqq0axqmVljIikKBraA6Q5rEnehlCmfwpU2UbBw1KpxXNvPP3FpPXcjfZP+p+dTSV4CepdnP6vGIlJR15sSKhgzqrxPo6lNaA0jOTE8+udJ9xqvlHvs8rkkx34WeaK6093vw9Q4iJ/Qu/FRxUBJJOxllJ2k9YuaRpInpyksBrQUcEtv4CSIrHPUlc8cDmA+lRu8y0rdOp4BOw5LUaBUhqL0Q2G925+K9V/1mFbeoIo3nj9VhZ1Ep9zjUd2aoFkVowbvSCoR1k/E9RJD8PMMxeLH5I6hKHNX0r0xwN+RTPDFRDZFXPOPqJvyBYyaAEqF3nXRPKNfE/5EwhhrOsH0S1lgN+jFdy+yma4GxTIYF9kVic6DDyQTOZvBibpofhMub/H58ALo1l fIpn6c3+ c8WKtRdJUJgfGc0ejNs+S4mTPVwgRNLqH5IeGPcl+2ON6BTed7HO9HJOzQA7IzCQlPJYPFT9qBUuXgRugULy7yDXnv3bThZsQ9UnR3qC9whHv6cj0uM9eJABabgGb9Yo6IMWYxGzfB/JgqjAEQ3vWCYYTfpvP9ARmDY1a9+h8wnRRnO/Rs8flZudYQruSVvkKSlMqMmol6xhbIaT08Vjd/32tfyMNHsWj6dbTmloMlp/CpgyuYcI2W7NGhrYw7XVnMpZuKC+z2rquQbusdSX9YInoo4ImuMVVktubPZ1QFNTA4uQjeedbIrHkg9mYSmXlHO7d53F34xcldeLrr+Xwt48XeZvk/UdnVLbh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/5/1 12:26, Lance Yang wrote: > In preparation for supporting try_to_unmap_one() to unmap PMD-mapped > folios, start the pagewalk first, then call split_huge_pmd_address() > to split the folio. > > Suggested-by: David Hildenbrand > Signed-off-by: Lance Yang > --- > include/linux/huge_mm.h | 20 ++++++++++++++++++++ > mm/huge_memory.c | 42 +++++++++++++++++++++-------------------- > mm/rmap.c | 24 +++++++++++++++++------ > 3 files changed, 60 insertions(+), 26 deletions(-) > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > index c8d3ec116e29..38c4b5537715 100644 > --- a/include/linux/huge_mm.h > +++ b/include/linux/huge_mm.h > @@ -409,6 +409,20 @@ static inline bool thp_migration_supported(void) > return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); > } > > +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, > + pmd_t *pmd, bool freeze, struct folio *folio); > + > +static inline void align_huge_pmd_range(struct vm_area_struct *vma, > + unsigned long *start, > + unsigned long *end) > +{ > + *start = ALIGN(*start, HPAGE_PMD_SIZE); > + *end = ALIGN_DOWN(*end, HPAGE_PMD_SIZE); > + > + VM_WARN_ON_ONCE(vma->vm_start > *start); > + VM_WARN_ON_ONCE(vma->vm_end < *end); > +} > + > #else /* CONFIG_TRANSPARENT_HUGEPAGE */ > > static inline bool folio_test_pmd_mappable(struct folio *folio) > @@ -471,6 +485,12 @@ static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, > unsigned long address, bool freeze, struct folio *folio) {} > static inline void split_huge_pmd_address(struct vm_area_struct *vma, > unsigned long address, bool freeze, struct folio *folio) {} > +static inline void split_huge_pmd_locked(struct vm_area_struct *vma, > + unsigned long address, pmd_t *pmd, > + bool freeze, struct folio *folio) {} > +static inline void align_huge_pmd_range(struct vm_area_struct *vma, > + unsigned long *start, > + unsigned long *end) {} > > #define split_huge_pud(__vma, __pmd, __address) \ > do { } while (0) > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 8261b5669397..145505a1dd05 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -2584,6 +2584,27 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, > pmd_populate(mm, pmd, pgtable); > } > > +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, > + pmd_t *pmd, bool freeze, struct folio *folio) > +{ > + VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio)); > + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); > + VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); > + VM_BUG_ON(freeze && !folio); > + > + /* > + * When the caller requests to set up a migration entry, we > + * require a folio to check the PMD against. Otherwise, there > + * is a risk of replacing the wrong folio. > + */ > + if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || > + is_pmd_migration_entry(*pmd)) { > + if (folio && folio != pmd_folio(*pmd)) > + return; > + __split_huge_pmd_locked(vma, pmd, address, freeze); > + } > +} > + > void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, > unsigned long address, bool freeze, struct folio *folio) > { > @@ -2595,26 +2616,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, > (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE); > mmu_notifier_invalidate_range_start(&range); > ptl = pmd_lock(vma->vm_mm, pmd); > - > - /* > - * If caller asks to setup a migration entry, we need a folio to check > - * pmd against. Otherwise we can end up replacing wrong folio. > - */ > - VM_BUG_ON(freeze && !folio); > - VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); > - > - if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || > - is_pmd_migration_entry(*pmd)) { > - /* > - * It's safe to call pmd_page when folio is set because it's > - * guaranteed that pmd is present. > - */ > - if (folio && folio != pmd_folio(*pmd)) > - goto out; > - __split_huge_pmd_locked(vma, pmd, range.start, freeze); > - } > - > -out: > + split_huge_pmd_locked(vma, range.start, pmd, freeze, folio); > spin_unlock(ptl); > mmu_notifier_invalidate_range_end(&range); > } > diff --git a/mm/rmap.c b/mm/rmap.c > index 7e2575d669a9..432601154583 100644 > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -1636,9 +1636,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, > if (flags & TTU_SYNC) > pvmw.flags = PVMW_SYNC; > > - if (flags & TTU_SPLIT_HUGE_PMD) > - split_huge_pmd_address(vma, address, false, folio); > - > /* > * For THP, we have to assume the worse case ie pmd for invalidation. > * For hugetlb, it could be much worse if we need to do pud > @@ -1650,6 +1647,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, > range.end = vma_address_end(&pvmw); > mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, > address, range.end); > + if (flags & TTU_SPLIT_HUGE_PMD) > + align_huge_pmd_range(vma, &range.start, &range.end); I am not sure why need this alignment? (1) For a partially mapped THP, 'range.start' and 'range.end' can beyond the VMA limits. For a PMD mapped THP, I think the address is already THP size alignment returned from vma_address(&folio->page, vma). (2) The range.end is not used. > if (folio_test_hugetlb(folio)) { > /* > * If sharing is possible, start and end will be adjusted > @@ -1664,9 +1663,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, > mmu_notifier_invalidate_range_start(&range); > > while (page_vma_mapped_walk(&pvmw)) { > - /* Unexpected PMD-mapped THP? */ > - VM_BUG_ON_FOLIO(!pvmw.pte, folio); > - > /* > * If the folio is in an mlock()d vma, we must not swap it out. > */ > @@ -1678,6 +1674,22 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, > goto walk_done_err; > } > > + if (!pvmw.pte && (flags & TTU_SPLIT_HUGE_PMD)) { > + /* > + * We temporarily have to drop the PTL and start once > + * again from that now-PTE-mapped page table. > + */ > + split_huge_pmd_locked(vma, range.start, pvmw.pmd, false, > + folio); > + pvmw.pmd = NULL; > + spin_unlock(pvmw.ptl); IMO, you should also make the 'pvmw.ptl = NULL;' after unlocking as page_vma_mapped_walk() did, in case some corner case met. > + flags &= ~TTU_SPLIT_HUGE_PMD; > + continue; > + } > + > + /* Unexpected PMD-mapped THP? */ > + VM_BUG_ON_FOLIO(!pvmw.pte, folio); > + > pfn = pte_pfn(ptep_get(pvmw.pte)); > subpage = folio_page(folio, pfn - folio_pfn(folio)); > address = pvmw.address;