From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AB81C433B4 for ; Wed, 19 May 2021 16:22:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CDB6960234 for ; Wed, 19 May 2021 16:22:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CDB6960234 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2F3036B0036; Wed, 19 May 2021 12:22:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A3C36B006E; Wed, 19 May 2021 12:22:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F6436B0070; Wed, 19 May 2021 12:22:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id CE53A6B0036 for ; Wed, 19 May 2021 12:22:36 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5D2579426 for ; Wed, 19 May 2021 16:22:36 +0000 (UTC) X-FDA: 78158498712.14.13A4221 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf11.hostedemail.com (Postfix) with ESMTP id 35E6F20007FA for ; Wed, 19 May 2021 16:22:34 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id q15so9811853pgg.12 for ; Wed, 19 May 2021 09:22:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=N1WhLH5NJ9UaJpTzcOfd4vqrvZqnaGoNvuxywAdx8ds=; b=LEtQwpwK+XAuFQ94W6UVlTFv8EG8QC6SeS3N9mo4T0CYTRRniNvwYGybTilAv0QBhr WmWV0ydah7TeAbWbwMNR8sHMiEZL2erSek+eqLexmf6aeZOKWK/LUOBnxeInnYf0lkY1 7CLKKmhVdFWLodPSa1UkfGnTKUhnHwJM7w6G+SfFlsaZCwUdlwqs7lwZ7xfIx1ryQegd l4F0AyQrNOMRvORXBMwgiv/OF1sZn9WXAeSFDnxKmduxSrrOtRcU4MpgvHFmzP2RUU+C dzNXEPwxEIQbuaGtj03bYOjKPwM2ONoMis/WhoCyoOdPBmJQKCfa1G0nakedqbM1UCYL 5Anw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=N1WhLH5NJ9UaJpTzcOfd4vqrvZqnaGoNvuxywAdx8ds=; b=pCSNiDziNhAID/Cfi+qpMnzDuBR0xfhEU3WKg+NUivf1odbz1dlv9ujYK4w2VUIM8p L4ic07OD9rA4jeGzWBIhdydC84BaAb8r4yUMmBCb4Wew0veaV80wYhqRHSeqQPhQH7Qk DVtkz5J67ithecYCEql1gTYcCfw5DeF2WyV0d7bN66IgCl/IkermcK7e+sXFmqJaocCi RWyBlX4O2M3MIX80wBhgXG1z0x9KikCPfO0X5ryAM8TqielaeV3OZBn8S/gCB9rQ3vZK PBuxTuN2kW/bgmdDH+k2Obv4Uf8d9ZsS3VCeGS7Avg7ngdecWyhc7lxHNHiniNEuTXz8 Y3+g== X-Gm-Message-State: AOAM532w2tJiOVrKIoelxgnNxhv0weE7tOpPDb3E4mDJLjLwsPGnUz6X zKNh68QS8cWK2y/qSXQDHicgIbjpR9jnBo+SdyBarg== X-Google-Smtp-Source: ABdhPJydyqLgiUphzSBR2+wh4S2N9ksRc5LSFg+dSGj8S6+oWaOPTEr57Kj+8ujuiJqTeQDKz1IYJt4rGthjXHiO0h0= X-Received: by 2002:aa7:88c8:0:b029:2de:7b37:7d23 with SMTP id k8-20020aa788c80000b02902de7b377d23mr8862758pff.59.1621441353905; Wed, 19 May 2021 09:22:33 -0700 (PDT) MIME-Version: 1.0 References: <20210518091826.36937-1-songmuchun@bytedance.com> <5ae7a4be-dfd5-faf6-a75c-a2adf5a344b2@arm.com> In-Reply-To: From: Muchun Song Date: Thu, 20 May 2021 00:21:57 +0800 Message-ID: Subject: Re: [External] Re: [PATCH] arm64: mm: hugetlb: add support for free vmemmap pages of HugeTLB To: Anshuman Khandual Cc: Will Deacon , Andrew Morton , David Hildenbrand , "Bodeddula, Balasubramaniam" , Oscar Salvador , Mike Kravetz , David Rientjes , linux-arm-kernel@lists.infradead.org, LKML , Linux Memory Management List , Xiongchun duan , fam.zheng@bytedance.com, zhengqi.arch@bytedance.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 35E6F20007FA Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=LEtQwpwK; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam04 X-Stat-Signature: wg1zwxtergxsz8m8cxwz366djwyg6f3o X-HE-Tag: 1621441354-235141 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 19, 2021 at 11:22 PM Muchun Song wrote: > > On Wed, May 19, 2021 at 10:43 PM Muchun Song wrote: > > > > On Wed, May 19, 2021 at 8:35 PM Anshuman Khandual > > wrote: > > > > > > > > > On 5/18/21 2:48 PM, Muchun Song wrote: > > > > The preparation of supporting freeing vmemmap associated with each > > > > HugeTLB page is ready, so we can support this feature for arm64. > > > > > > > > Signed-off-by: Muchun Song > > > > --- > > > > arch/arm64/mm/mmu.c | 5 +++++ > > > > fs/Kconfig | 2 +- > > > > 2 files changed, 6 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c > > > > index 5d37e461c41f..967b01ce468d 100644 > > > > --- a/arch/arm64/mm/mmu.c > > > > +++ b/arch/arm64/mm/mmu.c > > > > @@ -23,6 +23,7 @@ > > > > #include > > > > #include > > > > #include > > > > +#include > > > > > > > > #include > > > > #include > > > > @@ -1134,6 +1135,10 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, > > > > pmd_t *pmdp; > > > > > > > > WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END)); > > > > + > > > > + if (is_hugetlb_free_vmemmap_enabled() && !altmap) > > > > + return vmemmap_populate_basepages(start, end, node, altmap); > > > > + > > > > do { > > > > next = pmd_addr_end(addr, end); > > > > > > > > diff --git a/fs/Kconfig b/fs/Kconfig > > > > index 6ce6fdac00a3..02c2d3bf1cb8 100644 > > > > --- a/fs/Kconfig > > > > +++ b/fs/Kconfig > > > > @@ -242,7 +242,7 @@ config HUGETLB_PAGE > > > > > > > > config HUGETLB_PAGE_FREE_VMEMMAP > > > > def_bool HUGETLB_PAGE > > > > - depends on X86_64 > > > > + depends on X86_64 || ARM64 > > > > depends on SPARSEMEM_VMEMMAP > > > > > > > > config MEMFD_CREATE > > > > > > > > > > How does this interact with HugeTLB migration as such which might iterate > > > over individual constituent struct pages (overriding the same struct page > > > for all tail pages when this feature is enabled). A simple test involving > > > madvise(ptr, size, MADV_SOFT_OFFLINE) fails on various HugeTLB page sizes, > > > with this patch applied. Although I have not debugged this any further. > > > > It is weird. Actually, I didn't change the behaviour of the page migration. > > This feature is default off. If you want to enable this feature, you can pass > > "hugetlb_free_vmemmap=on" to the boot cmdline. Do you mean that the > > success rate of page migration will decrease when you enable this feature? > > The rate will increase if disbale. Right? > > I have done the test and found the issue. Because unmap_and_move_huge_page > always returns -EBUSY. I will look into this issue in depth. Thanks for your > report. > > The return point is as below: > > if (page_private(hpage) && !page_mapping(hpage)) { > rc = -EBUSY; > goto out_unlock; > } I know the issue. It was caused by commit d6995da31122 ("hugetlb: use page.private for hugetlb specific page flags"). The below patch can fix this issue. diff --git a/mm/migrate.c b/mm/migrate.c index e7a173da74ec..43419c4bb097 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1290,7 +1290,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, * page_mapping() set, hugetlbfs specific move page routine will not * be called and we could leak usage counts for subpools. */ - if (page_private(hpage) && !page_mapping(hpage)) { + if (hugetlb_page_subpool(hpage) && !page_mapping(hpage)) { rc = -EBUSY; goto out_unlock; } > > > > > Thanks. > > > > > > > > > > Soft offlining pfn 0x101c00 at process virtual address 0xffff7fa00000 > > > soft offline: 0x101c00: hugepage migration failed 1, type bfffc0000010006 > > > (referenced|uptodate|head|node=0|zone=2|lastcpupid=0xffff) > > >