From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F812C43334 for ; Tue, 12 Jul 2022 16:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDC389400B0; Tue, 12 Jul 2022 12:55:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B8C61940063; Tue, 12 Jul 2022 12:55:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2CB99400B0; Tue, 12 Jul 2022 12:55:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8C71E940063 for ; Tue, 12 Jul 2022 12:55:00 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 638F8807AE for ; Tue, 12 Jul 2022 16:55:00 +0000 (UTC) X-FDA: 79679047560.18.DAC0F03 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf01.hostedemail.com (Postfix) with ESMTP id D55C54003D for ; Tue, 12 Jul 2022 16:54:59 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id fz10so8292481pjb.2 for ; Tue, 12 Jul 2022 09:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=SNbvxQrAF0PGcWamYEyo8Q3smlt0K9Ue+RRXRcxSf94=; b=qoixECQBykn1whjmBUMxClVzmZaLfgU87MkzAi56yZgkw/6M1Jk+D/0kyqEJeVg79S qr1kuYjaaM0Pn1+fyw4B/hB41KAV3Y/6uwX+qSxouA0HfYLvoSTRywD/azAjOELNGd3s 5oRqb+W9jo6RZeQbPeG35GCgdoiR6IHrcD8TeRqQRtLXmYXSdqL1iQizZRsi7P07cLR6 X5a89k27O/E/aI9afoEpQ0Ezhl9t766rYc34qL8WxXUNC+20bDHIpYQSN6ERJk/4Bj+s cmzMDbQ2PqTINDPmm488KMxqLZu38TElVSFlc24eGFwIW2APhN76nPUt9VKkh3V4pNvl ozCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=SNbvxQrAF0PGcWamYEyo8Q3smlt0K9Ue+RRXRcxSf94=; b=a7kZuDWKuteXr7RyMm1qzShorZkUyC/kmfId9UfJvMYHo6pBxoQ/pKhAW1OJnU72OB T1HLXx6FDlSkWMXsiMbZJBQpPG+seb1cf/JxGy8FJ7bzZAjvI64pjqpCPg8VNLSARldw ZeECY48vB3oEfXqNDAzZJt4eiCNQ0zqWdt3k6HaElUD4fk7jP7j5v/vs0WTgOgILi2+8 TmueUXUWdnqlwpyIxkWeCtLXY9RzFx5vAZvMMVerqWITLgVh5Jpeo2mYTmlr5lO+q+pX 0ZSilsTyky1b5yav9/Ogc5MtN8SnomdGzP1FlJT0eNlXB6FItUQqN/kKUqpcQVvSKZWu xsTQ== X-Gm-Message-State: AJIora9ajFp46U76Gie/+jbFwcZ601VTdT0Pq5kMSMhc4Vryuxn/8nRB M1neQNuKZYB/jom3BUvacU5Juw== X-Google-Smtp-Source: AGRyM1sFGPWf6FKecDUbG+GZhrkpljlf9bfuS7/lNjUhV6QI55IwuKKqhqA5zXdKlU89oKIIvTUlzg== X-Received: by 2002:a17:90b:33c4:b0:1f0:3d9d:39ac with SMTP id lk4-20020a17090b33c400b001f03d9d39acmr5357409pjb.56.1657644898526; Tue, 12 Jul 2022 09:54:58 -0700 (PDT) Received: from google.com (55.212.185.35.bc.googleusercontent.com. [35.185.212.55]) by smtp.gmail.com with ESMTPSA id 64-20020a621943000000b00518764d09cdsm7059748pfz.164.2022.07.12.09.54.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Jul 2022 09:54:57 -0700 (PDT) Date: Tue, 12 Jul 2022 09:54:54 -0700 From: Zach O'Keefe To: Yang Shi Cc: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , Rongwei Wang , SeongJae Park , Song Liu , Vlastimil Babka , Zi Yan , Linux MM , Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , linux-api@vger.kernel.org Subject: Re: [mm-unstable v7 09/18] mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse Message-ID: References: <20220706235936.2197195-1-zokeefe@google.com> <20220706235936.2197195-10-zokeefe@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657644899; a=rsa-sha256; cv=none; b=AYG3xT4OkGmSf4da5uZtq5P/6+SJMj8kXsovqq6tvseGgHXXfqxVv5nRIUjFDQAzFd5AxR pswSGYop0pcRHCR/BOR+t4jHZvHywDDiOPAJYINRm1D1SVOLvi6xi/lxzznaNeVabp5Azb bNiXvL35mMmMVgwnbnhoXQwt7T3kdAM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qoixECQB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of zokeefe@google.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zokeefe@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657644899; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SNbvxQrAF0PGcWamYEyo8Q3smlt0K9Ue+RRXRcxSf94=; b=qDJyjRxOYfbA3ZSkuNXCIdoM02AxFYJ8z+NrcRP/lZtn9dj6V2MMakL49iN4+inKtcMhg0 +IGqjMvz84OVRgip7Sw4khP81YFJGgZEkg31RvVTgRHoZq0JuKkg9nls/CQTLPHeF+NHgd cS2EeIeRPrQ4WztRY7eKQYvJ3MoliPg= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qoixECQB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of zokeefe@google.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zokeefe@google.com X-Rspam-User: X-Stat-Signature: 496wzdurn3jcgjxrjzcm3ukt643fbocb X-Rspamd-Queue-Id: D55C54003D X-Rspamd-Server: rspam08 X-HE-Tag: 1657644899-287940 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Jul 11 14:22, Yang Shi wrote: > On Wed, Jul 6, 2022 at 5:06 PM Zach O'Keefe wrote: > > > > This idea was introduced by David Rientjes[1]. > > > > Introduce a new madvise mode, MADV_COLLAPSE, that allows users to request a > > synchronous collapse of memory at their own expense. > > > > The benefits of this approach are: > > > > * CPU is charged to the process that wants to spend the cycles for the > > THP > > * Avoid unpredictable timing of khugepaged collapse > > > > Semantics > > > > This call is independent of the system-wide THP sysfs settings, but will > > fail for memory marked VM_NOHUGEPAGE. If the ranges provided span > > multiple VMAs, the semantics of the collapse over each VMA is > > independent from the others. This implies a hugepage cannot cross a VMA > > boundary. If collapse of a given hugepage-aligned/sized region fails, > > the operation may continue to attempt collapsing the remainder of memory > > specified. > > > > The memory ranges provided must be page-aligned, but are not required to > > be hugepage-aligned. If the memory ranges are not hugepage-aligned, the > > start/end of the range will be clamped to the first/last > > hugepage-aligned address covered by said range. The memory ranges must > > span at least one hugepage-sized region. > > > > All non-resident pages covered by the range will first be > > swapped/faulted-in, before being internally copied onto a freshly > > allocated hugepage. Unmapped pages will have their data directly > > initialized to 0 in the new hugepage. However, for every eligible hugepage > > aligned/sized region to-be collapsed, at least one page must currently be > > backed by memory (a PMD covering the address range must already exist). > > > > Allocation for the new hugepage may enter direct reclaim and/or > > compaction, regardless of VMA flags. When the system has multiple NUMA > > nodes, the hugepage will be allocated from the node providing the most > > native pages. This operation operates on the current state of the > > specified process and makes no persistent changes or guarantees on how > > pages will be mapped, constructed, or faulted in the future > > > > Return Value > > > > If all hugepage-sized/aligned regions covered by the provided range were > > either successfully collapsed, or were already PMD-mapped THPs, this > > operation will be deemed successful. On success, process_madvise(2) > > returns the number of bytes advised, and madvise(2) returns 0. Else, -1 > > is returned and errno is set to indicate the error for the most-recently > > attempted hugepage collapse. Note that many failures might have > > occurred, since the operation may continue to collapse in the event a > > single hugepage-sized/aligned region fails. > > > > ENOMEM Memory allocation failed or VMA not found > > EBUSY Memcg charging failed > > EAGAIN Required resource temporarily unavailable. Try again > > might succeed. > > EINVAL Other error: No PMD found, subpage doesn't have Present > > bit set, "Special" page no backed by struct page, VMA > > incorrectly sized, address not page-aligned, ... > > > > Most notable here is ENOMEM and EBUSY (new to madvise) which are > > intended to provide the caller with actionable feedback so they may take > > an appropriate fallback measure. > > Don't forget to update man-pages. And cc'ed linux-api. > Thanks for the review, Yang. Yes, have plans to update the man-pages once things are fully ironed out. Also, thanks for tip to cc linux-api - I did not know about that. Best, Zach > > > > Use Cases > > > > An immediate user of this new functionality are malloc() implementations > > that manage memory in hugepage-sized chunks, but sometimes subrelease > > memory back to the system in native-sized chunks via MADV_DONTNEED; > > zapping the pmd. Later, when the memory is hot, the implementation > > could madvise(MADV_COLLAPSE) to re-back the memory by THPs to regain > > hugepage coverage and dTLB performance. TCMalloc is such an > > implementation that could benefit from this[2]. > > > > Only privately-mapped anon memory is supported for now, but additional > > support for file, shmem, and HugeTLB high-granularity mappings[2] is > > expected. File and tmpfs/shmem support would permit: > > > > * Backing executable text by THPs. Current support provided by > > CONFIG_READ_ONLY_THP_FOR_FS may take a long time on a large system which > > might impair services from serving at their full rated load after > > (re)starting. Tricks like mremap(2)'ing text onto anonymous memory to > > immediately realize iTLB performance prevents page sharing and demand > > paging, both of which increase steady state memory footprint. With > > MADV_COLLAPSE, we get the best of both worlds: Peak upfront performance > > and lower RAM footprints. > > * Backing guest memory by hugapages after the memory contents have been > > migrated in native-page-sized chunks to a new host, in a > > userfaultfd-based live-migration stack. > > > > [1] https://lore.kernel.org/linux-mm/d098c392-273a-36a4-1a29-59731cdf5d3d@google.com/ > > [2] https://github.com/google/tcmalloc/tree/master/tcmalloc > > > > Suggested-by: David Rientjes > > Signed-off-by: Zach O'Keefe > > Reviewed-by: Yang Shi > > > --- > > arch/alpha/include/uapi/asm/mman.h | 2 + > > arch/mips/include/uapi/asm/mman.h | 2 + > > arch/parisc/include/uapi/asm/mman.h | 2 + > > arch/xtensa/include/uapi/asm/mman.h | 2 + > > include/linux/huge_mm.h | 14 ++- > > include/uapi/asm-generic/mman-common.h | 2 + > > mm/khugepaged.c | 118 ++++++++++++++++++- > > mm/madvise.c | 5 + > > tools/include/uapi/asm-generic/mman-common.h | 2 + > > 9 files changed, 146 insertions(+), 3 deletions(-) > > > > diff --git a/arch/alpha/include/uapi/asm/mman.h b/arch/alpha/include/uapi/asm/mman.h > > index 4aa996423b0d..763929e814e9 100644 > > --- a/arch/alpha/include/uapi/asm/mman.h > > +++ b/arch/alpha/include/uapi/asm/mman.h > > @@ -76,6 +76,8 @@ > > > > #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ > > > > +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ > > + > > /* compatibility flags */ > > #define MAP_FILE 0 > > > > diff --git a/arch/mips/include/uapi/asm/mman.h b/arch/mips/include/uapi/asm/mman.h > > index 1be428663c10..c6e1fc77c996 100644 > > --- a/arch/mips/include/uapi/asm/mman.h > > +++ b/arch/mips/include/uapi/asm/mman.h > > @@ -103,6 +103,8 @@ > > > > #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ > > > > +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ > > + > > /* compatibility flags */ > > #define MAP_FILE 0 > > > > diff --git a/arch/parisc/include/uapi/asm/mman.h b/arch/parisc/include/uapi/asm/mman.h > > index a7ea3204a5fa..22133a6a506e 100644 > > --- a/arch/parisc/include/uapi/asm/mman.h > > +++ b/arch/parisc/include/uapi/asm/mman.h > > @@ -70,6 +70,8 @@ > > #define MADV_WIPEONFORK 71 /* Zero memory on fork, child only */ > > #define MADV_KEEPONFORK 72 /* Undo MADV_WIPEONFORK */ > > > > +#define MADV_COLLAPSE 73 /* Synchronous hugepage collapse */ > > + > > #define MADV_HWPOISON 100 /* poison a page for testing */ > > #define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */ > > > > diff --git a/arch/xtensa/include/uapi/asm/mman.h b/arch/xtensa/include/uapi/asm/mman.h > > index 7966a58af472..1ff0c858544f 100644 > > --- a/arch/xtensa/include/uapi/asm/mman.h > > +++ b/arch/xtensa/include/uapi/asm/mman.h > > @@ -111,6 +111,8 @@ > > > > #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ > > > > +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ > > + > > /* compatibility flags */ > > #define MAP_FILE 0 > > > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > > index 00312fc251c1..39193623442e 100644 > > --- a/include/linux/huge_mm.h > > +++ b/include/linux/huge_mm.h > > @@ -218,6 +218,9 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, > > > > int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags, > > int advice); > > +int madvise_collapse(struct vm_area_struct *vma, > > + struct vm_area_struct **prev, > > + unsigned long start, unsigned long end); > > void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, > > unsigned long end, long adjust_next); > > spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma); > > @@ -361,9 +364,16 @@ static inline void split_huge_pmd_address(struct vm_area_struct *vma, > > static inline int hugepage_madvise(struct vm_area_struct *vma, > > unsigned long *vm_flags, int advice) > > { > > - BUG(); > > - return 0; > > + return -EINVAL; > > } > > + > > +static inline int madvise_collapse(struct vm_area_struct *vma, > > + struct vm_area_struct **prev, > > + unsigned long start, unsigned long end) > > +{ > > + return -EINVAL; > > +} > > + > > static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, > > unsigned long start, > > unsigned long end, > > diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h > > index 6c1aa92a92e4..6ce1f1ceb432 100644 > > --- a/include/uapi/asm-generic/mman-common.h > > +++ b/include/uapi/asm-generic/mman-common.h > > @@ -77,6 +77,8 @@ > > > > #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ > > > > +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ > > + > > /* compatibility flags */ > > #define MAP_FILE 0 > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > index c7a09cc9a0e8..2b2d832e44f2 100644 > > --- a/mm/khugepaged.c > > +++ b/mm/khugepaged.c > > @@ -976,7 +976,8 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, > > struct collapse_control *cc) > > { > > /* Only allocate from the target node */ > > - gfp_t gfp = alloc_hugepage_khugepaged_gfpmask() | __GFP_THISNODE; > > + gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : > > + GFP_TRANSHUGE) | __GFP_THISNODE; > > int node = khugepaged_find_target_node(cc); > > > > if (!khugepaged_alloc_page(hpage, gfp, node)) > > @@ -2356,3 +2357,118 @@ void khugepaged_min_free_kbytes_update(void) > > set_recommended_min_free_kbytes(); > > mutex_unlock(&khugepaged_mutex); > > } > > + > > +static int madvise_collapse_errno(enum scan_result r) > > +{ > > + /* > > + * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide > > + * actionable feedback to caller, so they may take an appropriate > > + * fallback measure depending on the nature of the failure. > > + */ > > + switch (r) { > > + case SCAN_ALLOC_HUGE_PAGE_FAIL: > > + return -ENOMEM; > > + case SCAN_CGROUP_CHARGE_FAIL: > > + return -EBUSY; > > + /* Resource temporary unavailable - trying again might succeed */ > > + case SCAN_PAGE_LOCK: > > + case SCAN_PAGE_LRU: > > + return -EAGAIN; > > + /* > > + * Other: Trying again likely not to succeed / error intrinsic to > > + * specified memory range. khugepaged likely won't be able to collapse > > + * either. > > + */ > > + default: > > + return -EINVAL; > > + } > > +} > > + > > +int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, > > + unsigned long start, unsigned long end) > > +{ > > + struct collapse_control *cc; > > + struct mm_struct *mm = vma->vm_mm; > > + unsigned long hstart, hend, addr; > > + int thps = 0, last_fail = SCAN_FAIL; > > + bool mmap_locked = true; > > + > > + BUG_ON(vma->vm_start > start); > > + BUG_ON(vma->vm_end < end); > > + > > + cc = kmalloc(sizeof(*cc), GFP_KERNEL); > > + if (!cc) > > + return -ENOMEM; > > + cc->is_khugepaged = false; > > + cc->last_target_node = NUMA_NO_NODE; > > + > > + *prev = vma; > > + > > + /* TODO: Support file/shmem */ > > + if (!vma->anon_vma || !vma_is_anonymous(vma)) > > + return -EINVAL; > > + > > + hstart = (start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK; > > + hend = end & HPAGE_PMD_MASK; > > + > > + if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) > > + return -EINVAL; > > + > > + mmgrab(mm); > > + lru_add_drain_all(); > > + > > + for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { > > + int result = SCAN_FAIL; > > + > > + if (!mmap_locked) { > > + cond_resched(); > > + mmap_read_lock(mm); > > + mmap_locked = true; > > + result = hugepage_vma_revalidate(mm, addr, &vma, cc); > > + if (result != SCAN_SUCCEED) { > > + last_fail = result; > > + goto out_nolock; > > + } > > + } > > + mmap_assert_locked(mm); > > + memset(cc->node_load, 0, sizeof(cc->node_load)); > > + result = khugepaged_scan_pmd(mm, vma, addr, &mmap_locked, cc); > > + if (!mmap_locked) > > + *prev = NULL; /* Tell caller we dropped mmap_lock */ > > + > > + switch (result) { > > + case SCAN_SUCCEED: > > + case SCAN_PMD_MAPPED: > > + ++thps; > > + break; > > + /* Whitelisted set of results where continuing OK */ > > + case SCAN_PMD_NULL: > > + case SCAN_PTE_NON_PRESENT: > > + case SCAN_PTE_UFFD_WP: > > + case SCAN_PAGE_RO: > > + case SCAN_LACK_REFERENCED_PAGE: > > + case SCAN_PAGE_NULL: > > + case SCAN_PAGE_COUNT: > > + case SCAN_PAGE_LOCK: > > + case SCAN_PAGE_COMPOUND: > > + case SCAN_PAGE_LRU: > > + last_fail = result; > > + break; > > + default: > > + last_fail = result; > > + /* Other error, exit */ > > + goto out_maybelock; > > + } > > + } > > + > > +out_maybelock: > > + /* Caller expects us to hold mmap_lock on return */ > > + if (!mmap_locked) > > + mmap_read_lock(mm); > > +out_nolock: > > + mmap_assert_locked(mm); > > + mmdrop(mm); > > + > > + return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 > > + : madvise_collapse_errno(last_fail); > > +} > > diff --git a/mm/madvise.c b/mm/madvise.c > > index 851fa4e134bc..9f08e958ea86 100644 > > --- a/mm/madvise.c > > +++ b/mm/madvise.c > > @@ -59,6 +59,7 @@ static int madvise_need_mmap_write(int behavior) > > case MADV_FREE: > > case MADV_POPULATE_READ: > > case MADV_POPULATE_WRITE: > > + case MADV_COLLAPSE: > > return 0; > > default: > > /* be safe, default to 1. list exceptions explicitly */ > > @@ -1057,6 +1058,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma, > > if (error) > > goto out; > > break; > > + case MADV_COLLAPSE: > > + return madvise_collapse(vma, prev, start, end); > > } > > > > anon_name = anon_vma_name(vma); > > @@ -1150,6 +1153,7 @@ madvise_behavior_valid(int behavior) > > #ifdef CONFIG_TRANSPARENT_HUGEPAGE > > case MADV_HUGEPAGE: > > case MADV_NOHUGEPAGE: > > + case MADV_COLLAPSE: > > #endif > > case MADV_DONTDUMP: > > case MADV_DODUMP: > > @@ -1339,6 +1343,7 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, > > * MADV_NOHUGEPAGE - mark the given range as not worth being backed by > > * transparent huge pages so the existing pages will not be > > * coalesced into THP and new pages will not be allocated as THP. > > + * MADV_COLLAPSE - synchronously coalesce pages into new THP. > > * MADV_DONTDUMP - the application wants to prevent pages in the given range > > * from being included in its core dump. > > * MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump. > > diff --git a/tools/include/uapi/asm-generic/mman-common.h b/tools/include/uapi/asm-generic/mman-common.h > > index 6c1aa92a92e4..6ce1f1ceb432 100644 > > --- a/tools/include/uapi/asm-generic/mman-common.h > > +++ b/tools/include/uapi/asm-generic/mman-common.h > > @@ -77,6 +77,8 @@ > > > > #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ > > > > +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ > > + > > /* compatibility flags */ > > #define MAP_FILE 0 > > > > -- > > 2.37.0.rc0.161.g10f37bed90-goog > >