From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 452B4C433E0 for ; Wed, 24 Feb 2021 09:45:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 90F6364E85 for ; Wed, 24 Feb 2021 09:45:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 90F6364E85 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E70E86B006C; Wed, 24 Feb 2021 04:45:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DFBBA6B006E; Wed, 24 Feb 2021 04:45:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEA5A8D0001; Wed, 24 Feb 2021 04:45:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0108.hostedemail.com [216.40.44.108]) by kanga.kvack.org (Postfix) with ESMTP id B1A4B6B006C for ; Wed, 24 Feb 2021 04:45:07 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 73BF518050EBF for ; Wed, 24 Feb 2021 09:45:07 +0000 (UTC) X-FDA: 77852677854.25.C497798 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf13.hostedemail.com (Postfix) with ESMTP id 0BE33E0011ED for ; Wed, 24 Feb 2021 09:45:01 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=13;SR=0;TI=SMTPD_---0UPRGkQZ_1614159898; Received: from IT-FVFX43SYHV2H.local(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0UPRGkQZ_1614159898) by smtp.aliyun-inc.com(127.0.0.1); Wed, 24 Feb 2021 17:44:59 +0800 Subject: Re: [RFC] Hugepage collapse in process context To: David Rientjes , David Hildenbrand Cc: Vlastimil Babka , Michal Hocko , Hugh Dickins , Andrea Arcangeli , "Kirill A. Shutemov" , Song Liu , Matthew Wilcox , Minchan Kim , Chris Kennelly , linux-mm@kvack.org, linux-api@vger.kernel.org References: <0b51a213-650e-7801-b6ed-9545466c15db@suse.cz> <600ee57f-d839-d402-fb0f-e9f350114dce@redhat.com> <5127b9c-a147-8ef5-c942-ae8c755413d0@google.com> From: Alex Shi Message-ID: <25d9347b-9359-efab-e1e3-f98bd0012af9@linux.alibaba.com> Date: Wed, 24 Feb 2021 17:44:58 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.0; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <5127b9c-a147-8ef5-c942-ae8c755413d0@google.com> Content-Type: text/plain; charset=gbk X-Stat-Signature: 7yapp5iarcsa566irkzdzrq5iou4z483 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0BE33E0011ED Received-SPF: none (linux.alibaba.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=out30-54.freemail.mail.aliyun.com; client-ip=115.124.30.54 X-HE-DKIM-Result: none/none X-HE-Tag: 1614159901-292479 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: =D4=DA 2021/2/19 =C9=CF=CE=E76:34, David Rientjes =D0=B4=B5=C0: > On Thu, 18 Feb 2021, David Hildenbrand wrote: >=20 >>>>> Hi everybody, >>>>> >>>>> Khugepaged is slow by default, it scans at most 4096 pages every 10= s. >>>>> That's normally fine as a system-wide setting, but some application= s >>>>> would >>>>> benefit from a more aggressive approach (as long as they are willin= g to >>>>> pay for it). >>>>> >>>>> Instead of adding priorities for eligible ranges of memory to >>>>> khugepaged, >>>>> temporarily speeding khugepaged up for the whole system, or shardin= g its >>>>> work for memory belonging to a certain process, one approach would = be to >>>>> allow userspace to induce hugepage collapse. >>>>> >>>>> The benefit to this approach would be that this is done in process >>>>> context >>>>> so its cpu is charged to the process that is inducing the collapse. >>>>> Khugepaged is not involved. >>>> >>>> Yes, this makes a lot of sense to me. >>>> >>>>> Idea was to allow userspace to induce hugepage collapse through the= new >>>>> process_madvise() call. This allows us to collapse hugepages on be= half >>>>> of >>>>> current or another process for a vectored set of ranges. >>>> >>>> Yes, madvise sounds like a good fit for the purpose. >>> >>> Agreed on both points. >>> >>>>> This could be done through a new process_madvise() mode *or* it cou= ld be >>>>> a >>>>> flag to MADV_HUGEPAGE since process_madvise() allows for a flag >>>>> parameter >>>>> to be passed. For example, MADV_F_SYNC. >>>> >>>> Would this MADV_F_SYNC be applicable to other madvise modes? Most >>>> existing madvise modes do not seem to make much sense. We can argue = that >>>> MADV_PAGEOUT would guarantee the range was indeed reclaimed but I am= not >>>> sure we want to provide such a strong semantic because it can limit >>>> future reclaim optimizations. >>>> >>>> To me MADV_HUGEPAGE_COLLAPSE sounds like the easiest way forward. >>> >>> I guess in the old madvise(2) we could create a new combo of MADV_HUG= EPAGE | >>> MADV_WILLNEED with this semantic? But you are probably more intereste= d in >>> process_madvise() anyway. There the new flag would make more sense. B= ut >>> there's >>> also David H.'s proposal for MADV_POPULATE and there might be benefit= in >>> considering both at the same time? Should e.g. MADV_POPULATE with >>> MADV_HUGEPAGE >>> have the collapse semantics? But would MADV_POPULATE be added to >>> process_madvise() as well? Just thinking out loud so we don't end up = with >>> more >>> flags than necessary, it's already confusing enough as it is. >>> >> >> Note that madvise() eats only a single value, not flags. Combinations = as you >> describe are not possible. >> >> Something MADV_HUGEPAGE_COLLAPSE make sense to me that does not need t= he mmap >> lock in write and does not modify the actual VMA, only a mapping. >> >=20 > Agreed, and happy to see that there's a general consensus for the=20 > direction. Benefit of a new madvise mode is that it can be used for=20 > madvise() as well if you are interested in only a single range of your = own=20 > memory and then it doesn't need to reconcile with any of the already=20 > overloaded semantics of MADV_HUGEPAGE. It's a good idea to let process deal with its own THP policy. but current applications will miss the benefit w/o changes, and change is expensive for end users. So except this work, may a per memcg collapse be= nefit apps and free for them, we often deploy apps in cgroups on server now. Thanks Alex >=20 > Otherwise, process_madvise() can be used for other processes and/or=20 > vectored ranges. >=20 > Song's use case for this to prioritize thp usage is very important for = us=20 > as well. I hadn't thought of the madvise(MADV_HUGEPAGE) +=20 > madvise(MADV_HUGEPAGE_COLLAPSE) use case: I was anticipating the latter= =20 > would allocate the hugepage with khugepaged's gfp mask so it would alwa= ys=20 > compact. But it seems like this would actually be better to use the gf= p=20 > mask that would be used at fault for the vma and left to userspace to=20 > determine whether that's MADV_HUGEPAGE or not. Makes sense. >=20 > (Userspace could even do madvise(MADV_NOHUGEPAGE) +=20 > madvise(MADV_HUGEPAGE_COLLAPSE) to do the synchronous collapse but=20 > otherwise exclude it from khugepaged's consideration if it were incline= d.) >=20 > Two other minor points: >=20 > - Currently, process_madvise() doesn't use the flags parameter at all = so=20 > there's the question of whether we need generalized flags that apply= to=20 > most madvise modes or whether the flags can be specific to the mode=20 > being used. For example, a natural extension of this new mode would= be=20 > to determine the hugepage size if we were ever to support synchronou= s=20 > collapse into a 1GB gigantic page on x86 (MADV_F_1GB? :) >=20 > - We haven't discussed the future of khugepaged with this new mode: it= =20 > seems like we could simply implement khugepaged fully in userspace a= nd=20 > remove it from the kernel? :) >=20