From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5CB5C433F5 for ; Tue, 22 Mar 2022 15:54:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D01EB6B0071; Tue, 22 Mar 2022 11:54:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0536B0072; Tue, 22 Mar 2022 11:54:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B51C46B0074; Tue, 22 Mar 2022 11:54:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0147.hostedemail.com [216.40.44.147]) by kanga.kvack.org (Postfix) with ESMTP id A74276B0071 for ; Tue, 22 Mar 2022 11:54:15 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 5414CA0FAE for ; Tue, 22 Mar 2022 15:54:15 +0000 (UTC) X-FDA: 79272468870.21.ADBC708 Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by imf02.hostedemail.com (Postfix) with ESMTP id A690280010 for ; Tue, 22 Mar 2022 15:54:14 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id bn33so24539233ljb.6 for ; Tue, 22 Mar 2022 08:54:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=B+eH9tBhxkjOV8Krs3NkPJIK8N0DHHJwPoJEMLTmR0Y=; b=UnAujXu4WI8KnwoQtjsQ5oOBiElZ0lifjCTG9YraSaUsHApvR58/JSeaMUNGRsiX2L wH+Zb2XFRk62mN1IEZxNAI/rqvVa7js6aG1JKcDzpHmelKBTsh703tyJS/w2tNk2Mrg6 E5HANZf7W2u9ZftIIYf9P+3AwDEFgzTBvOxMFhjX6YB4ruDcPErfSfG/sSOtUwxmMIVM jjXTkr0F7PQj8aDFHKhmcRahEqR2Wt0B0DaPEr9eADMmCEMebkLmbZQqOJhD8haGJJde s4wPSsou2U439VyHMfNaQFqPL0THi+0erzRZyJXI5I3n6JdevQ3SlJUbLD9UUB1chrlL 2lVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=B+eH9tBhxkjOV8Krs3NkPJIK8N0DHHJwPoJEMLTmR0Y=; b=xtYO5oKVDr6880ei5GgV2UvvAPf/ohO5CUjatZuFKMCPva/Ugv7imsQQGlqYFaJXEM b5WrMSB2PU4dRjWq5kU+cSgtWAUgCRUaVHuo3KBoYgvQxd+wWjOKkTjktdDcbQTul2fy ObsoJ8bPiiIx8vn4POaY9ikhF7oSF46qp67M3urw3yL8lu7eyuyS2c5Z8SZdltBK/DMT aZkO+eRdCx2ERaSjZWdvd+73x+w6mTo02ZiyOrD14hWk/0lPJYlKJvwLVzegnCEzJ2Sa dTaX/X/TF98v2SPCfeovfP7Ou5ckmigKyIDNTA0piJomRNKmbCCoH4MrzmiF6OSn0Xo5 kwGQ== X-Gm-Message-State: AOAM532J0+t5NvTq7cJXmUSBsKvLbkwrUUh++ftXG1UJ3RgHqrPFGHrD ODy8HlSCQYmOE+T3qMpjsv83aVzWiD7jzmcS7S3WeA== X-Google-Smtp-Source: ABdhPJxS8UeX8/r9zzyEZqRm5sasitQghG+cgKBom5ZAJDz90gU30NF6t7LhLOt7lkXeG/6zwaOCfzd5mHAAqy8FTI4= X-Received: by 2002:a05:651c:992:b0:249:9dc2:9ede with SMTP id b18-20020a05651c099200b002499dc29edemr638526ljq.187.1647964452656; Tue, 22 Mar 2022 08:54:12 -0700 (PDT) MIME-Version: 1.0 References: <20220308213417.1407042-1-zokeefe@google.com> In-Reply-To: From: "Zach O'Keefe" Date: Tue, 22 Mar 2022 08:53:35 -0700 Message-ID: Subject: Re: [RFC PATCH 00/14] mm: userspace hugepage collapse To: Michal Hocko Cc: Alex Shi , David Hildenbrand , David Rientjes , Pasha Tatashin , SeongJae Park , Song Liu , Vlastimil Babka , Zi Yan , linux-mm@kvack.org, Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matthew Wilcox , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Peter Xu , Thomas Bogendoerfer , Yang Shi Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=UnAujXu4; spf=pass (imf02.hostedemail.com: domain of zokeefe@google.com designates 209.85.208.182 as permitted sender) smtp.mailfrom=zokeefe@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A690280010 X-Stat-Signature: 5mw9qjbineax5dq568teioeiazeif3ig X-HE-Tag: 1647964454-289176 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Mar 22, 2022 at 5:11 AM Michal Hocko wrote: > > On Mon 21-03-22 08:46:35, Zach O'Keefe wrote: > > Hey Michal, thanks for taking the time to review / comment. > > > > On Mon, Mar 21, 2022 at 7:38 AM Michal Hocko wrote: > > > > > > [ Removed Richard Henderson from the CC list as the delivery fails f= or > > > his address] > > > > Thank you :) > > > > > On Tue 08-03-22 13:34:03, Zach O'Keefe wrote: > > > > Introduction > > > > -------------------------------- > > > > > > > > This series provides a mechanism for userspace to induce a collapse= of > > > > eligible ranges of memory into transparent hugepages in process con= text, > > > > thus permitting users to more tightly control their own hugepage > > > > utilization policy at their own expense. > > > > > > > > This idea was previously introduced by David Rientjes, and thanks t= o > > > > everyone for your patience while I prepared these patches resulting= from > > > > that discussion[1]. > > > > > > > > [1] https://lore.kernel.org/all/C8C89F13-3F04-456B-BA76-DE2C378D30B= F@nvidia.com/ > > > > > > > > Interface > > > > -------------------------------- > > > > > > > > The proposed interface adds a new madvise(2) mode, MADV_COLLAPSE, a= nd > > > > leverages the new process_madvise(2) call. > > > > > > > > (*) process_madvise(2) > > > > > > > > Performs a synchronous collapse of the native pages mapped = by > > > > the list of iovecs into transparent hugepages. The default = gfp > > > > flags used will be the same as those used at-fault for the = VMA > > > > region(s) covered. > > > > > > Could you expand on reasoning here? The default allocation mode for #= PF > > > is rather light. Madvised will try harder. The reasoning is that we w= ant > > > to make stalls due to #PF as small as possible and only try harder fo= r > > > madvised areas (also a subject of configuration). Wouldn't it make mo= re > > > sense to try harder for an explicit calls like madvise? > > > > > > > The reasoning is that the user has presumably configured system/vmas > > to tell the kernel how badly they want thps, and so this call aligns > > with current expectations. I.e. a user who goes about the trouble of > > trying to fault-in a thp at a given memory address likely wants a thp > > "as bad" as the same user MADV_COLLAPSE'ing the same memory to get a > > thp. > > If the syscall tries only as hard as the #PF doesn't that limit the > functionality? I'd argue that, the various allocation semantics possible through existing thp knobs / vma flags, in addition to the proposed MADV_F_COLLAPSE_DEFRAG flag provides a flexible functional space to work with. Relatively speaking, in what way would we be lacking functionality? > I mean a non #PF can consume more resources to allocate > and collapse a THP as it won't inflict any measurable latency to the > targetting process (except for potential CPU contention). Sorry, I'm not sure I understand this. What latency are we discussing in this point? Do you mean to say that since MADV_COLLAPSE isn't in the fault path, it doesn't necessarily need to be fast / direct reclaim wouldn't be noticed? > From that > perspective madvise is much more similar to khugepaged. I would even > argue that it could try even harder because madvise is focused on a very > specific memory range and the execution is not shared among all > processes that are scanned by khugepaged. > Good point. Covered at the end. > > If this is not the case, then the MADV_F_COLLAPSE_DEFRAG flag could be > > used to explicitly request the kernel to try harder, as you mention. > > Do we really need that? How many do_harder levels do we want to support? > > What would be typical usecases for #PF based and DEFRAG usages? > Thanks for challenging this. Covered at the end. > [...] > > > > > Diverging from the at-fault semantics, VM_NOHUGEPAGE is ign= ored > > > > by default, as the user is explicitly requesting this actio= n. > > > > Define two flags to control collapse semantics, passed thro= ugh > > > > process_madvise(2)=E2=80=99s optional flags parameter: > > > > > > This part is discussed later in the thread. > > > > > > > > > > > MADV_F_COLLAPSE_LIMITS > > > > > > > > If supplied, collapse respects pte collapse limits set via > > > > sysfs: > > > > /transparent_hugepage/khugepaged/max_ptes_[none|swap|shared= ]. > > > > Required if calling on behalf of another process and not > > > > CAP_SYS_ADMIN. > > > > > > > > MADV_F_COLLAPSE_DEFRAG > > > > > > > > If supplied, permit synchronous compaction and reclaim, > > > > regardless of VMA flags. > > > > > > Why do we need this? > > > > Do you mean MADV_F_COLLAPSE_DEFRAG specifically, or both? > > > > * MADV_F_COLLAPSE_LIMITS is included because we'd like some form of > > inter-process protection for collapsing memory in another process' > > address space (which a malevolent program could exploit to cause oom > > conditions in another memcg hierarchy, for example), but we want > > privileged (CAP_SYS_ADMIN) users to otherwise be able to optimize thp > > utilization as they wish. > > Could you expand some more please? How is this any different from > khugepaged (well, except that you can trigger the collapsing explicitly > rather than rely on khugepaged to find that mm)? > MADV_F_COLLAPSE_LIMITS was motivated by being able to replicate & extend khugepaged in userspace, where the benefit is precisely that we can choose that mm/vma more intelligently. > > * MADV_F_COLLAPSE_DEFRAG is useful as mentioned above, where we want > > to explicitly tell the kernel to try harder to back this by thps, > > regardless of the current system/vma configuration. > > > > Note that when used together, these flags can be used to implement the > > exact behavior of khugepaged, through MADV_COLLAPSE. > > IMHO this is stretching the interface and this can backfire in the > future. The interface should be really trivial. I want to collapse a > memory area. Let the kernel do the right thing and do not bother with > all the implementation details. I would use the same allocation strategy > as khugepaged as this seems to be closesest from the latency and > application awareness POV. In a way you can look at the madvise call as > a way to trigger khugepaged functionality on he particular memory range. Trying to summarize a few earlier comments centering around MADV_F_COLLAPSE_DEFRAG and allocation semantics. This series presupposes the existence of an informed userspace agent that is aware of what processes/memory ranges would benefit most from thps. Such an agent might either be: (1) A system-level daemon optimizing thp utilization system-wide (2) A highly tuned process / malloc implementation optimizing their own thp usage The different types of agents reflects the divide between #PF and DEFRAG semantics. For (1), we want to view this exactly like triggering khugepaged functionality from userspace, and likely want DEFRAG semantics. For (2), I was viewing this as the "live" symmetric counterpart to at-fault thp allocation where the process has decided, at runtime, that this memory could benefit from thp backing, and so #PF semantics seemed like sane default. I'd worry that using DEFRAG semantics by default might deter adoption by users who might not be willing to wait an unbounded amount of time for direct reclaim. > -- > Michal Hocko > SUSE Labs