From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 578DDC83F10 for ; Thu, 31 Aug 2023 07:26:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80D968E000D; Thu, 31 Aug 2023 03:26:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BC9A8D0001; Thu, 31 Aug 2023 03:26:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ACEA8E000D; Thu, 31 Aug 2023 03:26:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5C4698D0001 for ; Thu, 31 Aug 2023 03:26:56 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 212C780167 for ; Thu, 31 Aug 2023 07:26:56 +0000 (UTC) X-FDA: 81183568032.18.ACCC7ED Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf10.hostedemail.com (Postfix) with ESMTP id 455F8C0004 for ; Thu, 31 Aug 2023 07:26:54 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf10.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693466814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=txaHzML0yWv15HChS3V9ev7TFkyJiRdxGujoxP82T4M=; b=km+s7IkrZIX3Lp0FmYiDKeNaJH5yRtiSN4RPZ8326xG1yia96sX13XG+V1bVOys1gEqD3C srN1ossmRHhVZD0D4dT+yVdwOEvlt7H2F+N6jhZ7HhWJAYACTANi6/3Aw8J7DjwkX4tM/t 4s5EUs59UVhqsvzswcY/v1NG2GjA4qQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf10.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693466814; a=rsa-sha256; cv=none; b=69SzDsKXD0EMVmAo9FuN+sBp2LD7Huvu1wYqbYeZf84vXe9Nz3Y0anp11+9F0M8JaHRAA3 jPj/97QD+PPPri3avJEAnhct5+GnYShZBBvUk3m2cewPbzE2YeIeN2Ir4OWgfr892pp1rJ HZemwJzxWttzWNeWRU+vZtHGdWdB2NY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BF84FC15; Thu, 31 Aug 2023 00:27:32 -0700 (PDT) Received: from [10.57.64.173] (unknown [10.57.64.173]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EAB0E3F64C; Thu, 31 Aug 2023 00:26:51 -0700 (PDT) Message-ID: <6e01c6fc-bbb7-47f5-9e2a-d476d2c3ef65@arm.com> Date: Thu, 31 Aug 2023 08:26:49 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Prerequisites for Large Anon Folios Content-Language: en-GB To: David Hildenbrand , "Yin, Fengwei" , Zi Yan , Matthew Wilcox , Yu Zhao , David Rientjes Cc: Linux-MM References: <7f66344b-bf63-41e0-ae79-0a0a1d4f2afd@arm.com> <43736fdb-1a9c-4ab4-bf9c-6e2052c6dfea@redhat.com> From: Ryan Roberts In-Reply-To: <43736fdb-1a9c-4ab4-bf9c-6e2052c6dfea@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: 3r65m8pnyt7ykw17jm1wgnmghi6szonj X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 455F8C0004 X-HE-Tag: 1693466814-884963 X-HE-Meta: U2FsdGVkX19g4l/4MYa+IJLRr2Mo4JT1+yUHXYcwU9EQqsCfsgWDMtOYncQCLv0arnv3fNwLcm2O66BdJlDW4NzQr4igenfenpuQ7tHyjgs5I013gmWYTFUpKRzasxWwY/pasNMSktkH1jT7HOacv8QQ99+lQGmvuizdJ09YneERA4sgjzCefCNm/jGjdf9vtSwAH8+gXmgDSRuk6Y9qsoOW/aOJWq9usVnxrLVXTi0L4iiqW+7SlbTFY9B2wTxJfqGIRPlZ0wR7heSPzGKPCc7chbMOBR5jvRojhTy/+ZhK+yajF9rGd7XL34RkcbTMEHQlhmAy4tyQv/aSc2ITiuKe/rDbGXFcRJMgJYYy/Vz6I1p/S5gYLOxNALiUxqVHl+GL0YcSwYZUDIoGr648gFA9UAssMGntW4QJoaJfxPsQZG85O/isaCQURSHwHeFeNXlG0bSmAMdHPge2TlrVD/+NAoRlMMpCoPnz24RhOjA3MoSKIyfDLF9Qv4V8PwEH+qkwvarMl07gw+wA6cwgWdiQtJhmy3X36j2KAaQPVyCchc4jeyi6vaccYGFa30jE0fKSCmZiwAlkA79nQrDIW1pTUTvVBh9KGkIuy0Uk0DDbB57HMD2yiN9nE6ACvHM8SsyXNZgNFmD1IVHfACnYHRrfJAFD43QYNHqa4FgPifCigxbcJYp6pNlHDSzvoKGHPF5Cgd0NjbuB8PY7pf4vAh26xANga/UpAdQ+ucem7/EUY1Mzaulkq8q6TYjswFkExz4GlRDIno1T5s+4HhEdR8yq+p5FSQD7Zr931zD/FL+rx/2QQvmD4jhcLi6l0P0PTobGCxQ47VQ2t/jCRp3PTtDyiqWv1F0bE+zV3qJX2OVjmG7SL3LYRnNtTFnrymUQalv30XKUC6ICpKqHHJas+PTgX30raBdSVbzdQU79aNrOEHn61+BoxtqjyY6MOCu77h22ey8ln65s/azhyFS cJBw47lh AbR/cTe3Hdlm/foj58/1kkn+SUipzE8gnJPCNfZPjGoWd+sbNSlEZZtIMVWt0CC5AabdQ18pscPsY61trqA3rOjTwlVj4isTswbkso6EkWmPEVy3d1C2uMFFeZTryhKxnEqeydBqBm4vfaZWbVKZoKNMdh6tzFdDROsIYANm1C8ra0tGqGoXYZt9nGcw7ePFczKa4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 30/08/2023 17:20, David Hildenbrand wrote: > On 30.08.23 12:44, Ryan Roberts wrote: >> Hi All, >> > > Hi Ryan, > > I'll be back from vacation next Wednesday. > > Note that I asked David R. to have large anon folios as topic for the next > bi-weekly mm meeting. Ahh great! I don't have an invite to this meeting - is that something I can get added to? > > There, we should discuss things like > * naming > * accounting (/proc/meminfo) > * required toggles (especially, to ways to disable it, as we want to >   keep toggles minimal) > > David R. raised that there are certainly workloads where the additional memory > overhead is usually not acceptable. So it will be valuable to get input from > others. > >> >> I want to get serious about getting large anon folios merged. To do that, there >> are a number of outstanding prerequistes. I'm hoping the respective owners may >> be able to provide an update on progress? > > I shared some details in the last meeting when you were on vacation :) > > High level update below. > > [...] > >>> >>> - item: >>>      shared vs exclusive mappings >>> >>>    priority: >>>      prerequisite >>> >>>    description: >- >>>      New mechanism to allow us to easily determine precisely whether a given >>>      folio is mapped exclusively or shared between multiple processes. Required >>>      for (from David H): >>> >>>      (1) Detecting shared folios, to not mess with them while they are shared. >>>      MADV_PAGEOUT, user-triggered page migration, NUMA hinting, khugepaged ... >>>      replace cases where folio_estimated_sharers() == 1 would currently be the >>>      best we can do (and in some cases, page_mapcount() == 1). >>> >>>      (2) COW improvements for PTE-mapped large anon folios after fork(). Before >>>      fork(), PageAnonExclusive would have been reliable, after fork() it's not. >>> >>>      For (1), "MADV_PAGEOUT" maps to the "madvise" item captured in this list. I >>>      *think* "NUMA hinting" maps to "numa balancing" (but need confirmation!). >>>      "user-triggered page migration" and "khugepaged" not yet captured (would >>>      appreciate someone fleshing it out). I previously understood migration >>> to be >>>      working for large folios - is "user-triggered page migration" some specific >>>      aspect that does not work? >>> >>>      For (2), this relates to Large Anon Folio enhancements which I plan to >>>      tackle after we get the basic series merged. >>> >>>    links: >>>      - 'email thread: Mapcount games: "exclusive mapped" vs. "mapped shared"' >>> >>>    location: >>>      - shrink_folio_list() >>> >>>    assignee: >>>      David Hildenbrand >> >> Any comment on this David? I think the last comment I saw was that you were >> planning to start an implementation a couple of weeks back? Did that get >> anywhere? > > The math should be solid at this point and I had a simple prototype running -- > including fairly clean COW reuse handling. > > I started cleaning it all up before my vacation. I'll first need the total > mapcount (which I sent), and might have to implement rmap patching during THP > split (easy), but I first have to do more measurements. > > Willies patches to free up space in the first tail page will be required. In > addition, my patches to free up ->private in tail pages for THP_SWAP. Both > things on their way upstream. > > Based on that, I need a bit spinlock to protect the total mapcount+tracking > data. There are things to measure (contention) and optimize (why even care about > tracking shared vs. exclusive if it's pretty guaranteed to always be shared -- > for example, shared libraries). > > So it looks reasonable at this point, but I'll have to look into possible > contentions and optimizations once I have the basics implemented cleanly. > > It's a shame we cannot get the subpage mapcount out of the way immediately, then > it wouldn't be "additional tracking" but "different tracking" :) > > Once back from vacation, I'm planning on prioritizing this. Shouldn't take ages > to get it cleaned up. Measurements and optimizations might take a bit longer. That's great - thanks for the update. I'm obviously happy to help with any benchmarking/testing - just shout. > > [...] > > >>> >>>    assignee: >>>      Yin, Fengwei >> >> As I understand it: initial solution based on folio_estimated_sharers() has gone >> into v6.5. Have a dependecy on David's precise shared vs exclusive work for an > > shared vs. exclusive in place would replace folio_estimated_sharers() users and > most sub-page mapcount users. > >> improved solution. And I think you mentioned you are planning to do a change >> that avoids splitting a large folio if it is entirely covered by the range? > > [..] >>> >>> - item: >>>      numa balancing >>> >>>    priority: >>>      prerequisite >>> >>>    description: >- >>>      Large, pte-mapped folios are ignored by numa-balancing code. Commit comment >>>      (e81c480): "We're going to have THP mapped with PTEs. It will confuse >>>      numabalancing. Let's skip them for now." Likely depends on "shared vs >>>      exclusive mappings". >> >>>    links: [] >>> >>>    location: >>>      - do_numa_page() >>> >>>    assignee: >>>      >>> >> >> Vaguely sounded like David might be planning to tackle this as part of his work >> on "shared vs exclusive mappings" ("NUMA hinting"??). David? > > It should be easy to handle it based on that. Similarly, khugepaged IIRC. OK that's good to hear. I missed it off the list, but I have a regression with large anon folios currently in the khugepaged mm selftest, which I think should be fixed by this. Thanks, Ryan >