From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79525C6FA8F for ; Wed, 30 Aug 2023 10:44:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA66028004C; Wed, 30 Aug 2023 06:44:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A56968E0009; Wed, 30 Aug 2023 06:44:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91EC328004C; Wed, 30 Aug 2023 06:44:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7D8AC8E0009 for ; Wed, 30 Aug 2023 06:44:31 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4B07280329 for ; Wed, 30 Aug 2023 10:44:31 +0000 (UTC) X-FDA: 81180437142.13.F5D80BD Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf18.hostedemail.com (Postfix) with ESMTP id A2DA71C002D for ; Wed, 30 Aug 2023 10:44:27 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693392269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i4F2D0p/SDUdGLrd6+xZoKtueQst+tSQNUO58EE0nmo=; b=oaxn3cfvyOQcgtdy6AOTbnONY9Bwhpd0C6QQtQuVY+lOq0dkrzH9SGpoJzPN7ayz6uF7Oz Zh5L4C7CYfGF7evrbOwRNGSEkXCP3SQcjaSxVc65QFNoJLet/83ohUgSztXIKMODt8qm4N WaQAsGcY5UjWilgwJs6NdzQtek2+XoQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693392269; a=rsa-sha256; cv=none; b=axWeBQztcHep7YLi7H9wIa8rjRsOuKYFwkJ+veGjaF+rEfCNmXsn5GoJb9fynokyhACQl6 zXQZc5H9lqRh73r/hv3lorw42+HMI1cqtrith7c9J6mdO96fPHlcJyilBmjHjqZJz1F8fq 6kC5AVlOnZflaHKhp9LBmiGswDm4mzE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 530162F4; Wed, 30 Aug 2023 03:45:06 -0700 (PDT) Received: from [10.57.64.173] (unknown [10.57.64.173]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E66DB3F64C; Wed, 30 Aug 2023 03:44:25 -0700 (PDT) Message-ID: <7f66344b-bf63-41e0-ae79-0a0a1d4f2afd@arm.com> Date: Wed, 30 Aug 2023 11:44:24 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Prerequisites for Large Anon Folios Content-Language: en-GB From: Ryan Roberts To: "Yin, Fengwei" , Zi Yan , Matthew Wilcox , David Hildenbrand , Yu Zhao Cc: Linux-MM References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A2DA71C002D X-Rspam-User: X-Stat-Signature: erazi1n8csby8n9h1uphg9cbha91f7pr X-Rspamd-Server: rspam01 X-HE-Tag: 1693392267-612754 X-HE-Meta: U2FsdGVkX18OlAbQ4gFA8bxsU6H3roCAyW94++EezeMCGpY2pS1IJfvBpbqDVe0un8h+4O3wpizkslBCzDlF+bhJOa7Zw7SWCtGCbdGf+7MsOxgXqlMMHGnsgHIK1/f3Or7qYHPX4zWZkL7JVwCO/e4Y8S7Zcxg5LoqnpbeNoAb9Sgl92hhbra0PvWO2oeRyTa/a5zgEP1BxZNvBqwrpKqv7N5jnJ82qiKWhwCZbgCBs2ZaOnYyL5kSJRRKD6m5AvOA8HfQEVKya46SQNhZcDcod5em8+lBR7rXPNtt7OABqu6S40GB9NGSsOhRmSGe0jVTcz7tKV3oKfs2cqKJiBYXy4bw/JDJOLEoTl8eMo565gdHkHL8dW0y1MYLQjJ5KyUiE1w6IxyBYLT4DAdZGrp3/Hfpdzo35SHLWE+0ML+u5QKazoZ2DnA+TPM1h9ZTDnuiZXGmGO5s96QfEdhRJEatpskY6QkUI3bWVwrQDvCfvPo9rmyPhoQRykGPr8DDuya4SEueV30yiebpdQPHa6a8M0KeyeV4pYQnn+JYQka417Wv3TC3C+hoSIrOjHzjTiOQLA0nx9MQc/Bk/x3YzKtW1p2exFvKi7WRlWlde54J7NK5Ldhi5DaGGWTbm7JguB+B3cMZdMiIaMlgodwgBAqjq/xSUQ4EX8nyD8hE0i9boMugbkGopf3IdmtjIOtIz5Cjxb7lnJ8UuCapPpwPfgFx5SwOk2PgIHSkN5PJAWMqQ27gyAjkHvJXVXCLI1oWnFJacIzq/dE8q/n/HovJM2Q288cJpFjuuak6pB0VtaISSuVbWzCQiMdAvNHRai7lQMruSbFn/aYyxW5z6v+SjubCMakPaiZX6jsUUiNohpWQmjp4pEbsbnnng0CI87orLSzfp8VY+E3osUIlR/FcXAHKSYSt1RPRjEo2Q1B3cfJ+ytkPRNC26H75JHWS9IWLAqTV+0rFsqPv84WelOLw f3p8Pyx3 XkmiA/+MlFwwIDRdrGkOdl7LysGSz+L/14qafuhRrd0GktQegt3OyOxcvsMYk1xpSs+r0nivAjhBzRck0Hu06etdUz3RtsX/zP5BN3kxw3J6Luytd53K4RD+YQQ1PAKy+Q4K25GjmaPTWYpwa5+1uApCr5TO9KzI0m9MTmu0HDGEFKbxi2M4ctuG7zBRJDBWm2qqjNwt7eTdapMNzgMoaVMB+cel/okHfNn/8BNqvWtIGgyK6XpCVEZjCCsJPF69tbsd8BI4kG0tXbrpoTL62EOS7rvd4IJd8RinR66PQ4gVlBW+xXF2b9+EU9SYwiIcyPiB2h98N63lN2zWkUJ+ntf304/1O9Xa+6UquOclAR3rUY4trhnQBVRUix9lCUvERHNWcJX0NjwRto90pqodECef/CYk+m/KP/GPj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi All, I want to get serious about getting large anon folios merged. To do that, there are a number of outstanding prerequistes. I'm hoping the respective owners may be able to provide an update on progress? I appreciate everyone is busy and likely juggling multiple things, so understand if no progress has been made or likely to be made - it would be good to know that though, so I can attempt to make alternative plans. See questions/comments below. Thanks! On 20/07/2023 10:41, Ryan Roberts wrote: > Hi All, > > As discussed at Matthew's call yesterday evening, I've put together a list of > items that need to be done as prerequisites for merging large anonymous folios > support. > > It would be great to get some review and confirmation as to whether anything is > missing or incorrect. Most items have an assignee - in that case it would be > good to check that my understanding that you are working on the item is correct. > > I think most things are independent, with the exception of "shared vs exclusive > mappings", which I think becomes a dependency for a couple of things (marked in > depender description); again would be good to confirm. > > Finally, although I'm concentrating on the prerequisites to clear the path for > merging an MVP Large Anon Folios implementation, I've included one "enhancement" > item ("large folios in swap cache"), solely because we explicitly discussed it > last night. My view is that enhancements can come after the initial large anon > folios merge. Over time, I plan to add other enhancements (e.g. retain large > folios over COW, etc). > > I'm posting the table as yaml as that seemed easiest for email. You can convert > to csv with something like this in Python: > > import yaml > import pandas as pd > pd.DataFrame(yaml.safe_load(open('work-items.yml'))).to_csv('work-items.csv') > > Thanks, > Ryan > > ----- > > - item: > shared vs exclusive mappings > > priority: > prerequisite > > description: >- > New mechanism to allow us to easily determine precisely whether a given > folio is mapped exclusively or shared between multiple processes. Required > for (from David H): > > (1) Detecting shared folios, to not mess with them while they are shared. > MADV_PAGEOUT, user-triggered page migration, NUMA hinting, khugepaged ... > replace cases where folio_estimated_sharers() == 1 would currently be the > best we can do (and in some cases, page_mapcount() == 1). > > (2) COW improvements for PTE-mapped large anon folios after fork(). Before > fork(), PageAnonExclusive would have been reliable, after fork() it's not. > > For (1), "MADV_PAGEOUT" maps to the "madvise" item captured in this list. I > *think* "NUMA hinting" maps to "numa balancing" (but need confirmation!). > "user-triggered page migration" and "khugepaged" not yet captured (would > appreciate someone fleshing it out). I previously understood migration to be > working for large folios - is "user-triggered page migration" some specific > aspect that does not work? > > For (2), this relates to Large Anon Folio enhancements which I plan to > tackle after we get the basic series merged. > > links: > - 'email thread: Mapcount games: "exclusive mapped" vs. "mapped shared"' > > location: > - shrink_folio_list() > > assignee: > David Hildenbrand Any comment on this David? I think the last comment I saw was that you were planning to start an implementation a couple of weeks back? Did that get anywhere? > > > > - item: > compaction > > priority: > prerequisite > > description: >- > Raised at LSFMM: Compaction skips non-order-0 pages. Already problem for > page-cache pages today. > > links: > - https://lore.kernel.org/linux-mm/ZKgPIXSrxqymWrsv@casper.infradead.org/ > - https://lore.kernel.org/linux-mm/C56EA745-E112-4887-8C22-B74FCB6A14EB@nvidia.com/ > > location: > - compaction_alloc() > > assignee: > Zi Yan > > Are you still planning to work on this, Zi? The last email I have is [1] where you agreed to take a look. [1] https://lore.kernel.org/linux-mm/4DD00BE6-4141-4887-B5E5-0B7E8D1E2086@nvidia.com/ > > - item: > mlock > > priority: > prerequisite > > description: >- > Large, pte-mapped folios are ignored when mlock is requested. Code comment > for mlock_vma_folio() says "...filter out pte mappings of THPs, which cannot > be consistently counted: a pte mapping of the THP head cannot be > distinguished by the page alone." > > location: > - mlock_pte_range() > - mlock_vma_folio() > > links: > - https://lore.kernel.org/linux-mm/20230712060144.3006358-1-fengwei.yin@intel.com/ > > assignee: > Yin, Fengwei > > series on list at [2]. Does this series cover everything? [2] https://lore.kernel.org/linux-mm/20230809061105.3369958-1-fengwei.yin@intel.com/ > > - item: > madvise > > priority: > prerequisite > > description: >- > MADV_COLD, MADV_PAGEOUT, MADV_FREE: For large folios, code assumes exclusive > only if mapcount==1, else skips remainder of operation. For large, > pte-mapped folios, exclusive folios can have mapcount upto nr_pages and > still be exclusive. Even better; don't split the folio if it fits entirely > within the range. Likely depends on "shared vs exclusive mappings". > > links: > - https://lore.kernel.org/linux-mm/20230713150558.200545-1-fengwei.yin@intel.com/ > > location: > - madvise_cold_or_pageout_pte_range() > - madvise_free_pte_range() > > assignee: > Yin, Fengwei As I understand it: initial solution based on folio_estimated_sharers() has gone into v6.5. Have a dependecy on David's precise shared vs exclusive work for an improved solution. And I think you mentioned you are planning to do a change that avoids splitting a large folio if it is entirely covered by the range? > > > > - item: > deferred_split_folio > > priority: > prerequisite > > description: >- > zap_pte_range() will remove each page of a large folio from the rmap, one at > a time, causing the rmap code to see the folio as partially mapped and call > deferred_split_folio() for it. Then it subsquently becmes fully unmapped and > it is removed from the queue. This can cause some lock contention. Proposed > fix is to modify to zap_pte_range() to "batch zap" a whole pte range that > corresponds to a folio to avoid the unneccessary deferred_split_folio() > call. > > links: > - https://lore.kernel.org/linux-mm/20230719135450.545227-1-ryan.roberts@arm.com/ > > location: > - zap_pte_range() > > assignee: > Ryan Roberts I have a series at [3] to solve this (different approach than described above). Although Yu has suggested this is not a prerequisite after all [4]. [3] https://lore.kernel.org/linux-mm/20230830095011.1228673-1-ryan.roberts@arm.com/ [4] https://lore.kernel.org/linux-mm/CAOUHufZr8ym0kzoa99=k3Gquc4AdoYXMaj-kv99u5FPv1KkezA@mail.gmail.com/ > > > > - item: > numa balancing > > priority: > prerequisite > > description: >- > Large, pte-mapped folios are ignored by numa-balancing code. Commit comment > (e81c480): "We're going to have THP mapped with PTEs. It will confuse > numabalancing. Let's skip them for now." Likely depends on "shared vs > exclusive mappings". > > links: [] > > location: > - do_numa_page() > > assignee: > > Vaguely sounded like David might be planning to tackle this as part of his work on "shared vs exclusive mappings" ("NUMA hinting"??). David? > > > - item: > large folios in swap cache > > priority: > enhancement > > description: >- > shrink_folio_list() currently splits large folios to single pages before > adding them to the swap cache. It would be preferred to add the large folio > as an atomic unit to the swap cache. It is still expected that each page > would use a separate swap entry when swapped out. This represents an > efficiency improvement. There is risk that this change will expose bad > assumptions in the swap cache that assume any large folio is pmd-mappable. > > links: > - https://lore.kernel.org/linux-mm/CAOUHufbC76OdP16mRsY3i920qB7khcu8FM+nUOG0kx5BMRdKXw@mail.gmail.com/ > > location: > - shrink_folio_list() > > assignee: > Not a prerequisite so not worrying about it for now. > > -----