From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD3AC0015E for ; Thu, 3 Aug 2023 10:24:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 022DF28022E; Thu, 3 Aug 2023 06:24:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EEE6828022C; Thu, 3 Aug 2023 06:24:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D67AA28022E; Thu, 3 Aug 2023 06:24:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C1C1A28022C for ; Thu, 3 Aug 2023 06:24:58 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4AC181A107D for ; Thu, 3 Aug 2023 10:24:58 +0000 (UTC) X-FDA: 81082410276.06.E9497DA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 8D71E1C0017 for ; Thu, 3 Aug 2023 10:24:55 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691058295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VviS0LM01VZE+HIUq3zjaZgTaQ8YTNq+sCdM8wCU140=; b=Jc8asRn3G63vA0y9nkm137kq/uE530occUmVCrF4afE1j+jcdDoGh1Gwa9Fy5BQJTeOpB6 1Q9Evx3YBuEk1YQ1e9e75nozdlt1GYsfZkJwsrrxnIAYg5DZssmqekIetqOpm05cYbw/x1 NVEjB+R3JhK0jy5YrGn+cpJSE4Lqt60= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691058295; a=rsa-sha256; cv=none; b=z6Jcje03sgDZqxwteRch4lELrHpqUNn163UEMuPSWm6LPx72F8iPmIY7ecUEVf8bi69Ulg 1QTxZKlzFhTSgPSfGsUTYNZRvjl1WRDb3jX/WsL+YtQRErhB3MoW3OSRBo/S1yY7LL5dnm 7gz5H5Zng34+/rNVHrm99vrwRVMnxPk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C929113E; Thu, 3 Aug 2023 03:25:37 -0700 (PDT) Received: from [10.1.35.53] (C02Z41KALVDN.cambridge.arm.com [10.1.35.53]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4B7243F5A1; Thu, 3 Aug 2023 03:24:52 -0700 (PDT) Message-ID: <5afdb9ee-e7c9-20c2-67c4-47b8ccab1661@arm.com> Date: Thu, 3 Aug 2023 11:24:50 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance To: Yu Zhao Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230726095146.2826796-1-ryan.roberts@arm.com> <20230726095146.2826796-3-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8D71E1C0017 X-Stat-Signature: zahrwcx7pupt5im7m8u83sq16gu9zjc1 X-Rspam-User: X-HE-Tag: 1691058295-571474 X-HE-Meta: U2FsdGVkX1/R05c2QlJaNmc6uKYQQX8BNN+LNZnntZS7UThj2mVq5t179QgCqqlDF2MGm+hkaZNu7ES7siWJ1srNV3VtsJz4iMPDmgqx8dfciNLPpcJF6bXEhKdDRgE2HA0JJAeIg7gXagThLGClDnNqxgqISXaCfI0gBrqfOQmbtvfBDsgYNH8OSgI43GhAXvkZmABTK7B2Qe/cR5hOnLiao8Hyv4mvNafA0jC9rUYtCVPsLVQFdsMD6n+t/jLW5Afzcj3bLuf0KZsyR8ft2TzqRBP1QAHSZjsecNSvhWUEvqSNFx1QJaQHrryok7+smjj7lHJaZ0BZK9vfX4yo5t1hBDSwMeq07rwZ0XhYIjvwKCjPAwalUICJ29GLQ+4I9d5mPH8FaA/zEOFr+Rz36rb2iSZdjrxavVZuS6vaLDF7oZztNlU4uuyRgmri9MjEphHqmEAWJTGpB0NMLWVNnUvihXiBIYhd1/FYQkv7llKQZonyC39UeVBaDt1TDZzkqknSQFo517NCihlnF5fGfC9N858lSwx6jibUwx1+Dp6favMb+RNRR/Cywd4wQD4weHSdGOabpX2eVCkUcWFabrRopQcVQ2DFA2j4DkprwC9IfZ22ac3aZfkiBHFQvPHhaHBFLHPZbeaRT3YMSc6ENoM2IwdD6tKjyVRQ+PpDHkInw3pfyVmNOXFvB3MMkNVK9GAhAO15j76zWzIhbD5KHO2l9qCmDmnZhmfUcyUanaQn2nkeDwzYO5HhAjPFT6kk6f1sFo0f7vWd07166A9YoYqtD5lSQrbkE3lfplkg8tBifX9F2R1Ot72bAUsG47xaYzEdlhWP6TXwFSjqmotBpXczaSZkPl6z+vh9xt69aA7yTbzNciukcjAKNV/SXxB7xE9zETQ4sSCsoFNGCQ4TbWGsmHfIbabuwA52RjIKH9qiIiYOXNS25wQmHmvjxLQRojqZ5ILozvbRtaNiqXZ xrNBqzEE EMVgaJCZqvAWnFhoAjn6cmLxMMmD5k74qS7E1YMSL5kZ/91Ym/oB7Nhtw5kLJof/xkp95e8iygYoWD1+LtgEkXIhze/lBbGmb0MMXwIqALPied4hS8TOvcm2DdVw5vSb22tIX176u5TpTfXFc9hf/fxNTLwnTg20HWU71EoNvyEYFJ9pXe8KGrnboA0SVhKpK4Uw5NE/dE30XWg47tBFdCK1Qj6NXyIMD0EvkB8HWGAl9OSwMTa6JrdJYXY63padWsG8hNEGmWZjXbiMLWWALaRmbGiC6D8afRzRx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 02/08/2023 22:05, Yu Zhao wrote: > On Wed, Aug 2, 2023 at 3:33 AM Ryan Roberts wrote: >> >> On 01/08/2023 07:18, Yu Zhao wrote: >>> On Wed, Jul 26, 2023 at 3:52 AM Ryan Roberts wrote: >>>> >>>> Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be >>>> allocated in large folios of a determined order. All pages of the large >>>> folio are pte-mapped during the same page fault, significantly reducing >>>> the number of page faults. The number of per-page operations (e.g. ref >>>> counting, rmap management lru list management) are also significantly >>>> reduced since those ops now become per-folio. >>>> >>>> The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, >>>> which defaults to disabled for now; The long term aim is for this to >>>> defaut to enabled, but there are some risks around internal >>>> fragmentation that need to be better understood first. >>>> >>>> When enabled, the folio order is determined as such: For a vma, process >>>> or system that has explicitly disabled THP, we continue to allocate >>>> order-0. THP is most likely disabled to avoid any possible internal >>>> fragmentation so we honour that request. >>>> >>>> Otherwise, the return value of arch_wants_pte_order() is used. For vmas >>>> that have not explicitly opted-in to use transparent hugepages (e.g. >>>> where thp=madvise and the vma does not have MADV_HUGEPAGE), then >>>> arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is >>>> bigger). This allows for a performance boost without requiring any >>>> explicit opt-in from the workload while limitting internal >>>> fragmentation. >>>> >>>> If the preferred order can't be used (e.g. because the folio would >>>> breach the bounds of the vma, or because ptes in the region are already >>>> mapped) then we fall back to a suitable lower order; first >>>> PAGE_ALLOC_COSTLY_ORDER, then order-0. >>>> >>>> arch_wants_pte_order() can be overridden by the architecture if desired. >>>> Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous >>>> set of ptes map physically contigious, naturally aligned memory, so this >>>> mechanism allows the architecture to optimize as required. >>>> >>>> Here we add the default implementation of arch_wants_pte_order(), used >>>> when the architecture does not define it, which returns -1, implying >>>> that the HW has no preference. In this case, mm will choose it's own >>>> default order. >>>> >>>> Signed-off-by: Ryan Roberts >>>> --- >>>> include/linux/pgtable.h | 13 ++++ >>>> mm/Kconfig | 10 +++ >>>> mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- >>>> 3 files changed, 172 insertions(+), 17 deletions(-) >>>> >>>> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h >>>> index 5063b482e34f..2a1d83775837 100644 >>>> --- a/include/linux/pgtable.h >>>> +++ b/include/linux/pgtable.h >>>> @@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) >>>> } >>>> #endif >>>> >>>> +#ifndef arch_wants_pte_order >>>> +/* >>>> + * Returns preferred folio order for pte-mapped memory. Must be in range [0, >>>> + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios >>>> + * to be at least order-2. Negative value implies that the HW has no preference >>>> + * and mm will choose it's own default order. >>>> + */ >>>> +static inline int arch_wants_pte_order(void) >>>> +{ >>>> + return -1; >>>> +} >>>> +#endif >>>> + >>>> #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR >>>> static inline pte_t ptep_get_and_clear(struct mm_struct *mm, >>>> unsigned long address, >>>> diff --git a/mm/Kconfig b/mm/Kconfig >>>> index 09130434e30d..fa61ea160447 100644 >>>> --- a/mm/Kconfig >>>> +++ b/mm/Kconfig >>>> @@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA >>>> >>>> source "mm/damon/Kconfig" >>>> >>>> +config LARGE_ANON_FOLIO >>>> + bool "Allocate large folios for anonymous memory" >>>> + depends on TRANSPARENT_HUGEPAGE >>>> + default n >>>> + help >>>> + Use large (bigger than order-0) folios to back anonymous memory where >>>> + possible, even for pte-mapped memory. This reduces the number of page >>>> + faults, as well as other per-page overheads to improve performance for >>>> + many workloads. >>>> + >>>> endmenu >>>> diff --git a/mm/memory.c b/mm/memory.c >>>> index 01f39e8144ef..64c3f242c49a 100644 >>>> --- a/mm/memory.c >>>> +++ b/mm/memory.c >>>> @@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) >>>> return ret; >>>> } >>>> >>>> +static bool vmf_pte_range_changed(struct vm_fault *vmf, int nr_pages) >>>> +{ >>>> + int i; >>>> + >>>> + if (nr_pages == 1) >>>> + return vmf_pte_changed(vmf); >>>> + >>>> + for (i = 0; i < nr_pages; i++) { >>>> + if (!pte_none(ptep_get_lockless(vmf->pte + i))) >>>> + return true; >>>> + } >>>> + >>>> + return false; >>>> +} >>>> + >>>> +#ifdef CONFIG_LARGE_ANON_FOLIO >>>> +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ >>>> + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) >>>> + >>>> +static int anon_folio_order(struct vm_area_struct *vma) >>>> +{ >>>> + int order; >>>> + >>>> + /* >>>> + * If THP is explicitly disabled for either the vma, the process or the >>>> + * system, then this is very likely intended to limit internal >>>> + * fragmentation; in this case, don't attempt to allocate a large >>>> + * anonymous folio. >>>> + * >>>> + * Else, if the vma is eligible for thp, allocate a large folio of the >>>> + * size preferred by the arch. Or if the arch requested a very small >>>> + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, >>>> + * which still meets the arch's requirements but means we still take >>>> + * advantage of SW optimizations (e.g. fewer page faults). >>>> + * >>>> + * Finally if thp is enabled but the vma isn't eligible, take the >>>> + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. >>>> + * This ensures workloads that have not explicitly opted-in take benefit >>>> + * while capping the potential for internal fragmentation. >>>> + */ >>>> + >>>> + if ((vma->vm_flags & VM_NOHUGEPAGE) || >>>> + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || >>>> + !hugepage_flags_enabled()) >>>> + order = 0; >>>> + else { >>>> + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); >>>> + >>>> + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) >>>> + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); >>>> + } >>>> + >>>> + return order; >>>> +} >>>> + >>>> +static int alloc_anon_folio(struct vm_fault *vmf, struct folio **folio) >>>> +{ >>>> + int i; >>>> + gfp_t gfp; >>>> + pte_t *pte; >>>> + unsigned long addr; >>>> + struct vm_area_struct *vma = vmf->vma; >>>> + int prefer = anon_folio_order(vma); >>>> + int orders[] = { >>>> + prefer, >>>> + prefer > PAGE_ALLOC_COSTLY_ORDER ? PAGE_ALLOC_COSTLY_ORDER : 0, >>>> + 0, >>>> + }; >>>> + >>>> + *folio = NULL; >>>> + >>>> + if (vmf_orig_pte_uffd_wp(vmf)) >>>> + goto fallback; >>> >>> I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here; >>> otherwise UFFD would miss VM_UFFD_MISSING/MINOR. >> >> I don't think this is the case. As far as I can see, VM_UFFD_MINOR only applies >> to shmem and hugetlb. > > Correct, but we don't have a helper to check against (VM_UFFD_WP | > VM_UFFD_MISSING). Reusing userfaultfd_armed() seems cleaner to me or > even future proof. > >> VM_UFFD_MISSING is checked under the PTL and if set on the >> VMA, then it is handled without mapping the folio that was just allocated: >> >> /* Deliver the page fault to userland, check inside PT lock */ >> if (userfaultfd_missing(vma)) { >> pte_unmap_unlock(vmf->pte, vmf->ptl); >> folio_put(folio); >> return handle_userfault(vmf, VM_UFFD_MISSING); >> } >> >> So we are racing to allocate a large folio; if the vma later turns out to have >> MISSING handling registered, we drop the folio and handle it, else we map the >> large folio. > > Yes, then we have allocated a large folio (with great effort if under > heavy memory pressure) for nothing. > >> The vmf_orig_pte_uffd_wp() *is* required because we need to individually check >> each PTE for the uffd_wp bit and fix it up. > > This is not correct: we cannot see a WP PTE before you see > VM_UFFD_WP. So checking VM_UFFD_WP is perfectly safe. I think you misunderstood me; I was trying to say that assuming we don't check userfaultfd_armed() then we need the vmf_orig_pte_uffd_wp() check because we need to ensure that the marker gets preserved for that specific pte and we can only do that if we are operating on a single pte. > > The reason we might want to check individual PTEs is because WP can be > done to a subrange of a VMA that has VM_UFFD_WP, which I don't think > is the common case and worth considering here. But if you want to keep > it, that's fine with me. Without some comments, the next person might > find these two checks confusing though, if you plan to add both. I'm not proposing we need both checks. > >> So I think the code is correct, but perhaps it is safer/simpler to always avoid >> allocating a large folio if the vma is registered for uffd in the way you >> suggest? I don't know enough about uffd to form a strong opinion either way. > > Yes, it's not about correctness. Just a second thought about not > allocating large folios unnecessarily when possible. OK, I misunderstood you; I thought your original point is about correctness. Anyway, you have convinced me that we should s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ on the grounds that trying hard to allocate a high order folio is almost always going to be a waste of effort. I'll change this in the next version. Thanks, Ryan