From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 374A5C04A6A for ; Thu, 10 Aug 2023 17:03:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADBF06B0072; Thu, 10 Aug 2023 13:03:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8C166B0075; Thu, 10 Aug 2023 13:03:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 953F08D0001; Thu, 10 Aug 2023 13:03:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 84E316B0072 for ; Thu, 10 Aug 2023 13:03:18 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 298621A112B for ; Thu, 10 Aug 2023 17:03:11 +0000 (UTC) X-FDA: 81108815382.30.B6E9F20 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 2C4AB40099 for ; Thu, 10 Aug 2023 17:02:03 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=og3puj3C; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691686924; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VgJtiDLaOXreWlpDuZr8GRrlR/LiYN/zYmNHHchv/Qw=; b=7MBTPazOFTe4wGcN1qgyGlo1czCmVfcRNGFvXGFkrquvHWlHkG2RkorwKADj1qg6DFUESL WQz0UcjVE6L3/IpHyZmDocFBp3Jt80HG4RGVAMyc7HJIZpCd8dggKi+gih22GZIM1XE+t4 qzx6Bw0v0JSWVNno274qRWSOB8Fu4yQ= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=og3puj3C; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691686924; a=rsa-sha256; cv=none; b=vudq81yulfSm9yPSnprEHZPz8KHdExFgmgWdOLiOofZmpioiPbwwwz9HccESee+kBSxMwA pDLDR1NYImD0JHUdsr61donY/i35pajLasG2yfdzAOFry+smaqmuTAEbWHi13bthFAXU77 vFN7kv/jOcnrzsSyFoLEy95Qnza+DtU= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-40c72caec5cso14961cf.0 for ; Thu, 10 Aug 2023 10:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691686923; x=1692291723; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=VgJtiDLaOXreWlpDuZr8GRrlR/LiYN/zYmNHHchv/Qw=; b=og3puj3CafyyWKCAiIg9b89DciXir2SQFxLRgBKas7ojE9Q/T9oQxRgV+a95R+3DYR sr5rNijrJQRvnfC1uOSkn0PMbZ1WORGkXmY+zEaCLE+Ad59+e8kvO0jEoKUclQBbaSc+ VmN/Wm+G6zwORTMsStCElnKq+GrwAE7zWaZrlb95zaw8VDwqGOj6YP47T73/KPnPwNma yi72fQIZ7k5ypE5t9IOnwoxWzlqPCUkd+5QE6bRhUJRKFqRq2Li/R9mY61VxBLqkuHt9 XRoFw0LGIwbuNMilIBX4IiOKLa7a7mc3U+//yaPoF4/m2rbl01imFVB4+HtKx2RIACNl UMuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691686923; x=1692291723; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VgJtiDLaOXreWlpDuZr8GRrlR/LiYN/zYmNHHchv/Qw=; b=VCVmBeYpru4lPQtMM+Nu9z4/6O11HhifOw+wp4WUBhz9PEgMXp4LhTxB9RVI3N/roJ VpNtgkgpD9uBcDJICimhieugh0RWpDommSKQj64xxUTp7dkBgMnnMbAqpVzTv9yWRHPN GtIYiGJ01wc2zqcrcvHq6fq6L3Qtqhe7zRwiC8Q7cYzdwVi9MhFE3AKG13p0RLgRrg7k kGfxzQdcQ0f2X5p/3zNnmgko5s/bOXDSv2X5l45E7kuE1Mk/TZCtLa8F8kj8wgcahu/w QQi8tXQiybKOA5H8GlbGRpX/zLEFwQlEdM5tkTWZCgkAihLfaa7n9R0s0P9Ipm8opDlm Vq+w== X-Gm-Message-State: AOJu0YyptFgFzVlGSAway4CVwfuHt+Y96+rXxa55jMeNazoiPDP6vuc9 o5c3btne8FdIau19qrT0SoXcdmDOgDwac/9B98YPqg== X-Google-Smtp-Source: AGHT+IFV6tfaj0UQ8ATzcITh54+A+d7nOCEmyXkaBukUcN1DOUffRbHtEL+OKi3cV6jmq7AY4LRd00Y/QQNAxU6IzHg= X-Received: by 2002:ac8:5a8c:0:b0:3f8:5b2:aef0 with SMTP id c12-20020ac85a8c000000b003f805b2aef0mr6997qtc.24.1691686923043; Thu, 10 Aug 2023 10:02:03 -0700 (PDT) MIME-Version: 1.0 References: <20230810142942.3169679-1-ryan.roberts@arm.com> <20230810142942.3169679-4-ryan.roberts@arm.com> In-Reply-To: <20230810142942.3169679-4-ryan.roberts@arm.com> From: Yu Zhao Date: Thu, 10 Aug 2023 11:01:26 -0600 Message-ID: Subject: Re: [PATCH v5 3/5] mm: LARGE_ANON_FOLIO for improved performance To: Ryan Roberts Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Catalin Marinas , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2C4AB40099 X-Stat-Signature: swdj8z8i71gid9qd7s5rxc65udk7s85w X-HE-Tag: 1691686923-108381 X-HE-Meta: U2FsdGVkX19BwzLuhJhvnXlEYInj0lSa1xEd/4ettOPOdQvqCpGbm1E2yYzULpfMtO/qMeblo8uZ9KkK3kCIhsUGPcL7TSHEmlxfIzau0LCu88zLafh6e5pbTxiQDrVJvywwEEH6zqcAadd+nAqA5L0HnCP0hCDb364kNWIevxtdaFryDFKt/8NiqMvKDxRdy8sCCxLlv1yl9icOu/TB/aDjk5yniodFSWw88j7YjV20QeEPCsn0ct7qTRzczQQj48oaX1bdshkBAKcIQQzuGFllE+xLyRpPUQfDM16RxDoEjDO6VBKZs2ZQ+Vy4Y98we5AVKXZC+szY/zoO99oh8obwoXMPqNiyM/SHn7FXeaJmpnxlDyWkCsb2mK3t0Oi32++JZjECN/Zho94LGLnLwcU1T21dEhpBb+zFYN4jOP9eLjx4uSm2JgLBkat08IcnpKBZn5ol27skmCJ96EoB/MdZN6I4eeP4fvN8UFtonCPJlHafx+F0dhJtsXX7FF8RyIHJP4ep/P1TLXIHibZZ1S45DY7f6obDeadRsP9xYeldevel2Fl1kHQ/vvcdbYBTCkLp15fgy2y8PxKm+bkBbNN/ekHPBJXoQ2/vbPalRBfLKy2A55GbTPDwO1Eu5yGH4MsuDbYF/ON0czGLQtLYmUWt2ZnoZmCPCkn26lwnbUEGwucEWDNHhgWNdemWXx0KQi+bGxak+yRl/Q7aur1GQYA9eXW0kecIWwriftMEK/IbbbGY2NvpNu+qjVLRtmul7ycheyKZ1XXrlv98Snz/lIwbJsFSCN3rDxxUVJVI4+qPgVZ7yDTnC70xHCHA5ofTFSYe0xGVmKIq6mpOq+DAolhLfbeqckd1RvBKqZRItZUQ/2MHaeQXOqEyzKhaOPZXilVpjk7qjHwXjGiypjO6aH7taShpHyowktdJ+5Swggef6LNLcxbXRFliP5VHPAgATJmi1T3XNeJOfGrvt4e vR/aNTt3 4tiZplo/JMCn+Jv65wnwupngUhCsCjkGwTjgzfBcEOTeZPM0EwFRLdNrJWpL78ah02pm8sKgVRLpbR19dRBaPXspKZrAYjoHfCUiwN12XD1u1jpMX5zLzDnXej0kJrehuEs6bE+gbcbrvgOMGRwn9uhrKjCUet5LmFWBZaDo2wset/v/XJIWc/UrnjUDjJYTc9/PL0BcmdoQy3GBRi8pRHGCgwIbAXq0odACUNZkRyLW84GgdOlHShqRCy37ozgL40gmVP066o/KP15Q8p5+USgbxwEwKQcmb0AhkBXKysQyhvLvGGll/feYtcqqCa4v74eVEjmNhGIuqgArhg+zupBMR1UXd2vmjUMhYutUKT1Y/zWEO3eNy2c/mJajRb6qsBxfP4wEJDiNKnDmJjQpfwLU8/9nRsQNUKI3t7Qipg2WuQ+f2wiKMQWSphtQSRLpOBkaA6Dq+nAkhhRiNKhV5vQd+Ky5mBrSlyG2Fi4DlAdJGXKY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 10, 2023 at 8:30=E2=80=AFAM Ryan Roberts = wrote: > > Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be > allocated in large folios of a determined order. All pages of the large > folio are pte-mapped during the same page fault, significantly reducing > the number of page faults. The number of per-page operations (e.g. ref > counting, rmap management lru list management) are also significantly > reduced since those ops now become per-folio. > > The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, > which defaults to disabled for now; The long term aim is for this to > defaut to enabled, but there are some risks around internal > fragmentation that need to be better understood first. > > Large anonymous folio (LAF) allocation is integrated with the existing > (PMD-order) THP and single (S) page allocation according to this policy, > where fallback (>) is performed for various reasons, such as the > proposed folio order not fitting within the bounds of the VMA, etc: > > | prctl=3Ddis | prctl=3Dena | prctl=3Dena | prctl= =3Dena > | sysfs=3DX | sysfs=3Dnever | sysfs=3Dmadvise | sysfs= =3Dalways > ----------------|-----------|-------------|---------------|------------- > no hint | S | LAF>S | LAF>S | THP>LAF>S > MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S > MADV_NOHUGEPAGE | S | S | S | S > > This approach ensures that we don't violate existing hints to only > allocate single pages - this is required for QEMU's VM live migration > implementation to work correctly - while allowing us to use LAF > independently of THP (when sysfs=3Dnever). This makes wide scale > performance characterization simpler, while avoiding exposing any new > ABI to user space. > > When using LAF for allocation, the folio order is determined as follows: > The return value of arch_wants_pte_order() is used. For vmas that have > not explicitly opted-in to use transparent hugepages (e.g. where > sysfs=3Dmadvise and the vma does not have MADV_HUGEPAGE or sysfs=3Dnever)= , > then arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever > is bigger). This allows for a performance boost without requiring any > explicit opt-in from the workload while limitting internal > fragmentation. > > If the preferred order can't be used (e.g. because the folio would > breach the bounds of the vma, or because ptes in the region are already > mapped) then we fall back to a suitable lower order; first > PAGE_ALLOC_COSTLY_ORDER, then order-0. > > arch_wants_pte_order() can be overridden by the architecture if desired. > Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous > set of ptes map physically contigious, naturally aligned memory, so this > mechanism allows the architecture to optimize as required. > > Here we add the default implementation of arch_wants_pte_order(), used > when the architecture does not define it, which returns -1, implying > that the HW has no preference. In this case, mm will choose it's own > default order. > > Signed-off-by: Ryan Roberts > --- > include/linux/pgtable.h | 13 ++++ > mm/Kconfig | 10 +++ > mm/memory.c | 144 +++++++++++++++++++++++++++++++++++++--- > 3 files changed, 158 insertions(+), 9 deletions(-) > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index 222a33b9600d..4b488cc66ddc 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -369,6 +369,19 @@ static inline bool arch_has_hw_pte_young(void) > } > #endif > > +#ifndef arch_wants_pte_order > +/* > + * Returns preferred folio order for pte-mapped memory. Must be in range= [0, > + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires larg= e folios > + * to be at least order-2. Negative value implies that the HW has no pre= ference > + * and mm will choose it's own default order. > + */ > +static inline int arch_wants_pte_order(void) > +{ > + return -1; > +} > +#endif > + > #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR > static inline pte_t ptep_get_and_clear(struct mm_struct *mm, > unsigned long address, > diff --git a/mm/Kconfig b/mm/Kconfig > index 721dc88423c7..a1e28b8ddc24 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -1243,4 +1243,14 @@ config LOCK_MM_AND_FIND_VMA > > source "mm/damon/Kconfig" > > +config LARGE_ANON_FOLIO > + bool "Allocate large folios for anonymous memory" > + depends on TRANSPARENT_HUGEPAGE > + default n > + help > + Use large (bigger than order-0) folios to back anonymous memory= where > + possible, even for pte-mapped memory. This reduces the number o= f page > + faults, as well as other per-page overheads to improve performa= nce for > + many workloads. > + > endmenu > diff --git a/mm/memory.c b/mm/memory.c > index d003076b218d..bbc7d4ce84f7 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -4073,6 +4073,123 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > return ret; > } > > +static bool vmf_pte_range_changed(struct vm_fault *vmf, int nr_pages) > +{ > + int i; > + > + if (nr_pages =3D=3D 1) > + return vmf_pte_changed(vmf); > + > + for (i =3D 0; i < nr_pages; i++) { > + if (!pte_none(ptep_get_lockless(vmf->pte + i))) > + return true; > + } > + > + return false; > +} > + > +#ifdef CONFIG_LARGE_ANON_FOLIO > +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ > + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SH= IFT) > + > +static int anon_folio_order(struct vm_area_struct *vma) > +{ > + int order; > + > + /* > + * If the vma is eligible for thp, allocate a large folio of the = size > + * preferred by the arch. Or if the arch requested a very small s= ize or > + * didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, which= still > + * meets the arch's requirements but means we still take advantag= e of SW > + * optimizations (e.g. fewer page faults). > + * > + * If the vma isn't eligible for thp, take the arch-preferred siz= e and > + * limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. This ensures worklo= ads > + * that have not explicitly opted-in take benefit while capping t= he > + * potential for internal fragmentation. > + */ > + > + order =3D max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); > + > + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) > + order =3D min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); > + > + return order; > +} I don't understand why we still want to keep ANON_FOLIO_MAX_ORDER_UNHINTED. 1. It's not used, since no archs at the moment implement arch_wants_pte_order() that returns >64KB. 2. As far as I know, there is no plan for any arch to do so. 3. Again, it seems to me the rationale behind ANON_FOLIO_MAX_ORDER_UNHINTED isn't convincing at all. Can we introduce ANON_FOLIO_MAX_ORDER_UNHINTED if/when needed please? Also you made arch_wants_pte_order() return -1, and I acknowledged [1]: Thanks: -1 actually is better than 0 (what I suggested) for the obvious reason. I thought we were on the same page, i.e., the "obvious reason" is that h/w might prefer 0. But here you are not respecting 0. But then why -1? [1] https://lore.kernel.org/linux-mm/CAOUHufZ7HJZW8Srwatyudf=3DFbwTGQtyq4Dy= L2SHwSg37N_Bo_A@mail.gmail.com/