From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 703EDC001DB for ; Fri, 4 Aug 2023 03:35:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF34F2802C2; Thu, 3 Aug 2023 23:35:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA4772802BC; Thu, 3 Aug 2023 23:35:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943D02802C2; Thu, 3 Aug 2023 23:35:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 825F22802BC for ; Thu, 3 Aug 2023 23:35:58 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 46F2340944 for ; Fri, 4 Aug 2023 03:35:58 +0000 (UTC) X-FDA: 81085008396.15.7DB9C9C Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf09.hostedemail.com (Postfix) with ESMTP id 86CA8140005 for ; Fri, 4 Aug 2023 03:35:55 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=VMdu7EnY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691120155; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/sb6r+VGT3TmM7MgKTQFNtqFfnqnNxxPfg5LfA7ZooI=; b=bH8H5qA5mVLp3GJ1Gqp19ifMwx42aPrUFhQQHJYiUf3P7z63TeUcM3/PUZmWpS0sUQj4OT eE1MeIKjZ1XmW5WUQCpzKAl0ups30QKbXXXb2Gl6KYsO+OG52RFQfBbTLPDMmZ3GJvY3IN b/2+fwyzfL58jGHrY6gS/Bu219X06I4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=VMdu7EnY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691120155; a=rsa-sha256; cv=none; b=Yw5pI5vYxfk9uMHjgcsh2PT0JHHFLY/+iNXptJ63bQL+SDhBbhrVfw97mPlwWxADF7lDL2 CWLnialF/3jTw7u+0pjwuSLSvHdV6linlwtErsUPd/A/ueOyIVFc5lorXwZVs/J2jtyRe0 0CAvIo25IJM98cW9Sz3lz85M+I70jqg= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-40c72caec5cso171391cf.0 for ; Thu, 03 Aug 2023 20:35:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691120154; x=1691724954; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/sb6r+VGT3TmM7MgKTQFNtqFfnqnNxxPfg5LfA7ZooI=; b=VMdu7EnYBlUsEreehBhYfn7bJdRZtpN/7z9eSJ7nrOK+7D71m/16Ejz+CN05lG3f31 rWyLzpBiNbFnFXBXD68ihHDUJ/kantz1hOauIc8/5T/SgZ+7rvQYNZQGNUNWGo0CbI43 4ZjqznBO8WaZmCdFHDAG0r+6cIPA8UmPgGCUPiPSVR6Lpfl5NTHmkWtPxXoDp8FW05H3 mrc0su3pzc2qpIOPbiYu69P5URJAq7gIYvQq/V+eIBfmrOWjTI48Qe1clF+FrgZZ2Huc CpMvZfpf+QSqieQbnnS0KOAc5T1Vc5Tii5cftwBdaSUTab7cbC8gW7TdB4JtyEUHzszx e8HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691120154; x=1691724954; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/sb6r+VGT3TmM7MgKTQFNtqFfnqnNxxPfg5LfA7ZooI=; b=FzBpgaRHI2aSWV1kXwVUYVL7zWK+KyE4mtMuEiA6bC0fpSmhg9JPGKBlDT8MyMoMHr gdccgOMXccHqYtYPSKNk2sWd3hh6k6VOF0W4q/ujWZIlLCOvG/AVeroN1JOa+O2SNNZb OeJnD5q41yt+O/FmCwX1GKWg+/38F6k9J8U1vtLKx3izXzvuugRNQGOrsu1D9+DGgRbO yAw9H0CFipXR2mmMDaowTq/37XDfKm1Wm7ChdBYlsGLfYLLmpqdtz6DgyXKnVc8QCuvP g7lOpkq8g4OiGzd+tvia9f5VXN2GXPYj7QDcDGiO14l9SxyjcNjxIEkp7acI8A2WhWuG y43A== X-Gm-Message-State: AOJu0YxKz+MRtXt7PpOAGLHrz1+eQMJKcdpclt+6n+H9n5U83fRrpUlH QTTHO/Xvrhlaxf/5lSCQRxUJbjxH+Bvwtk58BYQ8LA== X-Google-Smtp-Source: AGHT+IFJ1QV7iXowTdRosjlQw57fCZaf06cnVbK6VJ5K0A8byUES61rASuZorUZ6GaJqf+Ejaa8A47vrGV7+TJPzok4= X-Received: by 2002:ac8:5710:0:b0:3f9:56c:1129 with SMTP id 16-20020ac85710000000b003f9056c1129mr88588qtw.5.1691120154437; Thu, 03 Aug 2023 20:35:54 -0700 (PDT) MIME-Version: 1.0 References: <20230726095146.2826796-1-ryan.roberts@arm.com> <20230726095146.2826796-3-ryan.roberts@arm.com> <20230803142154.nvgkavg33uyn6f72@box.shutemov.name> <44C394AF-A9E1-499F-AE3F-7EAEA03B19DB@nvidia.com> In-Reply-To: <44C394AF-A9E1-499F-AE3F-7EAEA03B19DB@nvidia.com> From: Yu Zhao Date: Thu, 3 Aug 2023 21:35:18 -0600 Message-ID: Subject: Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance To: Zi Yan Cc: "Kirill A. Shutemov" , Ryan Roberts , Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Luis Chamberlain , Itaru Kitayama , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: rwqrc9d4atm69triqoaiwogb9mf9zptg X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 86CA8140005 X-HE-Tag: 1691120155-401836 X-HE-Meta: U2FsdGVkX1+2WAUo+fyGVKRklOA2Q1JaReg+w9UvR9BdfvhBXMS02FF5QGgpk05jTvSHyt1Z6AjmIrAMjmJJgbbo7quR0lPNJA1cNUjD1wmLGchK5tzWEDAIFpXTI0G8HS9NJ9spO4CbRYVaQLJYYsBkzC3Rt9rdabUbGHDtLf9+DJGOw9DoJg3qh/i0f3W4BuDPVK2ke0lkcwMDpilrPmrQvtYengpSc1Fb4igmGSXHoLdd5KrAaMtQFpQYXABnd0YQxjkYql6xVb2x5a5qFuLyil8umL36sOKW44d1yRzMg4O84EOnWQlZdnCGdLzAh9/FT9fzdtapmNg43cZSbxQh4R9S0TJWx5+TNZH75N3RG0GA6CLFWRPrdmJWdDqMi1/YKLUUrCsTT2B+FiPjEa9WI+Ti4JW/dcrns//qjc9Oe5Y/SEdtjpNo+LvJY6pAQ83Z7et8zSk8236Kqk+1ck35R/1+nQF0cdpjOj/jHJyIOauAxLxqsBYRnbGS1fFTUVhYnPEnaVyr/1NYWEyygZqgKVTVyZAKkQhHw2TGRi6cuxadpo72DNfP5QDfr3BEspY2WKIvvW48lSq2vlvgH4wr3S0HkQy7i9eWn56tjl3zk4ZBzhAbd6LvJvCK/MAK0QzGVW5vS8n6TypooR+T5oMdZHjujku0sy6ZnaGTk/4CylkMiV5s5OXCUztMGzP9DCbhFXkFg6NYS1B6eja+QypooE/tflCxe7XbVKTV7hsS8MMhlAilhyh9yciFKuKPZ8HMv1vUaSX9PIWNLlnZrGytOvzIK1Q7ucEbddCSiqTLO5Ef3iBNkpSkwSo17uUFoQLv/wdBsq+/ZJNxMzTAAuqqfNfggfu6dBPn5MGY0DivsZzfE66VQ/2XL6xju1vpg8RLQR3sauBNThEU8Q2NUlLqAHohhA+NeXR2AF8hfDxGevUyy8mui8DalT+bkZ6UDtEZyWaZWAwd3tbSBE1 zWPdx5fv KpezrMhcoesQ11WA4lYjvqnAFgRhOPxTTsUybpS08iJEFYS9BVcWMmlWACWGi4GBdpzn/jCqehypW14DtQWQVstsqqz6A9hUvL5uvwrs64g5ApQJr2xuvlMUj0Pw9mFRHwtkbUeFODdntw++XD0SaHKgm7FwQbseeo2bPZb+Ce4vGKdWA0ZRlZxBIbUBA8O8Yvp+TAQ26781r/DBXaYx3835uHT7bvI0+EA/j+vkB+GmeyChe7nJ+nqdgr8GRKE0JTeKLedgV0/YuIVqIOtM48J2eYO96IVWCAGU77TP1xeyZqAg6OIzNlEO/JOrQq3bshiOgaSaDlwb2y3rA0mad1KHT/tJxcYS7QpFWn4EH/BR9YZ9LjA/jWp3RE02E/sEwBYI/avpSEbNuL9M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 3, 2023 at 8:16=E2=80=AFPM Zi Yan wrote: > > On 3 Aug 2023, at 20:19, Yu Zhao wrote: > > > On Thu, Aug 3, 2023 at 8:27=E2=80=AFAM Kirill A. Shutemov > > wrote: > >> > >> On Thu, Aug 03, 2023 at 01:43:31PM +0100, Ryan Roberts wrote: > >>> + Kirill > >>> > >>> On 26/07/2023 10:51, Ryan Roberts wrote: > >>>> Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to= be > >>>> allocated in large folios of a determined order. All pages of the la= rge > >>>> folio are pte-mapped during the same page fault, significantly reduc= ing > >>>> the number of page faults. The number of per-page operations (e.g. r= ef > >>>> counting, rmap management lru list management) are also significantl= y > >>>> reduced since those ops now become per-folio. > >>>> > >>>> The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, > >>>> which defaults to disabled for now; The long term aim is for this to > >>>> defaut to enabled, but there are some risks around internal > >>>> fragmentation that need to be better understood first. > >>>> > >>>> When enabled, the folio order is determined as such: For a vma, proc= ess > >>>> or system that has explicitly disabled THP, we continue to allocate > >>>> order-0. THP is most likely disabled to avoid any possible internal > >>>> fragmentation so we honour that request. > >>>> > >>>> Otherwise, the return value of arch_wants_pte_order() is used. For v= mas > >>>> that have not explicitly opted-in to use transparent hugepages (e.g. > >>>> where thp=3Dmadvise and the vma does not have MADV_HUGEPAGE), then > >>>> arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is > >>>> bigger). This allows for a performance boost without requiring any > >>>> explicit opt-in from the workload while limitting internal > >>>> fragmentation. > >>>> > >>>> If the preferred order can't be used (e.g. because the folio would > >>>> breach the bounds of the vma, or because ptes in the region are alre= ady > >>>> mapped) then we fall back to a suitable lower order; first > >>>> PAGE_ALLOC_COSTLY_ORDER, then order-0. > >>>> > >>> > >>> ... > >>> > >>>> +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ > >>>> + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_S= HIFT) > >>>> + > >>>> +static int anon_folio_order(struct vm_area_struct *vma) > >>>> +{ > >>>> + int order; > >>>> + > >>>> + /* > >>>> + * If THP is explicitly disabled for either the vma, the process= or the > >>>> + * system, then this is very likely intended to limit internal > >>>> + * fragmentation; in this case, don't attempt to allocate a larg= e > >>>> + * anonymous folio. > >>>> + * > >>>> + * Else, if the vma is eligible for thp, allocate a large folio = of the > >>>> + * size preferred by the arch. Or if the arch requested a very s= mall > >>>> + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORD= ER, > >>>> + * which still meets the arch's requirements but means we still = take > >>>> + * advantage of SW optimizations (e.g. fewer page faults). > >>>> + * > >>>> + * Finally if thp is enabled but the vma isn't eligible, take th= e > >>>> + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHI= NTED. > >>>> + * This ensures workloads that have not explicitly opted-in take= benefit > >>>> + * while capping the potential for internal fragmentation. > >>>> + */ > >>>> + > >>>> + if ((vma->vm_flags & VM_NOHUGEPAGE) || > >>>> + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || > >>>> + !hugepage_flags_enabled()) > >>>> + order =3D 0; > >>>> + else { > >>>> + order =3D max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_= ORDER); > >>>> + > >>>> + if (!hugepage_vma_check(vma, vma->vm_flags, false, true,= true)) > >>>> + order =3D min(order, ANON_FOLIO_MAX_ORDER_UNHINT= ED); > >>>> + } > >>>> + > >>>> + return order; > >>>> +} > >>> > >>> > >>> Hi All, > >>> > >>> I'm writing up the conclusions that we arrived at during discussion i= n the THP > >>> meeting yesterday, regarding linkage with exiting THP ABIs. It would = be great if > >>> I can get explicit "agree" or disagree + rationale from at least Davi= d, Yu and > >>> Kirill. > >>> > >>> In summary; I think we are converging on the approach that is already= coded, but > >>> I'd like confirmation. > >>> > >>> > >>> > >>> The THP situation today > >>> ----------------------- > >>> > >>> - At system level: THP can be set to "never", "madvise" or "always" > >>> - At process level: THP can be "never" or "defer to system setting" > >>> - At VMA level: no-hint, MADV_HUGEPAGE, MADV_NOHUGEPAGE > >>> > >>> That gives us this table to describe how a page fault is handled, acc= ording to > >>> process state (columns) and vma flags (rows): > >>> > >>> | never | madvise | always > >>> ----------------|-----------|-----------|----------- > >>> no hint | S | S | THP>S > >>> MADV_HUGEPAGE | S | THP>S | THP>S > >>> MADV_NOHUGEPAGE | S | S | S > >>> > >>> Legend: > >>> S allocate single page (PTE-mapped) > >>> LAF allocate lage anon folio (PTE-mapped) > >>> THP allocate THP-sized folio (PMD-mapped) > >>>> fallback (usually because vma size/alignment insufficient for fo= lio) > >>> > >>> > >>> > >>> Principles for Large Anon Folios (LAF) > >>> -------------------------------------- > >>> > >>> David tells us there are use cases today (e.g. qemu live migration) w= hich use > >>> MADV_NOHUGEPAGE to mean "don't fill any PTEs that are not explicitly = faulted" > >>> and these use cases will break (i.e. functionally incorrect) if this = request is > >>> not honoured. > >>> > >>> So LAF must at least honour MADV_NOHUGEPAGE to prevent breaking exist= ing use > >>> cases. And once we do this, then I think the least confusing thing is= for it to > >>> also honor the "never" system/process state; so if either the system,= process or > >>> vma has explicitly opted-out of THP, then LAF should also be bypassed= . > >>> > >>> Similarly, any case that would previously cause the allocation of PMD= -sized THP > >>> must continue to be honoured, else we risk performance regression. > >>> > >>> That leaves the "madvise/no-hint" case, and all THP fallback paths du= e to the > >>> VMA not being correctly aligned or sized to hold a PMD-sized mapping.= In these > >>> cases, we will attempt to use LAF first, and fallback to single page = if the vma > >>> size/alignment doesn't permit it. > >>> > >>> | never | madvise | always > >>> ----------------|-----------|-----------|----------- > >>> no hint | S | LAF>S | THP>LAF>S > >>> MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S > >>> MADV_NOHUGEPAGE | S | S | S > >>> > >>> I think this (perhaps conservative) approach will be the least surpri= sing to > >>> users. And is the policy that is already implemented in this patch. > >> > >> This looks very reasonable. > >> > >> The only questionable field is no-hint/madvise. I can argue for both L= AF>S > >> and S here. I think LAF>S is fine as long as we are not too aggressive > >> with allocation order. > >> > >> I think we need to work on eliminating reasons for users to set 'never= '. > >> If something behaves better with 'never' kernel has failed user. > >> > >>> Downsides of this policy > >>> ------------------------ > >>> > >>> As Yu and Yin have pointed out, there are some workloads which do not= perform > >>> well with THP, due to large fault latency or memory wastage, etc. But= which > >>> _may_ still benefit from LAF. By taking the conservative approach, we= exclude > >>> these workloads from benefiting automatically. > >> > >> Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF = is > >> fine? > > > > No, it's not. And no one said order-8 LAF is fine :) The starting > > order for LAF that we have been discussing is at most 64KB (vs 2MB > > THP). For my taste, it's still too large. I'd go with 32KB/16KB. > > I guess it is because ARM64 supports contig PTE at 64KB, so getting > large anon folio at 64KB on ARM64 would have an extra perf boost when > set contig PTE bits patch is also in. > > On x86_64, 32KB might be better on AMD CPUs that support PTE clustering, > which would use a single TLB entry for 8 contiguous 4KB pages and is > done at microarchitecture level without additional software changes. > > > > > However, the same argument can be used to argue against the policy > > Ryan listed above: why order-10 LAF is ok for madvise but not order-11 > > (which becomes "always")? > > > > I'm strongly against this policy for two practical reasons I learned > > from tuning THPs in our data centers: > > Do you mind writing down your policy? That would help us see and discuss > the difference. > > > 1. By doing the above, we are blurring the lines between those values > > and making real-world performance tuning extremely hard if not > > impractice. > > 2. As I previously pointed out: if we mix LAFs with THPs, we actually > > risk causing performance regressions because giving smaller VMAs LAFs > > can deprive large VMAs of THPs. > > I think these two reasons are based on that we do not have a reasonable > LAF+THP allocation and management policy and we do not fully understand > the pros and cons of using LAF and mixing LAF with THP. It would be > safe to separate LAF and THP. By doing so, > > 1. for workloads do not benefit from THP, we can turn on LAF alone to > see if there is a performance boost and further understand if LAF > hurts, has no impactor , or improves the performance of these workloads. > > 2. for workloads benefit from THP, we can also turn on LAF separately > to understand the performance impact of LAF (hurt, no change, or improve)= . This is basically what I've been suggesting. We should have a separate knob, not overload the existing ones. And this separate knob should be able to take a list of fallback orders. After we have a wider deployment, we might gain a better understanding of the "cost function". Then we can try to build some in-kernel heuristics that automatically decides the best orders to fallback. If/when we get there, we can simply extend the knob by adding a new "magic word", e.g., "auto". > Ultimately, after we understand the performance impact of LAF, THP, and > mix of them and come up a reasonable kernel policy, a unified knob would > make sense. But we are not there yet. Exactly.