From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA2EAC001DF for ; Fri, 4 Aug 2023 00:19:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C7B82802A7; Thu, 3 Aug 2023 20:19:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2776A28022C; Thu, 3 Aug 2023 20:19:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1181A2802A7; Thu, 3 Aug 2023 20:19:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 01FF028022C for ; Thu, 3 Aug 2023 20:19:44 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CF9F1812E8 for ; Fri, 4 Aug 2023 00:19:44 +0000 (UTC) X-FDA: 81084513888.14.FB06771 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf29.hostedemail.com (Postfix) with ESMTP id 0A68B120020 for ; Fri, 4 Aug 2023 00:19:42 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PbTefRq9; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.182 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691108383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mA32fM9ZbOqMt7s1gfEqOlzSy+fN/PuS60CGam21os0=; b=NJ9Xm7ZRrhaynYBkGgu/cezkPtGmOmvMx29Dvbjb3SRYx/3apVYDyeLj+ur9SxJ7Z3nTEo /oglQ8OuaJd0gI6s99fBVklRvNe2g7ZSdIGLGUt/QxvlKvBiwPNFSrrlNijm2Jo+dVc4gJ y04lSNhiWk1LVNOQNMU579i8dgYOAz4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691108383; a=rsa-sha256; cv=none; b=ajKcHru76CzaLCrcgRfbj/UAHTMmGHOeBjEhJnweuyZTURWNIrBxp3b6KAtPUGoQWSMk+N B0nlSlcnGRJ3MY+Jr/4h6uVFvewXc4FEkKsunoigvmABR36Iyn8zwaTf/MXtLxjbFjN5Yz GM3ihKfPlKLr5iv4X6HXcBtxzLmcSKU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PbTefRq9; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.182 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-407db3e9669so72581cf.1 for ; Thu, 03 Aug 2023 17:19:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1691108382; x=1691713182; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=mA32fM9ZbOqMt7s1gfEqOlzSy+fN/PuS60CGam21os0=; b=PbTefRq9UzzDT25XxODA4kLZgtaubzDeg4foCwvMb2f8nS81Fq6JsW+YfaeyIooskj uvLYsfRkzXBTKPVf8djXijxIxNs5Rcdp9iGQZWSpIpQt+Dz9MbUjuI54cYFjv1G27Hto Iwl1It6sezNLVkHog+E+G5pB9EIIjZiwQt6gBw9dHbvw50bO8O38KjOLDgvDG4JaRevO 0Yi2L/SYMLEoJp87kkQrg8T/Ac0ukjeu7tMcYI5Hp++5WqeQuANjkR36khbSmbrn3NI8 EogsNIQXLWaufdp9TiBIE7KDcx6ah8Q65+ywMWIKeaQnbMyX+2q5rL9Gb7NNeBJch6Hl lbZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691108382; x=1691713182; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mA32fM9ZbOqMt7s1gfEqOlzSy+fN/PuS60CGam21os0=; b=lstR0yd1zefJONmsL87s3FEy/jP6++6DE9RDjrEdeJvajZrK+7SrSwjoQ9MH0n8yvK QBFSvgg81VtSEpZ7zlTBQp6gWFr6n20TPtzPpjSFIM8LnvuRxLgFlEEO9eDe9dbHEqsS GRpMEiUUZuRq4z/nznayV3j9K/CQXICGktPqNme6Mp4YllwOcEChxMV7PPnRqs7aZCva JHTi1KgshsTjYXcPR0IzZu+SYG5A2cPiRor5b4sTENnYdVMeX5Gzdfv7+LOeIyJmpJYu rwnsvBYIuJv/FRc95Xo0rcRCsEzJ28on8AdWz8FQ/UMRIcMXCUNB5/E4Ko2vbfLvuI3X PCBw== X-Gm-Message-State: AOJu0YxB1pqM+aY6nMPQLErJEtx31ylglG1SeiuNB7HKFtd8vM3rfENG bPzzmtEeHRSE9dnVXt1I+5PlNfutfa8AdToAo3Soow== X-Google-Smtp-Source: AGHT+IFy6SfL78iO/AoS+JkTZxkf9rSPiwDIAvWe4E9K9Ho7qgFmfL+RvoLDrQE4H/giQawLq7h21Dn16OgTMQqU3iw= X-Received: by 2002:a05:622a:15d4:b0:3f9:a770:7279 with SMTP id d20-20020a05622a15d400b003f9a7707279mr41199qty.9.1691108382007; Thu, 03 Aug 2023 17:19:42 -0700 (PDT) MIME-Version: 1.0 References: <20230726095146.2826796-1-ryan.roberts@arm.com> <20230726095146.2826796-3-ryan.roberts@arm.com> <20230803142154.nvgkavg33uyn6f72@box.shutemov.name> In-Reply-To: <20230803142154.nvgkavg33uyn6f72@box.shutemov.name> From: Yu Zhao Date: Thu, 3 Aug 2023 18:19:05 -0600 Message-ID: Subject: Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance To: "Kirill A. Shutemov" Cc: Ryan Roberts , Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi , "Huang, Ying" , Zi Yan , Luis Chamberlain , Itaru Kitayama , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 0A68B120020 X-Rspam-User: X-Stat-Signature: mf3yaqeyce557rbmg8wqopnj19o3shun X-Rspamd-Server: rspam03 X-HE-Tag: 1691108382-288717 X-HE-Meta: U2FsdGVkX18e+5n2yG+nebHXgi/WOV7E7SYro5Dh7uBo34M/zY4DMxHJ/NzjXQhEHum+jXIqN29AETpQdgG4Rtqm3DCOkt78daHJ1b2iw8sQKzCQhmFnuyedMD561b6bT/ZoYb3mvf9XAxJlfb+0kBinf/9V7FUMDj9rTz3y+iI/zU0yaQLmfEC/GcbRCDWamMcwShzOZ71WkvdApibiu9AAS11TPqygRhCzlU1R2x/KuYC4hmz3B6znCDAJbOYIp2tR/p3zGvKKDMSLqsbXznUwYroVi1HlLWkH9yg3EJLucJ9x5UiaYYoE32nY0NyEuAkTtdfyE16Wtd7AcOCauf0pRwq4W6L83Rn6Q5+l+UnBnsGBX/0nmYfbTKdFQ4Q1YVxDLXzWeLmAF5fhQV6znF5giUM9+X4GHvuz51DBNRgLIL1wNNm0LawZmN/989tnTEm+Zo30yzJEKAbWrM0eygCmgvASz1pMQPZNtc3I+FkWTCKruDeHJJEMFsK2z47ZbDVK/DdVDwzaKSiAKX4QpGuDn3i9RiGWbgJqYlk7Atcg1dI0m2gWic8LayoE747SxipTMwm1Far9fagxIyXnVzQvhCTBKb1vCZ59YMbG5JX7onRD3R7DmWRXdmlzQiT/rEvrTyE5kyIKaA6I5VmgOKSF1rf5sfcxgCBse1p7GYthaoBmzFAVcBT/kyN+ocAprqS+44qrraVIu9fWBDhfD90i3+iHmWiCIGQ2Nl/igf7HDiRh63Fygu8y4ysMb7wwL+cHCHUf0VQ5g/q1H2EnZSQKPAAr1YWrwdZZ6lkBnOfsu8wps78TfUkqSSE3Cb+oziqFmbP0tJf7vdX6zwoxS8HngrhNK2DfbDvp2LyrmBUZnnsxhynP1J5jmtj+MiQN/OdxzWvbv2KfsdadOueMR59pYO5YfSol4t3d1mRlmlg/j4IGtLfWxfPHoI771FiG8VVKoAHmqCbJkKk6sfN bTfo1gQS vEcLytYCPiL8020pSSxtJisiTsAAJ+Jil8DCJRWF1dkGN6YQgjDgHYuUD6xxyc018+n8dqQIXgLA8vLgOjFEn+YN1L7JGG4AwmJ7LRHXY6Y50thYVkKzay2JpG5OWY3CGjEloFJliw/tZij92TZa1MLIHHhikI3nGhM6NLnfLrJ2NgUwBnIno5pXnwtgfkpOoGDgjIqzLja8EbWpdHZeoV9UTNp9UtKOz316UcXeDmxces3+ww9XKYYOsP/hoZ5yDqOsiNvSaD0BmxdkLIapHvvyn6PoX+0NYI2VnHPSVWPtKOdMQYxd+MJsV/CQ7OxL7gwJ23vEZeWUHMYVh4T+Dq+rSyWank802L3RkEyMlNjQBw2FN50zggj06iA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Aug 3, 2023 at 8:27=E2=80=AFAM Kirill A. Shutemov wrote: > > On Thu, Aug 03, 2023 at 01:43:31PM +0100, Ryan Roberts wrote: > > + Kirill > > > > On 26/07/2023 10:51, Ryan Roberts wrote: > > > Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to = be > > > allocated in large folios of a determined order. All pages of the lar= ge > > > folio are pte-mapped during the same page fault, significantly reduci= ng > > > the number of page faults. The number of per-page operations (e.g. re= f > > > counting, rmap management lru list management) are also significantly > > > reduced since those ops now become per-folio. > > > > > > The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, > > > which defaults to disabled for now; The long term aim is for this to > > > defaut to enabled, but there are some risks around internal > > > fragmentation that need to be better understood first. > > > > > > When enabled, the folio order is determined as such: For a vma, proce= ss > > > or system that has explicitly disabled THP, we continue to allocate > > > order-0. THP is most likely disabled to avoid any possible internal > > > fragmentation so we honour that request. > > > > > > Otherwise, the return value of arch_wants_pte_order() is used. For vm= as > > > that have not explicitly opted-in to use transparent hugepages (e.g. > > > where thp=3Dmadvise and the vma does not have MADV_HUGEPAGE), then > > > arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is > > > bigger). This allows for a performance boost without requiring any > > > explicit opt-in from the workload while limitting internal > > > fragmentation. > > > > > > If the preferred order can't be used (e.g. because the folio would > > > breach the bounds of the vma, or because ptes in the region are alrea= dy > > > mapped) then we fall back to a suitable lower order; first > > > PAGE_ALLOC_COSTLY_ORDER, then order-0. > > > > > > > ... > > > > > +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ > > > + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SH= IFT) > > > + > > > +static int anon_folio_order(struct vm_area_struct *vma) > > > +{ > > > + int order; > > > + > > > + /* > > > + * If THP is explicitly disabled for either the vma, the process = or the > > > + * system, then this is very likely intended to limit internal > > > + * fragmentation; in this case, don't attempt to allocate a large > > > + * anonymous folio. > > > + * > > > + * Else, if the vma is eligible for thp, allocate a large folio o= f the > > > + * size preferred by the arch. Or if the arch requested a very sm= all > > > + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDE= R, > > > + * which still meets the arch's requirements but means we still t= ake > > > + * advantage of SW optimizations (e.g. fewer page faults). > > > + * > > > + * Finally if thp is enabled but the vma isn't eligible, take the > > > + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHIN= TED. > > > + * This ensures workloads that have not explicitly opted-in take = benefit > > > + * while capping the potential for internal fragmentation. > > > + */ > > > + > > > + if ((vma->vm_flags & VM_NOHUGEPAGE) || > > > + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || > > > + !hugepage_flags_enabled()) > > > + order =3D 0; > > > + else { > > > + order =3D max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_O= RDER); > > > + > > > + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, = true)) > > > + order =3D min(order, ANON_FOLIO_MAX_ORDER_UNHINTE= D); > > > + } > > > + > > > + return order; > > > +} > > > > > > Hi All, > > > > I'm writing up the conclusions that we arrived at during discussion in = the THP > > meeting yesterday, regarding linkage with exiting THP ABIs. It would be= great if > > I can get explicit "agree" or disagree + rationale from at least David,= Yu and > > Kirill. > > > > In summary; I think we are converging on the approach that is already c= oded, but > > I'd like confirmation. > > > > > > > > The THP situation today > > ----------------------- > > > > - At system level: THP can be set to "never", "madvise" or "always" > > - At process level: THP can be "never" or "defer to system setting" > > - At VMA level: no-hint, MADV_HUGEPAGE, MADV_NOHUGEPAGE > > > > That gives us this table to describe how a page fault is handled, accor= ding to > > process state (columns) and vma flags (rows): > > > > | never | madvise | always > > ----------------|-----------|-----------|----------- > > no hint | S | S | THP>S > > MADV_HUGEPAGE | S | THP>S | THP>S > > MADV_NOHUGEPAGE | S | S | S > > > > Legend: > > S allocate single page (PTE-mapped) > > LAF allocate lage anon folio (PTE-mapped) > > THP allocate THP-sized folio (PMD-mapped) > > > fallback (usually because vma size/alignment insufficient for fol= io) > > > > > > > > Principles for Large Anon Folios (LAF) > > -------------------------------------- > > > > David tells us there are use cases today (e.g. qemu live migration) whi= ch use > > MADV_NOHUGEPAGE to mean "don't fill any PTEs that are not explicitly fa= ulted" > > and these use cases will break (i.e. functionally incorrect) if this re= quest is > > not honoured. > > > > So LAF must at least honour MADV_NOHUGEPAGE to prevent breaking existin= g use > > cases. And once we do this, then I think the least confusing thing is f= or it to > > also honor the "never" system/process state; so if either the system, p= rocess or > > vma has explicitly opted-out of THP, then LAF should also be bypassed. > > > > Similarly, any case that would previously cause the allocation of PMD-s= ized THP > > must continue to be honoured, else we risk performance regression. > > > > That leaves the "madvise/no-hint" case, and all THP fallback paths due = to the > > VMA not being correctly aligned or sized to hold a PMD-sized mapping. I= n these > > cases, we will attempt to use LAF first, and fallback to single page if= the vma > > size/alignment doesn't permit it. > > > > | never | madvise | always > > ----------------|-----------|-----------|----------- > > no hint | S | LAF>S | THP>LAF>S > > MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S > > MADV_NOHUGEPAGE | S | S | S > > > > I think this (perhaps conservative) approach will be the least surprisi= ng to > > users. And is the policy that is already implemented in this patch. > > This looks very reasonable. > > The only questionable field is no-hint/madvise. I can argue for both LAF>= S > and S here. I think LAF>S is fine as long as we are not too aggressive > with allocation order. > > I think we need to work on eliminating reasons for users to set 'never'. > If something behaves better with 'never' kernel has failed user. > > > Downsides of this policy > > ------------------------ > > > > As Yu and Yin have pointed out, there are some workloads which do not p= erform > > well with THP, due to large fault latency or memory wastage, etc. But w= hich > > _may_ still benefit from LAF. By taking the conservative approach, we e= xclude > > these workloads from benefiting automatically. > > Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is > fine? No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB. However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")? I'm strongly against this policy for two practical reasons I learned from tuning THPs in our data centers: 1. By doing the above, we are blurring the lines between those values and making real-world performance tuning extremely hard if not impractice. 2. As I previously pointed out: if we mix LAFs with THPs, we actually risk causing performance regressions because giving smaller VMAs LAFs can deprive large VMAs of THPs.