From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A23ACC52D7C for ; Tue, 13 Aug 2024 07:14:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 288C16B008C; Tue, 13 Aug 2024 03:14:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 238CC6B0092; Tue, 13 Aug 2024 03:14:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 100936B0095; Tue, 13 Aug 2024 03:14:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E68906B008C for ; Tue, 13 Aug 2024 03:14:19 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 678FDA5282 for ; Tue, 13 Aug 2024 07:14:19 +0000 (UTC) X-FDA: 82446358638.21.1D3C067 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by imf21.hostedemail.com (Postfix) with ESMTP id 928931C000B for ; Tue, 13 Aug 2024 07:14:17 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="KD/sHgrn"; spf=pass (imf21.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723533223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jyz3DK8dqgTmgnmWwpr2k8InrcG4osaLvAAxFacRPCg=; b=Oy8fDPRAy/WW6oMPMTNDojusCNdBAJXrk4xswsPzkfDVmT2m6kimPZixDE7a35vtRgqwMn 5IeHarAskBPQvWy5nwLeFP/08t2vqfAvXrJ3lDo+Gka2dSudBSg/5V3u/q4jw/+7EzI5Pu 86Ku36xVaJdhrFyynzpHCkw5pJREdPo= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="KD/sHgrn"; spf=pass (imf21.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723533223; a=rsa-sha256; cv=none; b=GXMpW6RSMUP94dvNy9/xAzHflYhUpgsIpX7u9l32Z/yfPEGOTAG1PH1sTqmjmAJsqTEudf zdXsQwyN02LbRj0c7yCvUeI+SDMFh0A3xfU19AvQ3kHIm7ePXzIoURv8LFuPH97UxjG7oz EbBActKtKS1IHifi1YvS8mMlxxZD2Pg= Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-5afa207b8bfso5402695a12.0 for ; Tue, 13 Aug 2024 00:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723533256; x=1724138056; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=jyz3DK8dqgTmgnmWwpr2k8InrcG4osaLvAAxFacRPCg=; b=KD/sHgrnL8LyO55ACp/xeRbSHc3rVNmp3zVKoI7/KkZXjWGJdnDmcRhGrh+LK95tMC pe6wDQd6cYgImQbdmGZyf/XVmmXW65ohPoDS1qVi39BrNJmuSrxar1zD01vFJsPUjBdg 9NKOSziR8mOPxOqeO+gPGrvoEattc1eAv9s26mR8cx28YwszTGFrHZ/qwru072O1jEKG aSblW5bkxHV/qcP/hNlsgfXhkfgvolKbLVnq7XQ3ptYD5l01XKmZGv68S3DOpokhznhE 1KBUBDWSNnXbXUACGarKTXZwYj+85wPz9nAPIs/lliNym9VrfctuB8C69OjcsnH/BqO3 ODrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723533256; x=1724138056; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jyz3DK8dqgTmgnmWwpr2k8InrcG4osaLvAAxFacRPCg=; b=b0u8bME205XkxB22Um8Dhg3ljilBhgA5MhdKKqeSyAIKuHSEt50fYLUBBW7CL9X9YT ePtlErQWEoq4yOVKQPn7gEIUdQjmAS7ckQtlH+iJ4SEK1qiNs3K5xMBsp7FB+vg/GJDJ UfHeNTikXyX87l0bfpn1/q1XjUgDWetq+nzE1bVpA0UOi+mIz5kKRNItC95jp8h8XdZI BBoYQgmIv75lJ/KJgam4H3oMc4wZM3vEHCBQ3yxb6MVR1y/fVoRsnXyPPnMwyV/LVWyq rB8/VmYWMscCB6Y5Y486ZoMSjB3x9NGW98bHyG8pBolyNlisSW9HkgBygyGWm0iwcV7/ XvxA== X-Forwarded-Encrypted: i=1; AJvYcCW4sdyO7AjzJpQAe0DXgBTzKeVofWMHYyg+hbQ6ypuBZbs46YtL1YXT9+p8o1qwVd8g3ZmQM4FiNX/YvBU4gD4Fg3Y= X-Gm-Message-State: AOJu0YzMP69OK/bAAooQK/lJNV2n8Bexw2ZZFX35fCiiNFE4yDBJXqGJ NJy8ZP0yQ6dWan+kmZJUb/bc1UuWOZEV4GIa2aRPJBlCKQgoU6rivlZQmra+d9KVEBDXvUg8tX8 rwu5EeQG/Jb62y4EP1u3mCvICm4c= X-Google-Smtp-Source: AGHT+IEC6XO8vsLlXfWHSMDSyKsREiX+qwPLaicCQ1BlMeJyYZyVAL5hPQq9CLrVz05m6L+b8AKGJ/S4ZxgHnzO/6U8= X-Received: by 2002:a17:907:2d09:b0:a7d:34bf:600e with SMTP id a640c23a62f3a-a80ed2d7c6fmr136669266b.60.1723533255327; Tue, 13 Aug 2024 00:14:15 -0700 (PDT) MIME-Version: 1.0 References: <202407301049.5051dc19-oliver.sang@intel.com> <193e302c-4401-4756-a552-9f1e07ecedcf@redhat.com> <439265d8-e71e-41db-8a46-55366fdd334e@intel.com> <90477952-fde2-41d7-8ff4-2102c45e341d@redhat.com> <6uxnuf2gysgabyai2r77xrqegb7t7cc2dlzjz6upwsgwrnfk3x@cjj6on3wqm4x> <5a67c103-1d9d-440d-8bed-bbfa7d3ecf71@redhat.com> <5c0979a2-9a56-4284-82d2-42da62bda4a5@redhat.com> <817150f2-abf7-430f-9973-540bd6cdd26f@intel.com> In-Reply-To: <817150f2-abf7-430f-9973-540bd6cdd26f@intel.com> From: Mateusz Guzik Date: Tue, 13 Aug 2024 09:14:03 +0200 Message-ID: Subject: Re: [linus:master] [mm] c0bff412e6: stress-ng.clone.ops_per_sec -2.9% regression To: Yin Fengwei Cc: David Hildenbrand , kernel test robot , Peter Xu , oe-lkp@lists.linux.dev, lkp@intel.com, linux-kernel@vger.kernel.org, Andrew Morton , Huacai Chen , Jason Gunthorpe , Matthew Wilcox , Nathan Chancellor , Ryan Roberts , WANG Xuerui , linux-mm@kvack.org, ying.huang@intel.com, feng.tang@intel.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 928931C000B X-Stat-Signature: cp7rq819o5sxok4gmtgne6ngoa3nmnm3 X-HE-Tag: 1723533257-132265 X-HE-Meta: U2FsdGVkX1/NzM7R6BB9Fzxi0Hp0LJz4d72FUHKxlyaA92TPSLNX1BtotHeAyER9JjEW7/iW2HT6h1FcnX8e/5Tr0AFDDzW4pduJ7BfYjWGsRNIX4nkyGhfF3ZH2w0WVpX+769DZbuFnfrHD/oFqmWvowv7CyU+SM6oVrrihkau5rGU821aKxKSmsMtsbqWPuWFOH2tx3tS28ChpEx3RkTSS9mUNkwZ4ZifL5SwU1Wo55eQ7LuD1LViSdQF1divhaZx3t7+gvYbmnltU/PlzyPDJHCGNBAF9GEG3pFnPr6u+NQQ+piSEnbb2I+NsZYdJ49tdQFbb8nj4jwpdjFX+qQ25oSHj9af7zFIF65Ifnhg1DAkSd2WXsqeR7Dhs4Ts76outNCJGXp75Msnl3DUSAcD1e2qzfzKFmsbijOLTNDmlGJzq3XIbiQeYvKnb2Z4TWRNSOk6f05AD79cDLVuYCE9k99tUjvfZlHSv7QmzwCg4x5FPogsoe1mfEsczH8AFXtn7JnPzJZCGqo8/Gjeg4Z/jKe9lnzKJpr5zQzV+rL+nkVLDa0LrwveKex1LQH/wUVV17QDCj/tvdqKnWMjcxwxgAJStMWuvvmzNLNAhgl6Shf4MFWxZ05BctD3sllRH87NtgR6fuPau6IVKKfesde038fMi2BRVIu0PQTWLy6DNnFRajkYT4gAhTtdCZbGs8HGL0tY9xUWHYhByl1x2TfzJEaNfTmNREsXN2yRFda/PcnS1ueBe1vE7rcaIBN0XEKu6q54XEeEdRsbNe+5LCppYxp+kv/tvqrBBIlwnD7fxuLWx/ShkQ3Pv6SWC5f/NR7iYV3ndNC/ElOIynxgpBqFqVeL8s/VjpkK9biRBgDaMEMhG2yAyM6XyRhdlqyLidBhlDMYXSiV0nkKByjnJKeRI+Vlff9w88yVWtXKTtBJ+KBMQPtZvjDo9ULTaotg5Q0Kzfn1kb8pHD/VbO7J +RyAoKcL E/tEnkJXd9jh9Q9Zr7Jw0Kso/9jOYtCZR0Pi3mMrZMfXCCCET6+lX3t9cQM+Jx2rT4YiPRvhMfWj9qK2ro65ME/DwZ9d1Z2+FimGssEp7hZXxeGiTyupRhACHkazmPG96Qn92u2Rjgrh9CNv+PmDgxraoUmQP1ShWTG3Ssa1cc9I8pc9+5+gdqLtob0EBsgpUJ84SSUB1xpN/7OUagC0ggSlH6YalklYVnNLp2saJeebkwQvOxeSqCfrUXwE/LaLe4Sushj2Ehz8pRAR2+1XTYlH4Da7fbcfUvNwUyMdVKf2d15SpaA7RkYsgEg+eI1edTjbLcS5lBgAGrlPDySnqrxW2ONLu3Mg6zhA0fIKty2Vi8AgyynUbIIPsFCz3CgbqDP8CZ2AEe4tvHNXb6+TFnBu4HboJs8L8YbPdc964S/4E67ZU0Hd17QdJLyL1zd2+FEQOuQ8IBF/wtx1xW72iFYA5YjvviVAfWVOj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Aug 13, 2024 at 9:09=E2=80=AFAM Yin Fengwei = wrote: > > On 8/12/24 00:49, Mateusz Guzik wrote: > > On Mon, Aug 12, 2024 at 12:43:08PM +0800, Yin Fengwei wrote: > >> Hi David, > >> > >> On 8/1/24 09:44, David Hildenbrand wrote: > >>> On 01.08.24 15:37, Mateusz Guzik wrote: > >>>> On Thu, Aug 1, 2024 at 3:34=E2=80=AFPM David Hildenbrand > >>>> wrote: > >>>>> > >>>>> On 01.08.24 15:30, Mateusz Guzik wrote: > >>>>>> On Thu, Aug 01, 2024 at 08:49:27AM +0200, David Hildenbrand wrote: > >>>>>>> Yes indeed. fork() can be extremely sensitive to each > >>>>>>> added instruction. > >>>>>>> > >>>>>>> I even pointed out to Peter why I didn't add the > >>>>>>> PageHuge check in there > >>>>>>> originally [1]. > >>>>>>> > >>>>>>> "Well, and I didn't want to have runtime-hugetlb checks in > >>>>>>> PageAnonExclusive code called on certainly-not-hugetlb code paths= ." > >>>>>>> > >>>>>>> > >>>>>>> We now have to do a page_folio(page) and then test for hugetlb. > >>>>>>> > >>>>>>> return folio_test_hugetlb(page_folio(page)); > >>>>>>> > >>>>>>> Nowadays, folio_test_hugetlb() will be faster than at > >>>>>>> c0bff412e6 times, so > >>>>>>> maybe at least part of the overhead is gone. > >>>>>>> > >>>>>> > >>>>>> I'll note page_folio expands to a call to _compound_head. > >>>>>> > >>>>>> While _compound_head is declared as an inline, it ends up being bi= g > >>>>>> enough that the compiler decides to emit a real function instead a= nd > >>>>>> real func calls are not particularly cheap. > >>>>>> > >>>>>> I had a brief look with a profiler myself and for single-threaded = usage > >>>>>> the func is quite high up there, while it manages to get out with = the > >>>>>> first branch -- that is to say there is definitely performance los= t for > >>>>>> having a func call instead of an inlined branch. > >>>>>> > >>>>>> The routine is deinlined because of a call to page_fixed_fake_head= , > >>>>>> which itself is annotated with always_inline. > >>>>>> > >>>>>> This is of course patchable with minor shoveling. > >>>>>> > >>>>>> I did not go for it because stress-ng results were too unstable fo= r me > >>>>>> to confidently state win/loss. > >>>>>> > >>>>>> But should you want to whack the regression, this is what I would = look > >>>>>> into. > >>>>>> > >>>>> > >>>>> This might improve it, at least for small folios I guess: > >> Do you want us to test this change? Or you have further optimization > >> ongoing? Thanks. > > > > I verified the thing below boots, I have no idea about performance. If > > it helps it can be massaged later from style perspective. > > > > diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h > > index 5769fe6e4950..2d5d61ab385b 100644 > > --- a/include/linux/page-flags.h > > +++ b/include/linux/page-flags.h > > @@ -194,34 +194,13 @@ enum pageflags { > > #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP > > DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); > > > > -/* > > - * Return the real head page struct iff the @page is a fake head page,= otherwise > > - * return the @page itself. See Documentation/mm/vmemmap_dedup.rst. > > - */ > > +const struct page *_page_fixed_fake_head(const struct page *page); > > + > > static __always_inline const struct page *page_fixed_fake_head(const = struct page *page) > > { > > if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key)) > > return page; > > - > > - /* > > - * Only addresses aligned with PAGE_SIZE of struct page may be fa= ke head > > - * struct page. The alignment check aims to avoid access the fiel= ds ( > > - * e.g. compound_head) of the @page[1]. It can avoid touch a (pos= sibly) > > - * cold cacheline in some cases. > > - */ > > - if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && > > - test_bit(PG_head, &page->flags)) { > > - /* > > - * We can safely access the field of the @page[1] with PG= _head > > - * because the @page is a compound page composed with at = least > > - * two contiguous pages. > > - */ > > - unsigned long head =3D READ_ONCE(page[1].compound_head); > > - > > - if (likely(head & 1)) > > - return (const struct page *)(head - 1); > > - } > > - return page; > > + return _page_fixed_fake_head(page); > > } > > #else > > static inline const struct page *page_fixed_fake_head(const struct pa= ge *page) > > @@ -235,7 +214,7 @@ static __always_inline int page_is_fake_head(const = struct page *page) > > return page_fixed_fake_head(page) !=3D page; > > } > > > > -static inline unsigned long _compound_head(const struct page *page) > > +static __always_inline unsigned long _compound_head(const struct page = *page) > > { > > unsigned long head =3D READ_ONCE(page->compound_head); > > > > diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c > > index 829112b0a914..3fbc00db607a 100644 > > --- a/mm/hugetlb_vmemmap.c > > +++ b/mm/hugetlb_vmemmap.c > > @@ -19,6 +19,33 @@ > > #include > > #include "hugetlb_vmemmap.h" > > > > +/* > > + * Return the real head page struct iff the @page is a fake head page,= otherwise > > + * return the @page itself. See Documentation/mm/vmemmap_dedup.rst. > > + */ > > +const struct page *_page_fixed_fake_head(const struct page *page) > > +{ > > + /* > > + * Only addresses aligned with PAGE_SIZE of struct page may be fa= ke head > > + * struct page. The alignment check aims to avoid access the fiel= ds ( > > + * e.g. compound_head) of the @page[1]. It can avoid touch a (pos= sibly) > > + * cold cacheline in some cases. > > + */ > > + if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && > > + test_bit(PG_head, &page->flags)) { > > + /* > > + * We can safely access the field of the @page[1] with PG= _head > > + * because the @page is a compound page composed with at = least > > + * two contiguous pages. > > + */ > > + unsigned long head =3D READ_ONCE(page[1].compound_head); > > + > > + if (likely(head & 1)) > > + return (const struct page *)(head - 1); > > + } > > + return page; > > +} > > + > > /** > > * struct vmemmap_remap_walk - walk vmemmap page table > > * > > > > The change can resolve the regression (from -3% to 0.5%): > thanks for testing would you mind benchmarking the change which merely force-inlines _compund_= page? https://lore.kernel.org/linux-mm/66c4fcc5-47f6-438c-a73a-3af6e19c3200@redha= t.com/ > Please note: > 9cb28da54643ad464c47585cd5866c30b0218e67 is the parent commit > 3f16e4b516ef02d9461b7e0b6c50e05ba0811886 is the commit with above > patch > c0bff412e67b781d761e330ff9578aa9ed2be79e is the commit which > introduced regression > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > tbox_group/testcase/rootfs/kconfig/compiler/nr_threads/testtime/test/cpuf= req_governor/debug-setup: > > lkp-icl-2sp8/stress-ng/debian-12-x86_64-20240206.cgz/x86_64-rhel-8.3/gcc-= 12/100%/60s/clone/performance/yfw_test2 > > commit: > 9cb28da54643ad464c47585cd5866c30b0218e67 > 3f16e4b516ef02d9461b7e0b6c50e05ba0811886 > c0bff412e67b781d761e330ff9578aa9ed2be79e > > 9cb28da54643ad46 3f16e4b516ef02d9461b7e0b6c5 c0bff412e67b781d761e330ff95 > ---------------- --------------------------- --------------------------- > fail:runs %reproduction fail:runs %reproduction fail:runs > | | | | | > 3:3 0% 3:3 0% 3:3 > stress-ng.clone.microsecs_per_clone.pass > 3:3 0% 3:3 0% 3:3 > stress-ng.clone.pass > %stddev %change %stddev %change %stddev > \ | \ | \ > 2904 -0.6% 2886 +3.7% 3011 > stress-ng.clone.microsecs_per_clone > 563520 +0.5% 566296 -3.1% 546122 > stress-ng.clone.ops > 9306 +0.5% 9356 -3.0% 9024 > stress-ng.clone.ops_per_sec > > > BTW, the change needs to export symbol _page_fixed_fake_head otherwise > some modules hit build error. > ok, I'll patch that up if this approach will be the thing to do --=20 Mateusz Guzik