From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B42C3C3DA7F for ; Mon, 12 Aug 2024 04:49:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 429666B008A; Mon, 12 Aug 2024 00:49:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D91A6B008C; Mon, 12 Aug 2024 00:49:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27A5B6B0092; Mon, 12 Aug 2024 00:49:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0A18E6B008A for ; Mon, 12 Aug 2024 00:49:24 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7078F1C342E for ; Mon, 12 Aug 2024 04:49:23 +0000 (UTC) X-FDA: 82442364606.07.51005B5 Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) by imf12.hostedemail.com (Postfix) with ESMTP id 706B540011 for ; Mon, 12 Aug 2024 04:49:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nDAh8F2v; spf=pass (imf12.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.208.47 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723438127; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qpq3LPqm61lcZ3ArBnzJ/3LDps9wXJ9LbD1MIjBOKqk=; b=ZH19YfgrvyYY4LXfUK+eYPoaXFO23Rv7ZfRmd4upgQmQsb18hqGqypiIpDIIHDoaZka/nu lP5ic21j8tebW1oFPx6FIxgzu5/jvyDJnnyzguq/KULSuJEYzMysMBSCY9bMxTf+Tl+Fld dWJQaBjF44Z8iKNb46jnKhhNevHwkM4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nDAh8F2v; spf=pass (imf12.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.208.47 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723438127; a=rsa-sha256; cv=none; b=1x6MY+HYT1YQRKsF5dEsC4BORPVJ3NwQoJOdD2kGSHW3jV5YlVZfArvT7o1CiE+mJekmIx bLNIc5QAR7Qnnt2kP4M6iBLxwgvdgW2a+7ewI9RwxbLTmKmm1HUlvlFSeDdFY1k4wGwWHr eqwJn9VPUkf8bG3R7iIoFHM6GnMHQ30= Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-5a10835487fso5243125a12.1 for ; Sun, 11 Aug 2024 21:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723438160; x=1724042960; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=qpq3LPqm61lcZ3ArBnzJ/3LDps9wXJ9LbD1MIjBOKqk=; b=nDAh8F2vgOYQ33RB+Algo/Qp667OCxLIATYXBg4PUeb+5dOx4CwhgB/FiOWKN2TvRr sUxBdVQUA6oYKRTfO6+ECg5XI1mkTYgjPjqRwjKL2rwg3rndjKlB6ytayeQW7DlQte+q G7Ibcl8EipXBBNBHNFNgHD8BVG2UQ5X/Hu1y7wfmhQcth8qKB+ehcmlsiR/kd0Tw/Jm8 oQF0161lj2/AhHtdf7OSRu1Vl4dr3PQqHqMYF2qUggLUftEcEFOoys0gKaa2yUcEVbNL CNfr2fV0NhkRU4wFz4eEbnDoWxQK0dZfMcxEutdjz5oXaKOSrH0TldcDLUhrnQFcsAmv tyJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723438160; x=1724042960; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qpq3LPqm61lcZ3ArBnzJ/3LDps9wXJ9LbD1MIjBOKqk=; b=X3YcJHV1lDjST4EeDKg8FYeYgPyPT3R6msAiK1+Cne5LwtlMrI68df5gw9QxtXS0bq 4Q0AYzfGtjOuw5hz6TJmmhTvUyBZ3d0FS/JhJyRldjpdDQ3Jrb5sTnzDtGj5xQN1+oEm 9B92IOkyTlhbmPaanEk1UbzaHNQboe2Xna5snm1bij2lP586WpuvBphHRQdXhdhAvzbC 0JRSngWICDKFCzGD3N2b6UQLhnFGCJCfx/kagMFJhgpOoAEGGBPdgwTcOk9qc2W0i2P/ a3Kh5UuOvmkFvoeitV+N9DAeb3lLhNr+O53gOcdtEeHithKMjjkNw/qceZl3KZtfkXjD M34g== X-Forwarded-Encrypted: i=1; AJvYcCUQDeI+Vb4frEIyFfhS5CIxhkNW58X84bDpqRCOuz921quRRVJLvdeGzDDQHIVekL57NG1uyUwNu227g22JoEizpi0= X-Gm-Message-State: AOJu0Yz+T/HnXvjZ2S2QuTTP2F3v5xXlt3jAZ0ApsZDqp621iqIZ3pWK PnukEejzHfeAN0hQa0qQxxhH4+Jq3S56RXgwKdhsKJOJscmzQG8Q X-Google-Smtp-Source: AGHT+IE7qNtRHiWSXri3JkwIDsfJkJy1SPSZAsiJ/9zS+oo1xDwuodEP3kHjNAIedcu/4q/sCynWTQ== X-Received: by 2002:a17:907:1c0e:b0:a7a:9447:3e8b with SMTP id a640c23a62f3a-a80aa59eb36mr585387966b.25.1723438159639; Sun, 11 Aug 2024 21:49:19 -0700 (PDT) Received: from f (cst-prg-72-52.cust.vodafone.cz. [46.135.72.52]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a80bb11a533sm196522766b.96.2024.08.11.21.49.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Aug 2024 21:49:19 -0700 (PDT) Date: Mon, 12 Aug 2024 06:49:00 +0200 From: Mateusz Guzik To: Yin Fengwei Cc: David Hildenbrand , kernel test robot , Peter Xu , oe-lkp@lists.linux.dev, lkp@intel.com, linux-kernel@vger.kernel.org, Andrew Morton , Huacai Chen , Jason Gunthorpe , Matthew Wilcox , Nathan Chancellor , Ryan Roberts , WANG Xuerui , linux-mm@kvack.org, ying.huang@intel.com, feng.tang@intel.com Subject: Re: [linus:master] [mm] c0bff412e6: stress-ng.clone.ops_per_sec -2.9% regression Message-ID: References: <202407301049.5051dc19-oliver.sang@intel.com> <193e302c-4401-4756-a552-9f1e07ecedcf@redhat.com> <439265d8-e71e-41db-8a46-55366fdd334e@intel.com> <90477952-fde2-41d7-8ff4-2102c45e341d@redhat.com> <6uxnuf2gysgabyai2r77xrqegb7t7cc2dlzjz6upwsgwrnfk3x@cjj6on3wqm4x> <5a67c103-1d9d-440d-8bed-bbfa7d3ecf71@redhat.com> <5c0979a2-9a56-4284-82d2-42da62bda4a5@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 706B540011 X-Stat-Signature: zdjh97pjocp6wspibd1b6zaf9dpqzo73 X-HE-Tag: 1723438161-170078 X-HE-Meta: U2FsdGVkX18EbD+de8hn/41AJPgDG3r6LoUNROA4z3PLMOX8cMqPrYQEMvIf9EAXkrnWQRId8pWLvbbaNL8G8eXVINFanEEMzDWTmko68jbsRRty4rDF9UwQGAmJpu9bQgEGAymAbOIzjJi1Y5VGqvBwdBQDyackhlBJ5DcqEF7G4ubKVT7R1JyJpNcLLiSUMKSKB9QXQQhrW+2nLn5Qr5RvbN1KoyHA/9WtEvJm43XB8NRPTH3BjRLJ1hySOy8QqzpYZZVAa5tUP+a/9kc3ZIUVJMsIljPph3kvwS9tBq1APL+Q0BZtEqAvnJHooOFS2b6cAhYk5QtQDkml4bmmQrVIuXHFn2aSOH7QvwMGnqIKOUNS/5rQeRO8phHd2MxUanqky0nkyco7YBL+7Zio9Ai1gZnliHNoDhENLnoyRSm/bGqHDB7MUctiETj4FcRROVTZ/i0wuFgVymaJrgu0Y0fWxPh2qskcH+padzKt/oXaql5OOnf4tiBVE895UVNSG1ZepkbrAJ1iZPMx04XAirukl3E4Pm53VWMHuyjHp6+8W+L3doE213iHZWFKb/KCTUWj9kPnhwfwgbtAZt8QUexjka4QZqJzMouL59vpDleAb/PbIWRHSR8IN0ZuHLUJ3IK2ip326omvu8RgaFhxUhJEaVWjmc1Ac7Ur2iinPggK6L/0jamIpzlCjwElD59KPTKbYM2anjLlX//cuh9bQ9tcHI+kX5P7sy6YwRzbwRHKv/6N14/q0FaG/WHZkaqB2Gucx0BivLq4fPbb4wjx2g9W0vuhQABV/+ugKNAEsnPzz8dGU2nBK8GUplCvpbbwzuwH6fsCyJkEOW1I37ThnbsSoWgq89rhUiWnXyumguy0erwPvenBjKolu7juBHJCJfy74vgUTKYe378bg4TSGVHcgkOG5k+RppFRrbDXvJ5aLAGgdRPvaj+mXEpcaHaPoBvg9v6NfSHizVZQpmF xEvtUofS tezLWZF87uk3EaVrXOA9ZvNfbmrW8LJlIYL9OreaceqMoDxTF9rTn20HkKls2uCro7yh8L08zOpV0aPBx3BdKhXEs0RoAQRgOnmxFgv8ix1JW10awQF+vqB4lx2by+PgYE0zQN+jMlWFQWXXilHpJyS3bt/6RmeVRmfwdybe21NdQlyvldFXgWdSP0Bq/iVb1VPTEHgMO6no8ZLx62XEfuTD49nO0I7BPDDYdax1pdjEQGZ8ht5gBXa9NfynvBVlt+KQfUwNArCRYoNJRK2oSzyM/8wm2lCtFr7UXB0WZBC148r1bfeuDk/H1UmkHTgU7u5LqVMlcMMsD2RqtepgvgduUCH/7IsqfmPCjKMHXw1GxKwjpPb6phsQkr7/lrr0mOFVNIyHoha8kGIMq4zJFErqD2CwuTfyNWtDtSpKZqUGRHPNj54f+WdOKaI0RJiqi882QZgc4tVkETU1PBHqlaWIUrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Aug 12, 2024 at 12:43:08PM +0800, Yin Fengwei wrote: > Hi David, > > On 8/1/24 09:44, David Hildenbrand wrote: > > On 01.08.24 15:37, Mateusz Guzik wrote: > > > On Thu, Aug 1, 2024 at 3:34 PM David Hildenbrand > > > wrote: > > > > > > > > On 01.08.24 15:30, Mateusz Guzik wrote: > > > > > On Thu, Aug 01, 2024 at 08:49:27AM +0200, David Hildenbrand wrote: > > > > > > Yes indeed. fork() can be extremely sensitive to each > > > > > > added instruction. > > > > > > > > > > > > I even pointed out to Peter why I didn't add the > > > > > > PageHuge check in there > > > > > > originally [1]. > > > > > > > > > > > > "Well, and I didn't want to have runtime-hugetlb checks in > > > > > > PageAnonExclusive code called on certainly-not-hugetlb code paths." > > > > > > > > > > > > > > > > > > We now have to do a page_folio(page) and then test for hugetlb. > > > > > > > > > > > >       return folio_test_hugetlb(page_folio(page)); > > > > > > > > > > > > Nowadays, folio_test_hugetlb() will be faster than at > > > > > > c0bff412e6 times, so > > > > > > maybe at least part of the overhead is gone. > > > > > > > > > > > > > > > > I'll note page_folio expands to a call to _compound_head. > > > > > > > > > > While _compound_head is declared as an inline, it ends up being big > > > > > enough that the compiler decides to emit a real function instead and > > > > > real func calls are not particularly cheap. > > > > > > > > > > I had a brief look with a profiler myself and for single-threaded usage > > > > > the func is quite high up there, while it manages to get out with the > > > > > first branch -- that is to say there is definitely performance lost for > > > > > having a func call instead of an inlined branch. > > > > > > > > > > The routine is deinlined because of a call to page_fixed_fake_head, > > > > > which itself is annotated with always_inline. > > > > > > > > > > This is of course patchable with minor shoveling. > > > > > > > > > > I did not go for it because stress-ng results were too unstable for me > > > > > to confidently state win/loss. > > > > > > > > > > But should you want to whack the regression, this is what I would look > > > > > into. > > > > > > > > > > > > > This might improve it, at least for small folios I guess: > Do you want us to test this change? Or you have further optimization > ongoing? Thanks. I verified the thing below boots, I have no idea about performance. If it helps it can be massaged later from style perspective. diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5769fe6e4950..2d5d61ab385b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -194,34 +194,13 @@ enum pageflags { #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); -/* - * Return the real head page struct iff the @page is a fake head page, otherwise - * return the @page itself. See Documentation/mm/vmemmap_dedup.rst. - */ +const struct page *_page_fixed_fake_head(const struct page *page); + static __always_inline const struct page *page_fixed_fake_head(const struct page *page) { if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key)) return page; - - /* - * Only addresses aligned with PAGE_SIZE of struct page may be fake head - * struct page. The alignment check aims to avoid access the fields ( - * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly) - * cold cacheline in some cases. - */ - if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && - test_bit(PG_head, &page->flags)) { - /* - * We can safely access the field of the @page[1] with PG_head - * because the @page is a compound page composed with at least - * two contiguous pages. - */ - unsigned long head = READ_ONCE(page[1].compound_head); - - if (likely(head & 1)) - return (const struct page *)(head - 1); - } - return page; + return _page_fixed_fake_head(page); } #else static inline const struct page *page_fixed_fake_head(const struct page *page) @@ -235,7 +214,7 @@ static __always_inline int page_is_fake_head(const struct page *page) return page_fixed_fake_head(page) != page; } -static inline unsigned long _compound_head(const struct page *page) +static __always_inline unsigned long _compound_head(const struct page *page) { unsigned long head = READ_ONCE(page->compound_head); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 829112b0a914..3fbc00db607a 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -19,6 +19,33 @@ #include #include "hugetlb_vmemmap.h" +/* + * Return the real head page struct iff the @page is a fake head page, otherwise + * return the @page itself. See Documentation/mm/vmemmap_dedup.rst. + */ +const struct page *_page_fixed_fake_head(const struct page *page) +{ + /* + * Only addresses aligned with PAGE_SIZE of struct page may be fake head + * struct page. The alignment check aims to avoid access the fields ( + * e.g. compound_head) of the @page[1]. It can avoid touch a (possibly) + * cold cacheline in some cases. + */ + if (IS_ALIGNED((unsigned long)page, PAGE_SIZE) && + test_bit(PG_head, &page->flags)) { + /* + * We can safely access the field of the @page[1] with PG_head + * because the @page is a compound page composed with at least + * two contiguous pages. + */ + unsigned long head = READ_ONCE(page[1].compound_head); + + if (likely(head & 1)) + return (const struct page *)(head - 1); + } + return page; +} + /** * struct vmemmap_remap_walk - walk vmemmap page table *