From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE64CC4332F for ; Thu, 20 Oct 2022 15:28:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B4648E0002; Thu, 20 Oct 2022 11:28:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 265558E0001; Thu, 20 Oct 2022 11:28:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 154798E0002; Thu, 20 Oct 2022 11:28:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 03D2B8E0001 for ; Thu, 20 Oct 2022 11:28:32 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BB1A016074D for ; Thu, 20 Oct 2022 15:28:31 +0000 (UTC) X-FDA: 80041709622.21.5FD18EF Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf10.hostedemail.com (Postfix) with ESMTP id 4E0BDC0033 for ; Thu, 20 Oct 2022 15:28:31 +0000 (UTC) Received: from imladris.surriel.com ([96.67.55.152]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1olXSz-0008Af-26; Thu, 20 Oct 2022 11:28:17 -0400 Message-ID: <366045a27a96e01d0526d63fd78d4f3c5d1f530b.camel@surriel.com> Subject: Re: [mm] f35b5d7d67: will-it-scale.per_process_ops -95.5% regression From: Rik van Riel To: "Huang, Ying" , Nathan Chancellor Cc: kernel test robot , lkp@lists.01.org, lkp@intel.com, Andrew Morton , Yang Shi , Matthew Wilcox , linux-kernel@vger.kernel.org, linux-mm@kvack.org, feng.tang@intel.com, zhengjun.xing@linux.intel.com, fengwei.yin@intel.com Date: Thu, 20 Oct 2022 11:28:16 -0400 In-Reply-To: <871qr3nkw2.fsf@yhuang6-desk2.ccr.corp.intel.com> References: <202210181535.7144dd15-yujie.liu@intel.com> <87edv4r2ip.fsf@yhuang6-desk2.ccr.corp.intel.com> <871qr3nkw2.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-jyyb0dS8k9AkcqYqrs+D" User-Agent: Evolution 3.42.4 (3.42.4-2.fc35) MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=none (imf10.hostedemail.com: domain of riel@shelob.surriel.com has no SPF policy when checking 96.67.55.147) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666279711; a=rsa-sha256; cv=none; b=yww0+EXPE4PifYbikTnd//AJbLkQIcojeN+N2aGnpDPhrXsCAe1OclYNox5VgEtV+LX2et M99ZeCQAtOX+lQibxH8M0JT0QtyhwvTJvqyeW/gNnkPo7NJAufA8AQyAMY7oZHVcX7h2HX t8nxFybe4uOjJGGk9DbjuZOJwiQ41po= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666279711; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G+AKmQssqgWn324tTSkpngClD/00HGdjNTCBoZyO/RA=; b=za17EZIMFowBsbYu/Inm52WIGRqx/ruHJf4uwt5R2XDeqmT4PIRzrxrT+i+QyaRZ4gFy99 BMiH122RWnPJ5MMkiYImpSPgqPnsn9vL43idMGH3SDN66rTt21kGCKY+mujBZwT+gFHnhI XVveS7/oy4+wp0GHYRqAlAmC79sBTBw= Authentication-Results: imf10.hostedemail.com; dkim=none; spf=none (imf10.hostedemail.com: domain of riel@shelob.surriel.com has no SPF policy when checking 96.67.55.147) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none X-Stat-Signature: knf6ffucoeknt1q1ieeyhx9ji1ocu789 X-Rspamd-Queue-Id: 4E0BDC0033 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1666279711-263636 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --=-jyyb0dS8k9AkcqYqrs+D Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2022-10-20 at 13:07 +0800, Huang, Ying wrote: >=20 > Nathan Chancellor writes: > >=20 > > For what it's worth, I just bisected a massive and visible > > performance > > regression on my Threadripper 3990X workstation to commit > > f35b5d7d676e > > ("mm: align larger anonymous mappings on THP boundaries"), which > > seems > > directly related to this report/analysis. I initially noticed this > > because my full set of kernel builds against mainline went from 2 > > hours > > and 20 minutes or so to over 3 hours. Zeroing in on x86_64 > > allmodconfig, > > which I used for the bisect: > >=20 > > @ 7b5a0b664ebe ("mm/page_ext: remove unused variable in > > offline_page_ext"): > >=20 > > Benchmark 1: make -skj128 LLVM=3D1 allmodconfig all > > =C2=A0 Time (mean =C2=B1 =CF=83):=C2=A0=C2=A0=C2=A0=C2=A0 318.172 s =C2= =B1=C2=A0 0.730 s=C2=A0=C2=A0=C2=A0 [User: 31750.902 s, > > System: 4564.246 s] > > =C2=A0 Range (min =E2=80=A6 max):=C2=A0=C2=A0 317.332 s =E2=80=A6 318.6= 62 s=C2=A0=C2=A0=C2=A0 3 runs > >=20 > > @ f35b5d7d676e ("mm: align larger anonymous mappings on THP > > boundaries"): > >=20 > > Benchmark 1: make -skj128 LLVM=3D1 allmodconfig all > > Time (mean =C2=B1 =CF=83): 406.688 s =C2=B1 0.676 s [User: 31= 819.526 s, System: 16327.022 s] > > Range (min =E2=80=A6 max): 405.954 s =E2=80=A6 407.284 s 3 run >=20 > Have you tried to build with gcc?=C2=A0 Want to check whether is this > clang > specific issue or not. This may indeed be something LLVM specific. In previous tests, GCC has generally seen a benefit from increased THP usage. Many other applications also benefit from getting more THPs. LLVM showing 10% system time before this change, and a whopping 30% system time after that change, suggests that LLVM is behaving quite differently from GCC in some ways. If we can figure out what these differences are, maybe we can just fine tune the code to avoid this issue. I'll try to play around with LLVM compilation a little bit next week, to see if I can figure out what might be going on. I wonder if LLVM is doing lots of mremap calls or something... --=20 All Rights Reversed. --=-jyyb0dS8k9AkcqYqrs+D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEKR73pCCtJ5Xj3yADznnekoTE3oMFAmNRaRAACgkQznnekoTE 3oPi+QgAgglCDIr3nTtc9B3ddLnhdbmzp0chq5yKurOLQJ39FNC8HD7iaukRYXju toNVMhIfT5JulxIO/bpc1zP3tL1+JDZCBwtqyogLZfhdrQQKSaM4b+JqnAKKEDd7 KoZfirX7DoVxCCTZzduqntmhWUsh1xKJdoSuUGXrSa3PfBFaG8lv+VMQDPB45aNq SO3hZdgTEMm6u4LgsspE8Ak6oKLhRqgHtf2Kr2DujdQtLUvmn8HdKNROcZSaGpIJ dbf74SCUe7lZ3Ov3VZsTRR/CV6FeGiKQbKZOs3Va3Xima+aCkVTxKExyaeFShJyi FHKyytxJoZFGuxV0nKr7ZVQdTebHBA== =Eb0d -----END PGP SIGNATURE----- --=-jyyb0dS8k9AkcqYqrs+D--