From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83295C7EE23 for ; Thu, 8 Jun 2023 19:36:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 226A58E0001; Thu, 8 Jun 2023 15:36:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D6A56B0074; Thu, 8 Jun 2023 15:36:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09EC08E0001; Thu, 8 Jun 2023 15:36:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EB8456B0072 for ; Thu, 8 Jun 2023 15:36:47 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AE68716049C for ; Thu, 8 Jun 2023 19:36:47 +0000 (UTC) X-FDA: 80880588054.14.2F1639F Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf17.hostedemail.com (Postfix) with ESMTP id C0E4440012 for ; Thu, 8 Jun 2023 19:36:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="r/CLk2x/"; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686253005; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x/9rcr4cOQBqXolDSETy4vJh+qHPOC8JUf2wSjIeiXI=; b=ssWSjgEtzCF8Qxxxpd75nIv6K0WPZCl/Lki/rXCB/NuUj/eILMhJ5qQqkLENjMX+DxudSn AeC6KN82HN//0K6ZS1iq0wlKroDdtrRXl64RX8Aa6me4Zf76sSmIOwJDLCJk3e3vgiDc/n 40Hyc+0E2+XpXXfHJaNwxouOj09AmC0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686253005; a=rsa-sha256; cv=none; b=8q8aBQSYC3d8HOcRUnC0P7EOKngUXZ0BS1AXwTpWXKeD6xoBYdtQD3URqO5a0ixA03Btlm pU92QkNiWXQ3TY2zIHtKH73QDM301J+BBEj6caoULeD83zfoJMgGR02lprPm8BxYwo0J0+ aQQ+As9sbD3ufd5Av/0Uyka2eCP0F8k= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="r/CLk2x/"; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-3f9a81da5d7so296271cf.0 for ; Thu, 08 Jun 2023 12:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686253005; x=1688845005; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=x/9rcr4cOQBqXolDSETy4vJh+qHPOC8JUf2wSjIeiXI=; b=r/CLk2x/MAiwtT34jx28j8dudj2ty3X7Z6AW/HNLRvlNgNmfiqkDQ0/NNutw4FHBXb 0GxcudHIGySJJKR07rp//Sz8EQ8ZFfhocTJk00ey3mc5OjDl2IZWod798Lpi2zvx7U1V eeLJFrRsGd57HAlXrUnxfBEJ71W8PCIYWJAvxBTO4VOPK+cVTh/LUUkPCLYCD4Vjp8X6 JIMzSaDm7kpV42dMdfQEedA2k8yLvpDmj9vjD5yzKOYwPinSm/gwrlJWic3mfMndf033 sQHo8QrFG8aQwEaHW3hC8QOa+z/1temgzuPpED9PsDc4bQrb6wVCZrr40eFtB+46siaW 7ueA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686253005; x=1688845005; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x/9rcr4cOQBqXolDSETy4vJh+qHPOC8JUf2wSjIeiXI=; b=hC/oE6TBgm4n8RUxCoYSAw8NsS48vM28Y5/NA4PewaORsS+MjMrJX/ZqRqd0bqU2x2 U5+jNo6DBh8NSKN44PgAAKTx+wFj0hpbW0BeNOE2pI90aVFrXblvkgTGiZpQ1528gYej JPxKX3s9vbtsdK7FmrHi6Vs12q7mu6idYjux5Y5q5r0JPP0Ny4eGnJ/nZHYA9N7Li2sO 21UYylE0gefjoKJ0qLGjnAG4Iq3/wu9CdrdQ8NdVezCOLNEVZg+I58ZwuoUjqViCtjKh g5Xw/O+Cr7mzX1xJZ/t1CcUSz4oVklPLu5XtO+SIJSJJ6uIPydq0h+RTqh4GE9VGCzHY pZIw== X-Gm-Message-State: AC+VfDyQXzoxNnwHPwtHov9w94cZSsd7YAWub0k0Kn4M/njzGpUdCmRU wLS4aFV6UuIBcIsdCxb4D8w1WS2gtNajc5o9+OX6DA== X-Google-Smtp-Source: ACHHUZ4NMOg5fkuw5/WDThm2yIjlXlvQVqU1jaFR4IjPDZ+0uJvzlIRYOFqBOQTPtHistpqXx/ZVgwuR4r+QEAJARu8= X-Received: by 2002:ac8:5888:0:b0:3ef:2f55:2204 with SMTP id t8-20020ac85888000000b003ef2f552204mr330245qta.6.1686253004843; Thu, 08 Jun 2023 12:36:44 -0700 (PDT) MIME-Version: 1.0 References: <20221024052841.3291983-1-shakeelb@google.com> <20230608111408.s2minsenlcjow7q3@quack3> <20230608173700.wafw5tyw52gwoicu@google.com> In-Reply-To: From: Shakeel Butt Date: Fri, 9 Jun 2023 00:36:33 +0500 Message-ID: Subject: Re: [PATCH] mm: convert mm's rss stats into percpu_counter To: Dennis Zhou Cc: Jan Kara , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, mhocko@suse.cz, vbabka@suse.cz, regressions@lists.linux.dev, Yu Ma Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: C0E4440012 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8pu5bnk6f9j9nodq1i8f9o5wwjy6iqk8 X-HE-Tag: 1686253005-811868 X-HE-Meta: U2FsdGVkX1/nYfSSJi7bHdOQU+tvosz60JB8dHibUI0Myza6IJ8TCPbyUMBam5mIlcJWw0OAo6Ly8KfQhGrlzZeprkwYxIIt4d5j9mE5ZSnZ6WI2bcC+gcuWP1UjDUT5oyF8CnNevZn/n+P9ky9X7YKFitxfzIU9APSkByDu3YEww03xzjthG4Zi4RX5etF/+1rKiiOZosZcTVcNsikxviKBMEQd4F0cm73ROUfXlH0kAVIIcNHNa01D+Sz5SMiNiCdyd4Xztg6DMiTFZP1ev61HpUlL/IjYSbsBXrvmLxhrQQgyfzv2rFcYzeDZwpuBDS16SqZAPjqA5qqD0iaVxsyFidBeDn92dBOMIGU/HN+Pz8Sn3uVSqVP3k+WX0jIybnRLPMU4p8wFe6H4wwAq90y0VlogdSc4fnc46Cu1QQhrgp61gLclk02E/BTxGAdwjuMoQoCaHaEvEIpRutMSslNWQKl+PhxOcRfiEZh7JT+Yz/5YBe9NZw+8dxF7/LXZpLa/QtXUSGnlNFaNpsaELMH8G2h0ntp+uVj2Xadq3qhYtOJexqQgiDyd38nw9tl4SktaN+r8guLufub89boi0MXQe4ecwL0bwv3nIMcmvokJAXFTg3jarVpcJXRstS9ixKus/r0rEWb7g0tJOIHmPU+UgLN8WrG4pUseA+SJDAyyk24hLAXPNxpdocyMtaAA30Qdcj2Q0LOW7r5ihA+GmCUhEcg5b48mcMFb1ocUL2YhQHUjhj2N/AQnlcEpazIhTo0mzOMGK3VIZFnl0IC3yMCZ3tYaPA56F7b/SiVoohthTt29LDTo3ZnRrqjNOnvWcJigFtBvPc+3vi1IQR0thBekM+mUTZ80GpptoSKdmarnozOw2qCLBDiD5ZtaMpeWxxMLStuzqUX4MXH5bJZvPzv8vFpclssB+KsKY02TyljYrYyePbXPhU10hH/TaflCalJF7bu1a6YN6efUxzq 35xN1bw4 wWkbh3Zvmgg7pd/11eTeZ8FOHcbYgeu87C4hOab5ded4H1HpH1/efrlblfiwBY2u/nW6YDq7XQ4ouOKnRSrQc7sSRCDY8grWQ4qnX4eqRJwi6TKtGh+yCFml+ktsiJdJpMy104Ezjr8tEAbtaEVg+iGw58tVPVpBNikXXN1o35GGr7uW7SYR1mp6ogxMRcBBWRmRd0wcw259nhe5FZhx7b5NJGitfytUjHv52H7zKicNt47NhOrZMInjQ3njwKiAVKJQHEy0/BHCUo/IvnWrChhCW/udhHgsPP/affbC/zkv1gig= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jun 9, 2023 at 12:10=E2=80=AFAM Dennis Zhou = wrote: > > Hi Shakeel and Jan, > > On Thu, Jun 08, 2023 at 05:37:00PM +0000, Shakeel Butt wrote: > > On Thu, Jun 08, 2023 at 01:14:08PM +0200, Jan Kara wrote: > > [...] > > > > > > Somewhat late to the game but our performance testing grid has notice= d this > > > commit causes a performance regression on shell-heavy workloads. For > > > example running 'make test' in git sources on our test machine with 1= 92 > > > CPUs takes about 4% longer, system time is increased by about 9%: > > > > > > before (9cd6ffa6025) after (f1a7941243c1) > > > Amean User 471.12 * 0.30%* 481.77 * -1.96%* > > > Amean System 244.47 * 0.90%* 269.13 * -9.09%* > > > Amean Elapsed 709.22 * 0.45%* 742.27 * -4.19%* > > > Amean CPU 100.00 ( 0.20%) 101.00 * -0.80%* > > > > > > Essentially this workload spawns in sequence a lot of short-lived tas= ks and > > > the task startup + teardown cost is what this patch increases. To > > > demonstrate this more clearly, I've written trivial (and somewhat stu= pid) > > > benchmark shell_bench.sh: > > > > > > for (( i =3D 0; i < 20000; i++ )); do > > > /bin/true > > > done > > > > > > And when run like: > > > > > > numactl -C 1 ./shell_bench.sh > > > > > > (I've forced physical CPU binding to avoid task migrating over the ma= chine > > > and cpu frequency scaling interfering which makes the numbers much mo= re > > > noisy) I get the following elapsed times: > > > > > > 9cd6ffa6025 f1a7941243c1 > > > Avg 6.807429 7.631571 > > > Stddev 0.021797 0.016483 > > > > > > So some 12% regression in elapsed time. Just to be sure I've verified= that > > > per-cpu allocator patch [1] does not improve these numbers in any > > > significant way. > > > > > > Where do we go from here? I think in principle the problem could be f= ixed > > > by being clever and when the task has only a single thread, we don't = bother > > > with allocating pcpu counter (and summing it at the end) and just acc= ount > > > directly in mm_struct. When the second thread is spawned, we bite the > > > bullet, allocate pcpu counter and start with more scalable accounting= . > > > These shortlived tasks in shell workloads or similar don't spawn any > > > threads so this should fix the regression. But this is obviously easi= er > > > said than done... > > > > > > > Thanks Jan for the report. I wanted to improve the percpu allocation to > > eliminate this regression as it was reported by intel test bot as well. > > However your suggestion seems seems targetted and reasonable as well. A= t > > the moment I am travelling, so not sure when I will get to this. Do you > > want to take a stab at it or you want me to do it? Also how urgent and > > sensitive this regression is for you? > > > > thanks, > > Shakeel > > > > > > I _think_ I could probably spin you a percpu_alloc_bulk() series in a > couple days for percpu_counters. Let me try and find some time, unless > you had something different in mind. > That would be awesome and thanks a lot.