From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5B29C7EE29 for ; Thu, 8 Jun 2023 18:08:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 332068E0002; Thu, 8 Jun 2023 14:08:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E25B8E0001; Thu, 8 Jun 2023 14:08:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AA818E0002; Thu, 8 Jun 2023 14:08:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0A46B8E0001 for ; Thu, 8 Jun 2023 14:08:04 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CE5CA1203DE for ; Thu, 8 Jun 2023 18:08:03 +0000 (UTC) X-FDA: 80880364446.20.D96EE04 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf15.hostedemail.com (Postfix) with ESMTP id CB8F3A0004 for ; Thu, 8 Jun 2023 18:08:00 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=DzKX09a3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="Gzs5/PuX"; dmarc=none; spf=pass (imf15.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686247681; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KuVHWqF+Vmzlt+SUW4g6KuATrBuHpEsTJNICe6iwrHc=; b=LBZGY4ZRJBNMUxpJbQt+IiDqRM7VU4Ulbh84+H4Vb1GubGZnw1dZhFqViF/lhv1K8ZwPhm RoS/XPCL3e/KjiwVxZ/nKtCkTmLPu3tfO3RRP68swjZaV/RfPwkqGbp507XC6AXtokVjII wzLwrWqMgmqODEVNgcbV3N5/OU3fl2s= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=DzKX09a3; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="Gzs5/PuX"; dmarc=none; spf=pass (imf15.hostedemail.com: domain of jack@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=jack@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686247681; a=rsa-sha256; cv=none; b=zRXiZgDhtMJVMsjWXl3R47g7dENQg+bPYgnZG4pQwZn9QmXrH7fwJd2+NFIX1juMhpHOOq /XUv5heTV2CxNkYnBIis1SkzQLtA0cub0WokBdRGzqKVlGXOUhVMesAu5J5An203k7I2oY P2m18JNTGwXWd72Z9Jf8C/egXY06lH4= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5A8B41FD5F; Thu, 8 Jun 2023 18:07:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1686247679; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=KuVHWqF+Vmzlt+SUW4g6KuATrBuHpEsTJNICe6iwrHc=; b=DzKX09a3MIGcFGy/bs4aXveZ1NRbWdFmwIJGNRB9beMcnjg3/3rQybTWLiKEEr28ctuSsN ofr/NTBppLIfWVTmFNXZUx46CdJL0xK3Ddwm+KTSFVGpAx9e/RlIfnXRLns9g3ObZty3iL f+CdUzcXf1FewvV7CQfhmUyhAGPfQQo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1686247679; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=KuVHWqF+Vmzlt+SUW4g6KuATrBuHpEsTJNICe6iwrHc=; b=Gzs5/PuXubrvgfiZMpux40aW3clbzJ70xqZmpsjIQJjGKctEyd1ZInHwX45CvkQBIAO0YA iK5xiXdJ0Q+vtKAw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 4C8C6138E6; Thu, 8 Jun 2023 18:07:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id QcinEv8YgmSBYQAAMHmgww (envelope-from ); Thu, 08 Jun 2023 18:07:59 +0000 Received: by quack3.suse.cz (Postfix, from userid 1000) id DD23BA0749; Thu, 8 Jun 2023 20:07:58 +0200 (CEST) Date: Thu, 8 Jun 2023 20:07:58 +0200 From: Jan Kara To: Shakeel Butt Cc: Jan Kara , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, mhocko@suse.cz, vbabka@suse.cz, regressions@lists.linux.dev, Yu Ma Subject: Re: [PATCH] mm: convert mm's rss stats into percpu_counter Message-ID: <20230608180758.z4z4ijdjgfe4mbx4@quack3> References: <20221024052841.3291983-1-shakeelb@google.com> <20230608111408.s2minsenlcjow7q3@quack3> <20230608173700.wafw5tyw52gwoicu@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230608173700.wafw5tyw52gwoicu@google.com> X-Rspamd-Queue-Id: CB8F3A0004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 4r9f7doeg8a1ouaziitgt7h9cngbsk8p X-HE-Tag: 1686247680-361343 X-HE-Meta: U2FsdGVkX19Lb2ovqPpYdzG7WzpyvpkUQ3cG0lSAm9ULQ2VtbAMUe1Lw2pqG8YpY1lXECfoS/XDxTLz1LhKiKc5qpHyaBA2Q+b3ZOrO0RrkjETKOZDRc1nwdWHzOZnC/rflqMtCW3DS4Ly/zWwAapTpIT2w7XEw5e2/Kw5SHpjg4QBP2sXS8qSQqIOrq89OisiTbn3rYLpUFKmbcwr+8jVzz5dupYp+snSLMrv/aBIh0MiWppOGNj/RgQolz1mLKcj5LHfPCqNn0DaNPQy9l/JD+mOXNTm3fctMICbHsVYjPei2A+BoIx5PzVhPUdb1hZQvyDY0ym0kMpjXc/atab5zJo8nBpi00PHJf1S2cXqvvjcsNHvKOVNI1WG+g5EFwr/ndNzufMSmnWbaEAMxvWpgUcfxTidNtk/mvqkpZHBNev5L9diPg3ejgcZdETI/t3KEBwSqFBTD1kSURdRtywktXFpoT92jPq/BCtgRl8INhUb1idoewWXb0SsokcrpeDhgFgn4bBWOgp8VPLz1uNT8vNbRem7aeWyOiiom4s5lxtl4rmkcAjdTT/EQBlQz+BXTKPzki0RBdgqJRrwdh73Bk94bnF8fmQvpZrBvRBfWdrBIEdNfqIRDeQoHKwfAznlTu15F8d4s4/kfrCaqCJndgmi+ar9+DYLQdF886Lk1B9H9aC2EDFNUyZ7GIGPWn8+OZ+6+ywei9daJbiR+6bE8p9K1wzGL7I16l39qBNyA2yJRwXIdq143OJpRSSyDyDVtJVX4+UIEtVKlpJpHDr27zzOw4hGm8wTXGcHKbxw9ODkg0sYk7MnLni1payMhqVEDTPv1vHhkx6YLI8WVKjN1TVE+SWL6yLgDOiqbqvQrG1g0I8a/gUdCXQwEOpVihJLcgDVC6CSMvaXLRS7P64k0bD+NU3Wot5zahoIoHbyIP4S5GixH95XgrzqN7HgPJ0nLHm5TGmJLTml0MUiO cuzc+EKA fWLYqQjCnIBjOYD8bAJOYmrboh95eHoSpyKHdlJho6dYIJTt3qSG6VOSwwJDrsPbqHmnxrjkELS919v7uGm8ERkp/Yocs3c2yB4frq+Cy5S1SmDYQwc/ykE7P1JvZMg19H+IQ2lBm92APi1hySEMAcZEoGXrTcn3NMeEEREFaTiQn+OzTt43Bx1zcwaJHHnqROaWrDVqu6p8YHx7WiVXNvD0OpbpA54gJX7k/3uf+Jwuwwn0cMlDmxnmretBnt1WJUJ2Idb4GAttc6LBjxIxxJ87GAqDGhmJMBHw9VumruBbQ6xc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 08-06-23 17:37:00, Shakeel Butt wrote: > On Thu, Jun 08, 2023 at 01:14:08PM +0200, Jan Kara wrote: > [...] > > > > Somewhat late to the game but our performance testing grid has noticed this > > commit causes a performance regression on shell-heavy workloads. For > > example running 'make test' in git sources on our test machine with 192 > > CPUs takes about 4% longer, system time is increased by about 9%: > > > > before (9cd6ffa6025) after (f1a7941243c1) > > Amean User 471.12 * 0.30%* 481.77 * -1.96%* > > Amean System 244.47 * 0.90%* 269.13 * -9.09%* > > Amean Elapsed 709.22 * 0.45%* 742.27 * -4.19%* > > Amean CPU 100.00 ( 0.20%) 101.00 * -0.80%* > > > > Essentially this workload spawns in sequence a lot of short-lived tasks and > > the task startup + teardown cost is what this patch increases. To > > demonstrate this more clearly, I've written trivial (and somewhat stupid) > > benchmark shell_bench.sh: > > > > for (( i = 0; i < 20000; i++ )); do > > /bin/true > > done > > > > And when run like: > > > > numactl -C 1 ./shell_bench.sh > > > > (I've forced physical CPU binding to avoid task migrating over the machine > > and cpu frequency scaling interfering which makes the numbers much more > > noisy) I get the following elapsed times: > > > > 9cd6ffa6025 f1a7941243c1 > > Avg 6.807429 7.631571 > > Stddev 0.021797 0.016483 > > > > So some 12% regression in elapsed time. Just to be sure I've verified that > > per-cpu allocator patch [1] does not improve these numbers in any > > significant way. > > > > Where do we go from here? I think in principle the problem could be fixed > > by being clever and when the task has only a single thread, we don't bother > > with allocating pcpu counter (and summing it at the end) and just account > > directly in mm_struct. When the second thread is spawned, we bite the > > bullet, allocate pcpu counter and start with more scalable accounting. > > These shortlived tasks in shell workloads or similar don't spawn any > > threads so this should fix the regression. But this is obviously easier > > said than done... > > > > Thanks Jan for the report. I wanted to improve the percpu allocation to > eliminate this regression as it was reported by intel test bot as well. > However your suggestion seems seems targetted and reasonable as well. At > the moment I am travelling, so not sure when I will get to this. Do you > want to take a stab at it or you want me to do it? Also how urgent and > sensitive this regression is for you? It is not really urgent but eventually we'd like to get this fixed (like within couple of months). I have currently other stuff in progress so if you could get to it, it would be nice, otherwise I should be able to look into this in a week or two. Honza -- Jan Kara SUSE Labs, CR