From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D48D6C3DA59 for ; Mon, 22 Jul 2024 19:52:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B2806B0085; Mon, 22 Jul 2024 15:52:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 561386B0088; Mon, 22 Jul 2024 15:52:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 428D36B0089; Mon, 22 Jul 2024 15:52:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1DB336B0085 for ; Mon, 22 Jul 2024 15:52:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BBCBA1C4050 for ; Mon, 22 Jul 2024 19:52:00 +0000 (UTC) X-FDA: 82368434400.15.3263FC0 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) by imf08.hostedemail.com (Postfix) with ESMTP id EE6D516001E for ; Mon, 22 Jul 2024 19:51:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ITglKGYy; spf=pass (imf08.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721677873; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kXpFkO6uVCl721nl42w3K5FFXtikI48+rMf+SJkzYvU=; b=cKmniLEyW6Tl9h2OLUQ6+E9pPUozcKGnpbDix7eDGEF7t/LwNHK6t8caAp1IpbsaLwv5TU 3q+CrpcHGMPEa83uSfwMuEEouZiIc99EL0Guah4dQ9B9wEbmUHd/NXjGSUu21ZRRG0z9Eb cetpXHDd38wlM/MiqiKfRFanOYahoJM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721677873; a=rsa-sha256; cv=none; b=Ke0wxhIy7qVZWxLakFvIy589/L5kXcTqEhvMQaihj5fwKY5T0NWw92mx8HAWCGh0RoP+Ip 29NNflFKMplSdQycQUWmcyIczx1QPh48JhFeQQYU5oKidWvMdjUgbs6wBW9LQv9u0vJNlS BHoHS/ChBzbRkRXA9B2fh5TZhQOQKOM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ITglKGYy; spf=pass (imf08.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Envelope-To: roman.gushchin@linux.dev DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1721677915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kXpFkO6uVCl721nl42w3K5FFXtikI48+rMf+SJkzYvU=; b=ITglKGYyb1lXrwUVKVZGsl363SW1nV8/rrDKZtK20AOANCms50ADpRQ+QpigmbwmCHOSFG lX7WVgVCUc9WW+JvThFJe2h5OU09mOhq76FCQgdnp7qvTvyjAA+YP5DMue5thFFgqPYZy9 tWeKHb64TGFbIgXopPg+cmgFdKX2PW4= X-Envelope-To: oliver.sang@intel.com X-Envelope-To: oe-lkp@lists.linux.dev X-Envelope-To: lkp@intel.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: mhocko@kernel.org X-Envelope-To: muchun.song@linux.dev X-Envelope-To: cgroups@vger.kernel.org X-Envelope-To: ying.huang@intel.com X-Envelope-To: feng.tang@intel.com X-Envelope-To: fengwei.yin@intel.com Date: Mon, 22 Jul 2024 12:51:46 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Roman Gushchin Cc: Oliver Sang , oe-lkp@lists.linux.dev, lkp@intel.com, Linux Memory Management List , Andrew Morton , Johannes Weiner , Michal Hocko , Muchun Song , cgroups@vger.kernel.org, ying.huang@intel.com, feng.tang@intel.com, fengwei.yin@intel.com Subject: Re: [linux-next:master] [mm] 98c9daf5ae: aim7.jobs-per-min -29.4% regression Message-ID: References: <202407121335.31a10cb6-oliver.sang@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: EE6D516001E X-Stat-Signature: r9r46b8tuzn9iqzqrmmpe9987oq4hz7o X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1721677917-707876 X-HE-Meta: U2FsdGVkX19jwga++vASfE5WVjGalNmYvtNIRZehHJrMZ15+MihWrKIka5Q2B+VBV1DEjwztHa1sZnTwqxvizlTWQfLGs8+MP00hAeJNC7JhQ075aLlJ3TjUSxD8wGo/8NtoP5BMxVGHTk/V+QyfuwB6glUIcvzE0sAjYSNT8MbHh+KHnPYxEnf4g0sw7/4mfu5XbUqLsjJte/8k5w2jcIr+gdVD83G/lHaoU8u4jpsSlY5irh3Vf/eeOlcqqqkiaGUrXFYRHddEG6qOk2qzDP+SEHJPSrHRamM4t9xYtA56kiioDJNwbizF+m77xVFY54jVLlc1qTWXRCpdNPVvGpn3AbGA7QDQVtppiBaQ6c12xS+ryRQPKdAcWfTDVQ/2UWFTWAvvJF1LcWHqdsFOXRA3XD6mRdte2ZwwESYz9EiULRJJHFxKw7+hw7ouInUaLgMvL2U1+rBrbGUS60bpKL9etSuf6B1zArgGN3vFHjksxjf5E0LsLh0d8mqJNRro22uq3rUbkPFWrodJlLlFwz1QLM7EXMTUMYhkr3aQgXg4TUldpxf9IRjnf44FJB+4yUXNO4J0knoUyzS0aQiWlFMncL02otTdZEHoywb4hv/dPe8px+HQ0bZiUNDx2OZWQKCdOT0JYc8wg6UQZXjKkOl/tj9Hzmx9AaUJUhkEr983Ezml0Za77fAVEEmiJ6bhiE/XCaKt1bldMJs8yLMZ9KQvkB1QYHIgnSUxhz8XwIpSUGNGiVZhoQi/kXLQhmkX2Jl0KluBAGI7pzqPSWwbYUT1v4NhSMXgWZmwD6BL9nMNaaUnB/3mBjHRIEXT10AQa7EfI0SyUobm7evMzfEqEydEFxQaQNJ77WWje2PErMBPEAjef3whSNrfwOGWbExcvxhEoXinEoneV738hKCOzS9hVxyXfb/OuisLK2t+3/ilNQBlJvSGFKapPid9xWfzmJ5crMmXrcTb/MPAi1J XYaSsOkr uSwBZFvgojxt1w2us6MxtNkgUe54AhV2SsJW534ojJ1oTt3JpUe2XEMeJz32Y3vIThPGuIbzXdB/rmNqnpu4uxMZKDOwib+3Ya4iu+M9cZ3iJ1SjWNGtNtGoZ0x8C06SRRJXe/ra3tSM2dEy6cVOYdmAaZ4N5Y88gP3Tlenu0a77jwEs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 22, 2024 at 03:37:27PM GMT, Roman Gushchin wrote: > On Fri, Jul 19, 2024 at 03:38:26PM -0700, Shakeel Butt wrote: > > On Fri, Jul 19, 2024 at 05:14:16PM GMT, Roman Gushchin wrote: > > > On Tue, Jul 16, 2024 at 03:53:25PM +0800, Oliver Sang wrote: > > > > hi, Roman, > > > > > > > > On Mon, Jul 15, 2024 at 10:18:39PM +0000, Roman Gushchin wrote: > > > > > On Mon, Jul 15, 2024 at 10:14:31PM +0800, Oliver Sang wrote: > > > > > > hi, Roman Gushchin, > > > > > > > > > > > > On Fri, Jul 12, 2024 at 07:03:31PM +0000, Roman Gushchin wrote: > > > > > > > On Fri, Jul 12, 2024 at 02:04:48PM +0800, kernel test robot wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > kernel test robot noticed a -29.4% regression of aim7.jobs-per-min on: > > > > > > > > > > > > > > > > > > > > > > > > commit: 98c9daf5ae6be008f78c07b744bcff7bcc6e98da ("mm: memcg: guard memcg1-specific members of struct mem_cgroup_per_node") > > > > > > > > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > thank you for the report! > > > > > > > > > > > > > > I'd expect that the regression should be fixed by the commit > > > > > > > "mm: memcg: add cache line padding to mem_cgroup_per_node". > > > > > > > > > > > > > > Can you, please, confirm that it's not the case? > > > > > > > > > > > > > > Thank you! > > > > > > > > > > > > in our this aim7 test, we found the performance partially recovered by > > > > > > "mm: memcg: add cache line padding to mem_cgroup_per_node" but not fully > > > > > > > > > > Thank you for providing the detailed information! > > > > > > > > > > Can you, please, check if the following patch resolves the regression entirely? > > > > > > > > no. in our tests, the following patch has little impact. > > > > I directly apply it upon 6df13230b6 (if this is not the proper applyment, please > > > > let me know, thanks) > > > > > > Hm, interesting. And thank you for the confirmation, you did everything correct. > > > Because the only thing the original patch did was a removal of few fields from > > > the mem_cgroup_per_node struct, there are not many options left here. > > > Would you mind to try the following patch? > > > > > > Thank you and really appreciate your help! > > > > > > > > > diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h > > > index 7e2eb091049a..0e5bf25d324f 100644 > > > --- a/include/linux/memcontrol.h > > > +++ b/include/linux/memcontrol.h > > > @@ -109,6 +109,7 @@ struct mem_cgroup_per_node { > > > > > > /* Fields which get updated often at the end. */ > > > struct lruvec lruvec; > > > + CACHELINE_PADDING(_pad2_); > > > unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; > > > struct mem_cgroup_reclaim_iter iter; > > > }; > > > > > > > > > > I suspect we need padding in the struct mem_cgroup instead of in struct > > mem_cgroup_per_node. I am planning to run some experiments and will > > report back once I have some convincing numbers. > > You mean the regression was attributed to the wrong commit? > Because the blamed commit only removed few fields from the middle > of mem_cgroup_per_node. > The whole series changed the layout of multiple memcg structs, so focusing on a single patch is not necessarily correct. We need to accept the current status and work towards improving the charge and stats update path to similar or better than before performance.