From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8520AC25B75 for ; Thu, 23 May 2024 16:47:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD6B86B0085; Thu, 23 May 2024 12:47:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D86AC6B0088; Thu, 23 May 2024 12:47:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4E826B0089; Thu, 23 May 2024 12:47:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A70706B0085 for ; Thu, 23 May 2024 12:47:42 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ECF4F40684 for ; Thu, 23 May 2024 16:47:41 +0000 (UTC) X-FDA: 82150241922.14.08250E4 Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) by imf21.hostedemail.com (Postfix) with ESMTP id EF1D01C0008 for ; Thu, 23 May 2024 16:47:37 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vjsl3iax; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716482858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yxyw2zAEWN0qKyw9lUKHC/W91e6x9aQtxaZ2fTDMCmw=; b=P3bPI818lDHd9ANfvgDiF/Fn5MwJCQJfsjxJpkiI5kQxeJcToyUlJBJNQZFjyRpG/Dxke/ BAnLsHxZoLyz7Ybb2hB8Zv1OAoRwwNhFJLOCJmdnYV0N6HDH++YeX/5XdiQPQwrGe7npts 67xVP6ItrFECvmZTHpSAKIWi46+Xlfw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716482858; a=rsa-sha256; cv=none; b=Ib0CslliTl+YbKHze2xAs24Wns0UVIhUYKrbNp4jYmNtQxKuoiFoSn15jCCtQlILT/zXwZ 5nJ89VbdZlf0Vf1xrLoCqaeQIKVU7FccpHlC+kAngh4mePwefj7uXDCJ3rOX4qP2fPvjW7 xpz6O0jMpwnQoKtNUN0gWHC04LE8WUI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vjsl3iax; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev X-Envelope-To: oliver.sang@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1716482855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=yxyw2zAEWN0qKyw9lUKHC/W91e6x9aQtxaZ2fTDMCmw=; b=vjsl3iaxU+RHw65eH+palqJsGUULJewGhcQljNK+CS9L4f4UpVkRt9aNeVcTPJ4mbH6QdL Y6154HLxwqG3Ocg16EbQ5kfNs9AcaZcQCnGvmI3bTyFy94itcvMlPr62u4EzhHEMy8e14l 3yHgjhViAat5ZktvQg5ew7tSoD/fIeQ= X-Envelope-To: oe-lkp@lists.linux.dev X-Envelope-To: lkp@intel.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: yosryahmed@google.com X-Envelope-To: tjmercier@google.com X-Envelope-To: roman.gushchin@linux.dev X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: mhocko@kernel.org X-Envelope-To: muchun.song@linux.dev X-Envelope-To: cgroups@vger.kernel.org X-Envelope-To: ying.huang@intel.com X-Envelope-To: feng.tang@intel.com X-Envelope-To: fengwei.yin@intel.com Date: Thu, 23 May 2024 09:47:30 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Oliver Sang Cc: oe-lkp@lists.linux.dev, lkp@intel.com, Linux Memory Management List , Andrew Morton , Yosry Ahmed , "T.J. Mercier" , Roman Gushchin , Johannes Weiner , Michal Hocko , Muchun Song , cgroups@vger.kernel.org, ying.huang@intel.com, feng.tang@intel.com, fengwei.yin@intel.com Subject: Re: [linux-next:master] [memcg] 70a64b7919: will-it-scale.per_process_ops -11.9% regression Message-ID: References: <202405171353.b56b845-oliver.sang@intel.com> <20240520034933.wei3dffiuhq7uxhv@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Stat-Signature: thauh497me3j34x4d6fjhgrjocfemggt X-Rspamd-Queue-Id: EF1D01C0008 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1716482857-76219 X-HE-Meta: U2FsdGVkX19/f0joEwNoZHRijwCKmVBP5aNz9xv0iFGM1i1gZzG7xevhdv3NA+tu5iwnGes3AQVq46iTAKg6r4VwX3F3TJ8R9Q3wzbcUL++Vp9TTFtUbgDA2forg4jBmLidiDr+ZiafSCxtR0DFw8O0xHzUQjWRQBc3KumOZiABxiMKjdhMum0BTgxaXofe9VauWazEbdXB6c6RnuEIOOQnYOV/yoI9O0qbdoNqF4YxSd+0gpKV/R9JPjLW4M9CLdHmdBvJR9doMKwvplf8rlTCB9RCFy6Sr175CdswBRk2gHoY0VncJiSxiGEhik3xz9ZT4SHe1ddwZvR+Mu9Vtanu5QtnOvT+SI9BslT+vC6P4rHJeZDiTzGRJNcWS6Vub97zFAhP9gh7TVSA0QZDqiJZKikL6Y3Y8CkiqJnyip9AXa+j4GQ0XVn8s1AYLrBQW1R/2U5CRrY+EJFvO1tZV5bV6/wlIhaYEO6lCY6WCSHoT8ncp9b5w+NlSYGb87S7zAsFPE4p77LG/Lxp4mr086P8m6hMbU3vU+/DCXCuUPdZ5DAEOhzNYzZEdzB5MB2dO7mVkm6DHQlDNbK+BuduA4OE8ft+Jj8uwB9WNw+SV+tUF/T5RW2DR3wsvdH/qfmMtHduVvzc0Q591d3m6rcTqecK4fR857b5ZNDQYe80s0VQg/OfzJ7bh5k4OPXcLYTmfFxNfiUyQir14UGnPzgT6kGZsSKoeTsl5Amk66yGXeY3G40Littf74Tqf45IIZHXUV7f/QcUhxHeLr9eaORIPgbU5PHs/y8SNu6wXMcnO2ordzC6BUkLN9ThW4S0IQTVGGP7WEmmy7him6DwbkmrlKAvYfugIVpEKsKPo3yQ0jjm7ildmmxGmJwwSPi1Fiz6RzTUbbdknzYEoM66flbMIUhTPgvjQKh+D4j2VFJV7wq5BGBl+e+fITvYVMvcYGPIIzbWR4PZJtUCtchfs99F jVY3+SOl YBytbhP7N9XaYbdJ7kqkiB7RWUg0+hPLHWz1PNGdpSeeI7lvO4+TOYmUqdIeji9mpVNEihF7VbS9bhy25Fan/4qeOnm+vRkes2d7RXGrj7d/JiBEwawtf4ZKsFcTOnhiXYBfYNLuUJebMCxaPu+eRZCO8ETlGXtkR9DcltKd6JCUpTe803SCRM2snfMCZfwQmH83C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, May 23, 2024 at 03:48:40PM +0800, Oliver Sang wrote: > hi, Shakeel, > > On Tue, May 21, 2024 at 09:18:19PM -0700, Shakeel Butt wrote: > > On Tue, May 21, 2024 at 10:43:16AM +0800, Oliver Sang wrote: > > > hi, Shakeel, > > > > > [...] > > > > > > we reported regression on a 2-node Skylake server. so I found a 1-node Skylake > > > desktop (we don't have 1 node server) to check. > > > > > > > Please try the following patch on both single node and dual node > > machines: > > > the regression is partially recovered by applying your patch. > (but one even more regression case as below) > > details: > > since you mentioned the whole patch-set behavior last time, I applied the > patch upon > a94032b35e5f9 memcg: use proper type for mod_memcg_state > > below fd2296741e2686ed6ecd05187e4 = a94032b35e5f9 + patch > Thanks a lot Oliver. I have couple of questions and requests: 1. What is the baseline kernel you are using? Is it linux-next or linus? If linux-next, which one specifically? 2. What is the cgroup hierarchy where the workload is running? Is it running in the root cgroup? 3. For the followup experiments when needed, can you please remove the whole series (including 59142d87ab03b8ff) for the base numbers. 4. My experiment [1] on Cooper Lake (2 node) and Skylake (1 node) shows significant improvement but I noticed that I am directly running page_fault2_processes with -t equal nr_cpus but you are running through runtest.py. Also it seems like lkp has modified runtest.py. I will try to run the same setup as yours to repro. [1] https://lore.kernel.org/all/20240523034824.1255719-1-shakeel.butt@linux.dev thanks, Shakeel