From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51FF6C433F5 for ; Fri, 10 Sep 2021 02:34:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CDA8C61153 for ; Fri, 10 Sep 2021 02:34:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CDA8C61153 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 16AF1900002; Thu, 9 Sep 2021 22:34:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 11A926B0072; Thu, 9 Sep 2021 22:34:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00830900002; Thu, 9 Sep 2021 22:34:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id E1B306B0071 for ; Thu, 9 Sep 2021 22:34:23 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 86FA93209A for ; Fri, 10 Sep 2021 02:34:23 +0000 (UTC) X-FDA: 78570094806.06.AFDAA4B Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf06.hostedemail.com (Postfix) with ESMTP id A3F5F801A8A0 for ; Fri, 10 Sep 2021 02:34:22 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10102"; a="284674575" X-IronPort-AV: E=Sophos;i="5.85,282,1624345200"; d="scan'208";a="284674575" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2021 19:34:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,282,1624345200"; d="scan'208";a="540095773" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.146.151]) by FMSMGA003.fm.intel.com with ESMTP; 09 Sep 2021 19:34:16 -0700 Date: Fri, 10 Sep 2021 10:34:15 +0800 From: Feng Tang To: Shakeel Butt Cc: kernel test robot , Andrew Morton , 0day robot , Marek Szyprowski , Hillf Danton , Huang Ying , Johannes Weiner , Michal Hocko , Michal Koutn?? , Muchun Song , Roman Gushchin , Tejun Heo , LKML , lkp@lists.01.org, Xing Zhengjun , Linux MM , mm-commits@vger.kernel.org, Linus Torvalds Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression Message-ID: <20210910023415.GB94434@shbuild999.sh.intel.com> References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> <20210910010842.GA94434@shbuild999.sh.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A3F5F801A8A0 X-Stat-Signature: 7f7sbc87t5mdpt75yqptm1arfy4o6cma Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=intel.com (policy=none); spf=none (imf06.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.100) smtp.mailfrom=feng.tang@intel.com X-HE-Tag: 1631241262-383500 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 09, 2021 at 06:19:06PM -0700, Shakeel Butt wrote: [...] > > > > > I am looking into this. I was hoping we have resolution for [1] as > > > > > these patches touch similar data structures. > > > > > > > > > > [1] https://lore.kernel.org/all/20210811031734.GA5193@xsang-OptiPlex-9020/T/#u > > > > > > > > I tried 2 debug methods for that 36.4% vm-scalability regression: > > > > > > > > 1. Disable the HW cache prefetcher, no effect on this case > > > > 2. relayout and add padding to 'struct cgroup_subsys_state', reduce > > > > the regression to 3.1% > > > > > > > > > > Thanks Feng but it seems like the issue for this commit is different. > > > Rearranging the layout didn't help. Actually the cause of slowdown is > > > the call to queue_work() inside __mod_memcg_lruvec_state(). > > > > > > At the moment, queue_work() is called after 32 updates. I changed it > > > to 128 and the slowdown of will-it-scale:page_fault[1|2|3] halved > > > (from around 10% to 5%). I am unable to run reaim or > > > will-it-scale:fallocate2 as I was getting weird errors. > > > > > > Feng, is it possible for you to run these benchmarks with the change > > > (basically changing MEMCG_CHARGE_BATCH to 128 in the if condition > > > before queue_work() inside __mod_memcg_lruvec_state())? > > > > When I checked this, I tried different changes, including this batch > > number change :), but it didn't recover the regression (the regression > > is slightly reduced to about 12%) [...] > > Another change we can try is to remove this specific queue_work() > altogether because this is the only significant change for the > workload. That will give us the base performance number. If that also > has regression then there are more issues to debug. Thanks a lot for > your help. I just tested with patch removing the queue_work() in __mod_memcg_lruvec_state(), and the regression is gone. Also to avoid some duplication of debugging, here are some other tries I did: * add padding in 'struct lruvec' for 'lru_lock', no effect * add padding in 'mem_cgroup_per_node' between 'lruvec_stats_percpu' and 'lruvec_stats', no effect. Thanks, Feng