From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99EB3C433EF for ; Fri, 24 Jun 2022 07:07:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFD1E8E01EC; Fri, 24 Jun 2022 03:07:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E857A8E01E7; Fri, 24 Jun 2022 03:07:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFE4D8E01EC; Fri, 24 Jun 2022 03:07:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BA4CC8E01E7 for ; Fri, 24 Jun 2022 03:07:07 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8A7ACD95 for ; Fri, 24 Jun 2022 07:07:07 +0000 (UTC) X-FDA: 79612247694.22.CFD24BD Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 60A741A002B for ; Fri, 24 Jun 2022 07:07:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1656054425; x=1687590425; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=I94Jl9kETqNl2DSK3suPqG7/Oh9kRDrmgsP9ccLnRJ8=; b=a5RwlH18emUBIR94CXF/UMpEKLj4r1CibVoXH6HdCPMVWlybu3v20AUs Edi6K0m8gjBxK1CYinAyw26XT2ql1k/dSOpG3f/uwN7/yJvtbFxhkURi2 PUvJTxu5E9iEoDkRi2KwTDJwQvLQEndvTtb4+m6WTWtYPUy20m03oxEBk 8WznS62+IS7LHfbWDHxTr5ctL/WFuUfkstudgFqOh3USpkvl0+oV6dSf4 j4AZkMzANoh48qo7WEOjqDRZCDAmvtKNaB7zP4cIGDioJzKwLXJ0DHeCD xhnp9U2tYkJumjFZW2l4DBslMMpaaLFK9FZrPhLBs2UR7A8gW49NGKHxl A==; X-IronPort-AV: E=McAfee;i="6400,9594,10387"; a="282023274" X-IronPort-AV: E=Sophos;i="5.92,218,1650956400"; d="scan'208";a="282023274" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jun 2022 00:07:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,218,1650956400"; d="scan'208";a="691408221" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.146.138]) by fmsmga002.fm.intel.com with ESMTP; 24 Jun 2022 00:06:56 -0700 Date: Fri, 24 Jun 2022 15:06:56 +0800 From: Feng Tang To: Shakeel Butt Cc: Eric Dumazet , Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression Message-ID: <20220624070656.GE79500@shbuild999.sh.intel.com> References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> <20220623185730.25b88096@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656054425; a=rsa-sha256; cv=none; b=vdzIZ6BmcKGh9yTFwhROzORuMwgrfjx0b6ay2/sdBqxjxr4HwReuCtQtsma6A7b6LDpoRF H1O74dl1dN3+gIEVRq4JDtrVxSm/QyzEz7G4q9RSzCn0DEcXbpa7V6FFoEcj4eYLF+x1G0 hV035zn6tPtqTb2GkbwW3TQN0Gc7QYQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=a5RwlH18; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf19.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=feng.tang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656054425; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pHcutb6W6VJ95xyqIAbvT95f9jU95DQIQ756YBURjaM=; b=3fa9tY8NzN82HgVv1jGuq5NVvdn8T6fSrXAeoI3U5FsWPwXzPmcT1MCswHq/Tu1tYSCELT o6XlbYPLEKMhx4xJrC/j5UPz6lJXpqHlrlgjUCQe6PGWj3rr2/onyC7CeCITAiUaQAC8jf oL6KDAgPaQY2sN1XUxXSDuHNF+m809U= X-Stat-Signature: 4jdz1q9w5if7p4uiauy63om6e4c4hs45 X-Rspam-User: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 60A741A002B Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=a5RwlH18; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf19.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.65) smtp.mailfrom=feng.tang@intel.com X-HE-Tag: 1656054425-25258 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 23, 2022 at 11:34:15PM -0700, Shakeel Butt wrote: > CCing memcg folks. > > The thread starts at > https://lore.kernel.org/all/20220619150456.GB34471@xsang-OptiPlex-9020/ > > On Thu, Jun 23, 2022 at 9:14 PM Eric Dumazet wrote: > > > > On Fri, Jun 24, 2022 at 3:57 AM Jakub Kicinski wrote: > > > > > > On Thu, 23 Jun 2022 18:50:07 -0400 Xin Long wrote: > > > > From the perf data, we can see __sk_mem_reduce_allocated() is the one > > > > using CPU the most more than before, and mem_cgroup APIs are also > > > > called in this function. It means the mem cgroup must be enabled in > > > > the test env, which may explain why I couldn't reproduce it. > > > > > > > > The Commit 4890b686f4 ("net: keep sk->sk_forward_alloc as small as > > > > possible") uses sk_mem_reclaim(checking reclaimable >= PAGE_SIZE) to > > > > reclaim the memory, which is *more frequent* to call > > > > __sk_mem_reduce_allocated() than before (checking reclaimable >= > > > > SK_RECLAIM_THRESHOLD). It might be cheap when > > > > mem_cgroup_sockets_enabled is false, but I'm not sure if it's still > > > > cheap when mem_cgroup_sockets_enabled is true. > > > > > > > > I think SCTP netperf could trigger this, as the CPU is the bottleneck > > > > for SCTP netperf testing, which is more sensitive to the extra > > > > function calls than TCP. > > > > > > > > Can we re-run this testing without mem cgroup enabled? > > > > > > FWIW I defer to Eric, thanks a lot for double checking the report > > > and digging in! > > > > I did tests with TCP + memcg and noticed a very small additional cost > > in memcg functions, > > because of suboptimal layout: > > > > Extract of an internal Google bug, update from June 9th: > > > > -------------------------------- > > I have noticed a minor false sharing to fetch (struct > > mem_cgroup)->css.parent, at offset 0xc0, > > because it shares the cache line containing struct mem_cgroup.memory, > > at offset 0xd0 > > > > Ideally, memcg->socket_pressure and memcg->parent should sit in a read > > mostly cache line. > > ----------------------- > > > > But nothing that could explain a "-69.4% regression" > > > > memcg has a very similar strategy of per-cpu reserves, with > > MEMCG_CHARGE_BATCH being 32 pages per cpu. > > > > It is not clear why SCTP with 10K writes would overflow this reserve constantly. > > > > Presumably memcg experts will have to rework structure alignments to > > make sure they can cope better > > with more charge/uncharge operations, because we are not going back to > > gigantic per-socket reserves, > > this simply does not scale. > > Yes I agree. As you pointed out there are fields which are mostly > read-only but sharing cache lines with fields which get updated and > definitely need work. > > However can we first confirm if memcg charging is really the issue > here as I remember these intel lkp tests are configured to run in root > memcg and the kernel does not associate root memcg to any socket (see > mem_cgroup_sk_alloc()). > > If these tests are running in non-root memcg, is this cgroup v1 or v2? > The memory counter and the 32 pages per cpu stock are only used on v2. > For v1, there is no per-cpu stock and there is a separate tcpmem page > counter and on v1 the network memory accounting has to be enabled > explicitly i.e. not enabled by default. > > There is definite possibility of slowdown on v1 but let's first > confirm the memcg setup used for this testing environment. > > Feng, can you please explain the memcg setup on these test machines > and if the tests are run in root or non-root memcg? I don't know the exact setup, Philip/Oliver from 0Day can correct me. I logged into a test box which runs netperf test, and it seems to be cgoup v1 and non-root memcg. The netperf tasks all sit in dir: '/sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service' And the rootfs is a debian based rootfs Thanks, Feng > thanks, > Shakeel