From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71031CCA480 for ; Mon, 27 Jun 2022 02:38:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2D238E0001; Sun, 26 Jun 2022 22:38:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDDBB6B0072; Sun, 26 Jun 2022 22:38:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA4738E0001; Sun, 26 Jun 2022 22:38:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AB9D56B0071 for ; Sun, 26 Jun 2022 22:38:21 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7AA3F2100F for ; Mon, 27 Jun 2022 02:38:21 +0000 (UTC) X-FDA: 79622456802.19.7D22114 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf15.hostedemail.com (Postfix) with ESMTP id CEFE3A000E for ; Mon, 27 Jun 2022 02:38:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1656297499; x=1687833499; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=vqAGO4WsqTcOAwT6UqFx9w0d7monXGB2WJM15gkxapM=; b=YeTDQRoFR5yj+coI03TJylvUaruNIiiEZSYd7iVrVi6JMltxh3J6qUXe N3MskLtkgMocHjRt/vWBeRVGBGWjucxXlXK9Og12BKvFbSVG+xVRJZUSF 7WCF0E3bwbFq/eIo2toHrVF8yARb1QCgnN/mbYOq2nkzT42GJweqmzbl0 oRdALCxlFML6i+WcOsmwKZNQMep2RctZsFv16r4RotxdV82eTKtMM2pLd hAZcud6oIA3Q5GDH6OjTn+l2J5pcGvvLz/6yW3p6kXqFJLyD2nCJK9mR1 Gem8kQHQcYViHJ/LierZZHGO0fL1V9FiWpNlFTPCXgdNjQ2QYsJs5iTKT Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10390"; a="264381694" X-IronPort-AV: E=Sophos;i="5.92,225,1650956400"; d="scan'208";a="264381694" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Jun 2022 19:38:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,225,1650956400"; d="scan'208";a="646203825" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.146.138]) by fmsmga008.fm.intel.com with ESMTP; 26 Jun 2022 19:38:13 -0700 Date: Mon, 27 Jun 2022 10:38:12 +0800 From: Feng Tang To: Shakeel Butt , Eric Dumazet Cc: Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression Message-ID: <20220627023812.GA29314@shbuild999.sh.intel.com> References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> <20220623185730.25b88096@kernel.org> <20220624070656.GE79500@shbuild999.sh.intel.com> <20220624144358.lqt2ffjdry6p5u4d@google.com> <20220625023642.GA40868@shbuild999.sh.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220625023642.GA40868@shbuild999.sh.intel.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656297500; a=rsa-sha256; cv=none; b=O6lh1MzISg8w+AVPYnirowyvDTaVoSbmkPup4at8XMiTS5JrXRP6XVFiO6dTXw4rwX7o79 fijePL/LtDA4svDNoRk2wFVOTHIMrFbUPkbdTePao+zGyMGIzLFmBCtFSLEwaJuEzw40ye +bKTw52sJE68HysVuqzL2NSURLqffSs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YeTDQRoF; spf=none (imf15.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656297500; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TMJGJfUTeuXDKLM2CIZ5HsS9YkxR9Is+c3+TA6UvkaM=; b=IdUDVtFT2S0sea6CKwxRmxwTbA2awy6DAGpbRcedpMQIy9zD2ZOVE34Pv6Dl8QKwEy/iJQ 3oQ1WNOvgp2IsJRTr+XVQloxg7IhoG5FQEjztb0K9m1PTcvFxgdRW67efqhB3Ggq2QKyRh uH5kaQZvv8Ai4DIqxyoDxGu3x2/hjMU= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: CEFE3A000E Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YeTDQRoF; spf=none (imf15.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Stat-Signature: ob73g9rnfnwft5q3jquip68hm67jbydi X-HE-Tag: 1656297499-338374 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Jun 25, 2022 at 10:36:42AM +0800, Feng Tang wrote: > On Fri, Jun 24, 2022 at 02:43:58PM +0000, Shakeel Butt wrote: > > On Fri, Jun 24, 2022 at 03:06:56PM +0800, Feng Tang wrote: > > > On Thu, Jun 23, 2022 at 11:34:15PM -0700, Shakeel Butt wrote: > > [...] > > > > > > > > Feng, can you please explain the memcg setup on these test machines > > > > and if the tests are run in root or non-root memcg? > > > > > > I don't know the exact setup, Philip/Oliver from 0Day can correct me. > > > > > > I logged into a test box which runs netperf test, and it seems to be > > > cgoup v1 and non-root memcg. The netperf tasks all sit in dir: > > > '/sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service' > > > > > > > Thanks Feng. Can you check the value of memory.kmem.tcp.max_usage_in_bytes > > in /sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service after making > > sure that the netperf test has already run? > > memory.kmem.tcp.max_usage_in_bytes:0 Sorry, I made a mistake that in the original report from Oliver, it was 'cgroup v2' with a 'debian-11.1' rootfs. When you asked about cgroup info, I tried the job on another tbox, and the original 'job.yaml' didn't work, so I kept the 'netperf' test parameters and started a new job which somehow run with a 'debian-10.4' rootfs and acutally run with cgroup v1. And as you mentioned cgroup version does make a big difference, that with v1, the regression is reduced to 1% ~ 5% on different generations of test platforms. Eric mentioned they also got regression report, but much smaller one, maybe it's due to the cgroup version? Thanks, Feng > And here is more memcg stats (let me know if you want to check more) > > > If this is non-zero then network memory accounting is enabled and the > > slowdown is expected. > > >From the perf-profile data in original report, both > __sk_mem_raise_allocated() and __sk_mem_reduce_allocated() are called > much more often, which call memcg charge/uncharge functions. > > IIUC, the call chain is: > > __sk_mem_raise_allocated > sk_memory_allocated_add > mem_cgroup_charge_skmem > charge memcg->tcpmem (for cgroup v2) > try_charge memcg (for v1) > > Also from Eric's one earlier commit log: > > " > net: implement per-cpu reserves for memory_allocated > ... > This means we are going to call sk_memory_allocated_add() > and sk_memory_allocated_sub() more often. > ... > " > > So this slowdown is related to the more calling of charge/uncharge? > > Thanks, > Feng > > > > And the rootfs is a debian based rootfs > > > > > > Thanks, > > > Feng > > > > > > > > > > thanks, > > > > Shakeel