From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 600ADC43334 for ; Mon, 27 Jun 2022 08:46:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D152B8E0001; Mon, 27 Jun 2022 04:46:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC4DB6B0072; Mon, 27 Jun 2022 04:46:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B653A8E0001; Mon, 27 Jun 2022 04:46:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A499F6B0071 for ; Mon, 27 Jun 2022 04:46:34 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 7099D80EEC for ; Mon, 27 Jun 2022 08:46:34 +0000 (UTC) X-FDA: 79623384708.01.93C7A1E Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf10.hostedemail.com (Postfix) with ESMTP id BF0C8C0027 for ; Mon, 27 Jun 2022 08:46:33 +0000 (UTC) Received: by mail-yb1-f174.google.com with SMTP id i7so15479797ybe.11 for ; Mon, 27 Jun 2022 01:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4vsOnIc+C2gY1UVlbVz/JDciMvgYEap1WYGcSJm7nik=; b=izm5TvVaySgpwnEQF9/y67fAWYu9B2xhHcC5MrbsqfKRyBWcvUJOefttP3U8C7H5yK cf/c+G/MDFeL9d3jWR3ZM1W+uRZZqwp2hTAlphjRcQ4y5gzmknF6zrBjrzS5Oq9593TH prMhKu/+/nH//vvJzE+A8myFZw0Vf2YB5togDcAyh2pSj1lg7+NXGygkUac2Pq5BEBXO 5+wwtXrzIkpK6w7dEXtmMbM9imlU0I4M6KLt1i3bZoyGQ30Vb87yxIZpNyIgjCBXAtgw DSd6AsFRpnuauM1qBq8q0T/kLZUm8tLjA1oB1Fx/TPENkDLwNudIWg8vpilEWofAkp3b qS3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4vsOnIc+C2gY1UVlbVz/JDciMvgYEap1WYGcSJm7nik=; b=xxE3I3X0qRYpBQGyIizH0pJxlkbUbEWqtJoHXp5UWPGfXwDU5uUTv52X3C9CK5Sypb FM2HDqfM9P+2xcjcMIQeW/m0mCC+HQumMfCPtCR7NK2cTcgWDreAZPx4trAbIf1YcXbg cEEEJAW9lnmUZpUWFI4cBCPuaTx15iYB9DojqI+46Jh0wyGy/XxqszTP8Tl+ixfEFiIE 2N3hI072d938wRbtmhoMva77dGvBRSudF8ujbZEHTJEzVCfRp1uYPZYpu9/w+q2Dvnt0 wKGDnT3mWOYiPQ5hSaE3eiv9RhI6BxFQQYr6whCzPsx2RKYi6YvQ8XGzdX6seC2zLZ4o yFPg== X-Gm-Message-State: AJIora/yKgX4n3DwRvw82dxhP0oQ0FsLHJx8rMWLmvByM39fYiUQyn51 PmoJuWvlfF1LSnCLWIEh0l2gB3Md4zIeOEvg04sYvw== X-Google-Smtp-Source: AGRyM1tmdR+Xbj5kJ69vq4foOyHTSytqN33Xz6MWmkpMcamFreKrameDkaAL4/75vWqZ0Cye+/NLca9lf7qZmNQ/HPM= X-Received: by 2002:a25:6c5:0:b0:669:a17a:2289 with SMTP id 188-20020a2506c5000000b00669a17a2289mr12352386ybg.231.1656319592797; Mon, 27 Jun 2022 01:46:32 -0700 (PDT) MIME-Version: 1.0 References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> <20220623185730.25b88096@kernel.org> <20220624070656.GE79500@shbuild999.sh.intel.com> <20220624144358.lqt2ffjdry6p5u4d@google.com> <20220625023642.GA40868@shbuild999.sh.intel.com> <20220627023812.GA29314@shbuild999.sh.intel.com> In-Reply-To: <20220627023812.GA29314@shbuild999.sh.intel.com> From: Eric Dumazet Date: Mon, 27 Jun 2022 10:46:21 +0200 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Feng Tang Cc: Shakeel Butt , Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656319593; a=rsa-sha256; cv=none; b=K/kNMiP1gpfbxnSgdbU1Kz7gssVrWR65aaoxx1ubwCn39tZWC1NJHGQXEL18C3IMV3TlIE ovYCZkKLqeys9sYS6HFFwSwDCAWT0moovTfG9xjQC4HHLQh/GCkV4hzlJ3UcVfIvrhugZQ KHEni7Fbk5Wv7JRaWoGcAyky5sj8QUY= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=izm5TvVa; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of edumazet@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=edumazet@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656319593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4vsOnIc+C2gY1UVlbVz/JDciMvgYEap1WYGcSJm7nik=; b=gVffDWMU5ktG+U49SqGq34qoj37jgf6dtcjZsMNwjoA3atjbubHobqPUud9avOQUcYKO+i t+Fxv3H9XP8OkzOWfuGrR/Sk6G+DZPVeG/llLGlnFTQjuvi/VDHD11zbICQs0FFmufKMs3 RW9aqCX3LR7UiCSbLPdkkNdsDj/EmfA= X-Rspam-User: X-Stat-Signature: c6rxp61pqazn64m3a8j17znhegguu8ra X-Rspamd-Queue-Id: BF0C8C0027 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=izm5TvVa; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of edumazet@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=edumazet@google.com X-Rspamd-Server: rspam03 X-HE-Tag: 1656319593-768989 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 27, 2022 at 4:38 AM Feng Tang wrote: > > On Sat, Jun 25, 2022 at 10:36:42AM +0800, Feng Tang wrote: > > On Fri, Jun 24, 2022 at 02:43:58PM +0000, Shakeel Butt wrote: > > > On Fri, Jun 24, 2022 at 03:06:56PM +0800, Feng Tang wrote: > > > > On Thu, Jun 23, 2022 at 11:34:15PM -0700, Shakeel Butt wrote: > > > [...] > > > > > > > > > > Feng, can you please explain the memcg setup on these test machines > > > > > and if the tests are run in root or non-root memcg? > > > > > > > > I don't know the exact setup, Philip/Oliver from 0Day can correct me. > > > > > > > > I logged into a test box which runs netperf test, and it seems to be > > > > cgoup v1 and non-root memcg. The netperf tasks all sit in dir: > > > > '/sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service' > > > > > > > > > > Thanks Feng. Can you check the value of memory.kmem.tcp.max_usage_in_bytes > > > in /sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service after making > > > sure that the netperf test has already run? > > > > memory.kmem.tcp.max_usage_in_bytes:0 > > Sorry, I made a mistake that in the original report from Oliver, it > was 'cgroup v2' with a 'debian-11.1' rootfs. > > When you asked about cgroup info, I tried the job on another tbox, and > the original 'job.yaml' didn't work, so I kept the 'netperf' test > parameters and started a new job which somehow run with a 'debian-10.4' > rootfs and acutally run with cgroup v1. > > And as you mentioned cgroup version does make a big difference, that > with v1, the regression is reduced to 1% ~ 5% on different generations > of test platforms. Eric mentioned they also got regression report, > but much smaller one, maybe it's due to the cgroup version? This was using the current net-next tree. Used recipe was something like: Make sure cgroup2 is mounted or mount it by mount -t cgroup2 none $MOUNT_POINT. Enable memory controller by echo +memory > $MOUNT_POINT/cgroup.subtree_control. Create a cgroup by mkdir $MOUNT_POINT/job. Jump into that cgroup by echo $$ > $MOUNT_POINT/job/cgroup.procs. The regression was smaller than 1%, so considered noise compared to the benefits of the bug fix. > > Thanks, > Feng > > > And here is more memcg stats (let me know if you want to check more) > > > > > If this is non-zero then network memory accounting is enabled and the > > > slowdown is expected. > > > > >From the perf-profile data in original report, both > > __sk_mem_raise_allocated() and __sk_mem_reduce_allocated() are called > > much more often, which call memcg charge/uncharge functions. > > > > IIUC, the call chain is: > > > > __sk_mem_raise_allocated > > sk_memory_allocated_add > > mem_cgroup_charge_skmem > > charge memcg->tcpmem (for cgroup v2) > > try_charge memcg (for v1) > > > > Also from Eric's one earlier commit log: > > > > " > > net: implement per-cpu reserves for memory_allocated > > ... > > This means we are going to call sk_memory_allocated_add() > > and sk_memory_allocated_sub() more often. > > ... > > " > > > > So this slowdown is related to the more calling of charge/uncharge? > > > > Thanks, > > Feng > > > > > > And the rootfs is a debian based rootfs > > > > > > > > Thanks, > > > > Feng > > > > > > > > > > > > > thanks, > > > > > Shakeel