From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 506A4C43334 for ; Mon, 27 Jun 2022 14:53:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B96E76B0071; Mon, 27 Jun 2022 10:53:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B46F66B0072; Mon, 27 Jun 2022 10:53:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0E798E0001; Mon, 27 Jun 2022 10:53:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 93B696B0071 for ; Mon, 27 Jun 2022 10:53:08 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 669B26067E for ; Mon, 27 Jun 2022 14:53:08 +0000 (UTC) X-FDA: 79624308456.20.8C933C7 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf26.hostedemail.com (Postfix) with ESMTP id 7BF9814001C for ; Mon, 27 Jun 2022 14:53:07 +0000 (UTC) Received: by mail-pg1-f179.google.com with SMTP id z14so9383345pgh.0 for ; Mon, 27 Jun 2022 07:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=w4nnCuaaqVKXnI7zfu4XEe/EidzYbOyoiaXDjkUN60I=; b=gM2IEHiWITi7+4u4kxy5XXhdNF32YfAWT9wfPYpqKED+86BqP0njb8qUgEGsh9TRYl FRgvQrSgG3jwRpHz/uynTtwGUcE/VoJkOuXWiry7nkDqDl2zZnQnFn3X5plbLx3+56e4 hC0V9xPTZ2U0O8Y2L8ACjYHGOIzt7S5t8FpBOYyrGUe0ZInCXT+oy7CDYYs4Sf2DvCNY /KqagQAvzNqW1Eh3DsFCZ96+/e2yMMrqBvrtESk0Lw5VH/OFNxJ6iHniPwWMHoDSuUmt CAb+Lssqwd8d5t2N1o+ikpuuiFm7ysYla8g2bWw1dyswpjy7jf+F/+zei0aPAiIRJoaM cDSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=w4nnCuaaqVKXnI7zfu4XEe/EidzYbOyoiaXDjkUN60I=; b=PElhk1DUvSlG/AoCAvItwMp5X+B3GVPPupf6cF0faLmr+5KJ6CXLXff1l3YAnkpM9b VuSzGxGRtIVaHSRtzItsg+8o9sJQIy52q0wnTBkKlpl2KmL0ZQgNgrig2uPgAiqG4DBM bl4Uc6IwrXyGpCHjUvKTkBqTdLTR4M421QDov7pfUQIBQJmcaDVeRLa7u5aH5Ytwl/4L buJ0j4FbaZU8cQ6USy6WxGCfvlcwXKEssTdmNW5qIzXrJweP4RGnobDj6AswSGBkwuQS CutABqOBoHtQquXhcZtZy9Eu4hUUKEn6mvg++/FoytA/QwFLmwAcquIQsdW6AWmyUwMo FKFg== X-Gm-Message-State: AJIora9JbZjXHcat4DffpbaBMLeSgdG9kjwd2Dd8qEsXNjmgd8bmczT8 dZ3iYdJG3cOTxPBz5k4cB7wWdDUAv8mayqBX/6E1KQ== X-Google-Smtp-Source: AGRyM1vzP61WGqfu7fwvpR2g3YBsLeDPE49vmrsIKi0PKEuVDFQCZdthzssP0xkUNMYVlYw/hBLgJOeoW+fBSDeuNHo= X-Received: by 2002:a63:6cc8:0:b0:40d:e553:f200 with SMTP id h191-20020a636cc8000000b0040de553f200mr6417592pgc.166.1656341586171; Mon, 27 Jun 2022 07:53:06 -0700 (PDT) MIME-Version: 1.0 References: <20220623185730.25b88096@kernel.org> <20220624070656.GE79500@shbuild999.sh.intel.com> <20220624144358.lqt2ffjdry6p5u4d@google.com> <20220625023642.GA40868@shbuild999.sh.intel.com> <20220627023812.GA29314@shbuild999.sh.intel.com> <20220627123415.GA32052@shbuild999.sh.intel.com> In-Reply-To: <20220627123415.GA32052@shbuild999.sh.intel.com> From: Shakeel Butt Date: Mon, 27 Jun 2022 07:52:55 -0700 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Feng Tang Cc: Eric Dumazet , Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656341587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w4nnCuaaqVKXnI7zfu4XEe/EidzYbOyoiaXDjkUN60I=; b=Q3ggP3tDMr4UUROkHu5Hr5lin7ItK0KWgkp/nJ3dIKcjvk5IR36dXaJcduD/fxAb9kNkGc HSeB9nznD9THNZelzpwT3TnDlw4J+tJkPnIrl2l3ZftDzoBC1b7aUGkeiO1l/nKzQjGeTa 40YF7l6RG2u+sT/FS0LbswIX9eqeGMw= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gM2IEHiW; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of shakeelb@google.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656341587; a=rsa-sha256; cv=none; b=bJDO34xJNZc1C8GhWMxmNmb4U7Ag92y3c2xpdEbzT8pOutMk9WiyB4mrXqbyJelghoEnpL plMhUByUas/l7torzRJsYPY+QWqDmGug5DgrsXfAObMRYPFZB/lkigY5IyqMrxki18k0cQ LGUp3RFjDbip2XZGPQEsWRx94Ke/g3c= X-Rspamd-Queue-Id: 7BF9814001C Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gM2IEHiW; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of shakeelb@google.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: exkadyp5u1wdjfxui1aht5ghd8pb7uh8 X-HE-Tag: 1656341587-952847 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 27, 2022 at 5:34 AM Feng Tang wrote: > > On Mon, Jun 27, 2022 at 10:46:21AM +0200, Eric Dumazet wrote: > > On Mon, Jun 27, 2022 at 4:38 AM Feng Tang wrote: > [snip] > > > > > > > > > > Thanks Feng. Can you check the value of memory.kmem.tcp.max_usage_in_bytes > > > > > in /sys/fs/cgroup/memory/system.slice/lkp-bootstrap.service after making > > > > > sure that the netperf test has already run? > > > > > > > > memory.kmem.tcp.max_usage_in_bytes:0 > > > > > > Sorry, I made a mistake that in the original report from Oliver, it > > > was 'cgroup v2' with a 'debian-11.1' rootfs. > > > > > > When you asked about cgroup info, I tried the job on another tbox, and > > > the original 'job.yaml' didn't work, so I kept the 'netperf' test > > > parameters and started a new job which somehow run with a 'debian-10.4' > > > rootfs and acutally run with cgroup v1. > > > > > > And as you mentioned cgroup version does make a big difference, that > > > with v1, the regression is reduced to 1% ~ 5% on different generations > > > of test platforms. Eric mentioned they also got regression report, > > > but much smaller one, maybe it's due to the cgroup version? > > > > This was using the current net-next tree. > > Used recipe was something like: > > > > Make sure cgroup2 is mounted or mount it by mount -t cgroup2 none $MOUNT_POINT. > > Enable memory controller by echo +memory > $MOUNT_POINT/cgroup.subtree_control. > > Create a cgroup by mkdir $MOUNT_POINT/job. > > Jump into that cgroup by echo $$ > $MOUNT_POINT/job/cgroup.procs. > > > > > > > > The regression was smaller than 1%, so considered noise compared to > > the benefits of the bug fix. > > Yes, 1% is just around noise level for a microbenchmark. > > I went check the original test data of Oliver's report, the tests was > run 6 rounds and the performance data is pretty stable (0Day's report > will show any std deviation bigger than 2%) > > The test platform is a 4 sockets 72C/144T machine, and I run the > same job (nr_tasks = 25% * nr_cpus) on one CascadeLake AP (4 nodes) > and one Icelake 2 sockets platform, and saw 75% and 53% regresson on > them. > > In the first email, there is a file named 'reproduce', it shows the > basic test process: > > " > use 'performane' cpufre governor for all CPUs > > netserver -4 -D > modprobe sctp > netperf -4 -H 127.0.0.1 -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K & > netperf -4 -H 127.0.0.1 -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K & > netperf -4 -H 127.0.0.1 -t SCTP_STREAM_MANY -c -C -l 300 -- -m 10K & > (repeat 36 times in total) > ... > > " > > Which starts 36 (25% of nr_cpus) netperf clients. And the clients number > also matters, I tried to increase the client number from 36 to 72(50%), > and the regression is changed from 69.4% to 73.7% > Am I understanding correctly that this 69.4% (or 73.7%) regression is with cgroup v2? Eric did the experiments on v2 but on real hardware where the performance impact was negligible. BTW do you see similar regression for tcp as well or just sctp?