From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23905CDB47E for ; Thu, 12 Oct 2023 13:35:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91E8A8D0129; Thu, 12 Oct 2023 09:35:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CFDC8D0002; Thu, 12 Oct 2023 09:35:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BE2A8D0129; Thu, 12 Oct 2023 09:35:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6BBFD8D0002 for ; Thu, 12 Oct 2023 09:35:56 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3B0248061D for ; Thu, 12 Oct 2023 13:35:56 +0000 (UTC) X-FDA: 81336907512.17.1129028 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 6361D4000F for ; Thu, 12 Oct 2023 13:35:54 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lwWDLQru; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697117754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s/IHfa2K6n1mcSpH55sMGbJybaWLv+iEOtk/2jmt1/I=; b=ZvBAFO7UY3IZFTo20/5lgUFNmNA3hz9dsdRX05zc04wxUBX3lfx/REiRfb8R0rvELEFcv2 ke5tskU1n8dIFPw4qidPVcXVpwoemUqJmR3fLRqnLbw6dmbDEWMqnS2e7Afec5059HaLCn gcoFr/6giu5sn7kam5tXsM5j0kkKUck= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lwWDLQru; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697117754; a=rsa-sha256; cv=none; b=2eN05JFqR+rX8EwD5T8bVxrjItEKCvxUZADQxIpG2ldVoD7Zqw7Lqc4h0ssRVnw59/vyHn qQvNHDiU7qpGkdguIgFDweE+fCkCehsWxazxZFegwx9sszA9+KtipeTR1AIj1v/+Yj2NxB MhYM/ZWfr9+x9dEE0R740ShzITniVwg= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1c9c145bb5bso140085ad.1 for ; Thu, 12 Oct 2023 06:35:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697117753; x=1697722553; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=s/IHfa2K6n1mcSpH55sMGbJybaWLv+iEOtk/2jmt1/I=; b=lwWDLQru1JbAm9UqhtUcox0wNMNn54rLK9E+xD2haWvR72As+SGKIbv7CHk8fVeC6t ZEpAdJqAMtBgyt3rYFfuiECaGqSYMJLj6OPuYSWfXZ7IGUFkv8P9OW5aB68juFh6g5lv mIAoHzbC+DpEwPw4vsrpWKwXtjsYm7Tq7Yzlg2UegJX7sXjh4v3JJxuhpVMBAmuyZMaW 0fDMc6l3J7/7ks9UYp7zzQlNXa6Datv8AxuQbNS7rfhb2xqa9cfhwYgUIv1fWXU6WdQh v8eTmEJHxYTa4XrBk+aTvouD3wUUd/0OBlWF9MuTto8YvkNKDC9ZoTkNrjvYKBEriiLX PGOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697117753; x=1697722553; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s/IHfa2K6n1mcSpH55sMGbJybaWLv+iEOtk/2jmt1/I=; b=SQpooiuGCesJ/L1J5f/c4HpWUctTZVzUjL9o6NPRQGmJB3IsyTxV2J3Gwy8DBj2RSo UqK+H1iP0FX9ejas6wqDTBnhRCMNLLKMQ3Kkvi18K+VWoo1+opdp1BHkx7B8YOPoEtNa rF9P479gUUmJYGec0BIU4sTHCezu98VX7nkjIjLqM/0GbEZpM5Ey262g43lXtoGk28Jz f2NHNqsxf+PrAjR8WDECg5p3XphOtc0ruKIXFCf5je5GLJ881qXeDunL9+hvsSn4UEyI Ag+Rr3/OVBXIYU0/sf27aiYvQjoZI8YyLHiPIjYNaCxS/ehBk+6ycTGFLKXiQqQb/u24 6Ygg== X-Gm-Message-State: AOJu0YzY07g4Ms/Q8ZSBDQ5lB9//o/MA+85eGYke/nDSEyed8uVa9eZs MKmmRfKuXh8TWIvR7RnBg8MUQS7xfsac2leCaRLE8w== X-Google-Smtp-Source: AGHT+IFIQPiUjxAepWYWkZcNl2QcaX7znO99g+5O/6OHWHM+twceLTKy3udHT9xGrmOpBud3ydIlVyHq67mULq4BbXc= X-Received: by 2002:a17:903:2444:b0:1c3:3649:1f6a with SMTP id l4-20020a170903244400b001c336491f6amr444875pls.7.1697117752910; Thu, 12 Oct 2023 06:35:52 -0700 (PDT) MIME-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> <20231010032117.1577496-4-yosryahmed@google.com> <20231011003646.dt5rlqmnq6ybrlnd@google.com> In-Reply-To: From: Shakeel Butt Date: Thu, 12 Oct 2023 06:35:41 -0700 Message-ID: Subject: Re: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg To: Yosry Ahmed Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Ivan Babrou , Tejun Heo , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: azrinj17rwpgnbbch5fj8je8r1xzncc5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6361D4000F X-HE-Tag: 1697117754-3773 X-HE-Meta: U2FsdGVkX1+/gwGc1gXputbfMmriJ/eos7ShIh9fEmBT7g5KXp8HhzHf6gPBMSpeDkGSEzqjp0hl8GVXkMx67EL1KYWUauQWEcmf7dF+/w5tOAKc9rqNdizPPQXhAu0wZ2q32hX+juPEzjSdLuotJ9VT0UJaOSyCl/xGE1e3r6M9kz3hVER77GuRhXwMreKQvRVUf11eYuSGPaMW/euizORdsRGXrMUTuXhcjxJ4cN4cu7c+cMWqkMkN9o6KhTMJFUmsUegn9wC7BoXSeNA43acE1NHVon+7y/4huK73vUmeJ2/5j1xxiXlO/6XPeNSbm8WkpplRlXtnUiue61rt2A2G3C9XTGxrRlxzkz+Hn9OSkN/zd3pdyi5EHGnxSqS+YevRx0nZNrjYpmD+OGniS47EdP/qVDuxGeAVGWHBQFG1fe5HvL1zRQEjEX9hQ1ukq3DUe9ZjiiibJdP5qiokKpvE8/Rjgvy9xzw2jmeP7E3ATPegor9qIL7V25QFh/emxiWoL+xE/UsxY1dinNDE5UzR8dsmwOYTPoZkasdi8u5/3b4NZMHfWi+jkVWBpsHbLAsPPJNr8xjcbYiGSizmqDWdb9WLTCu4njmuYC8ITwy7zPhOqTmoYwkCjcP2EJfogWdsuUDSZpF3XuRBPOeQqgOYFpyt3rYuSBtinqdXJSXOFKN/BRrx1qUR2AArfNeJNfK0KRmZTYD7pPm4HaGZfMAwReuKj3dNbDss4wAEmn6+2Kjb+83KeIFx/Wk2RagdjA2885XdVvIUDXCwm0HMt6HSkfjWOQSp+/KtPBkt+NTxt2/1M2LjYndpOQ4ghHZuW+vec/INVmZ1Us24Vj+uzhz1s3jjEhAVRKKxh4p21rCCQ4VGN0tafadpg2PCZHR5yFLRk09mKM+Ko397iVgE14uGYg5m5BWmNLCWYbX7TkhOM24nDUKfpicO6cG54OatsLy5wDLxhZmcMv4rMQz 4SE/klUn Zns4Rh/ZgM8Klxt0tETQDqJk79UWV1Kzrpc/IJ6M4rwjPxomhxZi6VVTBHih6Rd3j2F6KFhhTMry8T6gxWuKlbJl0akK7ulSFupZwT9OX6nfoo9QhaatAOCpruq0wZfY8iN/TMdP6MsTKKfvScnE3agULYVZGrE7VR3HYk9euDkYuDKh4n2p9XAmIetfGbOyJC6Q3PNjyDYqO+1SeAQDQIiz8yC+tCb61mckoNiroe7BLY+GxDJTg6U/9yaZERUXKDff9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.046527, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 12, 2023 at 1:04=E2=80=AFAM Yosry Ahmed = wrote: > > On Wed, Oct 11, 2023 at 8:13=E2=80=AFPM Yosry Ahmed wrote: > > > > On Wed, Oct 11, 2023 at 5:46=E2=80=AFAM Shakeel Butt wrote: > > > > > > On Tue, Oct 10, 2023 at 6:48=E2=80=AFPM Yosry Ahmed wrote: > > > > > > > > On Tue, Oct 10, 2023 at 5:36=E2=80=AFPM Shakeel Butt wrote: > > > > > > > > > > On Tue, Oct 10, 2023 at 03:21:47PM -0700, Yosry Ahmed wrote: > > > > > [...] > > > > > > > > > > > > I tried this on a machine with 72 cpus (also ixion), running bo= th > > > > > > netserver and netperf in /sys/fs/cgroup/a/b/c/d as follows: > > > > > > # echo "+memory" > /sys/fs/cgroup/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a > > > > > > # echo "+memory" > /sys/fs/cgroup/a/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b/c > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/c/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b/c/d > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > # ./netserver -6 > > > > > > > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > # for i in $(seq 10); do ./netperf -6 -H ::1 -l 60 -t TCP_SENDF= ILE -- > > > > > > -m 10K; done > > > > > > > > > > You are missing '&' at the end. Use something like below: > > > > > > > > > > #!/bin/bash > > > > > for i in {1..22} > > > > > do > > > > > /data/tmp/netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K & > > > > > done > > > > > wait > > > > > > > > > > > > > Oh sorry I missed the fact that you are running instances in parall= el, my bad. > > > > > > > > So I ran 36 instances on a machine with 72 cpus. I did this 10 time= s > > > > and got an average from all instances for all runs to reduce noise: > > > > > > > > #!/bin/bash > > > > > > > > ITER=3D10 > > > > NR_INSTANCES=3D36 > > > > > > > > for i in $(seq $ITER); do > > > > echo "iteration $i" > > > > for j in $(seq $NR_INSTANCES); do > > > > echo "iteration $i" >> "out$j" > > > > ./netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K >> "out$j" = & > > > > done > > > > wait > > > > done > > > > > > > > cat out* | grep 540000 | awk '{sum +=3D $5} END {print sum/NR}' > > > > > > > > Base: 22169 mbps > > > > Patched: 21331.9 mbps > > > > > > > > The difference is ~3.7% in my runs. I am not sure what's different. > > > > Perhaps it's the number of runs? > > > > > > My base kernel is next-20231009 and I am running experiments with > > > hyperthreading disabled. > > > > Using next-20231009 and a similar 44 core machine with hyperthreading > > disabled, I ran 22 instances of netperf in parallel and got the > > following numbers from averaging 20 runs: > > > > Base: 33076.5 mbps > > Patched: 31410.1 mbps > > > > That's about 5% diff. I guess the number of iterations helps reduce > > the noise? I am not sure. > > > > Please also keep in mind that in this case all netperf instances are > > in the same cgroup and at a 4-level depth. I imagine in a practical > > setup processes would be a little more spread out, which means less > > common ancestors, so less contended atomic operations. > > > (Resending the reply as I messed up the last one, was not in plain text) > > I was curious, so I ran the same testing in a cgroup 2 levels deep > (i.e /sys/fs/cgroup/a/b), which is a much more common setup in my > experience. Here are the numbers: > > Base: 40198.0 mbps > Patched: 38629.7 mbps > > The regression is reduced to ~3.9%. > > What's more interesting is that going from a level 2 cgroup to a level > 4 cgroup is already a big hit with or without this patch: > > Base: 40198.0 -> 33076.5 mbps (~17.7% regression) > Patched: 38629.7 -> 31410.1 (~18.7% regression) > > So going from level 2 to 4 is already a significant regression for > other reasons (e.g. hierarchical charging). This patch only makes it > marginally worse. This puts the numbers more into perspective imo than > comparing values at level 4. What do you think? This is weird as we are running the experiments on the same machine. I will rerun with 2 levels as well. Also can you rerun the page fault benchmark as well which was showing 9% regression in your original commit message?