From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE095CDB46E for ; Thu, 12 Oct 2023 13:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 599BF8D0128; Thu, 12 Oct 2023 09:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 521538D0002; Thu, 12 Oct 2023 09:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C2068D0128; Thu, 12 Oct 2023 09:29:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2620E8D0002 for ; Thu, 12 Oct 2023 09:29:51 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E58C91204B2 for ; Thu, 12 Oct 2023 13:29:50 +0000 (UTC) X-FDA: 81336892140.03.6948B15 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf12.hostedemail.com (Postfix) with ESMTP id D52D440024 for ; Thu, 12 Oct 2023 13:29:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=gNIS7H5x; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697117389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ybk+sUTEHYxJD085NMJHTHmsqfChe2BT9mQZhimnZfU=; b=6jGkJH/wVNqVzSZXOKfSvM5fHR8xUrUwpZ7TvKcu749UY4HgmO4WAuY2xDIy3XZZ4YCuQr x4zR+JnXkj9b7AW0WlDUw08T1pLt3c+4W5xVS3k+a4ukiibvNu+lof0Ore42jqk1Yw+Oi0 G3uH8UfqnHimddjw6oelw9bbHvWOOZQ= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=gNIS7H5x; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697117389; a=rsa-sha256; cv=none; b=iGxNyzl3dXDpGebnMp0XlVdAJfryQMl1V/uvyKDpZEFtii4GDhVpkNV1wROaEyhu8gmQoI 7mkDcxJQSfW7toCHu/6UMMHRFVSoejEutoVvVKTcLhER+wtLtBuFI60lwGuuMO4EizN4oA +SoOiHsjxms92JSzCZv99Ccbz4edK9Q= Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-41959feaae2so5885001cf.0 for ; Thu, 12 Oct 2023 06:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1697117388; x=1697722188; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=Ybk+sUTEHYxJD085NMJHTHmsqfChe2BT9mQZhimnZfU=; b=gNIS7H5xfWBpM1KeUQBc9GLjiRNcXmnLMpg8Tr81PBvSXAYJLBGRi27pOmGhNBc6Xx 8oURuuQeGZKKYhr+pV40Z2ImC94/nGVJAjlbnV2HjFtNYQvisk7AZ+ee82hDNhJf4IFO sPJhWMErtjZw/EODhPOr5oLYd+4tbWOzwue0fPF2R7t4KFNDBiKQDxSkexKyGUEB1Cn+ zag8iObZvTs2uXYIixCWZiwkB0BLZNOz/ljNCQM2nEXDBAbBnxvCu3PCJmH0g/GXGc/a Cr7mvEaPj8vcKlq6fOaWDP51kstSQprn3+P1ZnlYn0gC2hH0idXfWjkDnbgwKeDwi9Yc gSNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697117388; x=1697722188; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ybk+sUTEHYxJD085NMJHTHmsqfChe2BT9mQZhimnZfU=; b=aqrjl59NT2hLleDtBwRHlurWD/0oeDGTshUgmRKS4tZObGbKFd/a8V1bOdNyZ+Ag12 YZ2vtvn1GJQrG7uwfjx5d9R0znWOa0ocAwtGNuetjdd4/JdSgbOZe3lJhsrvtvE6ae1C tZOLrM5KAn1bhjnORDzmUt9WhLA3XAj2xNXct5S4tMEIGCSgrFpOXAgouu7csJ7dsNb1 wA1AhQEgQebVVZqzp5V7aduhrJd2v5+L5da5qIRjdreGJaYlHLFmP5L4igApqbrDnphr ozJoUkvWBsjAnxrEAjO9W+Up07as9qdC7muAg4Bp9JLdVUq5bLReG1lOiBqVdBBua5hs 4a8w== X-Gm-Message-State: AOJu0YyQuFGOp/1BgARr7ScuXwilYubf4BJniYV2wgqrAhAd+HZpHkrK q47ORNMOsBXM58C9BwtZpJRysQ== X-Google-Smtp-Source: AGHT+IEu9EQEyDkwsQbhuGY67hZLMSeQYZ5ijIozLtPEEZ9plkLT155cbIZs7B+UAUVtPaIKEj7Fzg== X-Received: by 2002:a05:622a:11d4:b0:416:5e11:f7ec with SMTP id n20-20020a05622a11d400b004165e11f7ecmr31277304qtk.52.1697117387879; Thu, 12 Oct 2023 06:29:47 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:66a6]) by smtp.gmail.com with ESMTPSA id g26-20020ac870da000000b00400a99b8b38sm6146339qtp.78.2023.10.12.06.29.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Oct 2023 06:29:47 -0700 (PDT) Date: Thu, 12 Oct 2023 09:29:46 -0400 From: Johannes Weiner To: Yosry Ahmed Cc: Shakeel Butt , Andrew Morton , Michal Hocko , Roman Gushchin , Muchun Song , Ivan Babrou , Tejun Heo , Michal =?iso-8859-1?Q?Koutn=FD?= , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg Message-ID: <20231012132946.GA470544@cmpxchg.org> References: <20231010032117.1577496-1-yosryahmed@google.com> <20231010032117.1577496-4-yosryahmed@google.com> <20231011003646.dt5rlqmnq6ybrlnd@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: D52D440024 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 6dxc1hug5a8ww5kzj8kpy9nhyzbego4i X-HE-Tag: 1697117388-780783 X-HE-Meta: U2FsdGVkX1/D2sEmBg0pJjYmv1ltyNtkGbxbqi/D+gNoJF7WsuNLtHyszsdXgi27IlwLuqweCeGa+B+if6CicnRzAYCrmBIfoJrZ7zPsLRZScugsgBMW+s52LT0mr3mjBgzBUjZNulDjGUCmGCx89N/C13/8TQ2IfkrtrGOBKFv1jvE6BgAsgAouW/jnooIHYQUhIixAfZaXZ+zSV4MopekpK3GGlEVpzx/ESYaCPq61jWXolB/dngMzT82A6mCAsofX5fUgrgYWbMd0W2UeFtQ4N4bb3tnWCu1T9n/gxmxdjUR/SsebqJVwLkn/18DED8G286R2hbSUPKG4ymg/ayFAM3uQBrAxrpUcf4qwLIWCVAwvc9odWX32xjM+QYyansky/T7yMsnSsJi0zqT0b1CLYnh/ux3mmuD7kdIzjfzD/7MoDB8ayzdf2HrWTTEsj8KpnFjY0RJ+J43ewNrMaJVAmvfYXnRnqGmOHbI/WqIg28Ai9awRsx6tLH2I25KsXu3S0oSjbjCDtg5e/dNcdU7VFNZQ+gkC0Vky4p3pfzTFC4osJNfuvKwPiJeG46k+yG9CGou2/oRqffbfp8tpSdiPg5A7PpopTIDZJsUIt/7o9wY9wVIH95zFfpjrdjq1w+DTu7p+EjhIMDDNp3s8LDYluiIuTuyqEn2mnRV2IIA6SLsvI8mnnTVT4fBk2Ln54si8+iQXIcAVlz9PJ2opZBqAr9PGB+Z63S3IUv0rTt/OOPKAsATSyMydT5tULtqDGp+3QTqewpIYPFcwREYB2kGEM1rGvC2zPTm1rWyL/227XsKDs2NQCBb/U/ZaLAqK093qkv5L+dm/UISsXmUdEgaNtOKel4tabR0tRj/UAzaSjM5ysSJhhjRi65cMM9a2DONwyNl7Kiz4D11ODPsRDuZZMezy3VL+82curEQQfQW3uFQk1S323rGJ5+VaKVPQ21gjwOd6IodHt+p+X3M yitVL0RW MoACMsicxo2x5GQIRCcjZQNtIGAMrnN4EBwn53k9PtQ/rz7jtVYlzsAE08dystvdpYJS1EZMISJEx/+fFv57OtDnIh0SELMYZWhlTvAWGPyJ3q51u2dskbSIvBgEz8gbldEy31ZYb94UJBq+klwKtQlSoCPHSjwz9zQ6TpiVMB9x3HeI2phNIo3ZhVJeWTQLwXmaOO8zzYhvDQuyNBsWqLFiiHOisJqrhmdOK1EFRwz5Wi7MEizt+eByzp1jwOzoQeTD4L2RGySjuZk5XOoigYOI5498u8Q+OpdHUbN/pAnCTTPqhRMehlcoBiB2rLM0UUiUQZufPHWKFh0lJtNMn0Y8CFvidjNCKB6JiHJ3rRKrcE75a9tZaIFcuvbSJBpMzoDJGomsOI0ofPdMNvvl0IV5pQpn85wTxM5Fgt94aGHS3up4PqjftOlHWfHDOBjYWwy0a X-Bogosity: Ham, tests=bogofilter, spamicity=0.000234, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 12, 2023 at 01:04:03AM -0700, Yosry Ahmed wrote: > On Wed, Oct 11, 2023 at 8:13 PM Yosry Ahmed wrote: > > > > On Wed, Oct 11, 2023 at 5:46 AM Shakeel Butt wrote: > > > > > > On Tue, Oct 10, 2023 at 6:48 PM Yosry Ahmed wrote: > > > > > > > > On Tue, Oct 10, 2023 at 5:36 PM Shakeel Butt wrote: > > > > > > > > > > On Tue, Oct 10, 2023 at 03:21:47PM -0700, Yosry Ahmed wrote: > > > > > [...] > > > > > > > > > > > > I tried this on a machine with 72 cpus (also ixion), running both > > > > > > netserver and netperf in /sys/fs/cgroup/a/b/c/d as follows: > > > > > > # echo "+memory" > /sys/fs/cgroup/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a > > > > > > # echo "+memory" > /sys/fs/cgroup/a/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b/c > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/c/cgroup.subtree_control > > > > > > # mkdir /sys/fs/cgroup/a/b/c/d > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > # ./netserver -6 > > > > > > > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > # for i in $(seq 10); do ./netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- > > > > > > -m 10K; done > > > > > > > > > > You are missing '&' at the end. Use something like below: > > > > > > > > > > #!/bin/bash > > > > > for i in {1..22} > > > > > do > > > > > /data/tmp/netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K & > > > > > done > > > > > wait > > > > > > > > > > > > > Oh sorry I missed the fact that you are running instances in parallel, my bad. > > > > > > > > So I ran 36 instances on a machine with 72 cpus. I did this 10 times > > > > and got an average from all instances for all runs to reduce noise: > > > > > > > > #!/bin/bash > > > > > > > > ITER=10 > > > > NR_INSTANCES=36 > > > > > > > > for i in $(seq $ITER); do > > > > echo "iteration $i" > > > > for j in $(seq $NR_INSTANCES); do > > > > echo "iteration $i" >> "out$j" > > > > ./netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K >> "out$j" & > > > > done > > > > wait > > > > done > > > > > > > > cat out* | grep 540000 | awk '{sum += $5} END {print sum/NR}' > > > > > > > > Base: 22169 mbps > > > > Patched: 21331.9 mbps > > > > > > > > The difference is ~3.7% in my runs. I am not sure what's different. > > > > Perhaps it's the number of runs? > > > > > > My base kernel is next-20231009 and I am running experiments with > > > hyperthreading disabled. > > > > Using next-20231009 and a similar 44 core machine with hyperthreading > > disabled, I ran 22 instances of netperf in parallel and got the > > following numbers from averaging 20 runs: > > > > Base: 33076.5 mbps > > Patched: 31410.1 mbps > > > > That's about 5% diff. I guess the number of iterations helps reduce > > the noise? I am not sure. > > > > Please also keep in mind that in this case all netperf instances are > > in the same cgroup and at a 4-level depth. I imagine in a practical > > setup processes would be a little more spread out, which means less > > common ancestors, so less contended atomic operations. > > > (Resending the reply as I messed up the last one, was not in plain text) > > I was curious, so I ran the same testing in a cgroup 2 levels deep > (i.e /sys/fs/cgroup/a/b), which is a much more common setup in my > experience. Here are the numbers: > > Base: 40198.0 mbps > Patched: 38629.7 mbps > > The regression is reduced to ~3.9%. > > What's more interesting is that going from a level 2 cgroup to a level > 4 cgroup is already a big hit with or without this patch: > > Base: 40198.0 -> 33076.5 mbps (~17.7% regression) > Patched: 38629.7 -> 31410.1 (~18.7% regression) > > So going from level 2 to 4 is already a significant regression for > other reasons (e.g. hierarchical charging). This patch only makes it > marginally worse. This puts the numbers more into perspective imo than > comparing values at level 4. What do you think? I think it's reasonable. Especially comparing to how many cachelines we used to touch on the write side when all flushing happened there. This looks like a good trade-off to me.