From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D914CDB46E for ; Thu, 12 Oct 2023 15:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 079E88000F; Thu, 12 Oct 2023 11:10:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 028448D0002; Thu, 12 Oct 2023 11:10:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E320D8000F; Thu, 12 Oct 2023 11:10:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D3F218D0002 for ; Thu, 12 Oct 2023 11:10:49 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B37431CAFC9 for ; Thu, 12 Oct 2023 15:10:49 +0000 (UTC) X-FDA: 81337146618.12.A5A3E6C Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by imf16.hostedemail.com (Postfix) with ESMTP id C934418001C for ; Thu, 12 Oct 2023 15:10:47 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pYLaH3qY; spf=pass (imf16.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697123447; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bpDYHFY2dl9s+q1D4aeetz4SQmaXlkXA2zbUfRxzbYc=; b=PDjxbOuaQY47EyN/SgBKOmOUv5yrQGi1edi41dFOhLB4Wl2gRVydV7eCT+EIIB4X9LVPGU 6BSWRcP/zYMtWbngjai3gXuQ8bsC54SjP82w3fcr3Iu4D12tQ2qBzZa+bHEuScn84t6nw8 Dfa16A9ZwlmrNf4NY7fLQKm7DuDsWww= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697123447; a=rsa-sha256; cv=none; b=565FYw/R/tWrU5Hw3y+Vxj/4OMamSb+mq1CvMiGdMvZES2XoLNdt+LWHidNHQ2C3pripvN W5Z05SHs9eDSJWtQKSaVl3uJTe0hCzudieCGuIAyDzw93ALWGNizx5UkArAVCVp+Z1bzWB rNYRORbye2tqoWCUO48u/1BeUt+Sl6w= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=pYLaH3qY; spf=pass (imf16.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-538e8eca9c1so1886651a12.3 for ; Thu, 12 Oct 2023 08:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697123446; x=1697728246; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=bpDYHFY2dl9s+q1D4aeetz4SQmaXlkXA2zbUfRxzbYc=; b=pYLaH3qY/gFSieNqajMfDj2aDWOsOP8JXj5ECCg1n1jJ71ApvkWahv1CITkBs8/rUy xfxtDWVfxJoalY89CXLWxhw/bRplrBg1eb5+tFe9pNO9vXKFnK1NgGwEsQBFZPK8sUJK gMwG01Go7YQBirCPPUdZaNyZeQCrLTQwXJrj/CFCyPFBN9SV/MXhiqiryO/S8FUw4Z8t qUFhlfgO0wIk061pdj1jm7QnTMvsho2FwKsJkvX25BsfXL3V9H79WuwKui7TB7ZE6G/L w4R3m+sPFYU3MXpp5fnGRfRIwZSIOOhN5oZF643GqNePRNAqpbjg4zZy/qH+GZ41/zk0 YPxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697123446; x=1697728246; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bpDYHFY2dl9s+q1D4aeetz4SQmaXlkXA2zbUfRxzbYc=; b=KK2u+mBctWEQ6qeY85F5CrO/aFPaBTtjZYO34UzdhXN2gYT9gfvDVz5ZOhZJbdLNNN eB7CjQOburIn6zyrRA+fSnGJ8Qqy2tsCBwsBKhsYz6yBbkljm+NDwIQYXQRAHLWm6uDf HMxw8oExPtBJU1uD0fgdgRG/OfYxr8pel5P3xIoYpMK9BdjVOzG+qpIlDYsrSVncjnkT z1ligjUeFtBNhgmPX4tXKRb6Xb7sMtspI8OITEKF8qSNy3tPQWTRi4GfnwnCrj+c1+xp 9RBbCC500/mZjAhaqkhnIgWDxVQDxcezt2vauxMVOm58lLtQWWeUqJHDOpMT0B0vIUJ8 TLHQ== X-Gm-Message-State: AOJu0YyrFZcFhxpg4Z53PvzMfpOP/1M58qVWWlLeIiHK3gZfwfVlV0uN klkRmK3yhHux8/jDR8rr0StoJAjY3xoS7FoPkycThw== X-Google-Smtp-Source: AGHT+IG1NZsBF0x+ysdXPpuvixswdgMB2PNukO+5lnPWO4mvWEBQAq6PrIFm3ho2S4LstGS8XvPCoP0j7q2Idqu37TA= X-Received: by 2002:a17:907:b0d:b0:9ae:37c2:11b2 with SMTP id h13-20020a1709070b0d00b009ae37c211b2mr19347554ejl.15.1697123445857; Thu, 12 Oct 2023 08:10:45 -0700 (PDT) MIME-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> <20231010032117.1577496-4-yosryahmed@google.com> <20231011003646.dt5rlqmnq6ybrlnd@google.com> In-Reply-To: From: Yosry Ahmed Date: Thu, 12 Oct 2023 08:10:06 -0700 Message-ID: Subject: Re: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg To: Shakeel Butt Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Ivan Babrou , Tejun Heo , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: zf3ha67ccxo4i7ktkirts3ojfne5zxzh X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C934418001C X-Rspam-User: X-HE-Tag: 1697123447-105981 X-HE-Meta: U2FsdGVkX1+GpOCGvvyEbfRYRrhyJM8VHdQIBgQDZkebM8wqWzAfv5AwYmWlwhPcI++wS/DKe3v9UBknbuLadG6m09G65Lsq0VJmYEwQ/tZ19BE2sGcRwR6KOv914inlkocl86qYpU9/m/eyvyLyy5AFfpl4wz/B2+O6w4nqbAh3xXsxsVluE4coDcfbhf+rjvoIjioH7kCOz2H+oeZp3gucHUFxVSoC9K4XsG/1r3HtYyXxUwMUUMfskbxJuBYnWMtLGrZKGOA/9SE1tz40dWnS8AxlL/ralxjark+YwNyF9NFvlHQf8czwnFaJRwRN84bt29VgKx3VGLLCTnHhWwblLs61JN61tgW1/n/6OSCY6DFA2NpiwtiGpWgVE6XFf6/TXA0EfwDlj8rTs3oNs3GdCnFIM7A0smjHGh1zDdf4yTN4ys65SwWhNfrK73OMFvUH02bItVZ+6HbMc0O1xHsCylq1ZMfJ+vNV2p2nsTyD+lM0XiKK3PF5TUZUr8eVkBzaXqguZLfay9jlLEESndVeqctOUnvAalsr88lqAl4VpDbmmy15z+0C7U9YzcztWMzDPo9zQOSbsxF0CqX3oeWwnvbfsxx1UKbwk2qG6VsEcVI8GFvbp/F9G0KdMudivH2Dz4ITkSXy/Mu5gY5/8+JeiNVqo/fCFLXhguxN/yt6Kvs55N70oCSQZmJJr6lT+X8231PRNg0S6+nvGi/KUl+BTGeM/oCXwMZ58f9YHow3qqgrdzPiGOkSQK4xOVerRVVse9DXHqZeq7tWDCOQ2BMuVeTjTCgicmRut2CJRlcG4umczZqusRsYxLyS/XQ0wfF6OZsVmT8Rc/msMOIy51JTPmb1QMHP3agbuVX/LLiBL05RUZ+PaEM9Oq0NtpAdaJeXuYe9NvS3xHakSGlIHnYvZaMllZTKZaGjUe3/UdBxnJpNFJUQ4pgwIfvVrl2yjsAjS4Gshfh3bxsbp+9 S7LAiu43 QIJrhHc5od6pxyiMXkuQwO5NLiJmq9kaCA216t4aOHBvcrwO2bnjz3p+MXcetYtrLfh00TVhGv1AY1Fbiye+oucRc8oMFNYfatdNTgeBUK4R1XhpwYTFAAlWQu0UY9TvRBBurUdQZDGYA/dSnJzdeTHMxL54Ie7aLKFVghcH+le9arnN5GeSP3HAjRY0tMnLeXc2kTXjf0Ddb3aYo5hkPRX29lwjeCMtosBOlr5S/XJEt+d7n8mPstsWZdQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000206, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 12, 2023 at 6:35=E2=80=AFAM Shakeel Butt = wrote: > > On Thu, Oct 12, 2023 at 1:04=E2=80=AFAM Yosry Ahmed wrote: > > > > On Wed, Oct 11, 2023 at 8:13=E2=80=AFPM Yosry Ahmed wrote: > > > > > > On Wed, Oct 11, 2023 at 5:46=E2=80=AFAM Shakeel Butt wrote: > > > > > > > > On Tue, Oct 10, 2023 at 6:48=E2=80=AFPM Yosry Ahmed wrote: > > > > > > > > > > On Tue, Oct 10, 2023 at 5:36=E2=80=AFPM Shakeel Butt wrote: > > > > > > > > > > > > On Tue, Oct 10, 2023 at 03:21:47PM -0700, Yosry Ahmed wrote: > > > > > > [...] > > > > > > > > > > > > > > I tried this on a machine with 72 cpus (also ixion), running = both > > > > > > > netserver and netperf in /sys/fs/cgroup/a/b/c/d as follows: > > > > > > > # echo "+memory" > /sys/fs/cgroup/cgroup.subtree_control > > > > > > > # mkdir /sys/fs/cgroup/a > > > > > > > # echo "+memory" > /sys/fs/cgroup/a/cgroup.subtree_control > > > > > > > # mkdir /sys/fs/cgroup/a/b > > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/cgroup.subtree_control > > > > > > > # mkdir /sys/fs/cgroup/a/b/c > > > > > > > # echo "+memory" > /sys/fs/cgroup/a/b/c/cgroup.subtree_contro= l > > > > > > > # mkdir /sys/fs/cgroup/a/b/c/d > > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > > # ./netserver -6 > > > > > > > > > > > > > > # echo 0 > /sys/fs/cgroup/a/b/c/d/cgroup.procs > > > > > > > # for i in $(seq 10); do ./netperf -6 -H ::1 -l 60 -t TCP_SEN= DFILE -- > > > > > > > -m 10K; done > > > > > > > > > > > > You are missing '&' at the end. Use something like below: > > > > > > > > > > > > #!/bin/bash > > > > > > for i in {1..22} > > > > > > do > > > > > > /data/tmp/netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K = & > > > > > > done > > > > > > wait > > > > > > > > > > > > > > > > Oh sorry I missed the fact that you are running instances in para= llel, my bad. > > > > > > > > > > So I ran 36 instances on a machine with 72 cpus. I did this 10 ti= mes > > > > > and got an average from all instances for all runs to reduce nois= e: > > > > > > > > > > #!/bin/bash > > > > > > > > > > ITER=3D10 > > > > > NR_INSTANCES=3D36 > > > > > > > > > > for i in $(seq $ITER); do > > > > > echo "iteration $i" > > > > > for j in $(seq $NR_INSTANCES); do > > > > > echo "iteration $i" >> "out$j" > > > > > ./netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K >> "out$j= " & > > > > > done > > > > > wait > > > > > done > > > > > > > > > > cat out* | grep 540000 | awk '{sum +=3D $5} END {print sum/NR}' > > > > > > > > > > Base: 22169 mbps > > > > > Patched: 21331.9 mbps > > > > > > > > > > The difference is ~3.7% in my runs. I am not sure what's differen= t. > > > > > Perhaps it's the number of runs? > > > > > > > > My base kernel is next-20231009 and I am running experiments with > > > > hyperthreading disabled. > > > > > > Using next-20231009 and a similar 44 core machine with hyperthreading > > > disabled, I ran 22 instances of netperf in parallel and got the > > > following numbers from averaging 20 runs: > > > > > > Base: 33076.5 mbps > > > Patched: 31410.1 mbps > > > > > > That's about 5% diff. I guess the number of iterations helps reduce > > > the noise? I am not sure. > > > > > > Please also keep in mind that in this case all netperf instances are > > > in the same cgroup and at a 4-level depth. I imagine in a practical > > > setup processes would be a little more spread out, which means less > > > common ancestors, so less contended atomic operations. > > > > > > (Resending the reply as I messed up the last one, was not in plain text= ) > > > > I was curious, so I ran the same testing in a cgroup 2 levels deep > > (i.e /sys/fs/cgroup/a/b), which is a much more common setup in my > > experience. Here are the numbers: > > > > Base: 40198.0 mbps > > Patched: 38629.7 mbps > > > > The regression is reduced to ~3.9%. > > > > What's more interesting is that going from a level 2 cgroup to a level > > 4 cgroup is already a big hit with or without this patch: > > > > Base: 40198.0 -> 33076.5 mbps (~17.7% regression) > > Patched: 38629.7 -> 31410.1 (~18.7% regression) > > > > So going from level 2 to 4 is already a significant regression for > > other reasons (e.g. hierarchical charging). This patch only makes it > > marginally worse. This puts the numbers more into perspective imo than > > comparing values at level 4. What do you think? > > This is weird as we are running the experiments on the same machine. I > will rerun with 2 levels as well. Also can you rerun the page fault > benchmark as well which was showing 9% regression in your original > commit message? Thanks. I will re-run the page_fault tests, but keep in mind that the page fault benchmarks in will-it-scale are highly variable. We run them between kernel versions internally, and I think we ignore any changes below 10% as the benchmark is naturally noisy. I have a couple of runs for page_fault3_scalability showing a 2-3% improvement with this patch :)