From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC52CC02185 for ; Wed, 15 Jan 2025 21:37:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 618C9280004; Wed, 15 Jan 2025 16:37:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C8796B0088; Wed, 15 Jan 2025 16:37:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4909F280004; Wed, 15 Jan 2025 16:37:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2847C6B0085 for ; Wed, 15 Jan 2025 16:37:08 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E84A8AE70E for ; Wed, 15 Jan 2025 21:37:07 +0000 (UTC) X-FDA: 83010996894.28.254D838 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf11.hostedemail.com (Postfix) with ESMTP id 0A51C40014 for ; Wed, 15 Jan 2025 21:37:05 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yw2DueS2; spf=pass (imf11.hostedemail.com: domain of yosryahmed@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736977026; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7CizrBrhzcDgrTIqS9kYZgWrDglRMWmTQxzWdQcUOOw=; b=0Faw8lG70+SUfRkW9ehTcC7oNb7dR3dexuoAXwNZww3ORIIFCwMiYVQEuZ1XWis0sDWYpg j9hUhgRL89Y5BQL6IpthkA33rPo1pzZhCYbzY0hgkypFWQXvREIoDXyxypgz0awHqDxuza +DJFWuSgS8R3n7h7uRONcC8ZpESutXc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yw2DueS2; spf=pass (imf11.hostedemail.com: domain of yosryahmed@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736977026; a=rsa-sha256; cv=none; b=NgRNCCmRoNmfCK+D7fHGiGFNsWBrnhz3U0Aq1OYS25wfPZdqYAhl6eX/Q9iSdvJuYPy3WS vFf/z2Y3FISrGlK3O+BfUXCNNXEK6FVViJZTc7hX2+XN3JfZCEAhKL8CJgy8JyFT2S880e LZgi4zl7/d7WCKDLKheLT4HMM2rOEK0= Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-467a3f1e667so2224581cf.0 for ; Wed, 15 Jan 2025 13:37:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736977025; x=1737581825; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=7CizrBrhzcDgrTIqS9kYZgWrDglRMWmTQxzWdQcUOOw=; b=Yw2DueS2RhYyI2MvaBuXe61W09M3aCIkz79riMsc3/+OUz10nnZUDQRMPezroDKd6x KpwsBU0LjhJX0AxaJZADJKDp6aFUbVkIG8JI0bAJTEKkXeO3KqJ/WNiRcb6ADGTLARtM w7zIv24wlX4aRwP69vDZkVnRZlidQ8Se96NRpuoXsYbcOsb++k7m9weD6xDLAedguvd2 NXqYw1o5ClUht46GTKtv92uC9J82DQaSyKnIQntQgctCk5GA41jbcd2GYnkTHBLs+GAv KKTcG3ehq1IYhRbZK2Ot3TfHqDLprMDLf76k+dwLc3h7MapzW1V02K14QqZRM3Z5sLhk guLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736977025; x=1737581825; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7CizrBrhzcDgrTIqS9kYZgWrDglRMWmTQxzWdQcUOOw=; b=a6qivJl6RHB1CQkFxc5BqFUtiu9qBFzSd1xG4Mj7JznRvmk+3L30v6RNg25dBAkQqi 2bBVU6T4MDwuyiK3hBVbVf5usf1m5k0jXfwWIa7MUcFcXCcxbR5F4m1qTHRUTBFHTXKU rKACG1Gh4e6VhdXZpcMotpIY0iT4JeDdXwHYSQqNCgeFrJUTewhf6H9jVuUCSu3ig5il XBZ0ETneACl+AfiEQ2WlYbmjGQxU1SbCXxjiQDCOrTreB4LnJD2hEcUzJsrl9RpN7eRx Se2ihHwHofgd5CggA7PNyqcimuBtbJlbIOqzPqLLa3UtusB8UuCmZ7wdnhRs07Qulqx6 FYVw== X-Forwarded-Encrypted: i=1; AJvYcCUU5rqsJJpKfuG8Vy5LYfCYAw4TLfdDsEjzTjb+mEjn+LtWOpnFt8Dg/qyVSODfwC6dyIAjqzG4nQ==@kvack.org X-Gm-Message-State: AOJu0Yxf0V9LBA2AFg0lsPhwM9G6kD5hOvSReBqs+hIEOOMpf0xxK+WV sVUQ77O4IUi3XEeL5ZTolmuINUwakeJl1EbZAuSR68EeTm0iCD5SyB3yk4ejZ+fgQiMHqHWeX02 ViqRZ+lZzUUte61wNYEPcREPqyoxXChlNWhzB X-Gm-Gg: ASbGncv1PBrLQhPhpPdhFOgd9dPsOBfg2ArACyrTq3v6tT2ar7C8p7BaMqnFLzjqMKR IGCUKEdtyP4Wue5JlFB1F4QDrSrUthDUAkLY= X-Google-Smtp-Source: AGHT+IGadMPMDftUZZ1Cy9aXiE6CqdJ86cdL8uLSBK0+zEKACimk3BxqR9gWNGe1f/1AiI0+hqPp5ulJ5BoXOgeiZH8= X-Received: by 2002:ad4:5d4e:0:b0:6d8:899e:c3bf with SMTP id 6a1803df08f44-6df9b2d37a0mr500483466d6.34.1736977024924; Wed, 15 Jan 2025 13:37:04 -0800 (PST) MIME-Version: 1.0 References: <20241224011402.134009-1-inwardvessel@gmail.com> <3wew3ngaqq7cjqphpqltbq77de5rmqviolyqphneer4pfzu5h5@4ucytmd6rpfa> <3348742b-4e49-44c1-b447-b21553ff704a@gmail.com> In-Reply-To: From: Yosry Ahmed Date: Wed, 15 Jan 2025 13:36:28 -0800 X-Gm-Features: AbW1kvbW5LvYu8-fNVnjNchSkPu-0y59Nwd4frxavtQZbYj1NLjv2El8eSdshWk Message-ID: Subject: Re: [PATCH 0/9 RFC] cgroup: separate rstat trees To: JP Kobryn Cc: Shakeel Butt , =?UTF-8?Q?Michal_Koutn=C3=BD?= , hannes@cmpxchg.org, akpm@linux-foundation.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Tejun Heo Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0A51C40014 X-Stat-Signature: qsf4kmoxk5ej96grejp167py88u5j9yo X-HE-Tag: 1736977025-769883 X-HE-Meta: U2FsdGVkX19Y/LRp+Wg8QUJ3FfBUyg/ebdLfQlxr8+5TGeaqKqedwe9+2vheJsHXb9fwY4KmDU2zvU1cQ7ajbfEkyRXEmI+Z+LbHYxZxtuWLIgOFVNnGjEAlMehhoYufsFwCbXEkcx4OlR25f1nrevpTPl8xnaRW4Q01nhyHjzXIffgV6Dgk0bU2U5ZoRh4WEPWieVfPgZzycbCEnFFrThXvP8HhQuatwG6/9PKRyaH3F25UikIQ3WtgQe/2jy0bUiNIu66wLyR2JWCc9oPqUBkwQVJowN9lyzUTWchhS8TMzWhjG5QQfCw858HZADj2i38g0OIW8+a1gDfPr6czOKaDCu4SjkJ/EE+fz5lP3RnpPVhhaNd4Q1yJkVMn57rZ7FjBVEj9F/kttqvhZVk1m/2f19jpZkthYJSML9bSg6ASoJqhpu+f9k7uYRhfmgxG4L/+IaFWKi4x2m3qlFbJmsW1msMTFlYAp999dnc30OVIVTjTN+JYLvXiUvjnCqJ3i2X3HtKWPmZmq7ehwJC9FX2pxNc3XutcygZCZmt9+DKprDZbSGlqKUqanzeyyyrufT3fjEbrPL19oAf/er4ZyKKMWdKnTl9/o5i7dlDqXIlN4p/tr9YOTaIBGPBLq3YNkm17gG8KwE1u135NgdChMOakqxpori6eA3/ZR+5M8/x6NtHCwNfIrQ5Y8iL9QXZ2SpbSV1sKoh62+gQO7UEVmmYXUL4X29/FAlq2Lk+2pYwBXyIs/jqVvvwezGaFqqg1Y7q1P0k9oprBPSUH4eiJeVBCFGqJcsFobDnPq/rJ07cu0/IR/TZ+bS7o50zUMq4DdaX+9pZ3buyS+VsWUZBbm9Y3eNHUcx8Zt1BL9JDeibUMFJcLz20yjem9rFSGG8ND8FsdAOh8AvyH7MeMevzQBrNpXaLXKjYgCPE2SPy5RieZCYTAS8GHpO8muWrJNA+dtzkQ9U+tijkpXmrdQmg qQkt1gA/ JVjktIFKjCzn4bbfXos4+YhEPshTVBkpFzxd5EMJKXbxg8miOZxMWpHtW5FyjlPVx5ou5hpJxRui3o7pGIkDB2SPWtadYoZF1y7mYnjrzKQ7U1S6RQgMOLIK/DOrdzthgtYRVUEc735wZht8KHpPt5//5s6WHlVZgUXevxnVfqD5tn3zSskaTS+OdLEQOhUvdV3J/31cInAq2ocOL8ZC3hJ4wSheiIsSqI2qomWBjNMt8B3DmAnBuQMWWLdLNP+8THYt3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 15, 2025 at 11:39=E2=80=AFAM JP Kobryn = wrote: > > Hi Yosry, > > On 1/14/25 5:39 PM, Yosry Ahmed wrote: > > On Tue, Jan 14, 2025 at 5:33=E2=80=AFPM JP Kobryn wrote: > >> > >> Hi Michal, > >> > >> On 1/13/25 10:25 AM, Shakeel Butt wrote: > >>> On Wed, Jan 08, 2025 at 07:16:47PM +0100, Michal Koutn=C3=BD wrote: > >>>> Hello JP. > >>>> > >>>> On Mon, Dec 23, 2024 at 05:13:53PM -0800, JP Kobryn wrote: > >>>>> I've been experimenting with these changes to allow for separate > >>>>> updating/flushing of cgroup stats per-subsystem. > >>>> > >>>> Nice. > >>>> > >>>>> I reached a point where this started to feel stable in my local tes= ting, so I > >>>>> wanted to share and get feedback on this approach. > >>>> > >>>> The split is not straight-forwardly an improvement -- > >>> > >>> The major improvement in my opinion is the performance isolation for > >>> stats readers i.e. cpu stats readers do not need to flush memory stat= s. > >>> > >>>> there's at least > >>>> higher memory footprint > >>> > >>> Yes this is indeed the case and JP, can you please give a ballmark on > >>> the memory overhead? > >> > >> Yes, the trade-off is using more memory to allow for separate trees. > >> With these patches the changes in allocated memory for the > >> cgroup_rstat_cpu instances and their associated locks are: > >> static > >> reduced by 58% > >> dynamic > >> increased by 344% > >> > >> The threefold increase on the dynamic side is attributed to now having= 3 > >> rstat trees per cgroup (1 for base stats, 1 for memory, 1 for io), > >> instead of originally just 1. The number will change if more subsystem= s > >> start or stop using rstat in the future. Feel free to let me know if y= ou > >> would like to see the detailed breakdown of these values. > > > > What is the absolute per-CPU memory usage? > > This is what I calculate as the combined per-cpu usage. > before: > one cgroup_rstat_cpu instance for every cgroup > sizeof(cgroup_rstat_cpu) * nr_cgroups > after: > three cgroup_rstat_cpu instances for every cgroup + updater lock = for > every subsystem plus one for base stats > sizeof(cgroup_rstat_cpu) * 3 * nr_cgroups + > sizeof(spinlock_t) * (CGROUP_SUBSYS_COUNT + 1) > > Note that "every cgroup" includes the root cgroup. Also, 3 represents > the number of current rstat clients: base stats, memory, and io > (assuming all enabled). On a config I have at hand sizeof(cgroup_rstat_cpu) is 160 bytes. Ignoring the spinlock for a second because it doesn't scale with cgroups, that'd be an extra 320 * nr_cgroups * nr_cpus bytes. On a moderately large machine with 256 CPUs and 100 cgroups for example that's ~8MB. > > As I'm writing this, I realize I might need to include the bpf cgroups > as a fourth client and include this in my testing.