From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA6F9C3601B for ; Fri, 4 Apr 2025 01:11:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 700606B0006; Thu, 3 Apr 2025 21:11:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D6DF6B0007; Thu, 3 Apr 2025 21:11:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 465C96B0008; Thu, 3 Apr 2025 21:11:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2C3636B0006 for ; Thu, 3 Apr 2025 21:11:05 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 359A2121B14 for ; Fri, 4 Apr 2025 01:11:06 +0000 (UTC) X-FDA: 83294582532.22.CCD6758 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf24.hostedemail.com (Postfix) with ESMTP id 88327180004 for ; Fri, 4 Apr 2025 01:11:04 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hs3GuSly; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743729064; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mZsU5fSVi28PbSL6szY7mdlMdjJc5ZoxpLKyts0s7sA=; b=P1QpLScRRLUSVLgNYPh60riqWMJc25F/qYdlDAnRRV2yZ/TjW53LVuVJPbH4KJIcck9GUA 9OGocZcb7lBfn4lEZmNQKRsJmbujJErJCt4suO7Beuh33vjdvlF5b+QTSKyxHSs/8tApPc B+ssdkUUMh7Sn/2fLEzcxAxQIV3o93c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743729064; a=rsa-sha256; cv=none; b=K5LK5Vw5Vj62Sf3RMrUX+Sbqyhi5YT46RGcNjH+srmFtIREBmHDcND3EOw4zmUTX5QcNpe 8M1t/gz98awGpe/q31Isgkdax9Rej3cq1WziK8DrGcW0yxHW9B2Lu7gJt8d9mnFPjYsusK Vp6fJGI46XND7jaNlulwFgDbD5S6Zow= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hs3GuSly; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-223f4c06e9fso14358575ad.1 for ; Thu, 03 Apr 2025 18:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743729063; x=1744333863; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mZsU5fSVi28PbSL6szY7mdlMdjJc5ZoxpLKyts0s7sA=; b=hs3GuSlyBUn9tAo/lHnKpUzQpBK/U1CLDA4BxH3Jbh8aD/EWQPdUblx+rzJv45lQ2f enVHgsc/ud5eIYBLgh+r7Uu6fdDMkhf0Dj2FfWMRf1V+SM8UY6FHffublweZsw+TyxGm OJupc7wrsLFnjQkSu6zhntPsya6AMFFvCAKAqA045suvx3fp6935KzMwEtGLcthz7T1X +jCZTvaq2IFDCspWenxXrMu/y/m3PvTXPMzn7o4TpnZr4Srff8r5Qkf0U5pWK2as2N9H OJrgcGSctu4Fzfb2u5F4eg4D79OUhO+WS/rlapM7o2q98W3dziIbILl05iB8EzXg7m8u i22Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743729063; x=1744333863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mZsU5fSVi28PbSL6szY7mdlMdjJc5ZoxpLKyts0s7sA=; b=SnAUXD8B8MSYLjI9D5rzM0CjbtMbcrlnOxViJnh4tDX4TaxdwEwFho9cz92BA+TG3Z WRiPXnzlB1k2HhD3T8WS1Pfr+VzHyx1REgYGcDsj9ufS62KkmwNwt/C2OfNp2Ssj5XWs MS7z6CG8ctpyJIj15GFiQ09SO35jrNh4127e4//4xaWB9s8OEY5yk8TjsLkW8+6tgR7J JPH/z3q2y3TVZPGyNDlftdbJuquqNUFHrhzMOPtUma7d0mNcmuNvKKxGZnlFjc8kVnoL wknKkXNnCqHFYUeqGN9ZDJMQNvg8cna5lQ/2bzxsLp756RzLlvT9EYlizDWOvbJkHT9X AVHQ== X-Gm-Message-State: AOJu0Ywqv+vU01MHi3N9NtjcAHWhLvTZmiV1OcLh+JW+wjarAHgAT4Z5 yG36WnrwiQbrSDtlOwZ/4gfgyUAJsw8dnqzHDnNshdQuk/dUuUlZ X-Gm-Gg: ASbGnctygBpLUq6oqkIwHGSU9U+9U8jMoXDu1Ab5LukgfKmaYAMMhLtqQJXSH8yF+pl /k2fUvf84/2ssanRphjaYfJXEnrnrPuMfiI9prtFfKFaACaegmHmqmhYQS6B9f7mheo8Khkt6bj BQJdg2a7aompVVPiB2RIYB0kv5uDQCigiB72pzPB489BdjWzSToqEtezINJ1D0dAcyqyROYKPy6 MswtoEzhkMsQ7b89BGSjdP7cGZ4qu8/eAOsU7oiE9yFh5s/luW6BV64O6oGTDlcbCNEmsFE0NRj pyzurC4J0b0g/ImT/E7P17EonZS6sMGiWmA0KgD/+Dw3yMsJfJSoc3sngB48uQhF4S1IZj2o X-Google-Smtp-Source: AGHT+IF8Od4xvEKN3/RaUfeo4N1bACCSjzvUpvp6Hxw/d1Soi6qjdw8eOYD9IyiMEadxhY9T/76/Gg== X-Received: by 2002:a17:902:be15:b0:221:751f:cfbe with SMTP id d9443c01a7336-22a89b8e034mr16115945ad.19.1743729063215; Thu, 03 Apr 2025 18:11:03 -0700 (PDT) Received: from jpkobryn-fedora-PF5CFKNC.thefacebook.com ([2620:10d:c090:500::7:9b28]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-229785ad9a0sm21268675ad.39.2025.04.03.18.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 18:11:02 -0700 (PDT) From: JP Kobryn To: tj@kernel.org, shakeel.butt@linux.dev, yosryahmed@google.com, mkoutny@suse.com, hannes@cmpxchg.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 1/5] cgroup: move rstat base stat objects into their own struct Date: Thu, 3 Apr 2025 18:10:46 -0700 Message-ID: <20250404011050.121777-2-inwardvessel@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250404011050.121777-1-inwardvessel@gmail.com> References: <20250404011050.121777-1-inwardvessel@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 88327180004 X-Stat-Signature: cq1brmzxi4cd7469o7m5zbyuyjmxkc5k X-Rspam-User: X-HE-Tag: 1743729064-681674 X-HE-Meta: U2FsdGVkX18HE6OKwnPiOGAqYx5M4xneBEMKqvs1V2cyS8o2DFl8Pv8hoCkb3Tfy35h2Q4c2xNzwl5qOYRVjaP/GJRmJ5IF0bdeqUtsBC1QUna+5klgRsKoYdE0VZRqxDGX6tN+meW8sqO10HOLqCkR4R+BZAOi6KxZrMfmouR/py1vqQuw7X+2lbeLX3S6L1WqRSnTQfGL3mIlU7s8hYhTuy8aWqqjz6v9AQAhJhYu3qBn18Hy3g5ZYls2uoXZLVfptxTxmWcnoyASv9FAHKhcAdcKN6eXn61TVdBS+rYsvmf+uUjzAbmBbmlsnPNq2lz5n3QBWK2YkenpJUvpbgOG8vHshYvivCXYhNm4Y+V44xBGes1kNdgYlWX64o48sjt2Dd1b77BbWMnoPJaLdLPQligEgG5rr0243YW3r9l51PRgmPHxkMYFKGJRid3u3d5sfF9zYknHUG/fPSLhlHnMdPcPltEJ97Ch1M0ukVDv47C8E8RdurREf2dHynjkw0mI52etyUZ5cWELjh4QFbjCIoh0FUKtJZGzt75Hm85eCdnYoE2STUEmGJq6GjNu+xiBBE1u7TyycO1z3VvoKuYXEolDc3b8Kf5Z7hW5CRfBF+ea6Pxgld++28Hf7J6N41vD3uhGCCQjCmGeeTixPup+XSBLv8AEenG4QW6UiR5KE5AKiuSrna6XGQYyscXKSN9XZ0YunuZD4r2LKwzAB4vOHJDkzsV8rLRvblMdG6JnN6vhbEIpLN9o7NucTvO8w2AqG/kW74vWXBHL45vYphocu0fdJ5iyWotU0Ctj+eGpnwnd+yQpAQOfYfx+/zJlNgo3UV97l8Ut8VBu4ETSZP+pO1lJCwQ2uCn2mF+4sVKhyStR5gOBrMp+dQ71wshfXEqq8Mp3ULtIHGCsSVTVb9RR40fypOC0wFpqYu2dZbT9auRfYPO6ufqZxI09MmqRlUV/7UXFHnzyeJOdBPgC 5F7X2MhD pQvJs7lUPqticPTU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This non-functional change serves as preparation for moving to subsystem-based rstat trees. The base stats are not an actual subsystem, but in future commits they will have exclusive rstat trees just as other subsystems will. Moving the base stat objects into a new struct allows the cgroup_rstat_cpu struct to become more compact since it now only contains the minimum amount of pointers needed for rstat participation. Subsystems will (in future commits) make use of the compact cgroup_rstat_cpu struct while avoiding the memory overhead of the base stat objects which they will not use. An instance of the new struct cgroup_rstat_base_cpu was placed on the cgroup struct so it can retain ownership of these base stats common to all cgroups. A helper function was added for looking up the cpu-specific base stats of a given cgroup. Finally, initialization and variable names were adjusted where applicable. Signed-off-by: JP Kobryn --- include/linux/cgroup-defs.h | 38 ++++++++++------- kernel/cgroup/cgroup.c | 8 +++- kernel/cgroup/rstat.c | 84 ++++++++++++++++++++++--------------- 3 files changed, 79 insertions(+), 51 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 485b651869d9..6d177f770d28 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -344,10 +344,29 @@ struct cgroup_base_stat { * frequency decreases the cost of each read. * * This struct hosts both the fields which implement the above - - * updated_children and updated_next - and the fields which track basic - * resource statistics on top of it - bsync, bstat and last_bstat. + * updated_children and updated_next. */ struct cgroup_rstat_cpu { + /* + * Child cgroups with stat updates on this cpu since the last read + * are linked on the parent's ->updated_children through + * ->updated_next. + * + * In addition to being more compact, singly-linked list pointing + * to the cgroup makes it unnecessary for each per-cpu struct to + * point back to the associated cgroup. + * + * Protected by per-cpu cgroup_rstat_cpu_lock. + */ + struct cgroup *updated_children; /* terminated by self cgroup */ + struct cgroup *updated_next; /* NULL iff not on the list */ +}; + +/* + * This struct hosts the fields which track basic resource statistics on + * top of it - bsync, bstat and last_bstat. + */ +struct cgroup_rstat_base_cpu { /* * ->bsync protects ->bstat. These are the only fields which get * updated in the hot path. @@ -374,20 +393,6 @@ struct cgroup_rstat_cpu { * deltas to propagate to the per-cpu subtree_bstat. */ struct cgroup_base_stat last_subtree_bstat; - - /* - * Child cgroups with stat updates on this cpu since the last read - * are linked on the parent's ->updated_children through - * ->updated_next. - * - * In addition to being more compact, singly-linked list pointing - * to the cgroup makes it unnecessary for each per-cpu struct to - * point back to the associated cgroup. - * - * Protected by per-cpu cgroup_rstat_cpu_lock. - */ - struct cgroup *updated_children; /* terminated by self cgroup */ - struct cgroup *updated_next; /* NULL iff not on the list */ }; struct cgroup_freezer_state { @@ -518,6 +523,7 @@ struct cgroup { /* per-cpu recursive resource statistics */ struct cgroup_rstat_cpu __percpu *rstat_cpu; + struct cgroup_rstat_base_cpu __percpu *rstat_base_cpu; struct list_head rstat_css_list; /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index ac2db99941ca..77349d07b117 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -161,10 +161,14 @@ static struct static_key_true *cgroup_subsys_on_dfl_key[] = { }; #undef SUBSYS -static DEFINE_PER_CPU(struct cgroup_rstat_cpu, cgrp_dfl_root_rstat_cpu); +static DEFINE_PER_CPU(struct cgroup_rstat_cpu, root_rstat_cpu); +static DEFINE_PER_CPU(struct cgroup_rstat_base_cpu, root_rstat_base_cpu); /* the default hierarchy */ -struct cgroup_root cgrp_dfl_root = { .cgrp.rstat_cpu = &cgrp_dfl_root_rstat_cpu }; +struct cgroup_root cgrp_dfl_root = { + .cgrp.rstat_cpu = &root_rstat_cpu, + .cgrp.rstat_base_cpu = &root_rstat_base_cpu, +}; EXPORT_SYMBOL_GPL(cgrp_dfl_root); /* diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 4bb587d5d34f..a20e3ab3f7d3 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -19,6 +19,12 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) return per_cpu_ptr(cgrp->rstat_cpu, cpu); } +static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( + struct cgroup *cgrp, int cpu) +{ + return per_cpu_ptr(cgrp->rstat_base_cpu, cpu); +} + /* * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). * @@ -351,12 +357,22 @@ int cgroup_rstat_init(struct cgroup *cgrp) return -ENOMEM; } + if (!cgrp->rstat_base_cpu) { + cgrp->rstat_base_cpu = alloc_percpu(struct cgroup_rstat_base_cpu); + if (!cgrp->rstat_cpu) { + free_percpu(cgrp->rstat_cpu); + return -ENOMEM; + } + } + /* ->updated_children list is self terminated */ for_each_possible_cpu(cpu) { struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_base_cpu *rstatbc = + cgroup_rstat_base_cpu(cgrp, cpu); rstatc->updated_children = cgrp; - u64_stats_init(&rstatc->bsync); + u64_stats_init(&rstatbc->bsync); } return 0; @@ -379,6 +395,8 @@ void cgroup_rstat_exit(struct cgroup *cgrp) free_percpu(cgrp->rstat_cpu); cgrp->rstat_cpu = NULL; + free_percpu(cgrp->rstat_base_cpu); + cgrp->rstat_base_cpu = NULL; } void __init cgroup_rstat_boot(void) @@ -419,9 +437,9 @@ static void cgroup_base_stat_sub(struct cgroup_base_stat *dst_bstat, static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) { - struct cgroup_rstat_cpu *rstatc = cgroup_rstat_cpu(cgrp, cpu); + struct cgroup_rstat_base_cpu *rstatbc = cgroup_rstat_base_cpu(cgrp, cpu); struct cgroup *parent = cgroup_parent(cgrp); - struct cgroup_rstat_cpu *prstatc; + struct cgroup_rstat_base_cpu *prstatbc; struct cgroup_base_stat delta; unsigned seq; @@ -431,15 +449,15 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) /* fetch the current per-cpu values */ do { - seq = __u64_stats_fetch_begin(&rstatc->bsync); - delta = rstatc->bstat; - } while (__u64_stats_fetch_retry(&rstatc->bsync, seq)); + seq = __u64_stats_fetch_begin(&rstatbc->bsync); + delta = rstatbc->bstat; + } while (__u64_stats_fetch_retry(&rstatbc->bsync, seq)); /* propagate per-cpu delta to cgroup and per-cpu global statistics */ - cgroup_base_stat_sub(&delta, &rstatc->last_bstat); + cgroup_base_stat_sub(&delta, &rstatbc->last_bstat); cgroup_base_stat_add(&cgrp->bstat, &delta); - cgroup_base_stat_add(&rstatc->last_bstat, &delta); - cgroup_base_stat_add(&rstatc->subtree_bstat, &delta); + cgroup_base_stat_add(&rstatbc->last_bstat, &delta); + cgroup_base_stat_add(&rstatbc->subtree_bstat, &delta); /* propagate cgroup and per-cpu global delta to parent (unless that's root) */ if (cgroup_parent(parent)) { @@ -448,73 +466,73 @@ static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu) cgroup_base_stat_add(&parent->bstat, &delta); cgroup_base_stat_add(&cgrp->last_bstat, &delta); - delta = rstatc->subtree_bstat; - prstatc = cgroup_rstat_cpu(parent, cpu); - cgroup_base_stat_sub(&delta, &rstatc->last_subtree_bstat); - cgroup_base_stat_add(&prstatc->subtree_bstat, &delta); - cgroup_base_stat_add(&rstatc->last_subtree_bstat, &delta); + delta = rstatbc->subtree_bstat; + prstatbc = cgroup_rstat_base_cpu(parent, cpu); + cgroup_base_stat_sub(&delta, &rstatbc->last_subtree_bstat); + cgroup_base_stat_add(&prstatbc->subtree_bstat, &delta); + cgroup_base_stat_add(&rstatbc->last_subtree_bstat, &delta); } } -static struct cgroup_rstat_cpu * +static struct cgroup_rstat_base_cpu * cgroup_base_stat_cputime_account_begin(struct cgroup *cgrp, unsigned long *flags) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatbc; - rstatc = get_cpu_ptr(cgrp->rstat_cpu); - *flags = u64_stats_update_begin_irqsave(&rstatc->bsync); - return rstatc; + rstatbc = get_cpu_ptr(cgrp->rstat_base_cpu); + *flags = u64_stats_update_begin_irqsave(&rstatbc->bsync); + return rstatbc; } static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, - struct cgroup_rstat_cpu *rstatc, + struct cgroup_rstat_base_cpu *rstatbc, unsigned long flags) { - u64_stats_update_end_irqrestore(&rstatc->bsync, flags); + u64_stats_update_end_irqrestore(&rstatbc->bsync, flags); cgroup_rstat_updated(cgrp, smp_processor_id()); - put_cpu_ptr(rstatc); + put_cpu_ptr(rstatbc); } void __cgroup_account_cputime(struct cgroup *cgrp, u64 delta_exec) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatbc; unsigned long flags; - rstatc = cgroup_base_stat_cputime_account_begin(cgrp, &flags); - rstatc->bstat.cputime.sum_exec_runtime += delta_exec; - cgroup_base_stat_cputime_account_end(cgrp, rstatc, flags); + rstatbc = cgroup_base_stat_cputime_account_begin(cgrp, &flags); + rstatbc->bstat.cputime.sum_exec_runtime += delta_exec; + cgroup_base_stat_cputime_account_end(cgrp, rstatbc, flags); } void __cgroup_account_cputime_field(struct cgroup *cgrp, enum cpu_usage_stat index, u64 delta_exec) { - struct cgroup_rstat_cpu *rstatc; + struct cgroup_rstat_base_cpu *rstatbc; unsigned long flags; - rstatc = cgroup_base_stat_cputime_account_begin(cgrp, &flags); + rstatbc = cgroup_base_stat_cputime_account_begin(cgrp, &flags); switch (index) { case CPUTIME_NICE: - rstatc->bstat.ntime += delta_exec; + rstatbc->bstat.ntime += delta_exec; fallthrough; case CPUTIME_USER: - rstatc->bstat.cputime.utime += delta_exec; + rstatbc->bstat.cputime.utime += delta_exec; break; case CPUTIME_SYSTEM: case CPUTIME_IRQ: case CPUTIME_SOFTIRQ: - rstatc->bstat.cputime.stime += delta_exec; + rstatbc->bstat.cputime.stime += delta_exec; break; #ifdef CONFIG_SCHED_CORE case CPUTIME_FORCEIDLE: - rstatc->bstat.forceidle_sum += delta_exec; + rstatbc->bstat.forceidle_sum += delta_exec; break; #endif default: break; } - cgroup_base_stat_cputime_account_end(cgrp, rstatc, flags); + cgroup_base_stat_cputime_account_end(cgrp, rstatbc, flags); } /* -- 2.47.1