From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9230C3601B for ; Fri, 4 Apr 2025 01:11:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A41466B0008; Thu, 3 Apr 2025 21:11:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C7CD6B000C; Thu, 3 Apr 2025 21:11:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7397B6B000D; Thu, 3 Apr 2025 21:11:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 481166B0008 for ; Thu, 3 Apr 2025 21:11:08 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2AC0BC1ACE for ; Fri, 4 Apr 2025 01:11:09 +0000 (UTC) X-FDA: 83294582658.24.213B3F3 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf21.hostedemail.com (Postfix) with ESMTP id 66D281C0008 for ; Fri, 4 Apr 2025 01:11:07 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HVWo5XHm; spf=pass (imf21.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743729067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5vwLVvjlWc9f28WdHegVpOwSu3QlhWLOaVgP+ftxD/Q=; b=F6ss7RR5Yr0tycJcqSKI7qiCUDW4sR8h1dzFfkXnZW7pwBbgKVpU5lXyrMhPzN7HZ2s8rS cwoqGSz9tV1xeQN22r9pzYOp58Cvfl/RavlffMOnb/hz6M5WdatJeQ9d862xsWz+L/sx5+ 4xsrp+NCOEw/ofe66CiW9DBHCjZ93CM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743729067; a=rsa-sha256; cv=none; b=2pSW7KR19x9fWMJQ6UY8eh5wguRQlTBxj/gkiXCdCvmaPagWbOmpUPdecHsR/R6eTZugYj EmUECAewGVbMDoRvmzHKtDtZAcBxWUvLpp5Hxx+oy3XYjbCXdawAwtcnHyr3aoQXA4f+fV rhj+inHBXAs/AshZ8SrXA3plTIF6hTU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HVWo5XHm; spf=pass (imf21.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-223fb0f619dso16151635ad.1 for ; Thu, 03 Apr 2025 18:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743729066; x=1744333866; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5vwLVvjlWc9f28WdHegVpOwSu3QlhWLOaVgP+ftxD/Q=; b=HVWo5XHmSrD/5/Mx3RfbeParM6l787PZP1HKkVoQviDIdx5YNjA6kbt7sbIYvs0grt /O+Hs0WIFUNv8D6pF05mISW3SLbLKWzZAkHrts2TO7qa3+w5NtnddNIAdjeUGyUvaRtR 7nn7klHEHlfpXr77BT7MQ+ainDGTH+odaC3zPB7gcSdbyPVFpPZ2iELxByGAy2FwO0hj jqPgKhyZD8pLTLQ5EPxCWVCSc40oRl3pd6vZ1R3mhLG8jbVeYLKhsKPsAXg050cqdskN nDUt2uf66PVDedluLM9JQIegz5ulQa+xwH5aD4JDj3RJ/nt4+3BsdaJo9fLSlACRdf1m 30dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743729066; x=1744333866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5vwLVvjlWc9f28WdHegVpOwSu3QlhWLOaVgP+ftxD/Q=; b=tFJedrezRbI4gz9cI5GVuvfzV8I0kPB4jvsMR2fNqzGzAdlV6i/g5ALQPFWnWZ8j9R ckEujgjTotg99iueBAr3D6QNiHs1/KQ+pXzS0QCEh9hxugJv2DiDRbipiohSpUSi6RC/ Kc4xHEvawHFzk0W20fo6SKM51/PezLir5R9RVFvwzbAdwtXWeBga+8AFHX8TMJ480EAE MO6VEY1ZP41SHswuMmM+qcj50uILTy88W1bOBOs55Ymuj+Nr2g6W9hm9mSeAvdSs3H+6 JbUSLXbW6VBlo8oMbVu3nW3tUfIuRzQyPht6Nh8yiuft6NaapQ04iDwBKwqLJKNEbtYE x3vg== X-Gm-Message-State: AOJu0Ywn+vHw9EkZLtyR7FDB+b3zz/i6ve4v2dxlN3DDIAY9oTXLp8fY aE2PJAI9NgV7uFztNIL9nEA5IseEVM9y1nu1AG0T+foJBlFBEJTr X-Gm-Gg: ASbGncuPhzzOh/5MxFXgbKhR0QEqTPtuDMKxGJaQQ1BgRxmwsCoKaNBFgW49AF7DY1m 0ppHtOOOB5HZXywh+Ta+OCIu4IAQz4kXj7cMkLRw1w2p+jlV8UxLzN6fnfAD9PbWfyIM0WZ3Dc1 rgbuMKUkgDV9cVgADTzI2T5qDqXOrsIW/XV9LuoVp/agVzZL0tC+VQZOAjxSjdh8J4jopQ7AUZe 8jBGjOvDMrFN48XOsB7iVTas66Xyd1BYzIcVzJoKPfyJbzznwUXvxMze4z+eLE8uRs96NmkiT14 4SfJkB3hEgLwRwv/WQL0tqmeZN/0K1/jSVIsX5+6AFOl9QE6SkmqnlarEqXtG+Rjx4/eKKxj X-Google-Smtp-Source: AGHT+IG6qH+9OB7e70FWVAVKaHc6KOpA4SpNNU673ZRvU1zuYUnkNGSyYQV0Xdr1X73Ttnd1YAuaDw== X-Received: by 2002:a17:902:e5c4:b0:224:26fd:82e5 with SMTP id d9443c01a7336-22a8a0b40c8mr14656445ad.48.1743729066265; Thu, 03 Apr 2025 18:11:06 -0700 (PDT) Received: from jpkobryn-fedora-PF5CFKNC.thefacebook.com ([2620:10d:c090:500::7:9b28]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-229785ad9a0sm21268675ad.39.2025.04.03.18.11.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Apr 2025 18:11:05 -0700 (PDT) From: JP Kobryn To: tj@kernel.org, shakeel.butt@linux.dev, yosryahmed@google.com, mkoutny@suse.com, hannes@cmpxchg.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 3/5] cgroup: change rstat function signatures from cgroup-based to css-based Date: Thu, 3 Apr 2025 18:10:48 -0700 Message-ID: <20250404011050.121777-4-inwardvessel@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250404011050.121777-1-inwardvessel@gmail.com> References: <20250404011050.121777-1-inwardvessel@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: t9cg51fjwa16fnkunckdsxsrkkjyut69 X-Rspam-User: X-Rspamd-Queue-Id: 66D281C0008 X-Rspamd-Server: rspam08 X-HE-Tag: 1743729067-753360 X-HE-Meta: U2FsdGVkX18klsE9iS75W8tJFGLvw0BGM67i/1+snruHeYg3Hs1rtuXY56C6/HOlmKXfqoJUuo//9S3UNyDwJp5wdI1Gl09lmFpfZz7k7fLutFNU+kaFUKhESIwPErUVyuZ3Pbp1qPuB0zCzSwJwyQiQqeQaFKQTlG5KerJqTFumpyPXK9hqv5dLmwgEFG8Xsrhmc3593D1krYHzsR5YuLMjszHYdQQVah3KsSzw7cI+zJjMHCLWepWSXPXz1C16gCfB6GHGCvWQKB8KbVsEaExo9wG6UuK3j99Mq+AoFKBO0n9meINdG1ZjfVgbMAjTWiB5mMpRPn2pa4hm0ljqSbJMec8br9fYH9QR2PV8ryoHb8PD2J7av1Txt+foBnSpfsgL2nFEtE3kK1g7ybMU8uAmD41CNqFL/xReuRLD6VBdP+b0wL04I+PO3PEm66cXfGA1Fr7Rz+TcEgrKYmQk2alxMGSUB3AizMwaDH0SE+hjS0eMU9kE3BKVPCwqXiHsv9ZqNvaWpvcAOU1xPW/m5+w6RzClv6ICAQP7LW6AWGUharNv3HCwOKUqwIm0evAsJbcY+INgynFT3wamFtd27mKvvsDDFd63m2rNrEvPVK57iH0Ztt7GoYI0CYH4vJmoH/9JgpNMIYdJfKR9v6v/Rb7oP+G4KlqlSSPqHvPNyuosURK+flMSxOw1kpBtRRaAWClOQgz1GHYTIl0Tnsb3xxFuAA7gBcMQiQdB1bkpby/2uroqTa8tGd3bHdDVedoX0CHPGh1Tn+DWp6Rshtg4J14QRHjANl+4N7M0o7D7KotNj4Uy50UaboPMHEEXABCltMMOkqFj7H9pXWDj1pm51Ys5jE7a12KPyrlIjngdVIsFSP2Es/K1+JcFEvqAfrP+H7XzuDODX9F45a6WwJDxcTLS3Er44rIKhVCgqkyjTt68S430mPZlgMbEcm4bpwXOIVmFC51u6AErehbJAoC +SJhMKIK qo2UXimdOx8BDLUQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This non-functional change serves as preparation for moving to subsystem-based rstat trees. To simplify future commits, change the signatures of existing cgroup-based rstat functions to become css-based and rename them to reflect that. Though the signatures have changed, the implementations have not. Within these functions use the css->cgroup pointer to obtain the associated cgroup and allow code to function the same just as it did before this patch. At applicable call sites, pass the subsystem-specific css pointer as an argument or pass a pointer to cgroup::self if not in subsystem context. Note that cgroup_rstat_updated_list() and cgroup_rstat_push_children() are not altered yet since there would be a larger amount of css to cgroup conversions which may overcomplicate the code at this intermediate phase. Signed-off-by: JP Kobryn --- block/blk-cgroup.c | 6 +- include/linux/cgroup-defs.h | 2 +- include/linux/cgroup.h | 4 +- kernel/cgroup/cgroup-internal.h | 4 +- kernel/cgroup/cgroup.c | 30 ++++--- kernel/cgroup/rstat.c | 83 +++++++++++-------- mm/memcontrol.c | 4 +- .../bpf/progs/cgroup_hierarchical_stats.c | 9 +- 8 files changed, 80 insertions(+), 62 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 5905f277057b..0560ea402856 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1144,7 +1144,7 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu) /* * We source root cgroup stats from the system-wide stats to avoid * tracking the same information twice and incurring overhead when no - * cgroups are defined. For that reason, cgroup_rstat_flush in + * cgroups are defined. For that reason, css_rstat_flush in * blkcg_print_stat does not actually fill out the iostat in the root * cgroup's blkcg_gq. * @@ -1253,7 +1253,7 @@ static int blkcg_print_stat(struct seq_file *sf, void *v) if (!seq_css(sf)->parent) blkcg_fill_root_iostats(); else - cgroup_rstat_flush(blkcg->css.cgroup); + css_rstat_flush(&blkcg->css); rcu_read_lock(); hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) { @@ -2243,7 +2243,7 @@ void blk_cgroup_bio_start(struct bio *bio) } u64_stats_update_end_irqrestore(&bis->sync, flags); - cgroup_rstat_updated(blkcg->css.cgroup, cpu); + css_rstat_updated(&blkcg->css, cpu); put_cpu(); } diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 6d177f770d28..e4a9fb00b228 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -536,7 +536,7 @@ struct cgroup { /* * A singly-linked list of cgroup structures to be rstat flushed. * This is a scratch field to be used exclusively by - * cgroup_rstat_flush_locked() and protected by cgroup_rstat_lock. + * css_rstat_flush_locked() and protected by cgroup_rstat_lock. */ struct cgroup *rstat_flush_next; diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 7c120efd5e49..b906c53953b9 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -693,8 +693,8 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) /* * cgroup scalable recursive statistics. */ -void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); -void cgroup_rstat_flush(struct cgroup *cgrp); +void css_rstat_updated(struct cgroup_subsys_state *css, int cpu); +void css_rstat_flush(struct cgroup_subsys_state *css); /* * Basic resource stats. diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index 95ab39e1ec8f..c161d34be634 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -270,8 +270,8 @@ int cgroup_task_count(const struct cgroup *cgrp); /* * rstat.c */ -int cgroup_rstat_init(struct cgroup *cgrp); -void cgroup_rstat_exit(struct cgroup *cgrp); +int css_rstat_init(struct cgroup_subsys_state *css); +void css_rstat_exit(struct cgroup_subsys_state *css); void cgroup_rstat_boot(void); void cgroup_base_stat_cputime_show(struct seq_file *seq); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 00eb882dc6e7..f8cee7819642 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -1375,7 +1375,7 @@ static void cgroup_destroy_root(struct cgroup_root *root) cgroup_unlock(); - cgroup_rstat_exit(cgrp); + css_rstat_exit(&cgrp->self); kernfs_destroy_root(root->kf_root); cgroup_free_root(root); } @@ -2150,7 +2150,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) if (ret) goto destroy_root; - ret = cgroup_rstat_init(root_cgrp); + ret = css_rstat_init(&root_cgrp->self); if (ret) goto destroy_root; @@ -2192,7 +2192,7 @@ int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask) goto out; exit_stats: - cgroup_rstat_exit(root_cgrp); + css_rstat_exit(&root_cgrp->self); destroy_root: kernfs_destroy_root(root->kf_root); root->kf_root = NULL; @@ -5449,7 +5449,7 @@ static void css_free_rwork_fn(struct work_struct *work) cgroup_put(cgroup_parent(cgrp)); kernfs_put(cgrp->kn); psi_cgroup_free(cgrp); - cgroup_rstat_exit(cgrp); + css_rstat_exit(css); kfree(cgrp); } else { /* @@ -5479,7 +5479,7 @@ static void css_release_work_fn(struct work_struct *work) /* css release path */ if (!list_empty(&css->rstat_css_node)) { - cgroup_rstat_flush(cgrp); + css_rstat_flush(css); list_del_rcu(&css->rstat_css_node); } @@ -5507,7 +5507,7 @@ static void css_release_work_fn(struct work_struct *work) /* cgroup release path */ TRACE_CGROUP_PATH(release, cgrp); - cgroup_rstat_flush(cgrp); + css_rstat_flush(css); spin_lock_irq(&css_set_lock); for (tcgrp = cgroup_parent(cgrp); tcgrp; @@ -5700,17 +5700,13 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, if (ret) goto out_free_cgrp; - ret = cgroup_rstat_init(cgrp); - if (ret) - goto out_cancel_ref; - /* create the directory */ kn = kernfs_create_dir_ns(parent->kn, name, mode, current_fsuid(), current_fsgid(), cgrp, NULL); if (IS_ERR(kn)) { ret = PTR_ERR(kn); - goto out_stat_exit; + goto out_cancel_ref; } cgrp->kn = kn; @@ -5720,6 +5716,14 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, cgrp->root = root; cgrp->level = level; + /* + * Now that init_cgroup_housekeeping() has been called and cgrp->self + * is setup, it is safe to perform rstat initialization on it. + */ + ret = css_rstat_init(&cgrp->self); + if (ret) + goto out_stat_exit; + ret = psi_cgroup_alloc(cgrp); if (ret) goto out_kernfs_remove; @@ -5790,10 +5794,10 @@ static struct cgroup *cgroup_create(struct cgroup *parent, const char *name, out_psi_free: psi_cgroup_free(cgrp); +out_stat_exit: + css_rstat_exit(&cgrp->self); out_kernfs_remove: kernfs_remove(cgrp->kn); -out_stat_exit: - cgroup_rstat_exit(cgrp); out_cancel_ref: percpu_ref_exit(&cgrp->self.refcnt); out_free_cgrp: diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a20e3ab3f7d3..5bca55b4ec15 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -34,9 +34,10 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup *cgrp, const bool fast_path) +unsigned long _css_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, + struct cgroup_subsys_state *css, const bool fast_path) { + struct cgroup *cgrp = css->cgroup; unsigned long flags; bool contended; @@ -67,10 +68,12 @@ unsigned long _cgroup_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup *cgrp, unsigned long flags, +void _css_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, + struct cgroup_subsys_state *css, unsigned long flags, const bool fast_path) { + struct cgroup *cgrp = css->cgroup; + if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else @@ -80,16 +83,17 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, } /** - * cgroup_rstat_updated - keep track of updated rstat_cpu - * @cgrp: target cgroup + * css_rstat_updated - keep track of updated rstat_cpu + * @css: target cgroup subsystem state * @cpu: cpu on which rstat_cpu was updated * - * @cgrp's rstat_cpu on @cpu was updated. Put it on the parent's matching - * rstat_cpu->updated_children list. See the comment on top of + * @css->cgroup's rstat_cpu on @cpu was updated. Put it on the parent's + * matching rstat_cpu->updated_children list. See the comment on top of * cgroup_rstat_cpu definition for details. */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +__bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) { + struct cgroup *cgrp = css->cgroup; raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -104,7 +108,7 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) if (data_race(cgroup_rstat_cpu(cgrp, cpu)->updated_next)) return; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, cgrp, true); + flags = _css_rstat_cpu_lock(cpu_lock, cpu, css, true); /* put @cgrp and all ancestors on the corresponding updated lists */ while (true) { @@ -132,7 +136,7 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) cgrp = parent; } - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true); + _css_rstat_cpu_unlock(cpu_lock, cpu, css, flags, true); } /** @@ -213,7 +217,7 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) struct cgroup *head = NULL, *parent, *child; unsigned long flags; - flags = _cgroup_rstat_cpu_lock(cpu_lock, cpu, root, false); + flags = _css_rstat_cpu_lock(cpu_lock, cpu, &root->self, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -250,14 +254,14 @@ static struct cgroup *cgroup_rstat_updated_list(struct cgroup *root, int cpu) if (child != root) head = cgroup_rstat_push_children(head, child, cpu); unlock_ret: - _cgroup_rstat_cpu_unlock(cpu_lock, cpu, root, flags, false); + _css_rstat_cpu_unlock(cpu_lock, cpu, &root->self, flags, false); return head; } /* * A hook for bpf stat collectors to attach to and flush their stats. - * Together with providing bpf kfuncs for cgroup_rstat_updated() and - * cgroup_rstat_flush(), this enables a complete workflow where bpf progs that + * Together with providing bpf kfuncs for css_rstat_updated() and + * css_rstat_flush(), this enables a complete workflow where bpf progs that * collect cgroup stats can integrate with rstat for efficient flushing. * * A static noinline declaration here could cause the compiler to optimize away @@ -285,9 +289,11 @@ __bpf_hook_end(); * value -1 is used when obtaining the main lock else this is the CPU * number processed last. */ -static inline void __cgroup_rstat_lock(struct cgroup *cgrp, int cpu_in_loop) +static inline void __css_rstat_lock(struct cgroup_subsys_state *css, + int cpu_in_loop) __acquires(&cgroup_rstat_lock) { + struct cgroup *cgrp = css->cgroup; bool contended; contended = !spin_trylock_irq(&cgroup_rstat_lock); @@ -298,36 +304,41 @@ static inline void __cgroup_rstat_lock(struct cgroup *cgrp, int cpu_in_loop) trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended); } -static inline void __cgroup_rstat_unlock(struct cgroup *cgrp, int cpu_in_loop) +static inline void __css_rstat_unlock(struct cgroup_subsys_state *css, + int cpu_in_loop) __releases(&cgroup_rstat_lock) { + struct cgroup *cgrp = css->cgroup; + trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false); spin_unlock_irq(&cgroup_rstat_lock); } /** - * cgroup_rstat_flush - flush stats in @cgrp's subtree - * @cgrp: target cgroup + * css_rstat_flush - flush stats in @css->cgroup's subtree + * @css: target cgroup subsystem state * - * Collect all per-cpu stats in @cgrp's subtree into the global counters + * Collect all per-cpu stats in @css->cgroup's subtree into the global counters * and propagate them upwards. After this function returns, all cgroups in * the subtree have up-to-date ->stat. * - * This also gets all cgroups in the subtree including @cgrp off the + * This also gets all cgroups in the subtree including @css->cgroup off the * ->updated_children lists. * * This function may block. */ -__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) +__bpf_kfunc void css_rstat_flush(struct cgroup_subsys_state *css) { + struct cgroup *cgrp = css->cgroup; int cpu; might_sleep(); for_each_possible_cpu(cpu) { - struct cgroup *pos = cgroup_rstat_updated_list(cgrp, cpu); + struct cgroup *pos; /* Reacquire for each CPU to avoid disabling IRQs too long */ - __cgroup_rstat_lock(cgrp, cpu); + __css_rstat_lock(css, cpu); + pos = cgroup_rstat_updated_list(cgrp, cpu); for (; pos; pos = pos->rstat_flush_next) { struct cgroup_subsys_state *css; @@ -340,14 +351,15 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) css->ss->css_rstat_flush(css, cpu); rcu_read_unlock(); } - __cgroup_rstat_unlock(cgrp, cpu); + __css_rstat_unlock(css, cpu); if (!cond_resched()) cpu_relax(); } } -int cgroup_rstat_init(struct cgroup *cgrp) +int css_rstat_init(struct cgroup_subsys_state *css) { + struct cgroup *cgrp = css->cgroup; int cpu; /* the root cgrp has rstat_cpu preallocated */ @@ -378,11 +390,12 @@ int cgroup_rstat_init(struct cgroup *cgrp) return 0; } -void cgroup_rstat_exit(struct cgroup *cgrp) +void css_rstat_exit(struct cgroup_subsys_state *css) { + struct cgroup *cgrp = css->cgroup; int cpu; - cgroup_rstat_flush(cgrp); + css_rstat_flush(&cgrp->self); /* sanity check */ for_each_possible_cpu(cpu) { @@ -489,7 +502,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, unsigned long flags) { u64_stats_update_end_irqrestore(&rstatbc->bsync, flags); - cgroup_rstat_updated(cgrp, smp_processor_id()); + css_rstat_updated(&cgrp->self, smp_processor_id()); put_cpu_ptr(rstatbc); } @@ -591,12 +604,12 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq) struct cgroup_base_stat bstat; if (cgroup_parent(cgrp)) { - cgroup_rstat_flush(cgrp); - __cgroup_rstat_lock(cgrp, -1); + css_rstat_flush(&cgrp->self); + __css_rstat_lock(&cgrp->self, -1); bstat = cgrp->bstat; cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime, &bstat.cputime.utime, &bstat.cputime.stime); - __cgroup_rstat_unlock(cgrp, -1); + __css_rstat_unlock(&cgrp->self, -1); } else { root_cgroup_cputime(&bstat); } @@ -618,10 +631,10 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq) cgroup_force_idle_show(seq, &bstat); } -/* Add bpf kfuncs for cgroup_rstat_updated() and cgroup_rstat_flush() */ +/* Add bpf kfuncs for css_rstat_updated() and css_rstat_flush() */ BTF_KFUNCS_START(bpf_rstat_kfunc_ids) -BTF_ID_FLAGS(func, cgroup_rstat_updated) -BTF_ID_FLAGS(func, cgroup_rstat_flush, KF_SLEEPABLE) +BTF_ID_FLAGS(func, css_rstat_updated) +BTF_ID_FLAGS(func, css_rstat_flush, KF_SLEEPABLE) BTF_KFUNCS_END(bpf_rstat_kfunc_ids) static const struct btf_kfunc_id_set bpf_rstat_kfunc_set = { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 421740f1bcdc..140b31821b48 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -582,7 +582,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) if (!val) return; - cgroup_rstat_updated(memcg->css.cgroup, cpu); + css_rstat_updated(&memcg->css, cpu); statc = this_cpu_ptr(memcg->vmstats_percpu); for (; statc; statc = statc->parent) { stats_updates = READ_ONCE(statc->stats_updates) + abs(val); @@ -614,7 +614,7 @@ static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force) if (mem_cgroup_is_root(memcg)) WRITE_ONCE(flush_last_time, jiffies_64); - cgroup_rstat_flush(memcg->css.cgroup); + css_rstat_flush(&memcg->css); } /* diff --git a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c index c74362854948..ff189a736ad8 100644 --- a/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c +++ b/tools/testing/selftests/bpf/progs/cgroup_hierarchical_stats.c @@ -37,8 +37,9 @@ struct { __type(value, struct attach_counter); } attach_counters SEC(".maps"); -extern void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) __ksym; -extern void cgroup_rstat_flush(struct cgroup *cgrp) __ksym; +extern void css_rstat_updated( + struct cgroup_subsys_state *css, int cpu) __ksym; +extern void css_rstat_flush(struct cgroup_subsys_state *css) __ksym; static uint64_t cgroup_id(struct cgroup *cgrp) { @@ -75,7 +76,7 @@ int BPF_PROG(counter, struct cgroup *dst_cgrp, struct task_struct *leader, else if (create_percpu_attach_counter(cg_id, 1)) return 0; - cgroup_rstat_updated(dst_cgrp, bpf_get_smp_processor_id()); + css_rstat_updated(&dst_cgrp->self, bpf_get_smp_processor_id()); return 0; } @@ -141,7 +142,7 @@ int BPF_PROG(dumper, struct bpf_iter_meta *meta, struct cgroup *cgrp) return 1; /* Flush the stats to make sure we get the most updated numbers */ - cgroup_rstat_flush(cgrp); + css_rstat_flush(&cgrp->self); total_counter = bpf_map_lookup_elem(&attach_counters, &cg_id); if (!total_counter) { -- 2.47.1