From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AAB7C54754 for ; Thu, 15 May 2025 00:20:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B92686B00B4; Wed, 14 May 2025 20:19:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B451E6B00EB; Wed, 14 May 2025 20:19:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96CBB6B00EC; Wed, 14 May 2025 20:19:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 714916B00B4 for ; Wed, 14 May 2025 20:19:55 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3B3801A064A for ; Thu, 15 May 2025 00:19:56 +0000 (UTC) X-FDA: 83443234392.26.42A8959 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf25.hostedemail.com (Postfix) with ESMTP id 84611A0008 for ; Thu, 15 May 2025 00:19:54 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dPTIlpbr; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747268394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fWFox4LIjsqPkf8GOiIF7Krk9TUQS5EFTj7kh57y2mY=; b=zA4h1khkmiAd+KL/CX9C+P68BqeVK50K7oAIJbE6pOl1lXeqWsjqnsWoyaDGrJjn3SfbcL PijmD73CfTVSz08YehNG0nokvMGbQzJSGGoXqFmfRAb/in9s0hpZApQz74JbFgoXVpmuOz hzfcwX3e6SP2Cg9ec0WTC5BNbbOcvH4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747268394; a=rsa-sha256; cv=none; b=mgNqma2f41Or7IqqwiRCMPE08Gl8CwQYdVQK9QO97/V5JXVpRPqCRG3W9CybL5/L+v2br0 7xzC8Be04i/hkViJ4REZIOrG5aTmmtclta26WVBOSd+PCDwY+WOXBVuLnyffxrNGNCHZVj IAquxYkE/IgVva5AKW0JN7IL4YOlA0o= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dPTIlpbr; spf=pass (imf25.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-22fa48f7cb2so3950915ad.1 for ; Wed, 14 May 2025 17:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747268393; x=1747873193; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fWFox4LIjsqPkf8GOiIF7Krk9TUQS5EFTj7kh57y2mY=; b=dPTIlpbr+5auUzWmTitFpalhnmsp7u7hMj/OeeBX3C125yaxXk2uTwbvxZBbVCusDg YT1wXAFM6jUep0IkAjfiXXK/PJ1d4ppfkEYHhDc7pktcdDjPYOFFZFl66miebPej9tRY dlZSQnEMWi0lXWMe6874Y48+Dnzq5wqHMaj/m5XWW8I+0MK7CgpRTMO/AbSWLhsJAt5n NiAusbayOBZYoJpn0RvD6XLDTzHgU3EjtMBDH5eAF7/Ryf25R8DCOjks9oNn8zCDug9q /QmLi7t8rqVKGJlus49uVNp4OuUEti5us5KQB7FbGnUq/QvC7ayh3REdaN/Xy6zWOD+Q 1LFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747268393; x=1747873193; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fWFox4LIjsqPkf8GOiIF7Krk9TUQS5EFTj7kh57y2mY=; b=iwgc0S4ProD27alxP/q90BWfmq6QTu1ZFXb8IK8Wq5EYDT4s9Cb80mWfALfKfRCIj2 hlLPDc+2Mpy1E6xc8P4mdIXRkVRknF22c9z5CkKPeqCyjQwJ8x0x839mtq6DQiOnTO0H H8NW1HqUeTGnkYqJQfxYxEykQzQNZqVobIskM+LqTXvHzy7WGQdiji7nIj+gED7k0gpS 6bnRbXqGeV3J/h/VvGc0NBgj69VxYVIVAL4RwhzFYoUPNDYe62xNscur9Lrf+dbZd4yc /dokH2PwD0u4hhx7hMoCnIyZ7NOXYD5uVBUtaSOqsyIqgwxAvIQWGi0a64JbDU8ie0nZ k2XA== X-Gm-Message-State: AOJu0YySGYpe5A0hh0zG2I27Pgms/p+lgoS0zm9yPUney4HxmHF5rKtP jJ71GSeEFzc5CkGcM3iWeYI1G/3O63qPXoCV0zHS0n0GOWhAZlD5 X-Gm-Gg: ASbGncvSZWJKNnrz+rG9kQsf8jXuNK6p/i6tT/XSnA//e3Tga5WNTGxXdDUa/bYNpAv 5Qylmq89Q1Zxht4A1BMxqNpOBZsMldBqL4JOjoLkHwEk2lTX+4DopdqEPEv4dmmgaHS5ci5Env4 +q/wipDbbzgUTItpMgf2+cl7QGkhNTS7yeHghw1eIHfQG8kFjqn5GGxQ84nJ/nO28hFKgRmZMVy 8qxCacawJAddFOqnnrTDEJp1Ekss8n/DF0N/m7DQHsDblqU/ndxjuvpeg11BpG2Jo1YMUv25r5m /F0ptQ2UBxFPYFcuovH8iE1r8xb3t7sVN0eEaT16OnwSTlCaFjc47OkjWJAy/RODkYWqJwHUVFo lhRtzILtLiAXx/7mFnk2Pqkj0qlT0zy8QU0uW/IwhfZYRo7L9lQ== X-Google-Smtp-Source: AGHT+IEmWt7OzKVLMsDQaZtyJZngkoadbHoZBW1s+FCVw2/JSJ5mcNKu3bNimDj7381DTveNZAlOLA== X-Received: by 2002:a17:903:194f:b0:22f:a48f:7a9a with SMTP id d9443c01a7336-231981bf07emr67387955ad.37.1747268393381; Wed, 14 May 2025 17:19:53 -0700 (PDT) Received: from jpkobryn-fedora-PF5CFKNC.lan (c-67-188-127-15.hsd1.ca.comcast.net. [67.188.127.15]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22fc754785bsm105939545ad.20.2025.05.14.17.19.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 May 2025 17:19:52 -0700 (PDT) From: JP Kobryn To: tj@kernel.org, shakeel.butt@linux.dev, yosryahmed@google.com, mkoutny@suse.com, hannes@cmpxchg.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v6 4/6] cgroup: use subsystem-specific rstat locks to avoid contention Date: Wed, 14 May 2025 17:19:35 -0700 Message-ID: <20250515001937.219505-5-inwardvessel@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250515001937.219505-1-inwardvessel@gmail.com> References: <20250515001937.219505-1-inwardvessel@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: e93tf4n58pamxigeayjwgrgefin85qri X-Rspamd-Queue-Id: 84611A0008 X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1747268394-558450 X-HE-Meta: U2FsdGVkX1+p3FeKLtKuB8rnvfK2BbyuXNWe6NKCy3Z3wNRkVUUeSHIVRa0kxj5h9/4lv0I9gaAvlfyPbceRuTsW2zv3ErKVR6iF7QAOXw0YRoZ/RmrLDiy/RX+nZC0kIjTYPlv11/2zMGAYVp8H3ibJYMmgGUwG5nny7n0NNRv2Boxf4ITVA8fHe1GU9Nyq75oNjxLdUKgw4g0ljmuoVZTSha2Ad6fd2ZJCCkeCiMuGoPQXjLAIhZfGc2re1EtNcZln5T3C4XoxvJBFh2FB8Yg/1O4hvFLiGb+qAaYPLxp/pAfGCGyLDZX0KAC44aGgCAjwc6zQ22jB2ShjGJzNSNbT/ApEEnY3yM3SUlH3yiCVOoDN3kcuafzQkcKCZY5VeVakYuR6HdLQ+wGBTEpz1Tb7MdICVNt4DgCJzN8BE7WsMouXzUimTl8yQ2xQuKdoX9oxU4KLo9DvXbnWTQw5OcZ0FG+iOR6Jy2QYM973CNNz+OVNL5TEDGL71a6BKEbiV7urggBjBjCFufCatZSBQ1ZFJM9Hnhj2H+nSvm0Vl97LgHQKCUBzjtgHKZm5+4r31id7HBam8sNgeCZTl4Bjynm20MhFbFY3EQo0y8AcEgQ3SQuSxPv7KXHUgbPc3ecrNK+pdEg02L/i5KHiY2j5mVYqd6rqNqCTFLgJF3svTefY673kdNNQTZ2IzrYBg8/MKhQ/bLLXHcXV/CdppZi08ODR2aceENdKbJYsNhv4/3yUPGPwSYrRtkbb2ZkhLwD2jdK3iQIhcbUnVbDkk8KAORgRMwd7t6YZUw6m8TwBA0nXfE4nw+ybfCQNTLdFoqkglBv19wPOV0caFGCd3eCp4oV/XuPZ8CpnBTKHe5goJy/SR+1QtL0muFPt8amyLlSTu06d9baI613iiQBYNmusSsOKnKGkqfhkhkOBaLyrraWDQwOebCGCdDEJrl4IytVQioeh6G6dE9CY86+DDaD QQ0nFLPd i/DKJB3FyoGgbEUo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It is possible to eliminate contention between subsystems when updating/flushing stats by using subsystem-specific locks. Let the existing rstat locks be dedicated to the cgroup base stats and rename them to reflect that. Add similar locks to the cgroup_subsys struct for use with individual subsystems. Lock initialization is done in the new function ss_rstat_init(ss) which replaces cgroup_rstat_boot(void). If NULL is passed to this function, the global base stat locks will be initialized. Otherwise, the subsystem locks will be initialized. Change the existing lock helper functions to accept a reference to a css. Then within these functions, conditionally select the appropriate locks based on the subsystem affiliation of the given css. Add helper functions for this selection routine to avoid repeated code. Signed-off-by: JP Kobryn --- block/blk-cgroup.c | 4 +- include/linux/cgroup-defs.h | 10 +++- include/trace/events/cgroup.h | 12 +++- kernel/cgroup/cgroup-internal.h | 2 +- kernel/cgroup/cgroup.c | 3 +- kernel/cgroup/rstat.c | 98 +++++++++++++++++++++++---------- 6 files changed, 91 insertions(+), 38 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 0560ea402856..4dc10c1e97a4 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1074,8 +1074,8 @@ static void __blkcg_rstat_flush(struct blkcg *blkcg, int cpu) /* * For covering concurrent parent blkg update from blkg_release(). * - * When flushing from cgroup, cgroup_rstat_lock is always held, so - * this lock won't cause contention most of time. + * When flushing from cgroup, the subsystem rstat lock is always held, + * so this lock won't cause contention most of time. */ raw_spin_lock_irqsave(&blkg_stat_lock, flags); diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 17ecaae9c5f8..5b8127d29dc5 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -222,7 +222,10 @@ struct cgroup_subsys_state { /* * A singly-linked list of css structures to be rstat flushed. * This is a scratch field to be used exclusively by - * css_rstat_flush() and protected by cgroup_rstat_lock. + * css_rstat_flush(). + * + * Protected by rstat_base_lock when css is cgroup::self. + * Protected by css->ss->rstat_ss_lock otherwise. */ struct cgroup_subsys_state *rstat_flush_next; }; @@ -362,7 +365,7 @@ struct css_rstat_cpu { * the css makes it unnecessary for each per-cpu struct to point back * to the associated css. * - * Protected by per-cpu cgroup_rstat_cpu_lock. + * Protected by per-cpu css->ss->rstat_ss_cpu_lock. */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ @@ -792,6 +795,9 @@ struct cgroup_subsys { * specifies the mask of subsystems that this one depends on. */ unsigned int depends_on; + + spinlock_t rstat_ss_lock; + raw_spinlock_t __percpu *rstat_ss_cpu_lock; }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/include/trace/events/cgroup.h b/include/trace/events/cgroup.h index af2755bda6eb..7d332387be6c 100644 --- a/include/trace/events/cgroup.h +++ b/include/trace/events/cgroup.h @@ -231,7 +231,11 @@ DECLARE_EVENT_CLASS(cgroup_rstat, __entry->cpu, __entry->contended) ); -/* Related to global: cgroup_rstat_lock */ +/* + * Related to locks: + * global rstat_base_lock for base stats + * cgroup_subsys::rstat_ss_lock for subsystem stats + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), @@ -253,7 +257,11 @@ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_unlock, TP_ARGS(cgrp, cpu, contended) ); -/* Related to per CPU: cgroup_rstat_cpu_lock */ +/* + * Related to per CPU locks: + * global rstat_base_cpu_lock for base stats + * cgroup_subsys::rstat_ss_cpu_lock for subsystem stats + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index c161d34be634..b14e61c64a34 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -272,7 +272,7 @@ int cgroup_task_count(const struct cgroup *cgrp); */ int css_rstat_init(struct cgroup_subsys_state *css); void css_rstat_exit(struct cgroup_subsys_state *css); -void cgroup_rstat_boot(void); +int ss_rstat_init(struct cgroup_subsys *ss); void cgroup_base_stat_cputime_show(struct seq_file *seq); /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 45097dc9e099..44baa0318713 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6088,6 +6088,7 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss, bool early) css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); BUG_ON(css->id < 0); + BUG_ON(ss_rstat_init(ss)); BUG_ON(css_rstat_init(css)); } @@ -6167,7 +6168,7 @@ int __init cgroup_init(void) BUG_ON(cgroup_init_cftypes(NULL, cgroup_psi_files)); BUG_ON(cgroup_init_cftypes(NULL, cgroup1_base_files)); - cgroup_rstat_boot(); + BUG_ON(ss_rstat_init(NULL)); get_user_ns(init_cgroup_ns.user_ns); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 6ce134a7294d..0bb609e73bde 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -9,8 +9,8 @@ #include -static DEFINE_SPINLOCK(cgroup_rstat_lock); -static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +static DEFINE_SPINLOCK(rstat_base_lock); +static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -26,8 +26,24 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( return per_cpu_ptr(cgrp->rstat_base_cpu, cpu); } +static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) +{ + if (ss) + return &ss->rstat_ss_lock; + + return &rstat_base_lock; +} + +static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->rstat_ss_cpu_lock, cpu); + + return per_cpu_ptr(&rstat_base_cpu_lock, cpu); +} + /* - * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). + * Helper functions for rstat per CPU locks. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @fast_path determine the @@ -35,21 +51,23 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _css_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup_subsys_state *css, const bool fast_path) +unsigned long _css_rstat_cpu_lock(struct cgroup_subsys_state *css, int cpu, + const bool fast_path) { struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; unsigned long flags; bool contended; /* - * The _irqsave() is needed because cgroup_rstat_lock is - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring - * this lock with the _irq() suffix only disables interrupts on - * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables - * interrupts on both configurations. The _irqsave() ensures - * that interrupts are always disabled and later restored. + * The _irqsave() is needed because the locks used for flushing are + * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring this lock + * with the _irq() suffix only disables interrupts on a non-PREEMPT_RT + * kernel. The raw_spinlock_t below disables interrupts on both + * configurations. The _irqsave() ensures that interrupts are always + * disabled and later restored. */ + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); contended = !raw_spin_trylock_irqsave(cpu_lock, flags); if (contended) { if (fast_path) @@ -69,17 +87,18 @@ unsigned long _css_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _css_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup_subsys_state *css, unsigned long flags, - const bool fast_path) +void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu, + unsigned long flags, const bool fast_path) { struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else trace_cgroup_rstat_cpu_unlock(cgrp, cpu, false); + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); raw_spin_unlock_irqrestore(cpu_lock, flags); } @@ -94,7 +113,6 @@ void _css_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, */ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; /* @@ -115,7 +133,7 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) if (data_race(css_rstat_cpu(css, cpu)->updated_next)) return; - flags = _css_rstat_cpu_lock(cpu_lock, cpu, css, true); + flags = _css_rstat_cpu_lock(css, cpu, true); /* put @css and all ancestors on the corresponding updated lists */ while (true) { @@ -143,7 +161,7 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) css = parent; } - _css_rstat_cpu_unlock(cpu_lock, cpu, css, flags, true); + _css_rstat_cpu_unlock(css, cpu, flags, true); } /** @@ -171,11 +189,11 @@ static struct cgroup_subsys_state *css_rstat_push_children( child->rstat_flush_next = NULL; /* - * The cgroup_rstat_lock must be held for the whole duration from + * The subsystem rstat lock must be held for the whole duration from * here as the rstat_flush_next list is being constructed to when * it is consumed later in css_rstat_flush(). */ - lockdep_assert_held(&cgroup_rstat_lock); + lockdep_assert_held(ss_rstat_lock(head->ss)); /* * Notation: -> updated_next pointer @@ -245,12 +263,11 @@ static struct cgroup_subsys_state *css_rstat_push_children( static struct cgroup_subsys_state *css_rstat_updated_list( struct cgroup_subsys_state *root, int cpu) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct css_rstat_cpu *rstatc = css_rstat_cpu(root, cpu); struct cgroup_subsys_state *head = NULL, *parent, *child; unsigned long flags; - flags = _css_rstat_cpu_lock(cpu_lock, cpu, root, false); + flags = _css_rstat_cpu_lock(root, cpu, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -287,7 +304,7 @@ static struct cgroup_subsys_state *css_rstat_updated_list( if (child != root) head = css_rstat_push_children(head, child, cpu); unlock_ret: - _css_rstat_cpu_unlock(cpu_lock, cpu, root, flags, false); + _css_rstat_cpu_unlock(root, cpu, flags, false); return head; } @@ -314,7 +331,7 @@ __weak noinline void bpf_rstat_flush(struct cgroup *cgrp, __bpf_hook_end(); /* - * Helper functions for locking cgroup_rstat_lock. + * Helper functions for locking. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @cpu_in_loop indicate lock @@ -324,27 +341,31 @@ __bpf_hook_end(); */ static inline void __css_rstat_lock(struct cgroup_subsys_state *css, int cpu_in_loop) - __acquires(&cgroup_rstat_lock) + __acquires(ss_rstat_lock(css->ss)) { struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; bool contended; - contended = !spin_trylock_irq(&cgroup_rstat_lock); + lock = ss_rstat_lock(css->ss); + contended = !spin_trylock_irq(lock); if (contended) { trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, contended); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock_irq(lock); } trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended); } static inline void __css_rstat_unlock(struct cgroup_subsys_state *css, int cpu_in_loop) - __releases(&cgroup_rstat_lock) + __releases(ss_rstat_lock(css->ss)) { struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; + lock = ss_rstat_lock(css->ss); trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock_irq(lock); } /** @@ -466,12 +487,29 @@ void css_rstat_exit(struct cgroup_subsys_state *css) css->rstat_cpu = NULL; } -void __init cgroup_rstat_boot(void) +/** + * ss_rstat_init - subsystem-specific rstat initialization + * @ss: target subsystem + * + * If @ss is NULL, the static locks associated with the base stats + * are initialized. If @ss is non-NULL, the subsystem-specific locks + * are initialized. + */ +int __init ss_rstat_init(struct cgroup_subsys *ss) { int cpu; + if (ss) { + ss->rstat_ss_cpu_lock = alloc_percpu(raw_spinlock_t); + if (!ss->rstat_ss_cpu_lock) + return -ENOMEM; + } + + spin_lock_init(ss_rstat_lock(ss)); for_each_possible_cpu(cpu) - raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + + return 0; } /* -- 2.47.1