From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28235C3ABAA for ; Sat, 3 May 2025 00:19:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DCCD6B009B; Fri, 2 May 2025 20:19:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 289AF6B00A4; Fri, 2 May 2025 20:19:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 069CC6B00CC; Fri, 2 May 2025 20:19:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D65546B00C3 for ; Fri, 2 May 2025 20:19:31 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F3977C8632 for ; Sat, 3 May 2025 00:19:31 +0000 (UTC) X-FDA: 83399687742.13.9546555 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by imf24.hostedemail.com (Postfix) with ESMTP id 3094F180007 for ; Sat, 3 May 2025 00:19:30 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=alKFwxZa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.210.48 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746231570; a=rsa-sha256; cv=none; b=cirC7YDLodwU0UzcSly0AExk9cWk7XR9MCy5R1nhxtdmJ04ryjACNXcb3mUNn2MOZcTGpI uHZYMCaTe7zVQxhhvlpz2aqk5/dDF3/C3nU+owftg65OKu0yGZi2n3S57TkHMAe5I8E4Az plFMg4a8XwQmNzeZiVDZAzbdMV6BmD0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=alKFwxZa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of inwardvessel@gmail.com designates 209.85.210.48 as permitted sender) smtp.mailfrom=inwardvessel@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746231570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7gn9VQnWdJz4VhO77FIVJ04j98J5OLA080DsKiIUYkU=; b=2fEe6bm5+XoQUg4NaON/+3VEBdyYos7qQHNLxJE7yxKQiTWtGhA4c7UYjpjqLRtczQOlwY mEJVp9RyjSMZ1OUCf0widi2Uas3saZstxq7yOl1qooqaWhxUcFKI2Fg8NH533QMBCCQLkn 9i8Gr4d37boKdF6uCMthPc/e4nIIs5I= Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-72c3b863b8eso1790679a34.2 for ; Fri, 02 May 2025 17:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746231569; x=1746836369; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7gn9VQnWdJz4VhO77FIVJ04j98J5OLA080DsKiIUYkU=; b=alKFwxZanVC3XDfNDRLBh4xOF2yJn/V6GOuAiLnBCbHWeHbW6WbkGut7/uoY4Jaoas Q3YFiXvEuSmlOdyjkon7HoF7ikHLUCgt7fKU5hyOoIXjKEMQd4girOM4Pov7jQvfjBor vqZRvA9acwJ4gWkHVRvFgYRMuL2Cj85D9gp/J7SSOzwSjaYfVmaeDfR1oCadX915kKFy dMjESTAL0aK++VnzsxHzc6pXKUqS8cVsOmxuit2N2GHiSpcI5u9jlkuQxdyjZPuzQT7b HRWNJXy3jXHquxzUxmoQo84P9/N99+1EcHvyT3R6AdmjU9L0IeLvmisC3MRMHFeCCLLY J3Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746231569; x=1746836369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7gn9VQnWdJz4VhO77FIVJ04j98J5OLA080DsKiIUYkU=; b=rrjlNByl9g0CpBoVosLLQ39Tt617IIMwrsIVpBDdvPSk1MdmVSEDyPnF0fhE65uuz4 9NTCsR/IjeWa8uo5+Id+uxfSy6hMDHhnxPxM4+TNaIwlIGRAztjmG4Qd3L29NzLoyQyi PzTXBFa6KmPKKvDJFQhMzsYJth0vrOqFQyZbkVIf0W4ReXs/bPKzxRmbsDCql4QMHhnm xq1N32N3whvw7a8WA2QbA86RNPRO/tj6xml73w5jyLlY9KCDtsqdV8gR6KpRLjVn29qs 39oWRuyRqCY9N0JzWW0CtpOf0QJGCcAjUu5HnHprlIrjozdJNT3DSam8d9l8uCMAg+MG AGuA== X-Gm-Message-State: AOJu0YzLVfNqJaPFXqLu9hz6jKiF4cJpAUi6ROMzmGHkX6mPKyrIBCo5 ccrkTJtausfZDLYul00LZlLKn/1Oc1P7vRNqsToAOxW/UH5Vq39s0rxJNg== X-Gm-Gg: ASbGncsJtHrKm8HapyoQUXsw2dqo3fIHr9tG7bnJDNCEWokt4w3mawoKGoPCbxaatgw Zs4ISZrj/tSZSc4Lu5ZTBJCYLVxdwUzCsquW/JLfOC1MUEHwldfgfaJyQ4GO4HhEJx/p5qdmILK JDoc1wb579bCY2SUOrxT2rxuBD1hJU7J9BttjJSz/M+O+1Jty0FQ8vKNr4czSKXkybmZA/GlrBN Y6GoTylXofV1LMc54u3EJtzHaRlszIy/S78PMX1X33SEihs9cdq8DR6GY3ifXPCwa01GQ4yrqBG G7dVOnmzb4AGFulw1ypddsEIKWFpeAvAty7x4ajualAShP0pN9WFhkvzrBoytpBmz+Ln X-Google-Smtp-Source: AGHT+IEEvYbH8qiotsxVHATJlrvlyC+RXsviT//hx6CWKSGq6ssGakCaG62xqFdMk1ZhEFxbRffzyQ== X-Received: by 2002:a17:902:ea0e:b0:22d:e5ab:5525 with SMTP id d9443c01a7336-22e10393f2fmr75982785ad.38.1746231159700; Fri, 02 May 2025 17:12:39 -0700 (PDT) Received: from jpkobryn-fedora-PF5CFKNC.thefacebook.com ([2620:10d:c090:500::5:6a01]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22e15228f9csm13718635ad.178.2025.05.02.17.12.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 May 2025 17:12:39 -0700 (PDT) From: JP Kobryn To: tj@kernel.org, shakeel.butt@linux.dev, yosryahmed@google.com, mkoutny@suse.com, hannes@cmpxchg.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, cgroups@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v5 3/5] cgroup: use subsystem-specific rstat locks to avoid contention Date: Fri, 2 May 2025 17:12:20 -0700 Message-ID: <20250503001222.146355-4-inwardvessel@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250503001222.146355-1-inwardvessel@gmail.com> References: <20250503001222.146355-1-inwardvessel@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: 3094F180007 X-Rspamd-Server: rspam04 X-Stat-Signature: jmf6rd694xssbptb4hrq85kstp9qfup1 X-HE-Tag: 1746231570-979696 X-HE-Meta: U2FsdGVkX18xLUkobtkoReHbfTA5dCyERyGs0vj3qiJE+oIyw9wbNp8r8zRXdlqAohm7uIdMeUdb7iN2xBJOP01vgTUqUfr0pAWhGKzlhgaGMOX4qILnY/ZY9o89GISsUCjCZkMn7/QXxnXIHsGU1qkZFIJaovL6/tqIgGAgi5YuwX1g7ok1HAGZHhujlmiIdGmiu3ywWES+JSMO9glZjvTXD6K5I37PKNEwu35hAdSn2YR2w3Wc7eI/QRslsZskXM6LK1KoaaijS5FWibT3qEzN1YOz3P1G+a/2DDGCYg1hiaqMs9tNBU0fUzIWB2eWbQ6VVlsDOsdhEXP4igpk4gL+jq65M2/dwwGfwSfRQ5tpDzbkcBHn7l+slCIW5inqU18280/0Z3EOH45iTknR6R1OVdp+K2WNW6SWXJlAD8NZLwdzVolMPgwQ14+q0PiqbAZ+yUhWsZ0lBEyZsniHcOZsFhBYoGbIR3oFO5Khlv7pCMec4Zhq6dJ2QQ8HJeFOFePYW69fyWHzv/gNOFgqY/rBQSXnJi/CG4AiAKRapx2VT9q/k6GpxRZOEC2Dzuf7WJVtmokYQXxcTF/fd/oYKBRTuUEHbM3eowhG+UqenBew/Q/NYCwcyI5xAYusXTcgyRPgqNO6vD3p8jM5nlIbe9cclL42sgunJc6ns1j2YjEulLE/QF4tkLfX6mVLUVC3zUI0d+0uodVAN5unYMvDzDxS9dJTYNhE3C3BXk1vlSko50p0Y2RYRmCrYczwHqo9/icrwqlpeKSqAvUyAVlqsdrkcY6PahRHArWrLELWCqtMIeyFdeGtWQY6exWs4vvB6ZwoolHoRruq9fQKBmGkvjz+jgKc/NboZuxOO1ZKRZtv/hGN6es8zpfYPH6gJmFQylI4krBzJsSe7JBYeJrchruSufD6q2THDWMDz+eRWj6ExCbYJ1G02FaT/kYLSCAeVeYiJX3jZF7IhanXXai NukZAkyk lAucYJN6GrmEzPZ5i1B6AwBPfpUkaILurc9HWPhipb8Z38o7O84J46eoms2UHh9vkllfwUkZAee1RHOexFqiHUtOmDFt8e/ZoQnCZiFYQ77Vxe/au85tyQbn5POXlZBR0YAi0RcHT62tg1VmMQRImqqXAz8uD+zujlwKr+AmTGF5mrYQR4t986uZtFesRECVhYFVD2+HIjAXOeosoGMx63SsfuJ1u1heOs+w3AsRLAshWx0wldh1IoYLv6ERmds1YzeZTQ9T9d3B28SD6GxiZ979QeC/G705VqqirFA9U+4tzvPf/9vYM7lhAMVbHogK93xgGz1K6YvlEeOxEHwuLMeTEVcLTwGp5hFUJSr7pP9b5vpqmjVx1HzNBx6tVe7aUrYLD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It is possible to eliminate contention between subsystems when updating/flushing stats by using subsystem-specific locks. Let the existing rstat locks be dedicated to the cgroup base stats and rename them to reflect that. Add similar locks to the cgroup_subsys struct for use with individual subsystems. Lock initialization is done in the new function ss_rstat_init(ss) which replaces cgroup_rstat_boot(void). If NULL is passed to this function, the global base stat locks will be initialized. Otherwise, the subsystem locks will be initialized. Change the existing lock helper functions to accept a reference to a css. Then within these functions, conditionally select the appropriate locks based on the subsystem affiliation of the given css. Add helper functions for this selection routine to avoid repeated code. Signed-off-by: JP Kobryn --- block/blk-cgroup.c | 2 +- include/linux/cgroup-defs.h | 10 +++- include/trace/events/cgroup.h | 12 +++- kernel/cgroup/cgroup-internal.h | 2 +- kernel/cgroup/cgroup.c | 3 +- kernel/cgroup/rstat.c | 103 ++++++++++++++++++++++---------- 6 files changed, 95 insertions(+), 37 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 0560ea402856..62d0bf1e1a04 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1074,7 +1074,7 @@ static void __blkcg_rstat_flush(struct blkcg *blkcg, int cpu) /* * For covering concurrent parent blkg update from blkg_release(). * - * When flushing from cgroup, cgroup_rstat_lock is always held, so + * When flushing from cgroup, the subsystem lock is always held, so * this lock won't cause contention most of time. */ raw_spin_lock_irqsave(&blkg_stat_lock, flags); diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 17ecaae9c5f8..5b8127d29dc5 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -222,7 +222,10 @@ struct cgroup_subsys_state { /* * A singly-linked list of css structures to be rstat flushed. * This is a scratch field to be used exclusively by - * css_rstat_flush() and protected by cgroup_rstat_lock. + * css_rstat_flush(). + * + * Protected by rstat_base_lock when css is cgroup::self. + * Protected by css->ss->rstat_ss_lock otherwise. */ struct cgroup_subsys_state *rstat_flush_next; }; @@ -362,7 +365,7 @@ struct css_rstat_cpu { * the css makes it unnecessary for each per-cpu struct to point back * to the associated css. * - * Protected by per-cpu cgroup_rstat_cpu_lock. + * Protected by per-cpu css->ss->rstat_ss_cpu_lock. */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ @@ -792,6 +795,9 @@ struct cgroup_subsys { * specifies the mask of subsystems that this one depends on. */ unsigned int depends_on; + + spinlock_t rstat_ss_lock; + raw_spinlock_t __percpu *rstat_ss_cpu_lock; }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/include/trace/events/cgroup.h b/include/trace/events/cgroup.h index af2755bda6eb..7d332387be6c 100644 --- a/include/trace/events/cgroup.h +++ b/include/trace/events/cgroup.h @@ -231,7 +231,11 @@ DECLARE_EVENT_CLASS(cgroup_rstat, __entry->cpu, __entry->contended) ); -/* Related to global: cgroup_rstat_lock */ +/* + * Related to locks: + * global rstat_base_lock for base stats + * cgroup_subsys::rstat_ss_lock for subsystem stats + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), @@ -253,7 +257,11 @@ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_unlock, TP_ARGS(cgrp, cpu, contended) ); -/* Related to per CPU: cgroup_rstat_cpu_lock */ +/* + * Related to per CPU locks: + * global rstat_base_cpu_lock for base stats + * cgroup_subsys::rstat_ss_cpu_lock for subsystem stats + */ DEFINE_EVENT(cgroup_rstat, cgroup_rstat_cpu_lock_contended, TP_PROTO(struct cgroup *cgrp, int cpu, bool contended), diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h index c161d34be634..b14e61c64a34 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -272,7 +272,7 @@ int cgroup_task_count(const struct cgroup *cgrp); */ int css_rstat_init(struct cgroup_subsys_state *css); void css_rstat_exit(struct cgroup_subsys_state *css); -void cgroup_rstat_boot(void); +int ss_rstat_init(struct cgroup_subsys *ss); void cgroup_base_stat_cputime_show(struct seq_file *seq); /* diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index f96535009d56..8071be38a409 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6088,6 +6088,7 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss, bool early) css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); BUG_ON(css->id < 0); + BUG_ON(ss_rstat_init(ss)); BUG_ON(css_rstat_init(css)); } @@ -6165,7 +6166,7 @@ int __init cgroup_init(void) BUG_ON(cgroup_init_cftypes(NULL, cgroup_psi_files)); BUG_ON(cgroup_init_cftypes(NULL, cgroup1_base_files)); - cgroup_rstat_boot(); + BUG_ON(ss_rstat_init(NULL)); get_user_ns(init_cgroup_ns.user_ns); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 29ba804f6ccc..e1e9dd7de705 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -9,8 +9,8 @@ #include -static DEFINE_SPINLOCK(cgroup_rstat_lock); -static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); +static DEFINE_SPINLOCK(rstat_base_lock); +static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -26,8 +26,24 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( return per_cpu_ptr(cgrp->rstat_base_cpu, cpu); } +static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) +{ + if (ss) + return &ss->rstat_ss_lock; + + return &rstat_base_lock; +} + +static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->rstat_ss_cpu_lock, cpu); + + return per_cpu_ptr(&rstat_base_cpu_lock, cpu); +} + /* - * Helper functions for rstat per CPU lock (cgroup_rstat_cpu_lock). + * Helper functions for rstat per CPU locks. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @fast_path determine the @@ -35,21 +51,23 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu( * operations without handling high-frequency fast-path "update" events. */ static __always_inline -unsigned long _css_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup_subsys_state *css, const bool fast_path) +unsigned long _css_rstat_cpu_lock(struct cgroup_subsys_state *css, int cpu, + const bool fast_path) { struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; unsigned long flags; bool contended; /* - * The _irqsave() is needed because cgroup_rstat_lock is - * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring - * this lock with the _irq() suffix only disables interrupts on - * a non-PREEMPT_RT kernel. The raw_spinlock_t below disables - * interrupts on both configurations. The _irqsave() ensures - * that interrupts are always disabled and later restored. + * The _irqsave() is needed because the locks used for flushing are + * spinlock_t which is a sleeping lock on PREEMPT_RT. Acquiring this lock + * with the _irq() suffix only disables interrupts on a non-PREEMPT_RT + * kernel. The raw_spinlock_t below disables interrupts on both + * configurations. The _irqsave() ensures that interrupts are always + * disabled and later restored. */ + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); contended = !raw_spin_trylock_irqsave(cpu_lock, flags); if (contended) { if (fast_path) @@ -69,17 +87,18 @@ unsigned long _css_rstat_cpu_lock(raw_spinlock_t *cpu_lock, int cpu, } static __always_inline -void _css_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, - struct cgroup_subsys_state *css, unsigned long flags, - const bool fast_path) +void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu, + unsigned long flags, const bool fast_path) { struct cgroup *cgrp = css->cgroup; + raw_spinlock_t *cpu_lock; if (fast_path) trace_cgroup_rstat_cpu_unlock_fastpath(cgrp, cpu, false); else trace_cgroup_rstat_cpu_unlock(cgrp, cpu, false); + cpu_lock = ss_rstat_cpu_lock(css->ss, cpu); raw_spin_unlock_irqrestore(cpu_lock, flags); } @@ -94,7 +113,6 @@ void _css_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu, */ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; /* @@ -115,7 +133,7 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) if (data_race(css_rstat_cpu(css, cpu)->updated_next)) return; - flags = _css_rstat_cpu_lock(cpu_lock, cpu, css, true); + flags = _css_rstat_cpu_lock(css, cpu, true); /* put @css and all ancestors on the corresponding updated lists */ while (true) { @@ -143,7 +161,7 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) css = parent; } - _css_rstat_cpu_unlock(cpu_lock, cpu, css, flags, true); + _css_rstat_cpu_unlock(css, cpu, flags, true); } /** @@ -171,11 +189,11 @@ static struct cgroup_subsys_state *css_rstat_push_children( child->rstat_flush_next = NULL; /* - * The cgroup_rstat_lock must be held for the whole duration from + * The subsystem rstat lock must be held for the whole duration from * here as the rstat_flush_next list is being constructed to when * it is consumed later in css_rstat_flush(). */ - lockdep_assert_held(&cgroup_rstat_lock); + lockdep_assert_held(ss_rstat_lock(head->ss)); /* * Notation: -> updated_next pointer @@ -245,12 +263,11 @@ static struct cgroup_subsys_state *css_rstat_push_children( static struct cgroup_subsys_state *css_rstat_updated_list( struct cgroup_subsys_state *root, int cpu) { - raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); struct css_rstat_cpu *rstatc = css_rstat_cpu(root, cpu); struct cgroup_subsys_state *head = NULL, *parent, *child; unsigned long flags; - flags = _css_rstat_cpu_lock(cpu_lock, cpu, root, false); + flags = _css_rstat_cpu_lock(root, cpu, false); /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) @@ -287,7 +304,7 @@ static struct cgroup_subsys_state *css_rstat_updated_list( if (child != root) head = css_rstat_push_children(head, child, cpu); unlock_ret: - _css_rstat_cpu_unlock(cpu_lock, cpu, root, flags, false); + _css_rstat_cpu_unlock(root, cpu, flags, false); return head; } @@ -314,7 +331,7 @@ __weak noinline void bpf_rstat_flush(struct cgroup *cgrp, __bpf_hook_end(); /* - * Helper functions for locking cgroup_rstat_lock. + * Helper functions for locking. * * This makes it easier to diagnose locking issues and contention in * production environments. The parameter @cpu_in_loop indicate lock @@ -324,27 +341,31 @@ __bpf_hook_end(); */ static inline void __css_rstat_lock(struct cgroup_subsys_state *css, int cpu_in_loop) - __acquires(&cgroup_rstat_lock) + __acquires(ss_rstat_lock(css->ss)) { struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; bool contended; - contended = !spin_trylock_irq(&cgroup_rstat_lock); + lock = ss_rstat_lock(css->ss); + contended = !spin_trylock_irq(lock); if (contended) { trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, contended); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock_irq(lock); } trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended); } static inline void __css_rstat_unlock(struct cgroup_subsys_state *css, int cpu_in_loop) - __releases(&cgroup_rstat_lock) + __releases(ss_rstat_lock(css->ss)) { struct cgroup *cgrp = css->cgroup; + spinlock_t *lock; + lock = ss_rstat_lock(css->ss); trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock_irq(lock); } /** @@ -468,12 +489,34 @@ void css_rstat_exit(struct cgroup_subsys_state *css) css->rstat_cpu = NULL; } -void __init cgroup_rstat_boot(void) +/** + * ss_rstat_init - subsystem-specific rstat initialization + * @ss: target subsystem + * + * If @ss is NULL, the static locks associated with the base stats + * are initialized. If @ss is non-NULL, the subsystem-specific locks + * are initialized. + */ +int __init ss_rstat_init(struct cgroup_subsys *ss) { int cpu; + if (!ss) { + for_each_possible_cpu(cpu) + raw_spin_lock_init(per_cpu_ptr(&rstat_base_cpu_lock, cpu)); + + return 0; + } + + spin_lock_init(&ss->rstat_ss_lock); + ss->rstat_ss_cpu_lock = alloc_percpu(raw_spinlock_t); + if (!ss->rstat_ss_cpu_lock) + return -ENOMEM; + for_each_possible_cpu(cpu) - raw_spin_lock_init(per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu)); + raw_spin_lock_init(per_cpu_ptr(ss->rstat_ss_cpu_lock, cpu)); + + return 0; } /* -- 2.47.1