From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F39CEC61CE8 for ; Mon, 9 Jun 2025 22:56:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BE4A6B008C; Mon, 9 Jun 2025 18:56:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 895F46B0092; Mon, 9 Jun 2025 18:56:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ABCA6B0093; Mon, 9 Jun 2025 18:56:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 59BB46B008C for ; Mon, 9 Jun 2025 18:56:28 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1712CBA2C9 for ; Mon, 9 Jun 2025 22:56:28 +0000 (UTC) X-FDA: 83537372856.26.13C23A4 Received: from out-172.mta0.migadu.com (out-172.mta0.migadu.com [91.218.175.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 6754B1C0009 for ; Mon, 9 Jun 2025 22:56:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ukP1Hnw3; spf=pass (imf21.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749509786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vjwAclPLPNB0cOL9u2mwbYWPRL/XnTNpPecdd20u+W8=; b=i6T6hrm5Ujwkg1sXBwFS/OD1cvrSqStK9nJbtHEjzr4CJwQgg5yESsG3ahwt0BvTM2kS+G OKqh9292B1UZyiztQacXTNbXI4OxyxAIYqUj6UWAWl+TWoY1GeBPX0OqZI7mZowJCadZPH 6usFZVN4L7svtBwCShu4yLp0k+eFP/M= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ukP1Hnw3; spf=pass (imf21.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.172 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749509786; a=rsa-sha256; cv=none; b=TWz2js8Mj6lFUSgH/XyNOsvnW6pJQ7JaIx80mdM8+31h7pgbx713IDX+/d64e7tlDShtOK h1XxkWh8pX7hrXmLsVqP3gA3TBmDnaOXDHe2JAEyunTjqWLM90FIma+bricCw1vkc/RYC5 lBC8qqOa+xHoXCvAY8n5C19+l5pRQgM= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749509784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vjwAclPLPNB0cOL9u2mwbYWPRL/XnTNpPecdd20u+W8=; b=ukP1Hnw3lE2tb6d6IJxojqkr5GibDR6b2lbfMG8o+FPQKSs5saTqh6Blkbzlu4WZoQv1gm oB1mL/px5lNcl0rQQ3j/cQMrXM4WaTlFELDViUaOtAvwDugzHlhXr4OnNhIrDaQvoBN16c V4AmbNMSm2p9Tfqw+60SlWaHWLK0+qg= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 1/3] cgroup: support to enable nmi-safe css_rstat_updated Date: Mon, 9 Jun 2025 15:56:09 -0700 Message-ID: <20250609225611.3967338-2-shakeel.butt@linux.dev> In-Reply-To: <20250609225611.3967338-1-shakeel.butt@linux.dev> References: <20250609225611.3967338-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6754B1C0009 X-Stat-Signature: f873k3ppd186yfc5u3nkufec39sabaxo X-Rspam-User: X-HE-Tag: 1749509786-70530 X-HE-Meta: U2FsdGVkX18vQJedjB7rnzkMfLZk+qhRCEFT0dHrIPtm5WkVCW3mlLbn+gybP44uwqwRLWiK6wNj2wxry8Fe6JpoazVJWqLVlUuIcfYFkp+ayKj8BoMWn8bseu9GJgvU4G+qt/nTN0x8DchrqPYZT++xW23lazr1MzhuGwXV3m4y7f1fCCQnoUe4ybmqWwn9f09tmbul0+fgjbqFrkqWQIwNnYm8J7jNygm8lX2TweXcaIdxWeXZFcWNOdf81NtipgvbXFFROcBJRDJk567mAvesOU3OvdrXgP1LBHkohcG1x5+tGBBmsocGVVeVM2ihdeBCkwR/XOsgOt0Pm62YX5G02d5GNON60K6q2S+srx/JUuoAq5hEB4QXQwhIBqx9Sid9k/mZdQ+ElSCswUNsJWjr52bMMWQU4/rMw7YZX+6bIhvW/iRHBC/YWWRoycKY7F7Hc0tg57aj32iSzdwcEMk087NlP4UY58mOzJbPwdWv/+L8nN5bhlOZkXQItjqPd3IIVBHr4xxeVQDBWxy/ILkQfxHKVCqvIswnMx/h/bhtuk7MDPIm2kyvZxWLNT7n6igJkI3uCL+Gw1cOidL9o3Njs9sL0ddsb83QuS7XtJ+8Okg9dV9Q60j0x60lPqsW2DYi/kaiaOs3HBNAoBfhb3HfXKwHH9d9jr/EQmeJM5JgPmGDg2a7++OEpD1wcM3+Uo41BE2EZVnYuXkvoqKl1DDr7WxNiWJniYzYBBJ4oLx0ruV+m22mm2ZihLByPDlScS3aF+wgm1qAMYtolTJlHkA3c/2JPJIntbG0CFsdSP5DTdVdZG+h4BfT9IiM3H7ibbolzV0BjCiq16Suky57nU2HXuvZVwBM9pgluXrCJU767Lia+eJqDlV9Jrm0NILsJW4FYS2j1F2u+zsDTle2yNv6T402/wMEfZ3TNqJEyT6Ucb/y+5KhnEPJYdeTPWp/3ckVEnOal1O4U/MjrAQ pZzNGHmo DufoxI90buc1+I38SJOyjL+l4kRjTXhc3UxyBGpnHEtjEkcW0IYto+bdOKNMEQypIr6AolXB47Cf1PkhDMZ4uVSlmRJH6/Ms1PkCXaFVMl6M4YdYKw+FL+sn4kQgQbLdasWATFfuMArac0gWFeiBnTdTZp+jpE8pBPP9mggiNT5Wo3tp+QL211lXj/P7ZqzP3segzPBulSgxQUC8Ue7y39/aYwCUd7ejwzHvr6Avc0kcEhwgaZNok/brXZSBl8R2TCerW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add necessary infrastructure to enable the nmi-safe execution of css_rstat_updated(). Currently css_rstat_updated() takes a per-cpu per-css raw spinlock to add the given css in the per-cpu per-css update tree. However the kernel can not spin in nmi context, so we need to remove the spinning on the raw spinlock in css_rstat_updated(). To support lockless css_rstat_updated(), let's add necessary data structures in the css and ss structures. Signed-off-by: Shakeel Butt --- include/linux/cgroup-defs.h | 4 ++++ kernel/cgroup/rstat.c | 23 +++++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index e61687d5e496..45860fe5dd0c 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -384,6 +384,9 @@ struct css_rstat_cpu { */ struct cgroup_subsys_state *updated_children; struct cgroup_subsys_state *updated_next; /* NULL if not on the list */ + + struct llist_node lnode; /* lockless list for update */ + struct cgroup_subsys_state *owner; /* back pointer */ }; /* @@ -822,6 +825,7 @@ struct cgroup_subsys { spinlock_t rstat_ss_lock; raw_spinlock_t __percpu *rstat_ss_cpu_lock; + struct llist_head __percpu *lhead; /* lockless update list head */ }; extern struct percpu_rw_semaphore cgroup_threadgroup_rwsem; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index cbeaa499a96a..a5608ae2be27 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -11,6 +11,7 @@ static DEFINE_SPINLOCK(rstat_base_lock); static DEFINE_PER_CPU(raw_spinlock_t, rstat_base_cpu_lock); +static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list); static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); @@ -45,6 +46,13 @@ static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss) return &rstat_base_lock; } +static inline struct llist_head *ss_lhead_cpu(struct cgroup_subsys *ss, int cpu) +{ + if (ss) + return per_cpu_ptr(ss->lhead, cpu); + return per_cpu_ptr(&rstat_backlog_list, cpu); +} + static raw_spinlock_t *ss_rstat_cpu_lock(struct cgroup_subsys *ss, int cpu) { if (ss) { @@ -468,7 +476,8 @@ int css_rstat_init(struct cgroup_subsys_state *css) for_each_possible_cpu(cpu) { struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); - rstatc->updated_children = css; + rstatc->owner = rstatc->updated_children = css; + init_llist_node(&rstatc->lnode); if (is_self) { struct cgroup_rstat_base_cpu *rstatbc; @@ -532,9 +541,19 @@ int __init ss_rstat_init(struct cgroup_subsys *ss) return -ENOMEM; } + if (ss) { + ss->lhead = alloc_percpu(struct llist_head); + if (!ss->lhead) { + free_percpu(ss->rstat_ss_cpu_lock); + return -ENOMEM; + } + } + spin_lock_init(ss_rstat_lock(ss)); - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { raw_spin_lock_init(ss_rstat_cpu_lock(ss, cpu)); + init_llist_head(ss_lhead_cpu(ss, cpu)); + } return 0; } -- 2.47.1