From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 378A4C5B552 for ; Mon, 9 Jun 2025 22:56:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C19D16B0095; Mon, 9 Jun 2025 18:56:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCAC76B0096; Mon, 9 Jun 2025 18:56:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A931D6B0098; Mon, 9 Jun 2025 18:56:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 89B436B0095 for ; Mon, 9 Jun 2025 18:56:31 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4ED5E160509 for ; Mon, 9 Jun 2025 22:56:31 +0000 (UTC) X-FDA: 83537372982.04.F02AE05 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) by imf07.hostedemail.com (Postfix) with ESMTP id 8DFB740010 for ; Mon, 9 Jun 2025 22:56:29 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wpWU8hZ3; spf=pass (imf07.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749509789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8ipQfALEF/dBZznaVw0Klj9I7jvOAPbti7+InyzJ+44=; b=MoJkdaIaEw7Ueg5mmwMkiH/p5xHImDWjZ6ZFx/Sg92MTMPwy67iW19gO/8pnMQSMkUZGw8 D5mas+qZ9n+J7HkO5BgyE4wYw2ztqhpJQZ6y9KwWinGVEtxzoNanxfauw+Cu8D7/GXSkpa gyHa+xMktNrFp4aI/2ihhm0ZR5lkSfk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749509789; a=rsa-sha256; cv=none; b=IoS+LPuDAxcZ3U93GtLId9OUbM3u0C2NEWodighRoqMlQgP9ymP2tLDAk0A3yX/Ui/cEAo sHXSXfR3nYyWLzg7CNS86YdbSW2JtQ+hWV46kNTRXEtiNF6LGobnTVB0GdUON+jBPlMt80 cOXPOKtobsiOj0US18pUafmtjFUEiks= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wpWU8hZ3; spf=pass (imf07.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1749509788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8ipQfALEF/dBZznaVw0Klj9I7jvOAPbti7+InyzJ+44=; b=wpWU8hZ36G9p/4DRgAXIGESz1EZxzTdj//O6Df64GuE54onSIdS0slIj7gKAPNCTbjrS5E 2KlHWmrflLMQaEGV9qhrhaOUUtKUHOWD1/b0kpDDB9P0Am8exWCDg1FM8+obWe67EKAW5A 2sbpiDjBNYfA05yQIppOJCGTFdOS1jI= From: Shakeel Butt To: Tejun Heo , Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 2/3] cgroup: make css_rstat_updated nmi safe Date: Mon, 9 Jun 2025 15:56:10 -0700 Message-ID: <20250609225611.3967338-3-shakeel.butt@linux.dev> In-Reply-To: <20250609225611.3967338-1-shakeel.butt@linux.dev> References: <20250609225611.3967338-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8DFB740010 X-Stat-Signature: scj39yht49fxody74cbtttpzqaar5576 X-Rspam-User: X-HE-Tag: 1749509789-294733 X-HE-Meta: U2FsdGVkX1/tprM+YhuUUcmIf/oGb0CM/FuHoMBsDKl/NTSHzjuo0LHXBk2G9XKfbO0BxFtKvckodw/SNfqyVAWcIC2Z6l41jDgRbCfo30yt5/M/seNgVTTpIAUCyaIqtwHc2gNGdVwrDXmEgsG6FV1f2on73bkWDNs89D6iDL16GVOcujI2fa5PGD+FtytEQmOMNzdB3vibxxip7vYShJl1opoLow5I784atG2pePWIlKoRbKzcMromeNuPtggjh3qZi6UE3iKZvgQHj+VYJKuuYh6uPmOAdUEGA8Yh1DKEX8qR7bO4Eyz3g7UitzHKL/+kYtFmmfeNAY1MKaXS8Bj+V7GuuCKpJq0phETaml/sVMfFikZhsmG411gwn4TeDZnBceegjggsp17bRwxNqSyiiOaOtUC7JGV3eYD7iJizzNNGwVwZEcRtSQxTKVoBkp31V6GIxViLjEP0PNQEbCQ8wG8szAoDT102u2akmSjHwE3NJVVWxF4ZG74CuLCoXV0Y6MBri8amhT59RfecfmN7l9s/dr8imB6hzw/P3hShYUI+758WuYKi62wx5m7huJ4jO+84b2ed6kYhiymXOZ4o3eYTZv/7+rqPswmm8LoYp1dg/NVlrtjV427Y5FDexqOfW+uEzs+kPvHAxKDvWKQFuBTZ/Y37286BCUFQr6jwvUvpy2dmyIcAfpDWupqzHCoj2zXKq2JkX3RGhlg9twZXxz7mXZ1lF7KAxkZ+7vR8vLzAgu9cOrTGWs/CStgj29cj9JEC4QWglIRGxFXwjnew8gYhW3YwE3XgxUmKerhlB+bJpC0F4RBJSaKQEJujeFWphI3I3fSDxVkZvvsambl5A+82hsERQcnWxxNhVj7zQJb0wVKP9+Le6BeQMw9RQomDK3Q9Rhy6OHGhuH9uHz/RfBQG+l+3vl0pwVhlRyjWLtEfTXmTkvRgXYwy6dF5EE98tBnVpha1QsQBz0/ e0OVMFgo oBOxhY4L1fjemPH43cRAykk8Hredlu6Z4qbw7o/UCgrpeNJMlWramgEboXhx7vhHK+pn0RcmrEuTQPQQYR3IQ/FTP+kkyKpgIydvZfs1M0iCI4ufhbvT3TYHEv3I//L6DLxQ21WzG5O2vwJbgk44DZi5fXdeFI+kndSMiSlZCw1qc7zEWbXtvmRVsE4/pQ74jeF+IloIgZbJlYPI6ssCji8qOKfnM5oEtN/nSOKQpJyrzyAGmnDkSkwmiOpXO4mtiSdlC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To make css_rstat_updated() able to safely run in nmi context, let's move the rstat update tree creation at the flush side and use per-cpu lockless lists in struct cgroup_subsys to track the css whose stats are updated on that cpu. The struct cgroup_subsys_state now has per-cpu lnode which needs to be inserted into the corresponding per-cpu lhead of struct cgroup_subsys. Since we want the insertion to be nmi safe, there can be multiple inserters on the same cpu for the same lnode. The current llist does not provide function to protect against the scenario where multiple inserters can use the same lnode. So, using llist_node() out of the box is not safe for this scenario. However we can protect against multiple inserters using the same lnode by using the fact llist node points to itself when not on the llist and atomically reset it and select the winner as the single inserter. Signed-off-by: Shakeel Butt --- kernel/cgroup/rstat.c | 57 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 12 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index a5608ae2be27..4fabd7973067 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -138,13 +138,15 @@ void _css_rstat_cpu_unlock(struct cgroup_subsys_state *css, int cpu, * @css: target cgroup subsystem state * @cpu: cpu on which rstat_cpu was updated * - * @css's rstat_cpu on @cpu was updated. Put it on the parent's matching - * rstat_cpu->updated_children list. See the comment on top of - * css_rstat_cpu definition for details. + * Atomically inserts the css in the ss's llist for the given cpu. This is nmi + * safe. The ss's llist will be processed at the flush time to create the update + * tree. */ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) { - unsigned long flags; + struct llist_head *lhead = ss_lhead_cpu(css->ss, cpu); + struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); + struct llist_node *self; /* * Since bpf programs can call this function, prevent access to @@ -153,19 +155,37 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) if (!css_uses_rstat(css)) return; + lockdep_assert_preemption_disabled(); + + /* + * For arch that does not support nmi safe cmpxchg, we ignore the + * requests from nmi context for rstat update llist additions. + */ + if (!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) && in_nmi()) + return; + + /* If already on list return. */ + if (llist_on_list(&rstatc->lnode)) + return; + /* - * Speculative already-on-list test. This may race leading to - * temporary inaccuracies, which is fine. + * Make sure only one insert request can proceed on this cpu for this + * specific lnode and thus this needs to be safe against irqs and nmis. * - * Because @parent's updated_children is terminated with @parent - * instead of NULL, we can tell whether @css is on the list by - * testing the next pointer for NULL. + * Please note that llist_add() does not protect against multiple + * inserters for the same lnode. We use the fact that lnode points to + * itself when not on a list and then atomically set it to NULL to + * select the single inserter. */ - if (data_race(css_rstat_cpu(css, cpu)->updated_next)) + self = &rstatc->lnode; + if (!try_cmpxchg(&(rstatc->lnode.next), &self, NULL)) return; - flags = _css_rstat_cpu_lock(css, cpu, true); + llist_add(&rstatc->lnode, lhead); +} +static void __css_process_update_tree(struct cgroup_subsys_state *css, int cpu) +{ /* put @css and all ancestors on the corresponding updated lists */ while (true) { struct css_rstat_cpu *rstatc = css_rstat_cpu(css, cpu); @@ -191,8 +211,19 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu) css = parent; } +} + +static void css_process_update_tree(struct cgroup_subsys *ss, int cpu) +{ + struct llist_head *lhead = ss_lhead_cpu(ss, cpu); + struct llist_node *lnode; + + while ((lnode = llist_del_first_init(lhead))) { + struct css_rstat_cpu *rstatc; - _css_rstat_cpu_unlock(css, cpu, flags, true); + rstatc = container_of(lnode, struct css_rstat_cpu, lnode); + __css_process_update_tree(rstatc->owner, cpu); + } } /** @@ -300,6 +331,8 @@ static struct cgroup_subsys_state *css_rstat_updated_list( flags = _css_rstat_cpu_lock(root, cpu, false); + css_process_update_tree(root->ss, cpu); + /* Return NULL if this subtree is not on-list */ if (!rstatc->updated_next) goto unlock_ret; -- 2.47.1