From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9C87C27C4F for ; Fri, 21 Jun 2024 16:08:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AB236B0175; Fri, 21 Jun 2024 12:08:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 157F86B017B; Fri, 21 Jun 2024 12:08:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F12476B0175; Fri, 21 Jun 2024 12:08:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D38976B0171 for ; Fri, 21 Jun 2024 12:08:12 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5784D1A0E54 for ; Fri, 21 Jun 2024 16:08:12 +0000 (UTC) X-FDA: 82255377624.23.EE67835 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 49C424001D for ; Fri, 21 Jun 2024 16:08:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ui7cUZ3p; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of longman@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=longman@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718986082; a=rsa-sha256; cv=none; b=te6sp2zwKBbx0ZrHCGL2QLcvb/1IXbSHXSVaN3EbB4a0dX/WYSRrsXrZTlybuLtxiG35l2 ddGLTGuIS5Jh5HDT4D9G1u4ai6qhBAIIXKolOIs7RSt0IbL0Wl4nd7pVilXe8Moi08euNR 3nxEt+W+9+aPycsZXfWD7FOnhTM2tcI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ui7cUZ3p; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of longman@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=longman@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718986082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7KGBXm2hLDvnaGTtzdG+6MEjh0yCod7+iYftyydCF/o=; b=qVsXOAPWXmcwx8FJr6Re4hYBJV+VxXErqtqzJtgXrhwONIpa70Q6OYHErjKV3b7MClENhC a9AosmX2OndlAaypaIlOI2tBvrkZViueU+wNDgKRkc6WD8IrFPQFifHz4WgpUr6//W1mOt dpp4pV9n699EdxNgLWNVVDD9fIVQTKQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718986089; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7KGBXm2hLDvnaGTtzdG+6MEjh0yCod7+iYftyydCF/o=; b=Ui7cUZ3p9Ie9tC52buhMbIpP5jiK1+bbFwRbfIBQvYq4nxO/goD3pUBU94JTB1/SrbEP3B vABdUryA8DJd14h4WAbth4CMEeo0+JPRo5JvQzdNvk5drli9pZqRol6jDofnrZAlFrVlWt 3arOW/CxW+ceXXW8c8U3ydWdnucbdsQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-608-SDqec6jKNiCKuYjxs6g9rA-1; Fri, 21 Jun 2024 12:08:06 -0400 X-MC-Unique: SDqec6jKNiCKuYjxs6g9rA-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1A581195608C; Fri, 21 Jun 2024 16:08:04 +0000 (UTC) Received: from [10.22.17.209] (unknown [10.22.17.209]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B0DFD3000218; Fri, 21 Jun 2024 16:08:01 +0000 (UTC) Message-ID: <2de2850c-c844-4a75-884a-18d552fcb846@redhat.com> Date: Fri, 21 Jun 2024 12:08:00 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] cgroup/rstat: Avoid thundering herd problem by kswapd across NUMA nodes To: Jesper Dangaard Brouer , tj@kernel.org, cgroups@vger.kernel.org, yosryahmed@google.com, shakeel.butt@linux.dev Cc: hannes@cmpxchg.org, lizefan.x@bytedance.com, kernel-team@cloudflare.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <171898037079.1222367.13467317484793748519.stgit@firesoul> Content-Language: en-US From: Waiman Long In-Reply-To: <171898037079.1222367.13467317484793748519.stgit@firesoul> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Rspamd-Queue-Id: 49C424001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4sh631kc89ubwu8bdwzzb6g7434mcs49 X-HE-Tag: 1718986090-581481 X-HE-Meta: U2FsdGVkX19JjQmx3wHYoWjATdqB2v5DbJNL4k9xvGnraWHR1hLF0taycJoirVZOwHecY9UdxPXwRBtV1KWzqeQQ1fruKW7UczwSNNgmkwhLqJFF1ClAf1PczG5yS/wGamjGHwR7R3uSGRmBqo9xUW22okuHbTh+i54Ysv5qEaeWWTxIH/YlxKnqX+tqjiS8V6Cmd7a/7rHMLk0jpghUVzLMW6Oc2yo3S2QQQ4sE5+IC4LzVihLxjY8L9iLhWX01+f0Udw2XwItINve8UVH00ozl2S86Iv0HFDh/5romQOmfYD2ple/oOUB/KMwMvZimDkDFN6Lqt9I3gV4/LQ/kOzIEfO0Mjqs/1CJbT/YOuBQ1A5brVce+Y6H4IyXkw7F2Y5kFSXdZw61kcb1qfn6nIO7NI7YZmjjvRixMPKb9nu+oJUoTYvbcLjeqcR+NvtceihEQ8J/EH5lBYwBEvU1jrW2Pz5MMciO0rsIKmq42pZtSqSZLsJYrm/b9K4B69trWsPy1gfRzmyezRVtIVm5bbyI2K/g1Y2wRCy9y/EfZdjtc6S8JKNO7w1mfx9nCQu42NjAoitW1Rzr7dK0uJRgaVch68Y1xmSTe9caaluo8wf6V9lK9kTlR1pmMbFaw2QanVSn1Y/v6XxIgenQ7vx3ZZcc+oHEfKkeFDQ+vQr1Cltq56JBXKin5mD4bd5JQuqBtaLsuE2XV2RomL1tuG7/4nB76g6XoiMS6iWJtsp7lYl6YfNMFnZ3Ell3bBXZYxAl1S2UnEtUdrJYKwyp0HyBtEtFSgt/41K8Sclr/pZ8r3ZNITyC8N9UuUfHd6pMdt9xiedxezRvtj3D9wryDhx4Ki+tOb50/XCeLDJvwRdgN4I8csx1UztsSMCPMzzmeS809Z1+ofNZ2+CEN5k1gV4KMfVx32mJCoakiIzJinI+f1VMI1mElSn9+VR4nwo6UvkRcC2BhZ5HQDdFFWvE7LTq 7epYW3z9 8xVJxrra8OYPUWsQ7nnqb5Dq/k41QMQbxSu8VYpz07Pyrhf7eLppUNw+qqVaL66FLlr1sK7sT2IW/7186zxuHxEG9qJ0Nx1DRfArlUPscweQlwet4D8YR0tyURJyRCcW8VkCqTgI4JvQfxNmrDGbyJHJVFvhHqs6xJLgnXPWczhTScuGM1ejqqYhJaJYTJwjolvo57drYO/yrgNjM10h0p0M4QBUy/Sqys6FcikWIs4WNNpLV/ApK4LpC+xtoedVzK7b/0Bi/NfLc8y0Z/Qg+bIniQowwVCazBucM1lGUUkn3unz2+LcqocGXM679IUIRctdftBFYJoLzyTwrfSqEOMKaiVZghULIxD4R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/21/24 10:32, Jesper Dangaard Brouer wrote: > Avoid lock contention on the global cgroup rstat lock caused by kswapd > starting on all NUMA nodes simultaneously. At Cloudflare, we observed > massive issues due to kswapd and the specific mem_cgroup_flush_stats() > call inlined in shrink_node, which takes the rstat lock. > > On our 12 NUMA node machines, each with a kswapd kthread per NUMA node, > we noted severe lock contention on the rstat lock. This contention > causes 12 CPUs to waste cycles spinning every time kswapd runs. > Fleet-wide stats (/proc/N/schedstat) for kthreads revealed that we are > burning an average of 20,000 CPU cores fleet-wide on kswapd, primarily > due to spinning on the rstat lock. > > To help reviewer follow code: When the Per-CPU-Pages (PCP) freelist is > empty, __alloc_pages_slowpath calls wake_all_kswapds(), causing all > kswapdN threads to wake up simultaneously. The kswapd thread invokes > shrink_node (via balance_pgdat) triggering the cgroup rstat flush > operation as part of its work. This results in kernel self-induced rstat > lock contention by waking up all kswapd threads simultaneously. > Leveraging this detail: balance_pgdat() have NULL value in > target_mem_cgroup, this cause mem_cgroup_flush_stats() to do flush with > root_mem_cgroup. > > To resolve the kswapd issue, we generalized the "stats_flush_ongoing" > concept to apply to all users of cgroup rstat, not just memcg. This > concept was originally reverted in commit 7d7ef0a4686a ("mm: memcg: > restore subtree stats flushing"). If there is an ongoing rstat flush, > limited to the root cgroup, the flush is skipped. This is effective as > kswapd operates on the root tree, sufficiently mitigating the thundering > herd problem. > > This lowers contention on the global rstat lock, although limited to the > root cgroup. Flushing cgroup subtree's can still lead to lock contention. > > Fixes: 7d7ef0a4686a ("mm: memcg: restore subtree stats flushing"). > Signed-off-by: Jesper Dangaard Brouer > --- > include/linux/cgroup.h | 5 +++++ > kernel/cgroup/rstat.c | 28 ++++++++++++++++++++++++++++ > 2 files changed, 33 insertions(+) > > diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h > index 2150ca60394b..ad41cca5c3b6 100644 > --- a/include/linux/cgroup.h > +++ b/include/linux/cgroup.h > @@ -499,6 +499,11 @@ static inline struct cgroup *cgroup_parent(struct cgroup *cgrp) > return NULL; > } > > +static inline bool cgroup_is_root(struct cgroup *cgrp) > +{ > + return cgroup_parent(cgrp) == NULL; > +} > + > /** > * cgroup_is_descendant - test ancestry > * @cgrp: the cgroup to be tested > diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c > index fb8b49437573..5aba95e92d31 100644 > --- a/kernel/cgroup/rstat.c > +++ b/kernel/cgroup/rstat.c > @@ -11,6 +11,7 @@ > > static DEFINE_SPINLOCK(cgroup_rstat_lock); > static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); > +static atomic_t root_rstat_flush_ongoing = ATOMIC_INIT(0); > > static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); > > @@ -350,8 +351,25 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) > { > might_sleep(); > > + /* > + * This avoids thundering herd problem on global rstat lock. When an > + * ongoing flush of the entire tree is in progress, then skip flush. > + */ > + if (atomic_read(&root_rstat_flush_ongoing)) > + return; > + > + /* Grab right to be ongoing flusher, return if loosing race */ > + if (cgroup_is_root(cgrp) && > + atomic_xchg(&root_rstat_flush_ongoing, 1)) > + return; > + > __cgroup_rstat_lock(cgrp, -1); > + > cgroup_rstat_flush_locked(cgrp); > + > + if (cgroup_is_root(cgrp)) > + atomic_set(&root_rstat_flush_ongoing, 0); > + > __cgroup_rstat_unlock(cgrp, -1); > } > > @@ -362,13 +380,20 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) > * Flush stats in @cgrp's subtree and prevent further flushes. Must be > * paired with cgroup_rstat_flush_release(). > * > + * Current invariant, not called with root cgrp. > + * > * This function may block. > */ > void cgroup_rstat_flush_hold(struct cgroup *cgrp) > __acquires(&cgroup_rstat_lock) > { > might_sleep(); > + > __cgroup_rstat_lock(cgrp, -1); > + > + if (atomic_read(&root_rstat_flush_ongoing)) > + return; > + > cgroup_rstat_flush_locked(cgrp); > } > > @@ -379,6 +404,9 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) > void cgroup_rstat_flush_release(struct cgroup *cgrp) > __releases(&cgroup_rstat_lock) > { > + if (cgroup_is_root(cgrp)) > + atomic_set(&root_rstat_flush_ongoing, 0); > + > __cgroup_rstat_unlock(cgrp, -1); > } Since both cgroup_rstat_flush_hold() and cgroup_rstat_flush_release() are not called with root cgroup, the cgroup_rstat_flush_hold() hunk is essentially dead code. Cheers, Longman