From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FD8FC27C4F for ; Fri, 21 Jun 2024 15:13:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD5838D017D; Fri, 21 Jun 2024 11:13:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B85B98D017A; Fri, 21 Jun 2024 11:13:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A263D8D017D; Fri, 21 Jun 2024 11:13:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 848EB8D017A for ; Fri, 21 Jun 2024 11:13:50 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2F415160DFE for ; Fri, 21 Jun 2024 15:13:50 +0000 (UTC) X-FDA: 82255240620.28.9C80B23 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 43ED2140015 for ; Fri, 21 Jun 2024 15:13:48 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=StlKORTv; spf=pass (imf26.hostedemail.com: domain of longman@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718982818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EP/kWpZ/AZr0wzKgF+O2dWS3CSDCoLm/7KmRqzqigc4=; b=an+hnSBrpEPWhCWgVLunzECzNPeNZnkX55GQUGvJCUGbeZ2oZZyKLIbMGnbEclPaNI4Rfk t0AgrVgHOos5Wq45SBqTEqsowt9px/C+N5lhRF9wah+KsBYupHQ5dFoKGmsRL2ROMZhryn ILNciOwsF0VmjYYYJ8y+m+YeFMSghE8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718982818; a=rsa-sha256; cv=none; b=I0cbpkF9GvbdNWix+SWCypu/fVbrkG78CQk+jNoD7he/SnpGQOX/RrAwUNGH9oYen0DxJa X8rliiZzB58BNwu7Bf4wnXS6GqNWIKdYCz1cIo5yb8t/fDl9HNwevbvJ28QJD4aQvnMn2E EkLKsxiVRFo4IcUKzn/HHmWPe59d8/8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=StlKORTv; spf=pass (imf26.hostedemail.com: domain of longman@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718982827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EP/kWpZ/AZr0wzKgF+O2dWS3CSDCoLm/7KmRqzqigc4=; b=StlKORTvVCxHGNnGx8w+1SLOQAGC9MokDw9z5MkgTuCE/38SO0yll4ytSkfleNqimahfKs mIy06RJ8g3D/cVQTJNRu2tDDnszVu1URm85/2Eap4SY5VkeYMH2z544a3in8/Zj3eVIO5p hcgrvuXX5RN1BEOrK0RwJRkVujlTWMQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-52-qD7zYqEPO-GFO_glMF_BXw-1; Fri, 21 Jun 2024 11:13:44 -0400 X-MC-Unique: qD7zYqEPO-GFO_glMF_BXw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2273C1956055; Fri, 21 Jun 2024 15:13:42 +0000 (UTC) Received: from [10.22.17.209] (unknown [10.22.17.209]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id AC86B3000605; Fri, 21 Jun 2024 15:13:39 +0000 (UTC) Message-ID: Date: Fri, 21 Jun 2024 11:13:38 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1] cgroup/rstat: Avoid thundering herd problem by kswapd across NUMA nodes To: Jesper Dangaard Brouer , tj@kernel.org, cgroups@vger.kernel.org, yosryahmed@google.com, shakeel.butt@linux.dev Cc: hannes@cmpxchg.org, lizefan.x@bytedance.com, kernel-team@cloudflare.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <171898037079.1222367.13467317484793748519.stgit@firesoul> Content-Language: en-US From: Waiman Long In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Rspamd-Queue-Id: 43ED2140015 X-Stat-Signature: 8nwk16x1jnpd98nwotdjox4qskfismtr X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1718982828-748257 X-HE-Meta: U2FsdGVkX18BeukKTSUCLwJuOodaYfKCjy1BtCs6S23pz3ynfU9TYvHF/4bA4L0aQG1wkcwgCju5yFMe9+a85C4qnh5uvJJD17pJQRZEdmPONJ23dx0j8F65Q4IiMsTSXaT0J/mN8pK/2OCmUMwKwN/UtFv9755j9F9q/2uOV1yap1gDYgYJfJjk9WtLZzo775NPYehvyw5rbEd3hdHl3I1Svhi2aHSuy7a5ALa2KKAKfmWXkAVt/LHUefNtJ9UJIWF6Rs42BKhe40+j4pwY9qlLHTduWaP3mtSCoBycogP16uwP87hrLStOCxmRmzk3Fr3/MAHezD7K0fIEKnbsi1XDqtPp2EMqYexNf7/Eg5djOBJuf4Gq7alQHmCHLh+MQ0x2k1dIk/fzo4x3F3FhsADSu2DaSSyYI9DpE0mje1XtmxsDtHMjG7XoNApBa4Fk5BP3BZfckMSuP2LWM///TFrP66J/2ny7/t0mzMWv/8W9mEkXkHFJsI/ca5URW02zXvsx1QguH+isrzT9fvULE2G3eH7MzQgZ/saTKmgXDPdeGmjmA9Ch80CpQjSP1DQZXpccsiMK+rjN/OU1ybxQoJ5BI4Nlx1+jyQXYwzoruOxfEwl7KWifMAxOG6EIBQm/HYVYC27DavTvcVzHFDygX++SJXQ7RdN3OYZOsAyevG4Y1q0j5YT/JHSFlw5A66uCwMSqnVi0ZKnoaN1agTgnSxdQQKB62egrf9Knc9C14/SRWmcXAeIepnVGBT0ql6eZNzM7uFxkcgSpyJ3d5WPkHBpYB8LSSbl4R70JtT2oVeT7NXMPuvcJZ0N3OqBckaPoSz9VIZoZhlyWiqs7OBpcc5EvRfkymPXY/9TuKAWaVDsKgv6bLxahzYBk2rWhmw7DUarxDSmiYBsF25wEbN217Od9LSvbeBUv09Be9R9RNRDDJIx6OBVSp9Dvtmpy2UnZ8ueqgLZOgcmutdksL7D 7+sKZ3zd AK6EpWOfU5IYVTKc20H1DrfaSohOOnWG47gFqioEjBRFiZbenBB8N8Q5JOcgl6omjj4ClMQtvPSAI+fbha+HGdZfqwxVVDEkObXsB7GSlDqoDl02/g5SARLahE1irZMM4L9COfxqJ8t6OcxrjSM2QEfrQhM500rqSreLbthc0aoVkA4Q+e6phpezUj84yQtuEavVD0lS3QbhyeBLTKYM/LBwREfkgA39OUCHSqcDl7mAEU6uZ2yhErClewmNF9PmPibcZ/usge3o8K5LWE3zJCskf8/GHGzVjY8o9bMdJ/72GMy/qnW0ZJ70y+IUxFYE0ePmrca84iHhXNaDxm/fe2mRKB1BDOn0FjGCXZOyEs49UKUm5d+W8ZPxMIL390TcQoZcB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/21/24 10:47, Jesper Dangaard Brouer wrote: > > Hold off applying this patch, as test kernel didn't boot with this patch > applied on top of TJ's cgroup tree (on commit ec9eeb89e60d86).  I don't > know if this is related to this patch or not. Commit ec9eeb89e60d86 is just a snapshot of the Linus' tree. You should use for-next/for-6.11 tag as a base. Cheers, Longman > > --Jesper > > On 21/06/2024 16.32, Jesper Dangaard Brouer wrote: >> Avoid lock contention on the global cgroup rstat lock caused by kswapd >> starting on all NUMA nodes simultaneously. At Cloudflare, we observed >> massive issues due to kswapd and the specific mem_cgroup_flush_stats() >> call inlined in shrink_node, which takes the rstat lock. >> >> On our 12 NUMA node machines, each with a kswapd kthread per NUMA node, >> we noted severe lock contention on the rstat lock. This contention >> causes 12 CPUs to waste cycles spinning every time kswapd runs. >> Fleet-wide stats (/proc/N/schedstat) for kthreads revealed that we are >> burning an average of 20,000 CPU cores fleet-wide on kswapd, primarily >> due to spinning on the rstat lock. >> >> To help reviewer follow code: When the Per-CPU-Pages (PCP) freelist is >> empty, __alloc_pages_slowpath calls wake_all_kswapds(), causing all >> kswapdN threads to wake up simultaneously. The kswapd thread invokes >> shrink_node (via balance_pgdat) triggering the cgroup rstat flush >> operation as part of its work. This results in kernel self-induced rstat >> lock contention by waking up all kswapd threads simultaneously. >> Leveraging this detail: balance_pgdat() have NULL value in >> target_mem_cgroup, this cause mem_cgroup_flush_stats() to do flush with >> root_mem_cgroup. >> >> To resolve the kswapd issue, we generalized the "stats_flush_ongoing" >> concept to apply to all users of cgroup rstat, not just memcg. This >> concept was originally reverted in commit 7d7ef0a4686a ("mm: memcg: >> restore subtree stats flushing"). If there is an ongoing rstat flush, >> limited to the root cgroup, the flush is skipped. This is effective as >> kswapd operates on the root tree, sufficiently mitigating the thundering >> herd problem. >> >> This lowers contention on the global rstat lock, although limited to the >> root cgroup. Flushing cgroup subtree's can still lead to lock >> contention. >> >> Fixes: 7d7ef0a4686a ("mm: memcg: restore subtree stats flushing"). >> Signed-off-by: Jesper Dangaard Brouer >> --- >>   include/linux/cgroup.h |    5 +++++ >>   kernel/cgroup/rstat.c  |   28 ++++++++++++++++++++++++++++ >>   2 files changed, 33 insertions(+) >> >> diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h >> index 2150ca60394b..ad41cca5c3b6 100644 >> --- a/include/linux/cgroup.h >> +++ b/include/linux/cgroup.h >> @@ -499,6 +499,11 @@ static inline struct cgroup >> *cgroup_parent(struct cgroup *cgrp) >>       return NULL; >>   } >>   +static inline bool cgroup_is_root(struct cgroup *cgrp) >> +{ >> +    return cgroup_parent(cgrp) == NULL; >> +} >> + >>   /** >>    * cgroup_is_descendant - test ancestry >>    * @cgrp: the cgroup to be tested >> diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c >> index fb8b49437573..5aba95e92d31 100644 >> --- a/kernel/cgroup/rstat.c >> +++ b/kernel/cgroup/rstat.c >> @@ -11,6 +11,7 @@ >>     static DEFINE_SPINLOCK(cgroup_rstat_lock); >>   static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); >> +static atomic_t root_rstat_flush_ongoing = ATOMIC_INIT(0); >>     static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu); >>   @@ -350,8 +351,25 @@ __bpf_kfunc void cgroup_rstat_flush(struct >> cgroup *cgrp) >>   { >>       might_sleep(); >>   +    /* >> +     * This avoids thundering herd problem on global rstat lock. >> When an >> +     * ongoing flush of the entire tree is in progress, then skip >> flush. >> +     */ >> +    if (atomic_read(&root_rstat_flush_ongoing)) >> +        return; >> + >> +    /* Grab right to be ongoing flusher, return if loosing race */ >> +    if (cgroup_is_root(cgrp) && >> +        atomic_xchg(&root_rstat_flush_ongoing, 1)) >> +        return; >> + >>       __cgroup_rstat_lock(cgrp, -1); >> + >>       cgroup_rstat_flush_locked(cgrp); >> + >> +    if (cgroup_is_root(cgrp)) >> +        atomic_set(&root_rstat_flush_ongoing, 0); >> + >>       __cgroup_rstat_unlock(cgrp, -1); >>   } >>   @@ -362,13 +380,20 @@ __bpf_kfunc void cgroup_rstat_flush(struct >> cgroup *cgrp) >>    * Flush stats in @cgrp's subtree and prevent further flushes.  >> Must be >>    * paired with cgroup_rstat_flush_release(). >>    * >> + * Current invariant, not called with root cgrp. >> + * >>    * This function may block. >>    */ >>   void cgroup_rstat_flush_hold(struct cgroup *cgrp) >>       __acquires(&cgroup_rstat_lock) >>   { >>       might_sleep(); >> + >>       __cgroup_rstat_lock(cgrp, -1); >> + >> +    if (atomic_read(&root_rstat_flush_ongoing)) >> +        return; >> + >>       cgroup_rstat_flush_locked(cgrp); >>   } >>   @@ -379,6 +404,9 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) >>   void cgroup_rstat_flush_release(struct cgroup *cgrp) >>       __releases(&cgroup_rstat_lock) >>   { >> +    if (cgroup_is_root(cgrp)) >> +        atomic_set(&root_rstat_flush_ongoing, 0); >> + >>       __cgroup_rstat_unlock(cgrp, -1); >>   } >> >> >