From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 916B1C6FD19 for ; Wed, 8 Mar 2023 22:19:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BD236B0071; Wed, 8 Mar 2023 17:19:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26D766B0072; Wed, 8 Mar 2023 17:19:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 134A5280001; Wed, 8 Mar 2023 17:19:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 01FED6B0071 for ; Wed, 8 Mar 2023 17:19:06 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A48F31A109A for ; Wed, 8 Mar 2023 22:19:06 +0000 (UTC) X-FDA: 80547147492.07.68EDF2E Received: from forward501a.mail.yandex.net (forward501a.mail.yandex.net [178.154.239.81]) by imf17.hostedemail.com (Postfix) with ESMTP id 784A640005 for ; Wed, 8 Mar 2023 22:19:03 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=ya.ru header.s=mail header.b=eBn4qYnA; spf=pass (imf17.hostedemail.com: domain of tkhai@ya.ru designates 178.154.239.81 as permitted sender) smtp.mailfrom=tkhai@ya.ru; dmarc=pass (policy=none) header.from=ya.ru ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678313944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hY99/afLxsURtdLVxU9Hkmxvw2HCLwOpsC5xj/gX3VY=; b=4tsw8K3hGBeNms/Zxbrq75zFDheghLSGEJa/xuq5fGjMMXke1U/AV6KPyN2hjKcojIPRiS MjOngNn/6jTrFSuuo0X1jQSlS1GyV3KixTtLvDRy599fn3banphCBMhSjVqIf777VO8TIg GykpDl7kEUCdKdLCz5DY0/Pu3p6njjE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=ya.ru header.s=mail header.b=eBn4qYnA; spf=pass (imf17.hostedemail.com: domain of tkhai@ya.ru designates 178.154.239.81 as permitted sender) smtp.mailfrom=tkhai@ya.ru; dmarc=pass (policy=none) header.from=ya.ru ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678313944; a=rsa-sha256; cv=none; b=hWfCXCNTgnjQ/RwNWY8qg+aqEAsuRfOLhDpY1g086Kv75bBbNiFSWKOQjd+qysHKYbbGpp 25QyeHA9+UNTg0vaOHP1teTe86Zrn1g97oLCHuV0dfoQkuv4QU1ytRiY8xH4wl/xZrGg+Q AZuyebJ6qD+YiwpXmRxrbKSc2zsMk0o= Received: from vla3-fd3176e90be6.qloud-c.yandex.net (vla3-fd3176e90be6.qloud-c.yandex.net [IPv6:2a02:6b8:c15:2584:0:640:fd31:76e9]) by forward501a.mail.yandex.net (Yandex) with ESMTP id DF7285EF80; Thu, 9 Mar 2023 01:19:00 +0300 (MSK) Received: by vla3-fd3176e90be6.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id vIlinqCbCOs1-PIRG8Uyy; Thu, 09 Mar 2023 01:18:59 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ya.ru; s=mail; t=1678313939; bh=hY99/afLxsURtdLVxU9Hkmxvw2HCLwOpsC5xj/gX3VY=; h=From:In-Reply-To:Cc:Date:References:To:Subject:Message-ID; b=eBn4qYnAxKfGPFSTX0WfauAGTJQtkJby3kq2cBzTrDFbA3OQ+6NZ8eeDEuxtJ64SU j9FSIaISHaI7pUo+cVLKmm86gul7G3rLVf7H/v4jwVvCPMMkt8EdY/pVr3aDCcmWK/ 8zKJLIl3nsSNRcPuZOgfctBNRRvmSLz78E4AfaxE= Message-ID: <37c59b18-b853-b299-88d0-c9222de39af9@ya.ru> Date: Thu, 9 Mar 2023 01:18:57 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: [PATCH v4 2/8] mm: vmscan: make global slab shrink lockless Content-Language: en-US To: Qi Zheng , akpm@linux-foundation.org, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> <20230307065605.58209-3-zhengqi.arch@bytedance.com> From: Kirill Tkhai In-Reply-To: <20230307065605.58209-3-zhengqi.arch@bytedance.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 784A640005 X-Stat-Signature: sd5jpb9gt9dky8t43ifnn917d686jj6c X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678313943-574184 X-HE-Meta: U2FsdGVkX1/l57SPcLPmHic981c7cW20TmimIUAE+sPyIc8msSHfALYtEjJRkZyZ6a7VmDD11m2NSMsFaLDkXuvoDb/rvecydeOJtJFzurPVFVd2/s/oqQrlTDadJxHK5FexT3JGn7iN01NkrFfzs5+m2zTSB+6fZsj6C4la9E+8Po9LvFNk9m83PAKEpQaga9zRIXUZGv1eetMeI00n7AlsoIFneC6G9nscTaTgGJ96uX0Sb2k1rS4hjJuYaBud+RF1wuis1KbCAgtB/s+zzQyrljGoQIKucFCf/CCnBYArmIjult7W11FP1N8qCx3wJEvnT+5dDsHURGz9zvu3cVcepSnsTMND8KDA5LL1ibPN1pl7PpXPpwQ7uO1CylAzqmQDKs7cGr8Xe25x1nQPYjZHUALRtbnwfZl3Xr85Wweh7eZthuA31RpS+H203TVagyRSa/KVPG+shqru4c3a+ilOVEaP9R6W9taismIci6/WsfAMAcc5Q3lzFEGDXEJKsEj07x2P0OBeMQx5+xjAhAbILPuLKF1US0dRwEL7+bD+jiNfSAfBnPH7s9xIgjmvNHdsGKTf/DlMaK0CDC5FR0Gjdiler/ICK7yN81YwkbfsQMR6zaqKSMNpQlgi7PUXk7XNp3T0+j5iXAlsA+NjEtu4EgZEUiF1VMFbval9az40sZVdaTVaPYzOkSaePLK6f47SRCFkWSOt9IBVRPFM7M8kgMod23rhCG+/2g03mV7OqqV53NLKABivQmqHnuuq1hy/vzKMTKabSvT+tAT+BKvtMn9RN7Zczko/mamlfNlhTGb6hrnjds+jGo6q1xTcugZ3CT/V9fwmLn7/6fSqpAQdpUGkmz5wbZ3acvLTaIrTc0SaDEByL2prSm305gesIpukV3ojc0yPGNXOf3f7tro5tTHYfOsf5b8gj4bSLmli+i2AsqZzlOzIN+1v618IgPyXMZLV1ZmotfJAu7N 4CsA7VWK ZavQq9eaNIjyPMEnNLBCIZbca48NMjz+R8+2mly8n4BkwUNrnJ+5K4HrZ++dzUqxfhSogzKoEAZJmrYavIAVk15HFcMycwb1x/mws0Fy8UMoBmbSrLC2m3acu2XVg/9IDHKITcoRvVBKMyX6WVJjtrNU/lDzakCh4qcVSqBFOgQ6oRbiPnSrfbNI5hepZOJRInF2EOifjSbnJLeG2ZkXd9jkBzoT1BWPB9tiOYdDncRVeFQnc0yFXyl6VwEEchFzivStLVosKRK+J2JSI7UWb0XMvqdwm91KqfpB3G9uH8g9H7GFAmsntXsiF9eywcK+93I7HjDB2b0rL08ORGozCrX3FC/eE6z3jBHvEtldq4QgJgXpDXaaBjOv2IbIFK7hpGBWT4gINtXw8vicC2XoJY0oaucqZT1t+3YnIt5IN57NB0DnO9tobfFa6BMqjPlwHLPeA6XBOyhYTTJoQTjxcOJMh19GapLC5+SxU7CK8wNPOdUuCXPkOtdt5LB4JvLgm6zC1Mx1ovtdRM2Rfq+ct/GQtMXjvbxUcvpxjhggBbvky+l944eP9yMeOBFiQ/KqgbdCd/N2jOWNrnX2CZKaPV5lvBvsatbRaZE/i93f27e0lRBI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 07.03.2023 09:55, Qi Zheng wrote: > The shrinker_rwsem is a global read-write lock in > shrinkers subsystem, which protects most operations > such as slab shrink, registration and unregistration > of shrinkers, etc. This can easily cause problems in > the following cases. > > 1) When the memory pressure is high and there are many > filesystems mounted or unmounted at the same time, > slab shrink will be affected (down_read_trylock() > failed). > > Such as the real workload mentioned by Kirill Tkhai: > > ``` > One of the real workloads from my experience is start > of an overcommitted node containing many starting > containers after node crash (or many resuming containers > after reboot for kernel update). In these cases memory > pressure is huge, and the node goes round in long reclaim. > ``` > > 2) If a shrinker is blocked (such as the case mentioned > in [1]) and a writer comes in (such as mount a fs), > then this writer will be blocked and cause all > subsequent shrinker-related operations to be blocked. > > Even if there is no competitor when shrinking slab, there > may still be a problem. If we have a long shrinker list > and we do not reclaim enough memory with each shrinker, > then the down_read_trylock() may be called with high > frequency. Because of the poor multicore scalability of > atomic operations, this can lead to a significant drop > in IPC (instructions per cycle). > > So many times in history ([2],[3],[4],[5]), some people > wanted to replace shrinker_rwsem trylock with SRCU in > the slab shrink, but all these patches were abandoned > because SRCU was not unconditionally enabled. > > But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), > the SRCU is unconditionally enabled. So it's time to use > SRCU to protect readers who previously held shrinker_rwsem. > > This commit uses SRCU to make global slab shrink lockless, > the memcg slab shrink is handled in the subsequent patch. > > [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ > [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ > [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ > [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ > [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ > > Signed-off-by: Qi Zheng Acked-by: Kirill Tkhai > --- > mm/vmscan.c | 27 +++++++++++---------------- > 1 file changed, 11 insertions(+), 16 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 2dcc01682026..8515ac40bcaf 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, > > LIST_HEAD(shrinker_list); > DECLARE_RWSEM(shrinker_rwsem); > +DEFINE_SRCU(shrinker_srcu); > > #ifdef CONFIG_MEMCG > static int shrinker_nr_max; > @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) > void register_shrinker_prepared(struct shrinker *shrinker) > { > down_write(&shrinker_rwsem); > - list_add_tail(&shrinker->list, &shrinker_list); > + list_add_tail_rcu(&shrinker->list, &shrinker_list); > shrinker->flags |= SHRINKER_REGISTERED; > shrinker_debugfs_add(shrinker); > up_write(&shrinker_rwsem); > @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) > return; > > down_write(&shrinker_rwsem); > - list_del(&shrinker->list); > + list_del_rcu(&shrinker->list); > shrinker->flags &= ~SHRINKER_REGISTERED; > if (shrinker->flags & SHRINKER_MEMCG_AWARE) > unregister_memcg_shrinker(shrinker); > debugfs_entry = shrinker_debugfs_remove(shrinker); > up_write(&shrinker_rwsem); > > + synchronize_srcu(&shrinker_srcu); > + > debugfs_remove_recursive(debugfs_entry); > > kfree(shrinker->nr_deferred); > @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) > { > down_write(&shrinker_rwsem); > up_write(&shrinker_rwsem); > + synchronize_srcu(&shrinker_srcu); > } > EXPORT_SYMBOL(synchronize_shrinkers); > > @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, > { > unsigned long ret, freed = 0; > struct shrinker *shrinker; > + int srcu_idx; > > /* > * The root memcg might be allocated even though memcg is disabled > @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, > if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) > return shrink_slab_memcg(gfp_mask, nid, memcg, priority); > > - if (!down_read_trylock(&shrinker_rwsem)) > - goto out; > + srcu_idx = srcu_read_lock(&shrinker_srcu); > > - list_for_each_entry(shrinker, &shrinker_list, list) { > + list_for_each_entry_srcu(shrinker, &shrinker_list, list, > + srcu_read_lock_held(&shrinker_srcu)) { > struct shrink_control sc = { > .gfp_mask = gfp_mask, > .nid = nid, > @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, > if (ret == SHRINK_EMPTY) > ret = 0; > freed += ret; > - /* > - * Bail out if someone want to register a new shrinker to > - * prevent the registration from being stalled for long periods > - * by parallel ongoing shrinking. > - */ > - if (rwsem_is_contended(&shrinker_rwsem)) { > - freed = freed ? : 1; > - break; > - } > } > > - up_read(&shrinker_rwsem); > -out: > + srcu_read_unlock(&shrinker_srcu, srcu_idx); > cond_resched(); > return freed; > }