From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 907EFC3DA4A for ; Wed, 14 Aug 2024 12:57:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 296376B0083; Wed, 14 Aug 2024 08:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2467B6B0088; Wed, 14 Aug 2024 08:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 135FA6B0089; Wed, 14 Aug 2024 08:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E8A3F6B0083 for ; Wed, 14 Aug 2024 08:57:50 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 85D52C0E9A for ; Wed, 14 Aug 2024 12:57:50 +0000 (UTC) X-FDA: 82450853100.09.BD98F7E Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf14.hostedemail.com (Postfix) with ESMTP id 17E74100008 for ; Wed, 14 Aug 2024 12:57:46 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L+QLheuc; spf=pass (imf14.hostedemail.com: domain of hawk@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=hawk@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723640232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6P4b7w35wA3kPqYIXISiUSZtwIt4d5abdHs7ql+X7dM=; b=4WRI5loot3KMiGA3Ly5mT8Yfb+LMo3zX6URRFVwmK/AihnLb4lG4TTK5OpumVCpV2LKJbN hurMKu6sWsgWREz7oniTHS2p8kdh1LUX4bVGgfPwlPHcs/kcubPgFdpVCJmcZErpm4HK6B HPg1ENGFRVhV9lcRwfKJTeK+XsDZotI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L+QLheuc; spf=pass (imf14.hostedemail.com: domain of hawk@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=hawk@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723640232; a=rsa-sha256; cv=none; b=6Ae7/hNhe8O8q3AP7wa9p1lAjGnwhmxdbNz5jSXep5dVtigKdo+qpM2Rtvs5hSXIWWqgl/ 4w0T/s6eZ7rI7OE+Nna8b2y4aP2LhDkd4x8Zz4SOVmHX2xa0ssKx8wfhH9D0l6Cw778DfA qkDwYrvZNvXQRUNgCYDgUUpD4eO3Wrk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id C7C57CE1944; Wed, 14 Aug 2024 12:57:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3FD02C32786; Wed, 14 Aug 2024 12:57:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723640262; bh=aTyhtGtgdJk4Q5FCiryBiL3pR4RSq8FA/pjWp1jwZLw=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=L+QLheucYpn6NXe5AWVMiKgJkvZl34jrpH1MUqNKt60ApPTUOl9bXD6RvwZ0eNMkP 3j7/3KcLedGn8TwWpNNaSe584n/PaST/63jUJ1cHPh1FNUMivaxYEjythm6ZoF43hU qdIo5Dh1nsAl8b5xhTGeXPTVJp0oabFsyDK3mlW0uwvBxBemHLWfvpwREVTiCRSMyF IstVAl9ggvLcBY5ZW1IIKJAG4MJoZWtYB5AlhV4k2o79t2GQ8yRDQQ8y57ry/g9e3J NGM6l0WwLiAJMal052OcCDfkIfKXinbdJ9bTd5029Ht4ZWJL7KCWBpH8C8a7QBMVuQ 25tvTcoWy0q6w== Message-ID: Date: Wed, 14 Aug 2024 14:57:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] memcg: use ratelimited stats flush in the reclaim To: Shakeel Butt , Yosry Ahmed Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Meta kernel team , cgroups@vger.kernel.org References: <20240813215358.2259750-1-shakeel.butt@linux.dev> Content-Language: en-US From: Jesper Dangaard Brouer In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 17E74100008 X-Stat-Signature: i9d7ymxwqt71jak3uknh3den5djrp6uu X-HE-Tag: 1723640266-896051 X-HE-Meta: U2FsdGVkX19u3ptJp5ozv6iVecgpA5SyiQHufzrYqCdNZ/hdHVA6gEo9hnAkesF/DGDfuBkjB467/UQzGQjIk3Wn13hbqYy2DWT1Wg5UHvOUOdnP9D1gIYJh0bawgE6SOc/KkORHcW4kxwTstgYK67+vqrYsH/BodWd7dG0EAR4hER2MMKRw2cgZmsF6lbCgjHm0gMswE8bPP1SetSz/WfpTeBfy/3da+9+Vnrc2gUA9GuETVC4eXhhRkEQ49KI9EEFeA2aumYj4M/sqBV9YWe06nDT5rlDUOMCRp1IDzI6aIHzuGrgHQ8YF/vk0/+DPQJ7bAXdhUmVl868q3gwwlg4ZTgF9mE1/A7jkks9XHrcZpDRi7ln+jfFuTIb4agc07cUC0pUEXFAeCM7GvSHFrAm+c+vpCwN91sYwwf4OkY1MOt9PWUL/kfF1bnEG9InmNkjjQR8rQekMbWybaFQpnalBgR7N7rhKLEYAaSonKTC7+c1x0oOPphvnrgB73o0i8FCWBvrjpZzFhtKq64BdcXWd0ShZKRxWmCz2mcGdVA9DcQkbsOtEHdU3WrrWIHYqnf96oM6rX3ScyrSCfNiBQ3WeuAA7wQ1uLtq+jF0UkZ/0pHG0LILo4hJGV3Aukkzpus4gNXlilBBT/153F++RITeBT35dwD1tQOSDO45GcNXR/Cyxc7FMkgKEYio13AyakaRtZvUaHQfG9qOIQDG1/Nx7ypxTGeQ/XVMmDzE2ukMx3pA1ozXYdopVDv7ubvRBC+N1fbJ9YuMRpOkqOl8k5aRhKarTbKCXleacJ4hv0HWjowoN/UAVe7Tt2j2FJg3Q/zrJ4w+NFujSMumJj7fIJ21HwVk7adOA9T3fAg8z1WcL3ADC4AK5tGO7ZOn1aHgZaaX5XgSMbuf/WVevyR++N1ClMjn8suWo8Mb9aN350g5qhF+ubh3IWBSpmkJTH1pNeuPgCXZOW9Axh3+9xBU ZaWthOOI pHaYC5ndCD2Y/9NWKPA7BBS1/zMcHpkSFh+/8HYl0PNxfy3be5zyeyYJASpdaMxn19A3N8OyHvrnoM25QVdcnlzfBZ5otlqo/Em9ne9W+fuDOlL+oWo88qyDxxLTTZMhv/OzMjLJTq1dz+6NdhCje//LWRdKL/QSMMkrhPY76Cl/IFHK7RIJba2N7yhmhSSQT3zxBqZCXFnr/1fL00cyW1n60o7NJFkLUPUo91GYKmBeP4Q1sCvD2L7G+SVYL9CzDo1g4ArfgbEAB9BIeXJ4wGreIl1AAc0b0iXTk80WzJCgvfvARCvgRqy5XxRd2diPmGluvTfALPcKWUI8plNznL9s+5PxJGC8IlDlJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 14/08/2024 00.30, Shakeel Butt wrote: > On Tue, Aug 13, 2024 at 02:58:51PM GMT, Yosry Ahmed wrote: >> On Tue, Aug 13, 2024 at 2:54 PM Shakeel Butt wrote: >>> >>> The Meta prod is seeing large amount of stalls in memcg stats flush >>> from the memcg reclaim code path. At the moment, this specific callsite >>> is doing a synchronous memcg stats flush. The rstat flush is an >>> expensive and time consuming operation, so concurrent relaimers will >>> busywait on the lock potentially for a long time. Actually this issue is >>> not unique to Meta and has been observed by Cloudflare [1] as well. For >>> the Cloudflare case, the stalls were due to contention between kswapd >>> threads running on their 8 numa node machines which does not make sense >>> as rstat flush is global and flush from one kswapd thread should be >>> sufficient for all. Simply replace the synchronous flush with the >>> ratelimited one. >>> >>> One may raise a concern on potentially using 2 sec stale (at worst) >>> stats for heuristics like desirable inactive:active ratio and preferring >>> inactive file pages over anon pages but these specific heuristics do not >>> require very precise stats and also are ignored under severe memory >>> pressure. >>> >>> More specifically for this code path, the stats are needed for two >>> specific heuristics: >>> >>> 1. Deactivate LRUs >>> 2. Cache trim mode >>> >>> The deactivate LRUs heuristic is to maintain a desirable inactive:active >>> ratio of the LRUs. The specific stats needed are WORKINGSET_ACTIVATE* >>> and the hierarchical LRU size. The WORKINGSET_ACTIVATE* is needed to >>> check if there is a refault since last snapshot and the LRU size are >>> needed for the desirable ratio between inactive and active LRUs. See the >>> table below on how the desirable ratio is calculated. >>> >>> /* total target max >>> * memory ratio inactive >>> * ------------------------------------- >>> * 10MB 1 5MB >>> * 100MB 1 50MB >>> * 1GB 3 250MB >>> * 10GB 10 0.9GB >>> * 100GB 31 3GB >>> * 1TB 101 10GB >>> * 10TB 320 32GB >>> */ >>> >>> The desirable ratio only changes at the boundary of 1 GiB, 10 GiB, >>> 100 GiB, 1 TiB and 10 TiB. There is no need for the precise and accurate >>> LRU size information to calculate this ratio. In addition, if >>> deactivation is skipped for some LRU, the kernel will force deactive on >>> the severe memory pressure situation. >>> >>> For the cache trim mode, inactive file LRU size is read and the kernel >>> scales it down based on the reclaim iteration (file >> sc->priority) and >>> only checks if it is zero or not. Again precise information is not >>> needed. >>> >>> This patch has been running on Meta fleet for several months and we have >>> not observed any issues. Please note that MGLRU is not impacted by this >>> issue at all as it avoids rstat flushing completely. >>> >>> Link: https://lore.kernel.org/all/6ee2518b-81dd-4082-bdf5-322883895ffc@kernel.org [1] >>> Signed-off-by: Shakeel Butt >> >> Just curious, does Jesper's patch help with this problem? > > If you are asking if I have tested Jesper's patch in Meta's production > then no, I have not tested it. Also I have not taken a look at the > latest from Jesper as I was stuck in some other issues. > I see this patch as a whac-a-mole approach. But it should be applied as a stopgap, because my patches are still not ready to be merged. My patch is more generic, but *only* solves the rstat lock contention part of the issue. The remaining issue is that rstat is flushed too often, which I address in my other patch[2] "cgroup/rstat: introduce ratelimited rstat flushing". In [2], I explicitly excluded memcg as Shakeel's patch demonstrates memcg already have a ratelimit API specific to memcg. [2] https://lore.kernel.org/all/171328990014.3930751.10674097155895405137.stgit@firesoul/ I suspect the next whac-a-mole will be the rstat flush for the slab code that kswapd also activates via shrink_slab, that via shrinker->count_objects() invoke count_shadow_nodes(). --Jesper