From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 942B2C3DA5D for ; Mon, 22 Jul 2024 22:59:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CCB76B008C; Mon, 22 Jul 2024 18:59:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17D0A6B0092; Mon, 22 Jul 2024 18:59:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0453B6B0093; Mon, 22 Jul 2024 18:59:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DB6426B008C for ; Mon, 22 Jul 2024 18:59:09 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 53ADEC16DE for ; Mon, 22 Jul 2024 22:59:09 +0000 (UTC) X-FDA: 82368906018.27.FFE97B1 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) by imf30.hostedemail.com (Postfix) with ESMTP id E90128000A for ; Mon, 22 Jul 2024 22:59:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tSofeIJp; spf=pass (imf30.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721689123; a=rsa-sha256; cv=none; b=0Ika7SucSo4mqUdofAkWsl6AMVkXydJtFghFR4b0lgXVCJ2NHt9hbVQhBNjWC58FKDJD6Q 8fKg+ucdSrgWNVrPnVCOXkKte8HHhRjhl0wNXE7odXI2zourl5BVUubLC0vM7124kRIB1u k78lpYmqqwY+nU8B2g3LYkYsW5IlMjQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=tSofeIJp; spf=pass (imf30.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.171 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721689123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lnMFQME55Z/WVJKRcDsm0NmURfzs9UA9CICS+vyuQrk=; b=JrOEtm0eQHimie+T8pAk5t5t1CuBQD+DLD+UaHwOF2QDqBc6BJXWRo+DrjAiybXaWUKEvY 1BycM8eqZLdxlD9LEdCBhh1fBiChl7lHr48UkGYd9wHTgRAxorcuZKefu7BWHWonBv+thp sbY5pFEG7b1QTSQpzTaoO07KWKKctAk= X-Envelope-To: yosryahmed@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1721689142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lnMFQME55Z/WVJKRcDsm0NmURfzs9UA9CICS+vyuQrk=; b=tSofeIJp2Lu+QLtJIvS80D1fWbnhQuBb67hzp4bJWiWIeTu1CNfwp4UisFobiqpumLeoMB bN54JfBPFp+U0Bb4pNdzDWgSpt6jXrEetgNcFY6hq/E1PNxTs35LV3600sPXAyZ95IfE2y IYr/LNS8SYwzOZYxtN6bA1A5yzi5jGM= X-Envelope-To: hawk@kernel.org X-Envelope-To: tj@kernel.org X-Envelope-To: cgroups@vger.kernel.org X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: lizefan.x@bytedance.com X-Envelope-To: longman@redhat.com X-Envelope-To: kernel-team@cloudflare.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org Date: Mon, 22 Jul 2024 15:58:56 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Yosry Ahmed Cc: Jesper Dangaard Brouer , tj@kernel.org, cgroups@vger.kernel.org, hannes@cmpxchg.org, lizefan.x@bytedance.com, longman@redhat.com, kernel-team@cloudflare.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH V7 1/2] cgroup/rstat: Avoid thundering herd problem by kswapd across NUMA nodes Message-ID: References: <100caebf-c11c-45c9-b864-d8562e2a5ac5@kernel.org> <5ccc693a-2142-489d-b3f1-426758883c1e@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Stat-Signature: p7gfnmnrc7w8ihe5eae5g3ejqh1w9mzh X-Rspamd-Queue-Id: E90128000A X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721689146-386426 X-HE-Meta: U2FsdGVkX1/q7gOAcsUW+TpXr+ffTa71fMrzoLn2XGGMArUC1YQ3bPeFM6VXAHgt4X/+9+OS/cvyB7407pEj4icspfurjYvDJHtGj85XLIGEpwwk2LLH6sE10LKNY86fCp9rYzr4EQf/rrA3985tIOH0jiI8687PTqTWLWrs3OLwnZwWJuCjWVdChiGOpDNYLp6esJHZNOhSacEAiv0Hz+gVF3TQXYsuka7Srv+WXIg9WPSV7CYldZKL6FXpp94sAGaRoWyCq4itguFkq7+zba3UciNxZyRCustEF5u75zqWkNORRccBtERu1Xr3VnJspwrgZJAA9/lc8xUg71qpTVPEqIej/7ZDbGRYhQK7p8ziOIVatLZlw6TosHZDNS9w6XpolRe2Suv64G/HeNI25WPrE4Mpcr50ciUVarzyFENPqdnMK7lgup0Tf7uuOz5kpUVdlQIsbnV0z0VE1AtOmZo7iCDr5VaFZMHPAdW/lWNfju4yIKKNYD6qbX6kJ0qImLqhIu4/vBAyo7ty28qFrjeS5ux/19JFk102aZT16JuvtSqjP8gTHrSOi3AgSACgoBSOMhtWypROJEG1wln/IsPz6BxIexDOcM4vc45bjD0nFpV2jlAHU3KKnlbW0A23vvcCx2c+gs8ZYUSgTdJ5d7vuiWWcdHgqyBofHHKnSKW/rTgbOFdHu7s0Ax8O8+T1e3OtzqdOXF3oZh0NEbVr0FkH+4GtT91z3XwyINSLzOcDQsqNmeywDJBBHb5Y3cByHG4NmUERukF4+iLPEgWXugxTLf609hktpOAlrUhF1sf7qhV65ngFvJ9dqL17NDHd+Zvr8MCyD47zrb7vUnlFP2xycVaX8r1bBesFQj+bvmQ0tbJpoPXTQfF2LO8VosUbJwY8nLUa9Qxslk9LuzXPWci6H5WaGKjRszv6KCEhJYG9Yuh12kOemEhjLqifbKvFnTzqLrX59u00c4S5wcP GKj1oXWO Oq60Ya1Kb3MFIgk7rut301DY1Wd4saKdCQ4y76eRtZqFB49W1sNp+uxcJI+3hcB/yRFKE+SbFhlGgDGDOg6LUqR6UBHo2JFlPEVSJ5yvnc9BwWfa+z5DWPC4vS/TNdctiZcybWxOSYSZcyYca/zN0uyA+DrV19lCmRgTseiTBDe0mNIc7iIaHeNUs6Cel0RUVLuoBxigjmSPZKA0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000106, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 22, 2024 at 02:32:03PM GMT, Shakeel Butt wrote: > On Mon, Jul 22, 2024 at 01:12:35PM GMT, Yosry Ahmed wrote: > > On Mon, Jul 22, 2024 at 1:02 PM Shakeel Butt wrote: > > > > > > On Fri, Jul 19, 2024 at 09:52:17PM GMT, Yosry Ahmed wrote: > > > > On Fri, Jul 19, 2024 at 3:48 PM Shakeel Butt wrote: > > > > > > > > > > On Fri, Jul 19, 2024 at 09:54:41AM GMT, Jesper Dangaard Brouer wrote: > > > > > > > > > > > > > > > > > > On 19/07/2024 02.40, Shakeel Butt wrote: > > > > > > > Hi Jesper, > > > > > > > > > > > > > > On Wed, Jul 17, 2024 at 06:36:28PM GMT, Jesper Dangaard Brouer wrote: > > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > Looking at the production numbers for the time the lock is held for level 0: > > > > > > > > > > > > > > > > @locked_time_level[0]: > > > > > > > > [4M, 8M) 623 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > > > > > > > > [8M, 16M) 860 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| > > > > > > > > [16M, 32M) 295 |@@@@@@@@@@@@@@@@@ | > > > > > > > > [32M, 64M) 275 |@@@@@@@@@@@@@@@@ | > > > > > > > > > > > > > > > > > > > > > > Is it possible to get the above histogram for other levels as well? > > > > > > > > > > > > Data from other levels available in [1]: > > > > > > [1] > > > > > > https://lore.kernel.org/all/8c123882-a5c5-409a-938b-cb5aec9b9ab5@kernel.org/ > > > > > > > > > > > > IMHO the data shows we will get most out of skipping level-0 root-cgroup > > > > > > flushes. > > > > > > > > > > > > > > > > Thanks a lot of the data. Are all or most of these locked_time_level[0] > > > > > from kswapds? This just motivates me to strongly push the ratelimited > > > > > flush patch of mine (which would be orthogonal to your patch series). > > > > > > > > Jesper and I were discussing a better ratelimiting approach, whether > > > > it's measuring the time since the last flush, or only skipping if we > > > > have a lot of flushes in a specific time frame (using __ratelimit()). > > > > I believe this would be better than the current memcg ratelimiting > > > > approach, and we can remove the latter. > > > > > > > > WDYT? > > > > > > The last statement gives me the impression that you are trying to fix > > > something that is not broken. The current ratelimiting users are ok, the > > > issue is with the sync flushers. Or maybe you are suggesting that the new > > > ratelimiting will be used for all sync flushers and current ratelimiting > > > users and the new ratelimiting will make a good tradeoff between the > > > accuracy and potential flush stall? > > > > The latter. Basically the idea is to have more informed and generic > > ratelimiting logic in the core rstat flushing code (e.g. using > > __ratelimit()), which would apply to ~all flushers*. Then, we ideally > > wouldn't need mem_cgroup_flush_stats_ratelimited() at all. > > > > I wonder if we really need a universal ratelimit. As you noted below > there are cases where we want exact stats and then we know there are > cases where accurate stats are not needed but they are very performance > sensitive. Aiming to have a solution which will ignore such differences > might be a futile effort. > BTW I am not against it. If we can achieve this with minimal regression and maintainence burden then it would be preferable.