From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BDA6C3DA61 for ; Fri, 19 Jul 2024 00:41:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6FB26B0088; Thu, 18 Jul 2024 20:41:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D20876B0089; Thu, 18 Jul 2024 20:41:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE8016B008C; Thu, 18 Jul 2024 20:41:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9EAE96B0088 for ; Thu, 18 Jul 2024 20:41:08 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 16632120306 for ; Fri, 19 Jul 2024 00:41:08 +0000 (UTC) X-FDA: 82354647816.14.DB301ED Received: from out-183.mta1.migadu.com (out-183.mta1.migadu.com [95.215.58.183]) by imf16.hostedemail.com (Postfix) with ESMTP id A6A6E180006 for ; Fri, 19 Jul 2024 00:41:05 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Ad2SkKO7; spf=pass (imf16.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721349611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4qrSMnkpzFYj+5EVCkCG4KRZYIDNmGMzOcWNxn2WiHw=; b=0RqptsLcrlOiK4mT+3IUpWhRM+WLYOvRu0PoYUBqqDDi+tMGL/oLppwb7wsxBOV9XK3+A4 VsJ/We1zqIKSXBe++6ePBhvbwQYTOT2JRYhLF8vZYD6QR+rfI1oyAL4XI+jiylRUuIxfX0 LH6WuQnALL7M6MUNFzVe4M1EO8UC88c= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Ad2SkKO7; spf=pass (imf16.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721349611; a=rsa-sha256; cv=none; b=19IQY1fSFyWXaqV0PtQ1IdkxNpGWTI3MNWbYF7m8ugbooIbbZwLu3zQqY/8qEyr1Mu5cyY MW2n1s6r/kTAeF1YACFecU/rltHSWGmPNlTIy4JgpOKLZwVVbMN7bx53GXXkP+9TlylA05 A2Og3eWcV4nR/k1mcPVXfurLnKit+iY= X-Envelope-To: hawk@kernel.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1721349663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4qrSMnkpzFYj+5EVCkCG4KRZYIDNmGMzOcWNxn2WiHw=; b=Ad2SkKO70KGXQIEsX9jWjty3IeQRsSKTLMB7EQ0L2aVeL5+6PYyXlEEmvl8vBUVF2sVQeb Iufn63FaFjjDpyvIELUbpNT1aRSGcb/JCyHe29e3dKU1iSqzebUSCoa9YId061cm1bEbBx yzpffI9x9tMDUZ7CSTdb03bvzKbOqPM= X-Envelope-To: yosryahmed@google.com X-Envelope-To: tj@kernel.org X-Envelope-To: cgroups@vger.kernel.org X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: lizefan.x@bytedance.com X-Envelope-To: longman@redhat.com X-Envelope-To: kernel-team@cloudflare.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org Date: Thu, 18 Jul 2024 17:40:53 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Jesper Dangaard Brouer Cc: Yosry Ahmed , tj@kernel.org, cgroups@vger.kernel.org, hannes@cmpxchg.org, lizefan.x@bytedance.com, longman@redhat.com, kernel-team@cloudflare.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH V7 1/2] cgroup/rstat: Avoid thundering herd problem by kswapd across NUMA nodes Message-ID: References: <172070450139.2992819.13210624094367257881.stgit@firesoul> <100caebf-c11c-45c9-b864-d8562e2a5ac5@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <100caebf-c11c-45c9-b864-d8562e2a5ac5@kernel.org> X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A6A6E180006 X-Stat-Signature: cau1dq7srwfqhynsugpqcifxrteq4xh9 X-Rspam-User: X-HE-Tag: 1721349665-605105 X-HE-Meta: U2FsdGVkX1+kx5OFGtCplqdnweBbUyUdmSo2LUeCt+K2pQtnyyhJc4OvEjz/YA+GCbeFENxIMAwVlUT+vI9gkmZSMFSI4Scd6LJ8qTHIQKqtARB/4N8CjOQGr/743U+7VM6OB8yBWKvvxmlUfYz4b6zX4Rg/88dn/ozzMIMM6HDnwjWAAhQ6YUbGdQzKdhHn7yrE9x6orRNf9cdJqE5Lk+L+grTQdAJqPYjE2IMLmYvYJJ7b6fcIJGqN7GUvcWC3F67TfXpWKge8zLDc7wv0oPKOyMWlp6Ju8AYWUhG/GFl7mC3iQiJXv0/TRoLjLIJLhIpl4EY2KKQTYHuU9UJR2Fl81V9tZwXAWUnH2NI/gduzb80wTK2TBrJ3BlLtpqrz+DsryizhtPYmJA7a1atEPl+ytshfjqXzv1/8Gzf2wrdUTsqMxfV48oaQSeSLdjo8Md9T1j7LonvDAVliUaHprvzg0llYntOX1T4tSoT1UW7ifD5BbvZGXOLTIydlPXoh2w+/g2PU++7j15e3GbqOBFcevMj1CKK4M/jSV2tJzmaNOK3TrUvre3hjbKlOTss1Akpyj4Lk2IDskV4HX0PI1kxs989ELmRrsLtcSssLtnxbeWxib9kzfcVCA/ox1xNb1EQ4Hl+TUIBOOQqjVcKj8fjctBBP5GUgdeZWGDCCwxuySQsDtGILf/r1KsF45QYEOWD3ziPu/1O2VzHUfEci4E7AkVP/PS2Xn4KbjmPY/byCxjQO4xun6dRFZoPM0TbyY7PT5MjsP52zMmVBflZGs5kUcroJX5sRSSpgJIi5efX9XiGeD0ISJIX6KY/e8G0hKyXJWQ9Q7e3T0IsUIBCaN72B7NCMhBrW3vUE3AsOBCYk6PIhIf1w+OoeKLOfLoWuhrx3zrodf6QaaqkNRZzFpFAfJF3+FqFvIuUNapP5N5/agddQlGJeDMlY2cbizCUQZIZq1sc8DkB4bBH+lee l6e83RcE fdjXVR+qfclSsPkBTyYvugopDCOjJZR4eaqNPBI9y9MphReM46gVltOLsHt9vIG3uysv+g67lo62BE1YpwvWG2Vu2KFdBKMiaGHh2jIDtfrPTw2cclAnT8z82mA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.003419, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Jesper, On Wed, Jul 17, 2024 at 06:36:28PM GMT, Jesper Dangaard Brouer wrote: > [...] > > > Looking at the production numbers for the time the lock is held for level 0: > > @locked_time_level[0]: > [4M, 8M) 623 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > [8M, 16M) 860 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| > [16M, 32M) 295 |@@@@@@@@@@@@@@@@@ | > [32M, 64M) 275 |@@@@@@@@@@@@@@@@ | > Is it possible to get the above histogram for other levels as well? I know this is 12 numa node machine, how many total CPUs are there? > The time is in nanosec, so M corresponds to ms (milliseconds). > > With 36 flushes per second (as shown earlier) this is a flush every > 27.7ms. It is not unreasonable (from above data) that the flush time > also spend 27ms, which means that we spend a full CPU second flushing. > That is spending too much time flushing. One idea to further reduce this time is more fine grained flush skipping. At the moment we either skip the whole flush or not. How about we make this decision per-cpu? We already have per-cpu updates data and if it is less than MEMCG_CHARGE_BATCH, skip flush on that cpu.