From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B306ECD11DD for ; Thu, 28 Mar 2024 19:24:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CDBE6B009A; Thu, 28 Mar 2024 15:24:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47F166B009B; Thu, 28 Mar 2024 15:24:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36CB86B009C; Thu, 28 Mar 2024 15:24:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1ADDB6B009A for ; Thu, 28 Mar 2024 15:24:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 72A5914011A for ; Thu, 28 Mar 2024 19:24:43 +0000 (UTC) X-FDA: 81947424846.27.133840C Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) by imf29.hostedemail.com (Postfix) with ESMTP id 60B6C12001E for ; Thu, 28 Mar 2024 19:24:41 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=m2il9thC; spf=pass (imf29.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.181 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711653881; a=rsa-sha256; cv=none; b=BE0Njfp8oVPDbjh2wmcMEywmCOnjFYYPElPtkZ76LbgwUpMjkwsyKO/322KaSfjNcvEL6M GIc0euRREhpP+On6sq5+ze/0oECuPih58ZdrwLwEomcNh5LjqBlAbdRG58hJJYjBNjHnwg KUe8zlC0Zyc9manqBdDuTa8jyS11nMI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=m2il9thC; spf=pass (imf29.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.181 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711653881; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9m7kxahkpOfFeAppA/MMdv3HWeGeBpfeH3QIkwZrNkM=; b=SU/1ZNVVjaQHaBc7ol1Y6cO9PnYyHJNvqgNj7KlDnWbczkugPFitUsXxBcNT2FKMbVwuJE bjreyKo23UGwjGsZN/5E3KMvoWJ9SoUX0ZgrA6IwcVvGaI4ifjkM2SpIFaoZF1rLREtUMj CR6L3PQyNg/HHD7hJiLaDXHvDtA+DIo= Date: Thu, 28 Mar 2024 15:24:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1711653879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9m7kxahkpOfFeAppA/MMdv3HWeGeBpfeH3QIkwZrNkM=; b=m2il9thCRTKZ8stq4jpdCpEaq3OwikSMrJtYMV+72gTc058M+1PwAFvIrCoszbJ+fy+KJU K6rFI1eMkpzFAq6UxFVhHH5CHlbdHcdYHU6iejVcrvQxJr27MDCbiXoH+quam4IUmx6jx6 uCqiYYuf8PMX5e3gBLgHdFsiVe4+Ho0= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Kemeng Shi Cc: akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, tj@kernel.org, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240327155751.3536-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240327155751.3536-1-shikemeng@huaweicloud.com> X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 60B6C12001E X-Stat-Signature: tobgidxqsc6j5zxaubdfcgc19hz45bwu X-Rspam-User: X-HE-Tag: 1711653881-635554 X-HE-Meta: U2FsdGVkX1/cmaiMFQrs1zj/EUctdJyu2nfWdrCfI+1oWFIOWaOxVoLPLqanlMacXkpk9KpMKR4d4AlBHmTZb6SG64QUgl0qDc4RmUajsFb9FBEqMzVupszgsQmvtoYigzRU2EisELmBlErhadn+77XVNz7cy+YQVUh55T6bJsfwjf6IAKuQRd/vZnMzKQeqLNaNIdP8onWtxovwnUpTlL/ADA31+6bLAc5SFUcwqLD87qslXa7ExaM3bg8tc7W1nCoyU1LSBMzDC11FQMXmDc0FxXUc9LM3F0pCOK27sjDBKXx1DLeVoSEzIV+1CT47l22miVSlr7RdIUcs9zxmDrUHszkzXE8o2Ilq9SSid2ah2FuYwxVrdVHDpb36cNzsFNFVLnDwowkbGB/HRSCl9X2rFTfyYkJqtN4ocMzqsLQD+/5ktdqA3MxsOIlv6K0i8rco3RxkM8iArz7ASkNHNQdVRmM6IPQXhJNfksZ1jPGxfRCU2H4kS9DDrccAtKqfx3JUhC5QbuzC67Ks9TA9XupVVJ7Z00sEKK1OuFebY+vWXq76YpDwHdQKVzQXpjIzk7e97RCjziTaPL0IwRnpp7vxd9MTHVoWHa2y8js/1XLxIf4U9Ga3H7JI9iaIW2zBvcgURxatOuvQussCtF8SFwM66k/Aum6SS1lL/aqoZ67zxl0OuDQsRxiBmKpHK+g4yhO7vsho8LaKsUehcvcDr8X1e03v1sFwTVH0vEHzYEAUJvOV4TkK9E0Xl2OA6JB1anPKQvnt3KUvIh9G3i75SPCOdeiLZu3WyjnW/wzMH3pR6TRZkaqcH0mi9oOqfG9vbZu4/qJqme3CZLHOaXy2Rog5tPP1XCDcjWhjiUGhXjjt0bw1mmmatgPmWtPlSYcbVO7B6H5pdmeUmKdh/+BZVmlItvB6IcRNOu0lHileALOcHYTbymBvuYVem1N3lUUmlx3U9xdO3yNI3dEbxNA XmdLMa4e 3nZVYRWXWt6SfBZPBZUC5zJexIOUxHa3HQF2Xbox0BvLmTWqzasZUexKxMwAI5C9T/a75lzPuVziC12zsZQj/vR/vfPDY9u43XGvr0rhMp3MQf675A+y84rXC+rm/XFcQ10gXQrnuYxURT9ivvBYovvlNHpLJ4L4lUs0X0hj6qNk2UVn2YDM6CKbV6Dahs/eXPnJk17hL6eWlXHw9LQE6yHpLlg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 27, 2024 at 11:57:45PM +0800, Kemeng Shi wrote: > v1->v2: > -Send cleanup to wq_monitor.py separately. > -Add patch to avoid use after free of bdi. > -Rename wb_calc_cg_thresh to cgwb_calc_thresh as Tejun suggested. > -Use rcu walk to avoid use after free. > -Add debug output to each related patches. > > This series tries to improve visilibity of writeback. Patch 1 make > /sys/kernel/debug/bdi/xxx/stats show writeback info of whole bdi > instead of only writeback info in root cgroup. Patch 2 add a new > debug file /sys/kernel/debug/bdi/xxx/wb_stats to show per wb writeback > info. Patch 4 add wb_monitor.py to monitor basic writeback info > of running system, more info could be added on demand. Rest patches > are some random cleanups. More details can be found in respective > patches. Thanks! > This series is on top of patchset [1]. > > [1] https://lore.kernel.org/lkml/20240123183332.876854-1-shikemeng@huaweicloud.com/T/#mc6455784a63d0f8aa1a2f5aff325abcdf9336b76 Not bad I've been trying to improve our ability to debug latency issues - stalls of all sorts. While you're looking at all this code, do you think you could find some places to collect useful latency numbers? fs/bcachefs/time_stats.c has some code that's going to be moving out to lib/ at some point, after I switch it to MAD; if you could hook that up as well to a few points we could see at a glance if there are stalls happening in the writeback path. > > Following domain hierarchy is tested: > global domain (320G) > / \ > cgroup domain1(10G) cgroup domain2(10G) > | | > bdi wb1 wb2 > > /* all writeback info of bdi is successfully collected */ > # cat /sys/kernel/debug/bdi/252:16/stats: > BdiWriteback: 448 kB > BdiReclaimable: 1303904 kB > BdiDirtyThresh: 189914124 kB > DirtyThresh: 195337564 kB > BackgroundThresh: 32516508 kB > BdiDirtied: 3591392 kB > BdiWritten: 2287488 kB > BdiWriteBandwidth: 322248 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 2 > b_dirty_time: 0 > bdi_list: 1 > state: 1 > > /* per wb writeback info is collected */ > # cat /sys/kernel/debug/bdi/252:16/wb_stats: > cat wb_stats > WbCgIno: 1 > WbWriteback: 0 kB > WbReclaimable: 0 kB > WbDirtyThresh: 0 kB > WbDirtied: 0 kB > WbWritten: 0 kB > WbWriteBandwidth: 102400 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 0 > b_dirty_time: 0 > state: 1 > WbCgIno: 4284 > WbWriteback: 448 kB > WbReclaimable: 818944 kB > WbDirtyThresh: 3096524 kB > WbDirtied: 2266880 kB > WbWritten: 1447936 kB > WbWriteBandwidth: 214036 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 1 > b_dirty_time: 0 > state: 5 > WbCgIno: 4325 > WbWriteback: 224 kB > WbReclaimable: 819392 kB > WbDirtyThresh: 2920088 kB > WbDirtied: 2551808 kB > WbWritten: 1732416 kB > WbWriteBandwidth: 201832 kBps > b_dirty: 0 > b_io: 0 > b_more_io: 1 > b_dirty_time: 0 > state: 5 > > /* monitor writeback info */ > # ./wb_monitor.py 252:16 -c > writeback reclaimable dirtied written avg_bw > 252:16_1 0 0 0 0 102400 > 252:16_4284 672 820064 9230368 8410304 685612 > 252:16_4325 896 819840 10491264 9671648 652348 > 252:16 1568 1639904 19721632 18081952 1440360 > > > writeback reclaimable dirtied written avg_bw > 252:16_1 0 0 0 0 102400 > 252:16_4284 672 820064 9230368 8410304 685612 > 252:16_4325 896 819840 10491264 9671648 652348 > 252:16 1568 1639904 19721632 18081952 1440360 > ... > > Kemeng Shi (6): > writeback: protect race between bdi release and bdi_debug_stats_show > writeback: collect stats of all wb of bdi in bdi_debug_stats_show > writeback: support retrieving per group debug writeback stats of bdi > writeback: add wb_monitor.py script to monitor writeback info on bdi > writeback: rename nr_reclaimable to nr_dirty in balance_dirty_pages > writeback: define GDTC_INIT_NO_WB to null > > include/linux/writeback.h | 1 + > mm/backing-dev.c | 203 ++++++++++++++++++++++++++++++---- > mm/page-writeback.c | 31 ++++-- > tools/writeback/wb_monitor.py | 172 ++++++++++++++++++++++++++++ > 4 files changed, 378 insertions(+), 29 deletions(-) > create mode 100644 tools/writeback/wb_monitor.py > > -- > 2.30.0 >