From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EDC3CD11DD for ; Thu, 28 Mar 2024 19:40:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C79D46B00A0; Thu, 28 Mar 2024 15:40:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C02BF6B00A1; Thu, 28 Mar 2024 15:40:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA3646B00A2; Thu, 28 Mar 2024 15:40:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8AACC6B00A0 for ; Thu, 28 Mar 2024 15:40:13 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4E72B14024D for ; Thu, 28 Mar 2024 19:40:13 +0000 (UTC) X-FDA: 81947463906.15.5FD3D65 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf16.hostedemail.com (Postfix) with ESMTP id A953718000C for ; Thu, 28 Mar 2024 19:40:11 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=syslETbc; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711654811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4N45qEzZ33lTnFLFuDGUaQID+Buaapoa/kxgDVnQPks=; b=dz6sF495jxC+k1WG3G+H5cL24P5sROgJ7B1NUVq9olZyc2VgnaWH4TyiKLrrjvSDagddwR G1svaAnYESyWPMEGsMJpV2gr2pfYEzmmSq7WVN+Hk2Kr0M3YmElqpl20NYomyVPtf4pWRv crWRSsgytMWa8pPggWxTe9eBXyG1nho= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=syslETbc; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711654811; a=rsa-sha256; cv=none; b=Wi2L1U5uRt2dUNWFm/wC8BkVmrzFA/adCuC/Ryjic6wxOlBDMS7m/wJs2uQQmhIlmLXdMm 7mtY8OCNo8G5HzOQnR/8Ce1LtwMJ/Ecf7/Wgfn9eg7cy8jxp/Fkxp3NA8A4Mlkxf+Cb741 YSkzH1dcyd5TUEFiCDUcEUv9YrpaLc0= Date: Thu, 28 Mar 2024 15:40:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1711654805; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4N45qEzZ33lTnFLFuDGUaQID+Buaapoa/kxgDVnQPks=; b=syslETbcnuA4+bTAebP8fFAZFJWkAPqOUGHPC/c7H8OsQKPYKuszwdcPAPziypUvK7VRON FYz4SJudL+GgLZ2i6ep+wIfLO3fvs4ljgVuyJ9TDhKfjEErsFXReVX3yE0vUrczBvLYc6B CoGaZEpqCtASwzitAUPr+aUJWD0BhaY= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Tejun Heo Cc: Kemeng Shi , akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240327155751.3536-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: A953718000C X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ercia5smxiotxo3zdcdxi7x1ddcdzmfc X-HE-Tag: 1711654811-243305 X-HE-Meta: U2FsdGVkX1+dYI6+08Prs8f1FQugZbqYWW8jkwfVfje7vbEA/Y7WvwKNPmHyUNxxCBmTmDPItXHmEWL32vX7v/t7MRoOOngV2t0OFMJBCS/H297C8femlXN4p1FkdZp7uCuzAFh1xo3wdtM2xWPf37vKEzNWzwAQhCm76w3LmwGn5hbn17ZPZjW1M1UBXqGXj1SefsWW0epW+XymJyLJzDr3Qp32/1n+cx1ABMKvhsObsgZXVhzVLSJ0NX56vjjk1wQw5HUszX9wc8PiQEiTo81LjPnZSqKoHaGbSmRM70Gn4pqi7EAvPSI0INgi4DJyvYubNt+lLP4LFFz2y/sBJXutSxWwthUR641TwN2cRnMRDTjsUk+JWqe4iOvWEjTyvmFbOcMyhXx1JpodNgN8b115MtgIZjtrPAXgTTH7dQDALZd5lU7eqHtOLVJhR/Yigvsm24k3oytErQp4q27f7mfJ+GcfF8b8CtkknjK7Hu0smIIVc3nFKxJHDiK07vNJIWOrJ9ueAPMOiwwzsHIclLTWLDRaM7UZm9xVFWcsDjvbIQ4MlJi6HxnD1sYaTqzn77OGwmenRbZob5l1wp+2/nWAaiE6My0xNYiLbCMBC8tQTqJCh/opli8/yb+pLGbDvYqq0P+bvjYtlqWmcqf7kMaEOTJ9Aopeh6Fg7P+Km4gtv1KNh5HSKL2l9SYtGsiEUelcyApbASb4PTccf25pO4wh58RlN4hk5bktmcv/cRnxNNdyHlzn5HHwPj/VxyiHAv+EF33N+3xX94U8npygO4EurW0KTJmfp6VPisYfmVY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000080, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 28, 2024 at 09:31:57AM -1000, Tejun Heo wrote: > Hello, Kent. > > On Thu, Mar 28, 2024 at 03:24:35PM -0400, Kent Overstreet wrote: > > fs/bcachefs/time_stats.c has some code that's going to be moving out to > > lib/ at some point, after I switch it to MAD; if you could hook that up > > as well to a few points we could see at a glance if there are stalls > > happening in the writeback path. > > Using BPF (whether through bcc or bpftrace) is likely a better approach for > this sort of detailed instrumentation. Fixed debug information is useful and > it's also a common occurrence that they don't quite reveal the full picture > of what one's trying to understand and one needs to dig a bit deeper, wider, > aggregate data in a different way, or whatever. > > So, rather than adding more fixed infrastructure, I'd suggest adding places > which can easily be instrumented using the existing tools (they are really > great once you get used to them) whether that's tracepoints or just > strategically placed noinline functions. Collecting latency numbers at various key places is _enormously_ useful. The hard part is deciding where it's useful to collect; that requires intimate knowledge of the code. Once you're defining those collection poitns statically, doing it with BPF is just another useless layer of indirection. The time stats stuff I wrote is _really_ cheap, and you really want this stuff always on so that you've actually got the data you need when you're bughunting.