From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD222CD1288 for ; Wed, 3 Apr 2024 22:24:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 465F06B0088; Wed, 3 Apr 2024 18:24:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EF2B6B009B; Wed, 3 Apr 2024 18:24:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28FE56B009C; Wed, 3 Apr 2024 18:24:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 07FCC6B0088 for ; Wed, 3 Apr 2024 18:24:35 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C278BA0396 for ; Wed, 3 Apr 2024 22:24:34 +0000 (UTC) X-FDA: 81969650868.18.A2A1F37 Received: from out-174.mta0.migadu.com (out-174.mta0.migadu.com [91.218.175.174]) by imf15.hostedemail.com (Postfix) with ESMTP id D5959A000D for ; Wed, 3 Apr 2024 22:24:31 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="vjP/yYrv"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712183072; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m2bIqvAswIgDrceINf7iI+jp9YVPIpyS4ScRls8ALPo=; b=KOkCzJVmfU634euZxi51fL3ieftnv6HHBXk549B580XvecJ0rwbTwN4yiCw7FRNyt3FPWc 6VC/AmE1cNq/pgYiJvCU6liN6i/ks785e/s7Gs4IHDAaX0Mz2xR3+4p8UHApaAqzL6Rcqn lYES304MBhh9OhH+OVICakPQpwuy5L0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="vjP/yYrv"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.174 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712183072; a=rsa-sha256; cv=none; b=Ognm2ieB2kN+rthvM90LW9RBlE1SEtUY0DSI5alXzZ70ptfAOTvwzsw7DJ+6d8u3rx/AXq 4FbczZiDhBLaZXW/3iqkE2HDOVSj/QUjMR1xiPEVaqGYUFCmi/tSSrtCDyEc+zf1upqBqC XN5t8jaZL0Qj4G4jFOgOfk7j+2wMKoU= Date: Wed, 3 Apr 2024 18:24:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1712183069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=m2bIqvAswIgDrceINf7iI+jp9YVPIpyS4ScRls8ALPo=; b=vjP/yYrvtF19e9GfRt/XiqymxaisEEbkdtyuuCJTe5x3Tzrxf7uToW6LSjpKH1aM+fIeK0 P9E8Bz3M+ETRXVDhyGNKvROLYhHwrlU+63KTdjYO4b8Kviblvao04uSo5uDEb+Xz95wKRS QhWTv+BTLyRzNRskMf78GaGa3slcvU8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Tejun Heo Cc: Jan Kara , Kemeng Shi , akpm@linux-foundation.org, willy@infradead.org, bfoster@redhat.com, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240403162716.icjbicvtbleiymjy@quack3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: D5959A000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: rsz5rek51b6fdsbcmqbz7m8eirnpifng X-HE-Tag: 1712183071-828491 X-HE-Meta: U2FsdGVkX18E4nVFaTYvWOFMbqN6JJ5bIcZywAnknFVCPgTtjBW6u1rmL+NGUR/vBC4NvNFmcCCwHI7Gh2sdZtKwcFFfm+4ZaNl4FGmLQ6OSQTvEnQafOtDAgWvdIA+/f8n/H4Kdx/RawY6+FTIaV/PPup6QW6flgeXS+Tbh3seyinPn97CvAXCUHBsb6ejjhUUKVMbm8NPJikY+LiNRsUmwKaH9k2hpZma/YJA0So0b+StdJu7pAr1iTotxoO60+rvnBWIzWTLlQHaumOrsPTQbtGpI8AG594J2QA2t6HBWxKMoL5+aeLzCPIuqzoZr2oj6OWl/gjB/XaqGJqek45rFxbz9Xy5NnTS7pmQCsT6439U1Ny/VpBqQEU/iEuhU3SCk6tKoBMfIwBHPb119Ks0XMznJNEqjrD1JeFhoyD6st8Ro1Rw94NZpfzZhnggdOdleDwNdXrW4UOpXdvwFlatNOVjUSQECZYqt9r+WILa7jkFe0G3BEWsC2vacAt2TQJIhHT3wIToBknUxjgP63EGLfKgPd6bUg40IKvt8/eMvplzayiT8qauJYSnXT6lpK2M242gzQ8shl+fTNn/EYjKMTiEek72xh31wXhf8Z6v871zwmMHEcbLjN9F8+YP0/J7O7ipTLdmI9nsUCgPsYW59+y0Q0zormxnyl15wttYyhnEZyBLY/iBd1CPtTCgVEz0kollceZzRZSRZn7rsGjTYz1qMSN9TrKCNyL9t2E2RlBJl6uvHr0L04yiP9yx27YfCiMCQtDpO09z1Bwpoqq7TwyUl/xsTaH6L/J6pgviQhDTZ1Ro7IUHkOa2mwN9nzbPKdvDI6jvuGj+ZMZtHr88qUR8NOZjVG34mKteoSuXAqAMc5bhF/8liBaRdiAgjonHGOTB/Juy/5vxQ9SxOFU51DLNgPrm1f7st+z4xCXmunC03ZUWlhWuDEpFIM5qsWME868W4uqactylhJxv R1o6dLYX yuHx2w/afqVAV1boO9/6M8x4NdgEpjKjWN6D7nD3YBfAaZtn73g1VRP5KLqKIiSEqVVpOOLp9bIrfOPx2j80n4Gll1cGJgGon6hT5bwjf/DpBPFioxnmZA7LDUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Apr 03, 2024 at 09:21:37AM -1000, Tejun Heo wrote: > Hello, > > On Wed, Apr 03, 2024 at 03:06:56PM -0400, Kent Overstreet wrote: > ... > > That's how it should be if you just make a point of making your internal > > state easy to view and introspect, but when I'm debugging issues that > > run into the wider block layer, or memory reclaim, we often hit a wall. > > Try drgn: > > https://drgn.readthedocs.io/en/latest/ > > I've been adding drgn scripts under tools/ directory for introspection. > They're easy to write, deploy and ask users to run. Which is still inferior to simply writing to_text() functions for all your objects and exposing them under sysfs/debugfs. Plus, it's a whole new language/system for boths devs and users to learn. And having to_text() functions makes your log and error messages way better. "But what about code size/overhead?" - bullshit, we're talking about a couple percent of .text for the code itself; we blow more memory on permament dentries/inodes due to the way our virtual filesystems work but that's more of a problem for tracefs. > > Writeback throttling was buggy for _months_, no visibility or > > introspection or concerns for debugging, and that's a small chunk of > > code. io_uring - had to disable it. I _still_ have people bringing > > issues to me that are clearly memory reclaim related but I don't have > > the tools. > > > > It's not like any of this code exports much in the way of useful > > tracepoints either, but tracepoints often just aren't what you want; > > what you want just to be able to see internal state (_without_ having to > > use a debugger, because that's completely impractical outside highly > > controlled environments) - and tracing is also never the first thing you > > want to reach for when you have a user asking you "hey, this thing went > > wonky, what's it doing?" - tracing automatically turns it into a multi > > step process of decide what you want to look at, run the workload more > > to collect data, iterate. > > > > Think more about "what would make code easier to debug" and less about > > "how do I shove this round peg through the square tracing/BPF slot". > > There's _way_ more we could be doing that would just make our lives > > easier. > > Maybe it'd help classifying visibility into the the following categories: > > 1. Current state introspection. > 2. Dynamic behavior tracing. > 3. Accumluative behavior profiling. > > drgn is great for #1. Tracing and BPF stuff is great for #2 especially when > things get complicated. #3 is the trickest. Static stuff is useful in a lot > of cases but BPF can also be useful in other cases. > > I agree that it's all about using the right tool for the problem. Yeah, and you guys are all about the nerdiest and most overengineered tools and ignoring the basics. Get the simple stuff right, /then/ if there's stuff you still can't do, that's when you start looking at the more complicated stuff.