From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5799CD128A for ; Wed, 3 Apr 2024 19:07:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D99D6B0083; Wed, 3 Apr 2024 15:07:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1637A6B0085; Wed, 3 Apr 2024 15:07:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 003FD6B0087; Wed, 3 Apr 2024 15:07:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D2FD46B0083 for ; Wed, 3 Apr 2024 15:07:04 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6ED44A0D80 for ; Wed, 3 Apr 2024 19:07:04 +0000 (UTC) X-FDA: 81969153168.06.CDD8907 Received: from out-175.mta0.migadu.com (out-175.mta0.migadu.com [91.218.175.175]) by imf25.hostedemail.com (Postfix) with ESMTP id 84AD8A002C for ; Wed, 3 Apr 2024 19:07:02 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ab29Awgi; spf=pass (imf25.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712171222; a=rsa-sha256; cv=none; b=T1oTlQLK4TrrBk+D3gwa3Oo4zFNqgjCn0o4JypbzhWUKOSgajFcaIwxE+FeAur7edmwp9g t8B63oXm89Vv+4fBeQOiWJhf6M0RiuSxrhy4/f0G8kOTlQnCGlWDFmN3YaHyO56yEQBUOo OlGtIakoOH2WFh9IUa7GWuureoP5+7E= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ab29Awgi; spf=pass (imf25.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.175 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712171222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MPufZuIJuTLIDu10Urps9fsUQXIz9rznyNkrrZnyWEM=; b=F5nsXFqmI4S1QixJRXabQ6AWYjJXnjE9L0cX5BN6rX57Mt2mjQQdICtGDg5YNC6XONsWiP VZ54h7jcyUvQ+uU9earuEMKoAfG8vBdlqmKlWRjhybBMZNjgpQmlKzbzN8ksogjaYFuN9y KdTOXOKrbVcGH3sxH48bEn4mZB23Vrs= Date: Wed, 3 Apr 2024 15:06:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1712171220; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=MPufZuIJuTLIDu10Urps9fsUQXIz9rznyNkrrZnyWEM=; b=ab29AwgidGYXl/9pYqDFJfdDItXW7t4LUhM3Qj9YVwhcTDaF8J7UEropu/nN8V+SnlrPp/ +SrSQ2f18s9M0jQ8SgPaaB8J5ogz82i+9eN08TaGx4gkgjN0F4Euhr/MbcHZN6rMiDbbtt T9FLbyOJvoCIqIyAx5v/kURkMzbxliU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Tejun Heo Cc: Jan Kara , Kemeng Shi , akpm@linux-foundation.org, willy@infradead.org, bfoster@redhat.com, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240403162716.icjbicvtbleiymjy@quack3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 84AD8A002C X-Stat-Signature: gfxdzp4himxs8ygafnp1xw4nn4dhgzt7 X-Rspam-User: X-HE-Tag: 1712171222-974868 X-HE-Meta: U2FsdGVkX1+V2JAx+q/s5wqUgGYAZ0SbcIf/Cn7qmWS9+r1afUCotSBvBb9lEpng8VfjiIOuhXBeCCDVq31ToAZdwgvWY+rrqtuUS+1wbVrAjA0CMVMUiNqc2GTF93GgqRwi6Hyxov60nQN1mZr5FjDRtnQB6KtyYJo7xJCCj8gmIHup/Xf19KzMkrbqgWkw8s/kjFGJBxYOqp73/AyDhdG5N7XGHJtECwOVpvJ3AznfoB53ozy/sx3GpF1eHx0JWyzI1GWPvdFhFJVeVIR3x/ygfgzaR8XTreFo31GdhlcsjqwFTdhgx8qFnwMAeBFFeWK3Ef/zWg0tM+mTH2YMouxokBEKUmENROs/qKgcrwyFvD4JDcViHvw3re6MesK3g2lSpZCBrkzvuKg3Ux13M6pXFjiinJNnO2F7nGONgJHqIvEjlWyZi40quwKD7ACGLXZtsVCT1VoHwpkFgyWYGRHuTX76jqHKvXnAOxseXkXoerohiii7tPjfeMYxatPUy3kEcKifOkh3LltJohJs1NtsvI65JrLpeySmewT0qow7Vbu/30w4s/JbhF3Ss4QhdlcmwWx3vHTUIahvrp0FClXRR20/PunXRfgMuPm+dSLOc1cZDTVZYcDIsnsovk8+uiTbTQ4ecwc7b5YUFsUTHHx8FJWZNstt5ERLtCt/GmXlstf0lNOlIkE4ieZnC+D+4riXR8wwd8eHZFSQqR1ujia0BNJ7asQODaM5KYBzCIW5yuTUcR0PVYTzMkodQ3DDNvKQTMIczPBwzUBsbgP5d0Cks8k9r+R4OSGH+rHmckJ8UZDJodtFdRpMicW2rGv2pwhR7yzsrvallTur2sGCLnVeTpMX4lgXdyKJtfyrgEaoQ482ojD14Qsjf2ii6TcjkRdm+i4TJblXRzjadthZe+vIDJJoGX4UwGyhOLyWatqllot/eIMfv+U4QPhuGBINs2mIbLQ4kKApRilw1v2 V+K/GSk+ 1wBGbbxjErP0RJwd3QQiXKcvXWjvI333H52qmJdGwlAGFmHtloDU7vhyQFeRLWEOc+hy+ZCDqM52j0i/rsrPpgkHy6g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Apr 03, 2024 at 08:44:05AM -1000, Tejun Heo wrote: > Hello, > > On Wed, Apr 03, 2024 at 06:27:16PM +0200, Jan Kara wrote: > > Yeah, BPF is great and I use it but to fill in some cases from practice, > > there are sysadmins refusing to install bcc or run your BPF scripts on > > their systems due to company regulations, their personal fear, or whatever. > > So debugging with what you can achieve from a shell is still the thing > > quite often. > > Yeah, I mean, this happens with anything new. Tracing itself took quite a > while to be adopted widely. BPF, bcc, bpftrace are all still pretty new and > it's likely that the adoption line will keep shifting for quite a while. > Besides, even with all the new gizmos there definitely are cases where good > ol' cat interface makes sense. > > So, if the static interface makes sense, we add it but we should keep in > mind that the trade-offs for adding such static infrastructure, especially > for the ones which aren't *widely* useful, are rather quickly shfiting in > the less favorable direction. A lot of our static debug infrastructure isn't that useful because it just sucks. Every time I hit a sysfs or procfs file that's just a single integer, and nothing else, when clearly there's internal structure and description that needs to be there I die a little inside. It's lazy and amateurish. I regularly debug things in bcachefs over IRC in about 5-10 minutes of asking to check various files and pastebin them - this is my normal process, I pretty much never have to ssh and touch the actual machines. That's how it should be if you just make a point of making your internal state easy to view and introspect, but when I'm debugging issues that run into the wider block layer, or memory reclaim, we often hit a wall. Writeback throttling was buggy for _months_, no visibility or introspection or concerns for debugging, and that's a small chunk of code. io_uring - had to disable it. I _still_ have people bringing issues to me that are clearly memory reclaim related but I don't have the tools. It's not like any of this code exports much in the way of useful tracepoints either, but tracepoints often just aren't what you want; what you want just to be able to see internal state (_without_ having to use a debugger, because that's completely impractical outside highly controlled environments) - and tracing is also never the first thing you want to reach for when you have a user asking you "hey, this thing went wonky, what's it doing?" - tracing automatically turns it into a multi step process of decide what you want to look at, run the workload more to collect data, iterate. Think more about "what would make code easier to debug" and less about "how do I shove this round peg through the square tracing/BPF slot". There's _way_ more we could be doing that would just make our lives easier.