From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B37E3CD11DD for ; Thu, 28 Mar 2024 20:22:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BB956B0095; Thu, 28 Mar 2024 16:22:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16CC66B0096; Thu, 28 Mar 2024 16:22:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 034036B0098; Thu, 28 Mar 2024 16:22:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D87EF6B0095 for ; Thu, 28 Mar 2024 16:22:21 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 711D180DF8 for ; Thu, 28 Mar 2024 20:22:21 +0000 (UTC) X-FDA: 81947570082.11.B028044 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf07.hostedemail.com (Postfix) with ESMTP id 78A3D40016 for ; Thu, 28 Mar 2024 20:22:19 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="t1xfM/sk"; spf=pass (imf07.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711657339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SITu3wP6qaILu57znUskVSN6H9osyOHT4Gb41FteW/Y=; b=bPrI2ozMi2D5TMQn9QhHud+iCB6NYpuk/vpc4HnkOYwJb8V1cJ8DiP2cZ+Biexj0GF4z1w QbW3Xk3FEYfj/IVZRkYw+7lvhoXORtkdxPT+dNEcQkcHTgtze54cFGX5aVhzGamTyBzvrN ohx1OV4vQbmKGA7CFO+fwE01jUgMQ9Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711657339; a=rsa-sha256; cv=none; b=zpDunfZz6HG19i04DU1ETzAUxiygZF9ru09dSIBq2X//XXHuaKfpntJjhpMJ1vcp/Z6uqu 2eNCqNxPCTrYtxzjG7D4XxCq7kKqRwLrDZeMwKcyh/lPGU4/SUjMj5Y+T1vXR6d6ISQA/B bf39K+R0au8pyvQ4mou2EYOOodMOjJY= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="t1xfM/sk"; spf=pass (imf07.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev Date: Thu, 28 Mar 2024 16:22:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1711657337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=SITu3wP6qaILu57znUskVSN6H9osyOHT4Gb41FteW/Y=; b=t1xfM/sk4la00t065yw/dchJtSd86DyHJ5hZsC77hI3tD6MWdTT1Xhr+DJZxHogBRqAlv2 VvI/3ZhGyUghQnky940nynaLU+ZcO+Dg2ZNUQXtqMkuZiWU+UN7Tob6gPhBCiYaT9Nv7UV zNUs/Hgip5OP/EHxeC07CyVObPP73YQ= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Tejun Heo Cc: Kemeng Shi , akpm@linux-foundation.org, willy@infradead.org, jack@suse.cz, bfoster@redhat.com, dsterba@suse.com, mjguzik@gmail.com, dhowells@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 0/6] Improve visibility of writeback Message-ID: References: <20240327155751.3536-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 78A3D40016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: jts6ifxdttd5kiugo6ddtdunkhcnw1f4 X-HE-Tag: 1711657339-738468 X-HE-Meta: U2FsdGVkX19a6SPXLlCfXgLLpp8YPwcse+m43SSkJNtLJ/nuuIV58Tjj991ZrKSLk3Swrc+fnuDyS5D43J7FrUpgzGXei3OOT/YMV0Z56sH9lnZmQUqJoa7YEhaoAK+8ahC/vMUmjP2rDpmD87/zTt5EFHlD8/wbmZrWWzDpPQv32FYu3YMCtgoUCcdZlheaRkHZRhJe1BlvLaASFfmQXJG/krH8aXj4pv04zeJlySy9C5VTVHj1N6TyoG47w/HIftZxDvdh1PVtulVKe7YU8QNZFy1yP+Wjxf+VkwtJEobbNT+UAZ/r6S9cgSm+f1DYyYDgozmYbxe0zaLP9kbQ3BLsuTaNnaBzzaWmGghqb7THd3yMKLYigbsuDOdTmNgFNzA86ViMqmNeaju4ZF7d8OHOJJMnMHZ9p8su3tje18q8OzHyNfl5GZxf3fANSKFyAvfJl+dE5bWMsPUQMgsLvJK+duuePUpJcVbvokNU1E6TSRKo6OwO6CwxW8iKFfNGK1IxGHxoi8L8Pg/o/Hf75/lVbQqcywg6gNOZ/1yWC/abCyDwFAuSzGcUU2ax3VrHOje+8ANwHQcj7DXgSuAtwi1dpdXuzRrbYvFj89RD3ldK6bjp5DKRdodX55ahy6HN9tqkb0J9bWKvKUC4ibKqtF21RqpzdRtHzRBE7uaSUZMNZtRoJmpH+bttn+Co8HhiEXAvlf6h0moPHZ8Aceb57Rc8G6dpt7/IXD9nwjyhBsZthAvH2/7/vvUqSjV4md2iFaKA2aCqxeVv+kGRLbMZ7K968f2pSF6N4m92C8THUeu1hQCCFbiEqxO9ByD0sebf5rJffSAEj0WkAlFYnb3a+izWKQbZ1EgB+p5Kl92xKSG7SwJbdR0F6Ao3nUyYX48YjBbRYyRIfttl0fEyCjEmIExWCRtx3sschC8sUCyg1j1k/WVHudUxE1qESjvWgu+aA0PacRBQBtcxMpq0Lh7 uWFccJk/ 0AzQenrIQ5tUMmubNYkCBuJMUJpGMnOekZM8uI/58AfDmdB2aBmqaCjnthuM9umd9f9TYdYrTIqszEvQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 28, 2024 at 10:13:27AM -1000, Tejun Heo wrote: > Hello, > > On Thu, Mar 28, 2024 at 03:55:32PM -0400, Kent Overstreet wrote: > > > On Thu, Mar 28, 2024 at 03:40:02PM -0400, Kent Overstreet wrote: > > > > Collecting latency numbers at various key places is _enormously_ useful. > > > > The hard part is deciding where it's useful to collect; that requires > > > > intimate knowledge of the code. Once you're defining those collection > > > > poitns statically, doing it with BPF is just another useless layer of > > > > indirection. > > > > > > Given how much flexibility helps with debugging, claiming it useless is a > > > stretch. > > > > Well, what would it add? > > It depends on the case but here's an example. If I'm seeing occasional tail > latency spikes, I'd want to know whether there's any correation with > specific types or sizes of IOs and if so who's issuing them and why. With > BPF, you can detect those conditions to tag and capture where exactly those > IOs are coming from and aggregate the result however you like across > thousands of machines in production without anyone noticing. That's useful, > no? That's cool, but really esoteric. We need to be able to answer basic questions and build an overall picture of what the system is doing without having to reach for the big stuff. Most users are never going to touch tracing, let alone BPF; that's too much setup. But I can and do regularly tell users "check this, this and this" and debug things on that basis without ever touching their machine. And basic latency numbers are really easy for users to understand, that makes them doubly worthwhile to collect and make visible. > Also, actual percentile disribution is almost always a lot more insightful > than more coarsely aggregated numbers. We can't add all that to fixed infra. > In most cases not because runtime overhead would be too hight but because > the added interface and code complexity and maintenance overhead isn't > justifiable given how niche, adhoc and varied these use cases get. You can't calculate percentiles accurately and robustly in one pass - that only works if your input data obeys a nice statistical distribution, and the cases we care about are the ones where it doesn't. > > > > > The time stats stuff I wrote is _really_ cheap, and you really want this > > > > stuff always on so that you've actually got the data you need when > > > > you're bughunting. > > > > > > For some stats and some use cases, always being available is useful and > > > building fixed infra for them makes sense. For other stats and other use > > > cases, flexibility is pretty useful too (e.g. what if you want percentile > > > distribution which is filtered by some criteria?). They aren't mutually > > > exclusive and I'm not sure bdi wb instrumentation is on top of enough > > > people's minds. > > > > > > As for overhead, BPF instrumentation can be _really_ cheap too. We often run > > > these programs per packet. > > > > The main things I want are just > > - elapsed time since last writeback IO completed, so we can see at a > > glance if it's stalled > > - time stats on writeback io initiation to completion > > > > The main value of this one will be tracking down tail latency issues and > > finding out where in the stack they originate. > > Yeah, I mean, if always keeping those numbers around is useful for wide > enough number of users and cases, sure, go ahead and add fixed infra. I'm > not quite sure bdi wb stats fall in that bucket given how little attention > it usually gets. I think it should be getting a lot more attention given that memory reclaim and writeback are generally implicated whenever a user complains about their system going out to lunch.