From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E152FECAAD3 for ; Fri, 2 Sep 2022 01:35:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7246480094; Thu, 1 Sep 2022 21:35:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AD018008D; Thu, 1 Sep 2022 21:35:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5266F80094; Thu, 1 Sep 2022 21:35:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3CF328008D for ; Thu, 1 Sep 2022 21:35:41 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 082C9C1101 for ; Fri, 2 Sep 2022 01:35:41 +0000 (UTC) X-FDA: 79865428482.09.905A80D Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by imf23.hostedemail.com (Postfix) with ESMTP id 6BBC814005A for ; Fri, 2 Sep 2022 01:35:40 +0000 (UTC) Date: Thu, 1 Sep 2022 21:35:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1662082539; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DKbeuQ0KEVmGJmd1nxH6IkYl6hlcE8fBK782zCfa3dY=; b=djK/rzc16o2I5st6iLU+mmFycopk8nXeACxAiaLCYMzWJYltipsNT6kJ+bcD6p5dv0fhQK tsO6ER4nJ6etmNpcfsjSsEGbvTpJzvOlYT1SEGzRruYg/z2yOc4vkCLRQmY8hz09hO4/JQ EYL/angxfkhwhSI7Gv5opBj0F7OyMnA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Steven Rostedt Cc: Suren Baghdasaryan , akpm@linux-foundation.org, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, changbin.du@intel.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, arnd@arndb.de, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-mm@kvack.org, iommu@lists.linux.dev, kasan-dev@googlegroups.com, io-uring@vger.kernel.org, linux-arch@vger.kernel.org, xen-devel@lists.xenproject.org, linux-bcache@vger.kernel.org, linux-modules@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 27/30] Code tagging based latency tracking Message-ID: <20220902013532.6n5cyf3oofntljho@moria.home.lan> References: <20220830214919.53220-1-surenb@google.com> <20220830214919.53220-28-surenb@google.com> <20220901173844.36e1683c@gandalf.local.home> <20220901215438.gy3bgqa4ghhm6ztm@moria.home.lan> <20220901183430.120311ce@gandalf.local.home> <20220901225515.ogg7pyljmfzezamr@moria.home.lan> <20220901202311.546a53b5@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220901202311.546a53b5@gandalf.local.home> X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662082540; a=rsa-sha256; cv=none; b=0soDhb3LirpgVfON5l59yoYtGv+y/nrl872SP/wRXe9ED+O1COyVzPJzrVpGoyq/qXCmKA 6Qk7g+wJVR9YWCLFeV39F7J0KOwZzo/q1IoYCbNdY9AJ4qucVNsXXaeebQALWpBWzdaFNs AaAbFfAMAnSkcbLhJzOGLpnn8wB8K9Y= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="djK/rzc1"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf23.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.121.223.63 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662082540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DKbeuQ0KEVmGJmd1nxH6IkYl6hlcE8fBK782zCfa3dY=; b=n5tr/dCQdvyMGReFUya+hWiTDqm7H+qEP+5nAeLN3QjPRGFhO5QZsUBUI8xa2LyLT7ktYF 48FZAiXMYHnbPPl/k7t/F12ulSeouOexpzMFOt0mWLh7zI9vEPDO5oh5XhEpCg/42CgDjI hW6lzebh+bGvEQDbsr1q0zprBQUzO9s= X-Rspamd-Server: rspam09 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="djK/rzc1"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf23.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.121.223.63 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev X-Rspam-User: X-Stat-Signature: 3456k9c1ggerkb18fq6g1em6t8qdbfbd X-Rspamd-Queue-Id: 6BBC814005A X-HE-Tag: 1662082540-147037 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 01, 2022 at 08:23:11PM -0400, Steven Rostedt wrote: > If ftrace, perf, bpf can't do what you want, take a harder look to see if > you can modify them to do so. Maybe we can use this exchange to make both of our tools better. I like your histograms - the quantiles algorithm I've had for years is janky, I've been meaning to rip that out, I'd love to take a look at your code for that. And having an on/off switch is a good idea, I'll try to add that at some point. Maybe you got some ideas from my stuff too. I'd love to get better tracepoints for measuring latency - what I added to init_wait() and finish_wait() was really only a starting point. Figuring out the right places to measure is where I'd like to be investing my time in this area, and there's no reason we couldn't both be making use of that. e.g. with kernel waitqueues, I looked at hooking prepare_to_wait() first but not all code uses that, init_wait() got me better coverage. But I've already seen that that misses things, too, there's more work to be done. random thought: might try adding a warning in schedule() any time it's called and codetag_time_stats_start() hasn't been called, that'll be a starting point...