From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36206C433E3 for ; Fri, 29 May 2020 15:05:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7B1220814 for ; Fri, 29 May 2020 15:05:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="mOccdqbR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7B1220814 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3269C8001A; Fri, 29 May 2020 11:05:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D75380010; Fri, 29 May 2020 11:05:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1ED188001A; Fri, 29 May 2020 11:05:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id 077D180010 for ; Fri, 29 May 2020 11:05:55 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A42CBFB6B for ; Fri, 29 May 2020 15:05:54 +0000 (UTC) X-FDA: 76870081428.15.quill86_5fad6b79b240f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 2CFC3180D0FA1 for ; Fri, 29 May 2020 15:04:18 +0000 (UTC) X-HE-Tag: quill86_5fad6b79b240f X-Filterd-Recvd-Size: 5694 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 29 May 2020 15:04:17 +0000 (UTC) Received: from devnote2 (NE2965lan1.rev.em-net.ne.jp [210.141.244.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 96E8120776; Fri, 29 May 2020 15:04:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1590764656; bh=FciU3g6BFTgiqn2lv6NpBYEKauejyEmufm3hNL8Ykhw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=mOccdqbR4q2OaPnq+gKf0CMp04pxPP9dRjCqJkjvJEm5BcIIeYQmSgcFOwyVWbsbJ vdDlM5hq7eOD3+Nqh3E4yDijWUSIjHRF0EfTJcRZh6WVD7WknH498tsz7vXJZQ5vyU KoHxpv5og7v7KHsIufyikSgBzfg9QtczKaEeVLDE= Date: Sat, 30 May 2020 00:03:59 +0900 From: Masami Hiramatsu To: Peter Zijlstra Cc: Axel Rasmussen , Steven Rostedt , Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Vlastimil Babka , Will Deacon , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Tal Gilboa , Thomas Gleixner , Uwe =?UTF-8?B?S2xlaW5lLUvDtm5pZw==?= , Vincenzo Frascino , Yang Shi , Yu Zhao , Tom Zanussi Subject: Re: [PATCH v2 0/7] Add histogram measuring mmap_lock contention latency Message-Id: <20200530000359.519f38720ab457435ecf7b6f@kernel.org> In-Reply-To: <20200529080957.GI706495@hirez.programming.kicks-ass.net> References: <20200528235238.74233-1-axelrasmussen@google.com> <20200528202435.65396221@oasis.local.home> <20200529080957.GI706495@hirez.programming.kicks-ass.net> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 2CFC3180D0FA1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 29 May 2020 10:09:57 +0200 Peter Zijlstra wrote: > On Thu, May 28, 2020 at 06:39:08PM -0700, Axel Rasmussen wrote: > > > The use case we have in mind for this is to enable this instrumentation > > widely in Google's production fleet. Internally, we have a userspace thing > > which scrapes these metrics and publishes them such that we can look at > > aggregate metrics across our fleet. The thinking is that mechanisms like > > lockdep or getting histograms with e.g. BPF attached to the tracepoint > > introduces too much overhead for this to be viable. (Although, granted, I > > don't have benchmarks to prove this - if there's skepticism, I can produce > > such a thing - or prove myself wrong and rethink my approach. :) ) > > Whichever way around; I don't believe in special instrumentation like > this. We'll grow a thousand separate pieces of crap if we go this route. > > Next on, someone will come and instrument yet another lock, with yet > another 1000 lines of gunk. > > Why can't you kprobe the mmap_lock things and use ftrace histograms? +1. As far as I can see the series, if you want to make a histogram of the duration of acquiring locks, you might only need 7/7 (but this is a minimum subset.) I recommend you to introduce a set of tracepoints -- start-locking, locked, and released so that we can investigate which process is waiting for which one. Then you can use either bpf or ftrace to make a histogram easily. Thank you, -- Masami Hiramatsu