From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05C0CC3DA59 for ; Mon, 22 Jul 2024 22:15:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 752DA6B0089; Mon, 22 Jul 2024 18:15:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 702736B008A; Mon, 22 Jul 2024 18:15:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CA296B008C; Mon, 22 Jul 2024 18:15:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B95C6B0089 for ; Mon, 22 Jul 2024 18:15:44 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D9F6D41A89 for ; Mon, 22 Jul 2024 22:15:43 +0000 (UTC) X-FDA: 82368796566.09.095D524 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf13.hostedemail.com (Postfix) with ESMTP id 0AAF620011 for ; Mon, 22 Jul 2024 22:15:41 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QFRJubF+; spf=pass (imf13.hostedemail.com: domain of schatzberg.dan@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=schatzberg.dan@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721686518; a=rsa-sha256; cv=none; b=V4RCrsCWNQTKJ6892AhDxsdVkh93cn9z0fAvzMxrDhTcRkYpadklMqRnc/6R170YocoVgf ubx+4D1FAUq8b1cWhwnIQ8VUfs76rPEwks7GUbADp+hq52SMW05mcOg5QdyzmggGgMOZ90 5sOikO+Xr4yI3UsR2WFUW/8v5xo93js= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QFRJubF+; spf=pass (imf13.hostedemail.com: domain of schatzberg.dan@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=schatzberg.dan@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721686518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1dBf+57GJV8yrC/Dvux4h+ZiwhwmUiRNCMjM/iqEHfI=; b=KX+LttXX1KGfCXoExWXvSyHyNzcEPdw1baWNeVuD/bV+JAEQnuM8t6iHiY09JgELGLlOI0 6aZar2w9o5KQ6Oymbm9KmRllDd+M/wtKiebOYQKbmAtWEBXAvssOhtpUY0vaUf73vs+7+Q OT5cGP3pHAj5JGe/WKp6+YJVqqFW4xY= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1fc56fd4de1so1496515ad.0 for ; Mon, 22 Jul 2024 15:15:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721686541; x=1722291341; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=1dBf+57GJV8yrC/Dvux4h+ZiwhwmUiRNCMjM/iqEHfI=; b=QFRJubF+gNNCFxinJ8ojumDIryFgltqUwZfO9bcKSZfPhWxdXScpVwuBZvvNDwEZGq l9qsOJ5nrYW19RSUH1y3V8uJ4Dhr6DQtRWqAsG06i9luwf/8uqPRwaHd/QwT7AYMNQ2y YGoWM87uuPVh1tSngz07LkFuYK+2g5lz+5PKMzgdc90gYITROKpX6k2q1PtHj63YExFJ 5kbMGyz7So+h40c1bd5vd5cuDdxmnQjT62IVdmUPB6if3aiTeRGRONM3BT5Ylb/GvlPT nW6nDJRVH19LhhdlXbKFZC/Cb8tIf5qYWbnhvlM3et0SQ6K4GdXZusIBeqmc2zGBTTDD 1kBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721686541; x=1722291341; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=1dBf+57GJV8yrC/Dvux4h+ZiwhwmUiRNCMjM/iqEHfI=; b=S6qlHep4ALMKKCW5/fA76omzyM81q+YAyQY3KbS0WBe+G+Kv4SG6DaVIPT3y5frgKT wOIUQYPQisW8mqA8mOq7El3U6WJQJrks4lcDsV79W4yvcyYkYQ2nuXyq8aP9hBDNez2Y 8FD69XN4U+0J02Vtve+D5H6ubtRZFucV4nTWRALGDGjy6surQ2XIvmjg2y1t8NN7F9lM lJGpi4LLLswippS5awQICTjcpcqgn9UjxQYjh55+stmYlUcCrza2sKma0ea8XdbCDWWO Ldcw0LBMLN77jkIIHa9bmBw9zjpusrtDP/WiLUt4nVFtCmBRe0Jvr6UN1bQSbi0faIHW ITPQ== X-Forwarded-Encrypted: i=1; AJvYcCUx/vEGatxZ4xBC2vRKsXBX6j6fweYX8jLjWkQ7CwgQvXL2YGPFrl7rVeLgW/PhQsCNo/fAmk2GmCI6f4aTLy4CY9M= X-Gm-Message-State: AOJu0YxEUrZCs3IXbUeTB+7woy3HJpsS6Hp1irdXG5UFSBc+I3bk0Cjw nvahHJCA9I7qVIDkg1xQueX6QzHTQQR+1k18WV5jQ1cbMrG4A3LQ X-Google-Smtp-Source: AGHT+IHk0XvkbpWGkRtH+ZS8BzVZyoTYbmOO2AJmzjl5P8DFexiKxzco5WRPX5Y3cOv/ENR/RiXZ5g== X-Received: by 2002:a17:903:120b:b0:1fb:2ebc:d17a with SMTP id d9443c01a7336-1fdb95814f0mr4152205ad.23.1721686540588; Mon, 22 Jul 2024 15:15:40 -0700 (PDT) Received: from dschatzberg-fedora-PF3DHTBV ([2620:10d:c090:500::7:101f]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fd6f318457sm60180855ad.134.2024.07.22.15.15.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jul 2024 15:15:40 -0700 (PDT) Date: Mon, 22 Jul 2024 18:15:37 -0400 From: Dan Schatzberg To: David Rientjes Cc: Michal Hocko , Andrew Morton , Mel Gorman , Balbir Singh , Peter Zijlstra , linux-mm@kvack.org Subject: Re: Tools for explaining memory mappings/usage/pressure Message-ID: References: <29c27dab-a590-5df2-c840-279bf9dff090@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: os8qtngkphco6aopuntyrz9scwcq7oy5 X-Rspamd-Queue-Id: 0AAF620011 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721686541-905614 X-HE-Meta: U2FsdGVkX19pDbEMbu5aRkNJM1eO4LB8tf1dr2qeFp05Y8oakTaCMmjvL883flC7iuCfbPjXwUyOlVQl0VCwMhWQI2WvX/qnuS5ScTEczVFrcy0Fk44S+WdvTsMgjmH+ScICTNcytp7rqj8X0ZqXd4CyyFTw4p11MgZtX2458lDFTNvlb6Of74ILlV7/N015ikvsvCa6h2dVNAgzK/mP1u3oqAajtpIm8cTwMPYqd7pPB/BrC0qUGpu4V3R6zazGH0aRUIesswBGkVyHAHXxS3QjRar7/pbd9UYS00KeCLn4GxY+CTOFYjz0eL5hhj+r3RF4fm3YF4dNRwNtTtXQrlpxfqkH7sUgbPBOM9+mJYOn1dGjEoOLCJLWnORWDE+FKtbX6V9avc6u+A++0xPhRyNHByepzSwv+yozlqQewhRJ2df5A34MHhykdL/fbKpOhDCsuBZ2ZFo99z23g1cj9OlY4xG9QyPBaSLuByHCpPaH0W+roumtIjoy7rG2Pn+X+J6hqGKoz+J5THhMBheKOKN9U+6AQwUKvtP3RbTc2cErJj0jozhdVPvR1z5kiktC54yF71vTaD14YDtU7/qlq1qD8MnkooFDohS2nF3fn1KGQYRsVV737XwylR3c6N0yTukSrps2ijNamYBGkne0SBOhNvYBKbaALtDFziHHGAlSRUAddxmoFcm/wraYGfT20hcbnXtv6nqqKmVZm/+Js83kXdGDJGRMWKaOoeh7fZ5twGSwISu7xrVLREEMHmw52FnTolgQ60FVnSi46v3QKg3XyXfR3FEHSMotyaQEaD9IbL4ZQyxK2kr7Op8YFYzSpPB87oxmHnQ4oMRCeOCIdY8ZNW0ZgzduBfajFQGnRxwjKi63Y1q4XxDzptX/3sU3Lk4GPvhO1pO9QTh49+91FIHWsGYhUDrfBm+7Y9FYtNUCrNb6YKfVGIYRwMDEU0l5v/8X8IttJENevYk1jju RmGOP5yw XwC55DKGthmIqNNQ88IsmPZTOkM7TSJEzgCwwdrweWxtLGgVO9dui8ZE7kyFRL6wiLBIl+1wDKFM0fyeOrjquBMoFO3+TDlZrnSgBnn7Uf055mN2chjxWEOliYnrp1Ak6hmNBQgjGs0s9jizGVj6oTOo9y+HxwoyLnUArEOK81wStH2queAem+ZF4qEJ+Y+r5dCk1BFt8ICSX0ac9ag2C2hU28dscRN0T3GyG/a7ummfteL7JEbxD1z+olaJCfReB2rD3DdzYiEfQTOsh6hHpHuH/MCebp363TJybWdZD4qGaeMbNjXbJnQtqtjRWqHj6XCgnfxGkbOLNSnU1uOOPCgG8L1372X2P/FlAM/++fkaH/sgDlfNpLX0HkpI/EOzlnm6sqCMgok/uAzI6+/+9bn09//4odBP4QKyGSJ6jl/n3FpVubO3vGniZxan0eSeenitiqxvqhMmMavW10qOaP2irux4nsCbxSck1fG42iuXxQ48ldVHXmL/wlz9jZBuI9Ndv2iH8MB8CjlV3nX0Ca3Fd5aYipAFqAtzqXA99tYM7U4k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.016540, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Jul 21, 2024 at 04:05:57PM -0700, David Rientjes wrote: > Thanks Dan, this is fantastic! I've been playing with it locally. > > This does indeed appear to meet the exact needs of what I was referring to > above, I'm excited that this already exists. > > Few questions for you: Just a brief preface to my answers: Below is maintained by just a couple engineers and our primary focus is internal debugging use-cases. We welcome contributions as expanding Below's user base leads to benefits for our internal use-cases. I'll try and speak to what we would and wouldn't welcome, but before embarking on some more specific work it may be worth circling back with us to avoid misalignment. > > - Do you know of anybody who has deployed this in their guest when > running on a public cloud? I believe so, engineers from Aviatrix have been contributing to Below recently as they have their customers use below to collect data for off-host debugging. I've heard anecdotally that Netflix has been using Below, but not entirely confident in that still being true. > > - Is there a motivation to add this to well known distros so it is "just > there" and can run out of the box? There's some configuration and > setup that it requires https://github.com/facebookincubator/below?tab=readme-ov-file#installing It's packaged for Below, Alpine Linux and Gentoo already. We'd welcome any additional contributions to package Below for other distros so long as the maintenance burden is not too high. > > - How receptive are the maintainers to adding new data points, things > like additional fields from vmstat, adding in /proc/pagetypeinfo, etc? In general, we welcome contributions adding additional data collection, so long as it is sufficiently performant (e.g. collecting data for each thread in the system may require more rigour to ensure it doesn't blow up storage costs or cpu overhead of running Below) or at least made optional. Of course we expect this to be done in a fashion that doesn't overly burden the maintenance of the codebase as well. We're a bit more scrutinizing about adding data to the TUI (more specifically, we scrutinize where the data gets added) just because adding everyone's personal favorite metric in the most prominent spot leads to UI clutter and devalues the tool as a visual guide to debugging. > > - Any plans to support cgroup v1? :) Would that be nacked outright? > Some customers still run this in their guest No plans, but we're not opposed to contributions. I don't think it would be too challenging, just need to make sure there's some (github) testing setup for it since we are not running cgroup v1 in our internal CI. > > - For the "/usr/bin/below record --retain-for-s 604800 --compress" > support, is there an appetite for separating this out into its own > non-systemd managed process? IOW, the ability to tell the customer > "go run 'mini-below' and send over the data" that *just* does the > record operation and doesn't require installing/configuring anything? I think I follow what you're suggesting here - basically something fully self-contained (relies on no external configuration) to run below record followed by below snapshot or some way to record directly to a snapshot so data can be analyzed off-host. That seems perfectly reasonable. I believe Aviatrix would benefit from making this easier for their customers as well.