From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CC17C3DA61 for ; Sun, 21 Jul 2024 23:06:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BAB76B0085; Sun, 21 Jul 2024 19:06:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16ACC6B0089; Sun, 21 Jul 2024 19:06:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 031A06B008C; Sun, 21 Jul 2024 19:06:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D81C26B0085 for ; Sun, 21 Jul 2024 19:06:02 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8ADC7802BE for ; Sun, 21 Jul 2024 23:06:02 +0000 (UTC) X-FDA: 82365294564.29.3411FC6 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf08.hostedemail.com (Postfix) with ESMTP id B0BA9160008 for ; Sun, 21 Jul 2024 23:06:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ivvBoDJE; spf=pass (imf08.hostedemail.com: domain of rientjes@google.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721603138; a=rsa-sha256; cv=none; b=FvB1hvT/jEmkOQXQVhfof2CUR/48qqFHPG5M2Ncn0SD7HAvmvhstry64TN0YW3TIAhvsaP HzTk/rlq0r1Sm7jPdQ1AP3Vk4aLnnWBCM8kAP2aInxyouaqehcw/oTTw9yFMYx702zHT02 xMC5PlO/H6PkoRLnB42AYob59PB+TV8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ivvBoDJE; spf=pass (imf08.hostedemail.com: domain of rientjes@google.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721603138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bf73fbCctAgmy0JhWA+NdnZlwOWLSEFP0MCIt/7TNSI=; b=kBvkt5+W5FKs7hl9huPBMRb4cb53UJe2IG2TFrpTH/r3gOi5AwZrVDU1S3cwunkUn9yYGN OejiikRepYingNS/Ma5FLk6C1FsOE8LE9PGoBfWg2n2MbvcpcZcQH1vgZbDQ9NW3W3zacV zXwUlOMXECFiTsugY4h51bsoEwrwtls= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1fd657c9199so410785ad.1 for ; Sun, 21 Jul 2024 16:06:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721603159; x=1722207959; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=bf73fbCctAgmy0JhWA+NdnZlwOWLSEFP0MCIt/7TNSI=; b=ivvBoDJERfdhYB7USgK+9FwNwd52EnI0VpRv0NcntAqCkezMtlWgE6oGN6+yoEAOKG vMT9eOz5Fm3WCRK7Uoge0oNdxU7lK6X/n6RbjYL3MJ+r7B9pqoxjt1lKIaDU5NA2hyiA O2wuC9GQNtx5PHVVpTWHSbzNEM8iHX8GpgZo4dABgLsY2mMoIkyBpv2jwiaw1bwf9Vf+ at3ScXToCj6Cu9zZ4Zs2FuKHQMt2k6bA5NGV7ihRe2G37BZTRKR8trAjEpMZnsEC4nNt iOOhmYk4W9monpOFMLsL+clZCDL1JHcSb1YRrLLi5F2FRhOhtyNLT+LxqWzXoImD1yy0 /euw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721603159; x=1722207959; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bf73fbCctAgmy0JhWA+NdnZlwOWLSEFP0MCIt/7TNSI=; b=s7Tp/DLCDxmXTQKyvRWPB3dBhBR2TafRnoTjPW1vyl5GZQCde+d9/4SU+X9afeOLjl eDYAoXs06EaXXmCWeYekA0wvAfBVWcbmCWO0ApTfakM7hPClDbNd2WYtJlqNss9ezxeP 7NP2IXnj458QFVekUiLaq2aqBE2g4DrkVPy8rwFE9Ty9yZxIORgp1kOZl7fn8LAfkYiU 8ipHZ3t9rSHdob4j/tNoWkGMoVxCxnTDDxbaWCyRgunezRxdioZm/f5AaErxcncWsT4V 7In2h4suO+rLf0rLyG/Scdj8vEOeJPMOm09ZDjjtEEeGEAqt608whurt3ZYaYRvxddk/ CFMA== X-Forwarded-Encrypted: i=1; AJvYcCUBXBmuvUFNoYNy67L4RMGiosLn0qSAfhNdxZ27a61k6uxp2RFF88mZ9EayWGWb1JL+Q4GHyW7hVqaPbRJyi76wIgw= X-Gm-Message-State: AOJu0Yxy9UqKKbKILYGa7I9mNiVE4Fm81XyWJO2kpyIIMRMmTajSSq2l Xu+aJPt9i4cqM+lDl5w4h7I1sHZuSkgc54f0POWbZOrCeqxnxvCTUIMNOOiiQg== X-Google-Smtp-Source: AGHT+IFJfqyEFlVDD0CbxcRC1ngRemmq0C25ORsa94YVT7m2Uj7YRQWSUrGOaExLyjbBu/E89vb5Yg== X-Received: by 2002:a17:902:ce89:b0:1fc:60f2:a089 with SMTP id d9443c01a7336-1fd7ff46556mr2251795ad.17.1721603158839; Sun, 21 Jul 2024 16:05:58 -0700 (PDT) Received: from [2620:0:1008:15:4c7f:e46f:6233:fd11] ([2620:0:1008:15:4c7f:e46f:6233:fd11]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-79f0520a8d0sm3059893a12.0.2024.07.21.16.05.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 Jul 2024 16:05:58 -0700 (PDT) Date: Sun, 21 Jul 2024 16:05:57 -0700 (PDT) From: David Rientjes To: Dan Schatzberg cc: Michal Hocko , Andrew Morton , Mel Gorman , Balbir Singh , Peter Zijlstra , linux-mm@kvack.org Subject: Re: Tools for explaining memory mappings/usage/pressure In-Reply-To: Message-ID: References: <29c27dab-a590-5df2-c840-279bf9dff090@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Stat-Signature: g9ats8embdzncpunt8bdyuj4pmceks6r X-Rspamd-Queue-Id: B0BA9160008 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721603160-921957 X-HE-Meta: U2FsdGVkX1/l+V0KuXnm56eAqKtxsWLYgQcOHCeYZg2Iyk3+5hR5M5OsPE/TOoeAbudVrtHffAEhA0uOKlUwR8J9eXQpxEREO7x/HsDy17idRLcZFv5N5klyjz/iiY5l/IhCbHhkWr4HwxZNVsI7XH0Uph1XyrfVqUGFjJsu6fbz+xdxvTmbOB4CL3CG//1yKBxl7BFMXzxTfSmhfQYf0YMnua3CCIy0qTXnOLP7zhR3PGNVG5jjDbscscflZl24xP1MBIo2MomDik5LFGj+y4J1fWQdDCYeX8oVo9x4r2/otOrNKdEwo79xLlyBtOquBOctR0NDpn6W4ONfeDsiRYgBp1Pv+AYkst5aXTJw1kfSh1U/NDUrZTngQtyX0uC4nxG+TdeYkMypjnxqP5KazPv2l78poMccMJs3gXcGSRBpl+tQHA07Nf4HRxifzWeLgtJ3bSPGSf8jxUbtQ4sAvadCfdlvxJLoJYbGI5XBrCRam06g/S9G+X+kgDou81N4IK5bs+xU6ntV3ZqBAiHZYGtZr2xpOeSWWEhIPPSRshzqBdkMvUY77abY5rhFB5SxWDha70wYSq/AnRodC28ReaW5ji9KFYksVGfWjMVPrMx/rw0BKBWDoAc6WeawajG1gt1c/586KoeMKYoViBG5cTOSKJ9qXT1sLBO3NgegfCm0OiJ0oeGc73w1dkK5JJT9t775XdJ38ipb4EHE8ZB8f3XnXBJeeZZ5YLb8l3VUH92mr+pvxNhhkELRvfo3njXl8Bvi2IE3G93jSY1mGp3Q8WvLcGJvcLcLx+HFxdPg6bnV6fAZplHzGjgSzsSdbjYyUUpaE47aRB3HSaG1dsiEgORS0GfnUn4iBybVZhJ4ZFxKKM2CPRKQOt7g7wD92iVBbIlHWEV/UVxfjbmxLM9wcntFjKgrST4A3Vjk+E7O39BBgH1OLXLjPviER5EeqHVfp3Uza28sr6B2WY6Y6Qr qEhQ9fwQ JPZDX3ho9XD5NbcJpN2lmFTnnevHVt+2LD4m1O4P3/l4FCLqs6obCZYIxX1hlp/Po2zCWyvC+hy2w/38V0DGeCqGmb6VXEm+f8UKBK5KuYjPXmQT0+40Ul9JQZ6Wxonveh4mrLWLsJu6lJ05sua/FcC/eE02m3BMX/8ucCzipItOr9nkwtFRUFk7p2/njJwqThrNTjLPKRQutGRRF1yPRwb5TkSzg/AzWIzj8yXn9bbpMoFT3XfqGMdlQbC9kSMNRvtT/GlvEqnUW1bXs5HsX3NxJAsPqQteP420wgVbYvjV7oOu86NKlxwM+SqJUzk8Xb8WxygtGaNG8h28I3yZmOeX3YUeC4cFfSumDEbHFVvFCWM7LON6dM/W2WjMiHaAf8x1X6QTy9MVOJB7je7jjhs5jesIF9/zHYi9aCIS+EEn64UcwTdkz4K+sqE4G2236p/cBZRaZUOjvsLOx57+0I2wZOkD5IwhxJ6gT X-Bogosity: Ham, tests=bogofilter, spamicity=0.001090, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 8 Jul 2024, Dan Schatzberg wrote: > On Sat, Jul 06, 2024 at 01:55:11PM -0700, David Rientjes wrote: > > Rather than hacky scripts that collect things like vmstat, memory.stat, > > buddyinfo, etc, at regular intervals, it would be preferable to hand off > > something more complete. Idea is an open source tool that can be run in > > the background to collect metrics for the system, NUMA nodes, and memcg > > hierarchies, as well as potentially from subsystems in the kernel like > > delay accounting. IOW, I want to be able to say "install ${tool} and send > > over the log file." > > > > Are thre any open source tools that do a good job of this today that I can > > latch onto? If not, sounds like I'll be writing one from scratch. Let me > > know if there's interest in this as well. > > > > Thanks! > > > > Hi David, > > At meta we have built and deployed Below[1] for this purpose. It's a > tool similar to `top` or others, but can record system state > periodically and allow for replaying. We run this on our production > fleet, periodically recording system state to the local disk. When we > need to debug a machine at a point in the past, we can log in and > replay the state. This uses a TUI (see the link for a demo) to make > navigating the data more natural. > > I'm aware of a few other organizations who have also deployed Below, > but tend to run it more in the manner you suggest - have it record > data but then use the snapshot command to export the state (e.g. as if > it was a log file) that can then be viewed off-host. Some > organizations eschew the TUI altogether and export the data to > Prometheus/Grafana. > > I'll caution though that having the data is one thing, being able to > interpret it is entirely different. While we try and put the most > useful and easily-understood metrics front-and-center in the TUI, > debugging an issue like you describe would probably require some > domain-expertise. > > [1] https://github.com/facebookincubator/below > Thanks Dan, this is fantastic! I've been playing with it locally. This does indeed appear to meet the exact needs of what I was referring to above, I'm excited that this already exists. Few questions for you: - Do you know of anybody who has deployed this in their guest when running on a public cloud? - Is there a motivation to add this to well known distros so it is "just there" and can run out of the box? There's some configuration and setup that it requires - How receptive are the maintainers to adding new data points, things like additional fields from vmstat, adding in /proc/pagetypeinfo, etc? - Any plans to support cgroup v1? :) Would that be nacked outright? Some customers still run this in their guest - For the "/usr/bin/below record --retain-for-s 604800 --compress" support, is there an appetite for separating this out into its own non-systemd managed process? IOW, the ability to tell the customer "go run 'mini-below' and send over the data" that *just* does the record operation and doesn't require installing/configuring anything? This could be potentially very exciting. Happy to take this discussion off-list as well: if anybody else from this thread (or not yet on this thread) is interested, please let me know so I include you. Thanks!