Re: Tools for explaining memory mappings/usage/pressure

[David Rientjes <rientjes@google.com>]

On Mon, 8 Jul 2024, Dan Schatzberg wrote:

> On Sat, Jul 06, 2024 at 01:55:11PM -0700, David Rientjes wrote:
> > Rather than hacky scripts that collect things like vmstat, memory.stat, 
> > buddyinfo, etc, at regular intervals, it would be preferable to hand off 
> > something more complete.  Idea is an open source tool that can be run in 
> > the background to collect metrics for the system, NUMA nodes, and memcg 
> > hierarchies, as well as potentially from subsystems in the kernel like 
> > delay accounting.  IOW, I want to be able to say "install ${tool} and send 
> > over the log file."
> > 
> > Are thre any open source tools that do a good job of this today that I can 
> > latch onto?  If not, sounds like I'll be writing one from scratch.  Let me 
> > know if there's interest in this as well.
> > 
> > Thanks!
> > 
> 
> Hi David,
> 
> At meta we have built and deployed Below[1] for this purpose. It's a
> tool similar to `top` or others, but can record system state
> periodically and allow for replaying. We run this on our production
> fleet, periodically recording system state to the local disk. When we
> need to debug a machine at a point in the past, we can log in and
> replay the state. This uses a TUI (see the link for a demo) to make
> navigating the data more natural.
> 
> I'm aware of a few other organizations who have also deployed Below,
> but tend to run it more in the manner you suggest - have it record
> data but then use the snapshot command to export the state (e.g. as if
> it was a log file) that can then be viewed off-host. Some
> organizations eschew the TUI altogether and export the data to
> Prometheus/Grafana.
> 
> I'll caution though that having the data is one thing, being able to
> interpret it is entirely different. While we try and put the most
> useful and easily-understood metrics front-and-center in the TUI,
> debugging an issue like you describe would probably require some
> domain-expertise.
> 
> [1] https://github.com/facebookincubator/below
> 

Thanks Dan, this is fantastic!  I've been playing with it locally.

This does indeed appear to meet the exact needs of what I was referring to 
above, I'm excited that this already exists.

Few questions for you:

 - Do you know of anybody who has deployed this in their guest when 
   running on a public cloud?

 - Is there a motivation to add this to well known distros so it is "just
   there" and can run out of the box?  There's some configuration and 
   setup that it requires

 - How receptive are the maintainers to adding new data points, things 
   like additional fields from vmstat, adding in /proc/pagetypeinfo, etc?

 - Any plans to support cgroup v1? :)  Would that be nacked outright?  
   Some customers still run this in their guest

 - For the "/usr/bin/below record --retain-for-s 604800 --compress" 
   support, is there an appetite for separating this out into its own
   non-systemd managed process?  IOW, the ability to tell the customer
   "go run 'mini-below' and send over the data" that *just* does the
   record operation and doesn't require installing/configuring anything?

This could be potentially very exciting.

Happy to take this discussion off-list as well: if anybody else from this 
thread (or not yet on this thread) is interested, please let me know so I 
include you.

Thanks!