Re: Tools for explaining memory mappings/usage/pressure

[Dan Schatzberg <schatzberg.dan@gmail.com>]

On Sat, Jul 06, 2024 at 01:55:11PM -0700, David Rientjes wrote:
> Rather than hacky scripts that collect things like vmstat, memory.stat, 
> buddyinfo, etc, at regular intervals, it would be preferable to hand off 
> something more complete.  Idea is an open source tool that can be run in 
> the background to collect metrics for the system, NUMA nodes, and memcg 
> hierarchies, as well as potentially from subsystems in the kernel like 
> delay accounting.  IOW, I want to be able to say "install ${tool} and send 
> over the log file."
> 
> Are thre any open source tools that do a good job of this today that I can 
> latch onto?  If not, sounds like I'll be writing one from scratch.  Let me 
> know if there's interest in this as well.
> 
> Thanks!
> 

Hi David,

At meta we have built and deployed Below[1] for this purpose. It's a
tool similar to `top` or others, but can record system state
periodically and allow for replaying. We run this on our production
fleet, periodically recording system state to the local disk. When we
need to debug a machine at a point in the past, we can log in and
replay the state. This uses a TUI (see the link for a demo) to make
navigating the data more natural.

I'm aware of a few other organizations who have also deployed Below,
but tend to run it more in the manner you suggest - have it record
data but then use the snapshot command to export the state (e.g. as if
it was a log file) that can then be viewed off-host. Some
organizations eschew the TUI altogether and export the data to
Prometheus/Grafana.

I'll caution though that having the data is one thing, being able to
interpret it is entirely different. While we try and put the most
useful and easily-understood metrics front-and-center in the TUI,
debugging an issue like you describe would probably require some
domain-expertise.

[1] https://github.com/facebookincubator/below