Tools for explaining memory mappings/usage/pressure

[David Rientjes <rientjes@google.com>]

Hi all,

I'm trying to crowdsource information on open source tools that can be 
used directly by customers to explain memory mappings, usage, pressure, 
etc.

We encounter both internal and external users that are looking for this 
insight and it often requires significant engineering time to collect data 
to make any conclusions.

A recent example is an external customer that recently upgraded their 
userspace and started to run into memcg constrained memory pressure that 
wasn't previously observed.  After handing off a hacky script to run in 
the background, it was immediately obvious that the source of the direct 
reclaim was all of the MADV_FREE memory that was sitting around.  
Converting to MADV_DONTNEED solved their issue.

A month ago, a different external customer was concerned about increased 
memory access latency in their guest on some instances although there 
were no issues observable on the host.  After handing off a hacky script 
to run in the background, it was immediately obvious that memory 
fragmentation was resulting in a large disparity in the number of 
hugepages that were available on some instances.

Rather than hacky scripts that collect things like vmstat, memory.stat, 
buddyinfo, etc, at regular intervals, it would be preferable to hand off 
something more complete.  Idea is an open source tool that can be run in 
the background to collect metrics for the system, NUMA nodes, and memcg 
hierarchies, as well as potentially from subsystems in the kernel like 
delay accounting.  IOW, I want to be able to say "install ${tool} and send 
over the log file."

Are thre any open source tools that do a good job of this today that I can 
latch onto?  If not, sounds like I'll be writing one from scratch.  Let me 
know if there's interest in this as well.

Thanks!