Re: Tools for explaining memory mappings/usage/pressure

["Vlastimil Babka (SUSE)" <vbabka@kernel.org>]

On 7/6/24 10:55 PM, David Rientjes wrote:
> Hi all,
> 
> I'm trying to crowdsource information on open source tools that can be 
> used directly by customers to explain memory mappings, usage, pressure, 
> etc.
> 
> We encounter both internal and external users that are looking for this 
> insight and it often requires significant engineering time to collect data 
> to make any conclusions.
> 
> A recent example is an external customer that recently upgraded their 
> userspace and started to run into memcg constrained memory pressure that 
> wasn't previously observed.  After handing off a hacky script to run in 
> the background, it was immediately obvious that the source of the direct 
> reclaim was all of the MADV_FREE memory that was sitting around.  
> Converting to MADV_DONTNEED solved their issue.

BTW, was this reported/fixed upstream? Sounds like a bug to me that would
better be fixed than suggesting the MADV_DONTNEED workaround to everyone
from now on.

> A month ago, a different external customer was concerned about increased 
> memory access latency in their guest on some instances although there 
> were no issues observable on the host.  After handing off a hacky script 
> to run in the background, it was immediately obvious that memory 
> fragmentation was resulting in a large disparity in the number of 
> hugepages that were available on some instances.
> 
> Rather than hacky scripts that collect things like vmstat, memory.stat, 
> buddyinfo, etc, at regular intervals, it would be preferable to hand off 
> something more complete.  Idea is an open source tool that can be run in 
> the background to collect metrics for the system, NUMA nodes, and memcg 
> hierarchies, as well as potentially from subsystems in the kernel like 
> delay accounting.  IOW, I want to be able to say "install ${tool} and send 
> over the log file."
> 
> Are thre any open source tools that do a good job of this today that I can 
> latch onto?  If not, sounds like I'll be writing one from scratch.  Let me 
> know if there's interest in this as well.
> 
> Thanks!
>