Re: Tools for explaining memory mappings/usage/pressure

[Dan Schatzberg <schatzberg.dan@gmail.com>]

On Sun, Jul 21, 2024 at 04:05:57PM -0700, David Rientjes wrote:
> Thanks Dan, this is fantastic!  I've been playing with it locally.
> 
> This does indeed appear to meet the exact needs of what I was referring to 
> above, I'm excited that this already exists.
> 
> Few questions for you:

Just a brief preface to my answers: Below is maintained by just a
couple engineers and our primary focus is internal debugging
use-cases. We welcome contributions as expanding Below's user base
leads to benefits for our internal use-cases. I'll try and speak to
what we would and wouldn't welcome, but before embarking on some more
specific work it may be worth circling back with us to avoid
misalignment.

> 
>  - Do you know of anybody who has deployed this in their guest when 
>    running on a public cloud?

I believe so, engineers from Aviatrix have been contributing to Below
recently as they have their customers use below to collect data for
off-host debugging. I've heard anecdotally that Netflix has been using
Below, but not entirely confident in that still being true.

> 
>  - Is there a motivation to add this to well known distros so it is "just
>    there" and can run out of the box?  There's some configuration and 
>    setup that it requires

https://github.com/facebookincubator/below?tab=readme-ov-file#installing

It's packaged for Below, Alpine Linux and Gentoo already. We'd welcome
any additional contributions to package Below for other distros so
long as the maintenance burden is not too high.

> 
>  - How receptive are the maintainers to adding new data points, things 
>    like additional fields from vmstat, adding in /proc/pagetypeinfo, etc?

In general, we welcome contributions adding additional data
collection, so long as it is sufficiently performant (e.g. collecting
data for each thread in the system may require more rigour to ensure
it doesn't blow up storage costs or cpu overhead of running Below) or
at least made optional. Of course we expect this to be done in a
fashion that doesn't overly burden the maintenance of the codebase as
well.

We're a bit more scrutinizing about adding data to the TUI (more
specifically, we scrutinize where the data gets added) just because
adding everyone's personal favorite metric in the most prominent spot
leads to UI clutter and devalues the tool as a visual guide to
debugging.

> 
>  - Any plans to support cgroup v1? :)  Would that be nacked outright?  
>    Some customers still run this in their guest

No plans, but we're not opposed to contributions. I don't think it
would be too challenging, just need to make sure there's some (github)
testing setup for it since we are not running cgroup v1 in our
internal CI.

> 
>  - For the "/usr/bin/below record --retain-for-s 604800 --compress" 
>    support, is there an appetite for separating this out into its own
>    non-systemd managed process?  IOW, the ability to tell the customer
>    "go run 'mini-below' and send over the data" that *just* does the
>    record operation and doesn't require installing/configuring anything?

I think I follow what you're suggesting here - basically something
fully self-contained (relies on no external configuration) to run
below record followed by below snapshot or some way to record directly
to a snapshot so data can be analyzed off-host. That seems perfectly
reasonable. I believe Aviatrix would benefit from making this easier
for their customers as well.