Re: [LSF/MM/BPF TOPIC] Single Owner Memory

[Matthew Wilcox <willy@infradead.org>]

On Mon, Feb 20, 2023 at 02:10:24PM -0500, Pasha Tatashin wrote:
> Within Google the vast majority of memory, over 90% has a single
> owner. This is because most of the jobs are not multi-process but
> instead multi-threaded. The examples of single owner memory
> allocations are all tcmalloc()/malloc() allocations, and
> mmap(MAP_ANONYMOUS | MAP_PRIVATE) allocations without forks. On the
> other hand, the struct page metadata that is shared for all types of
> memory takes 1.6% of the system memory. It would be reasonable to find
> ways to optimize memory such that the common som case has a reduced
> amount of metadata.
> 
> This would be similar to HugeTLB and DAX that are treated as special
> cases, and can release struct pages for the subpages back to the
> system.

DAX can't, unless something's changed recently.  You're referring to
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP

> The proposal is to discuss a new som driver that would use HugeTLB as
> a source of 2M chunks. When user creates a som memory, i.e.:
> 
> mmap(MAP_ANONYMOUS | MAP_PRIVATE);
> madvise(mem, length, MADV_DONTFORK);
> 
> A vma from the som driver is used instead of regular anon vma.

That's going to be "interesting".  The VMA is already created with
the call to mmap(), and madvise has not traditionally allowed drivers
to replace a VMA.  You might be better off creating a /dev/som and
hacking the malloc libraries to pass an fd from that instead of passing
MAP_ANONYMOUS.

> The discussion should include the following topics:
> -  Interaction with folio and the proposed struct page {memdesc}.
> - Handling for migrate_pages() and friends.
> - Handling for FOLL_PIN and FOLL_LONGTERM.
> - What type of madvise() properties the som memory should handle

Obviously once we get to dynamically allocated memdescs, this whole
thing goes away, so I'm not excited about making big changes to the
kernel to support this.

The savings you'll see are 6 pages (24kB) per 2MB allocated (1.2%).
That's not nothing, but it's not huge either.