memfd and transparent hugepages

memfd and transparent hugepages

[Avi Kivity]

#define _GNU_SOURCE
#include <sys/mman.h>
#include <linux/memfd.h>
#include <unistd.h>
#include <stddef.h>

int main(int ac, char** av) {
    size_t memsz = (size_t)2048 << 20;
    int fd = memfd_create("memory", MFD_CLOEXEC);
    ftruncate(fd, memsz);
    void* p = mmap((void*)0x40000000, memsz, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FILE, fd, 0);
    madvise(p, memsz, MADV_HUGEPAGE);
    madvise(p, memsz, MADV_POPULATE_WRITE);
    madvise(p, memsz, MADV_COLLAPSE);
    pause();
    return 0;
}

While memfd is documented as using anonymous pages, AnonHugePages shows
as zero.

If memfd incompatible with transparent hugepages?

Re: memfd and transparent hugepages

[David Hildenbrand]

On 18.11.24 15:13, Avi Kivity wrote:
> #define _GNU_SOURCE
> #include <sys/mman.h>
> #include <linux/memfd.h>
> #include <unistd.h>
> #include <stddef.h>
> 
> int main(int ac, char** av) {
>      size_t memsz = (size_t)2048 << 20;
>      int fd = memfd_create("memory", MFD_CLOEXEC);
>      ftruncate(fd, memsz);
>      void* p = mmap((void*)0x40000000, memsz, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FILE, fd, 0);
>      madvise(p, memsz, MADV_HUGEPAGE);
>      madvise(p, memsz, MADV_POPULATE_WRITE);
>      madvise(p, memsz, MADV_COLLAPSE);
>      pause();
>      return 0;
> }

Hi,

> 
> While memfd is documented as using anonymous pages, AnonHugePages shows
> as zero.

it's an anonymous file, not anonymous memory. memfd is shmem in diguise.

> 
> If memfd incompatible with transparent hugepages?

Shmem supports THP.

But you have one flaws in your program: you have to mmap it MAP_SHARED. 
It's shmem.

Otherwise during MADV_POPULATE_WRITE you're simply allocating a shmem 
page (pageache), and the COW it to map an anonymous page in your page 
tables, resulting in a double memory consumption (pagecache+anonymous 
memory). khugepaged does currently not operate on such VMAs.

If you use MAP_SHARED in your example above:

$ cat /proc/`pgrep tst`/smaps_rollup
00400000-7ffc2770d000 ---p 00000000 00:00 0 
[rollup]
Rss:             2098460 kB
Pss:             2097269 kB
Pss_Dirty:       2097248 kB
Pss_Anon:             96 kB
Pss_File:             21 kB
Pss_Shmem:       2097152 kB
Shared_Clean:       1200 kB
Shared_Dirty:          0 kB
Private_Clean:        12 kB
Private_Dirty:   2097248 kB
Referenced:      2098460 kB
Anonymous:            96 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:  2097152 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB



-- 
Cheers,

David / dhildenb

Re: memfd and transparent hugepages

[Avi Kivity]

On Mon, 2024-11-18 at 20:42 +0100, David Hildenbrand wrote:
> On 18.11.24 15:13, Avi Kivity wrote:
> > #define _GNU_SOURCE
> > #include <sys/mman.h>
> > #include <linux/memfd.h>
> > #include <unistd.h>
> > #include <stddef.h>
> > 
> > int main(int ac, char** av) {
> >      size_t memsz = (size_t)2048 << 20;
> >      int fd = memfd_create("memory", MFD_CLOEXEC);
> >      ftruncate(fd, memsz);
> >      void* p = mmap((void*)0x40000000, memsz, PROT_READ|PROT_WRITE,
> > MAP_PRIVATE|MAP_FILE, fd, 0);
> >      madvise(p, memsz, MADV_HUGEPAGE);
> >      madvise(p, memsz, MADV_POPULATE_WRITE);
> >      madvise(p, memsz, MADV_COLLAPSE);
> >      pause();
> >      return 0;
> > }
> 
> Hi,
> 
> > 
> > While memfd is documented as using anonymous pages, AnonHugePages
> > shows
> > as zero.
> 
> it's an anonymous file, not anonymous memory. memfd is shmem in
> diguise.
> 
> > 
> > If memfd incompatible with transparent hugepages?
> 
> Shmem supports THP.
> 
> But you have one flaws in your program: you have to mmap it
> MAP_SHARED. 
> It's shmem.
> 
> Otherwise during MADV_POPULATE_WRITE you're simply allocating a shmem
> page (pageache), and the COW it to map an anonymous page in your page
> tables, resulting in a double memory consumption (pagecache+anonymous
> memory). khugepaged does currently not operate on such VMAs.
> 
> If you use MAP_SHARED in your example above:
> 


Thanks a lot - this works.

In case anyone's interested, I'm using this to provide the equivalent
of vmalloc() in my application. It usually uses a large THP arena
(consuming all of the machine's memory) to allocate from, but sometimes
I need large contiguous memory areas for third-party libraries. I can't
easily allocate large contiguous areas, but 128kB allocations are fine.
So I allocate a few of those, and use mmap(MAP_FIXED) to stitch
together several 128kB allocations into a single large contiguous map.