[Issue] mprotect+madvise may be better than mmap for permission changes and page zeroing

[Issue] mprotect+madvise may be better than mmap for permission changes and page zeroing

[Lance Yang]

Hello Everyone,

I've noticed that using mprotect(PROT_NONE) with
madvise(MADV_DONTNEED) is much faster than
mmap(PROT_NONE, MAP_FIXED) alone for changing
permissions and zeroing pages.

I have maintained a chunk-allocator internally at the
company. It allocates a chunk using
mmap(PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0),
releases a chunk using mmap(PROT_NONE, MAP_ANON|MAP_FIXED|MAP_PRIVATE, -1, 0),
and reuses a chunk using mprotect(PROT_READ|PROT_WRITE).
Recently, I replaced mmap(PROT_NONE, MAP_FIXED) with
mprotect(PROT_NONE) + madvise(MADV_DONTNEED) to
reduce the latency of releasing chunks.

Test code:
https://github.com/ioworker0/mmapvsmprotect/blob/main/test2.c

Here are the test results on my machine:
CPU: AMD EPYC 7R13 Processor
Kernel: 6.2.0
Elapsed Time for mprotect+madvise: 3670 nanoseconds
Elapsed Time for mmap: 5520 nanoseconds

Thanks for your time!
Lance Yang

Re: [Issue] mprotect+madvise may be better than mmap for permission changes and page zeroing

[David Hildenbrand]

On 25.01.24 13:28, Lance Yang wrote:
> Hello Everyone,
> 
> I've noticed that using mprotect(PROT_NONE) with
> madvise(MADV_DONTNEED) is much faster than
> mmap(PROT_NONE, MAP_FIXED) alone for changing
> permissions and zeroing pages.
> 
> I have maintained a chunk-allocator internally at the
> company. It allocates a chunk using
> mmap(PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0),
> releases a chunk using mmap(PROT_NONE, MAP_ANON|MAP_FIXED|MAP_PRIVATE, -1, 0),
> and reuses a chunk using mprotect(PROT_READ|PROT_WRITE).
> Recently, I replaced mmap(PROT_NONE, MAP_FIXED) with
> mprotect(PROT_NONE) + madvise(MADV_DONTNEED) to
> reduce the latency of releasing chunks.
> 
> Test code:
> https://github.com/ioworker0/mmapvsmprotect/blob/main/test2.c
> 
> Here are the test results on my machine:
> CPU: AMD EPYC 7R13 Processor
> Kernel: 6.2.0
> Elapsed Time for mprotect+madvise: 3670 nanoseconds
> Elapsed Time for mmap: 5520 nanoseconds

mprotect+madvise won't free page tables, mmap will. That's the biggest 
difference.

-- 
Cheers,

David / dhildenb

Re: [Issue] mprotect+madvise may be better than mmap for permission changes and page zeroing

[Lance Yang]

Thanks!

On Thu, Jan 25, 2024 at 8:44 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 25.01.24 13:28, Lance Yang wrote:
> > Hello Everyone,
> >
> > I've noticed that using mprotect(PROT_NONE) with
> > madvise(MADV_DONTNEED) is much faster than
> > mmap(PROT_NONE, MAP_FIXED) alone for changing
> > permissions and zeroing pages.
> >
> > I have maintained a chunk-allocator internally at the
> > company. It allocates a chunk using
> > mmap(PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0),
> > releases a chunk using mmap(PROT_NONE, MAP_ANON|MAP_FIXED|MAP_PRIVATE, -1, 0),
> > and reuses a chunk using mprotect(PROT_READ|PROT_WRITE).
> > Recently, I replaced mmap(PROT_NONE, MAP_FIXED) with
> > mprotect(PROT_NONE) + madvise(MADV_DONTNEED) to
> > reduce the latency of releasing chunks.
> >
> > Test code:
> > https://github.com/ioworker0/mmapvsmprotect/blob/main/test2.c
> >
> > Here are the test results on my machine:
> > CPU: AMD EPYC 7R13 Processor
> > Kernel: 6.2.0
> > Elapsed Time for mprotect+madvise: 3670 nanoseconds
> > Elapsed Time for mmap: 5520 nanoseconds
>
> mprotect+madvise won't free page tables, mmap will. That's the biggest
> difference.
>
> --
> Cheers,
>
> David / dhildenb
>