Re: mmap_sem bottleneck

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Oct 17, 2016 at 02:33:53PM +0200, Laurent Dufour wrote:
> Hi all,
> 
> I'm sorry to resurrect this topic, but with the increasing number of
> CPUs, this becomes more frequent that the mmap_sem is a bottleneck
> especially between the page fault handling and the other threads memory
> management calls.
> 
> In the case I'm seeing, there is a lot of page fault occurring while
> other threads are trying to manipulate the process memory layout through
> mmap/munmap.
> 
> There is no *real* conflict between these operations, the page fault are
> done a different page and areas that the one addressed by the mmap/unmap
> operations. Thus threads are dealing with different part of the
> process's memory space. However since page fault handlers and mmap/unmap
> operations grab the mmap_sem, the page fault handling are serialized
> with the mmap operations, which impact the performance on large system.
> 
> For the record, the page fault are done while reading data from a file
> system, and I/O are really impacted by this serialization when dealing
> with a large number of parallel threads, in my case 192 threads (1 per
> online CPU). But the source of the page fault doesn't really matter I guess.
> 
> I took time trying to figure out how to get rid of this bottleneck, but
> this is definitively too complex for me.
> I read this mailing history, and some LWN articles about that and my
> feeling is that there is no clear way to limit the impact of this
> semaphore. Last discussion on this topic seemed to happen last march
> during the LSFMM submit (https://lwn.net/Articles/636334/). But this
> doesn't seem to have lead to major changes, or may be I missed them.
> 
> I'm now seeing that this is a big thing and that it would be hard and
> potentially massively intrusive to get rid of this bottleneck, and I'm
> wondering what could be to best approach here, RCU, range locks, etc..
> 
> Does anyone have an idea ?

If its really just the pagefaults you care about you can have a look at
my speculative page fault stuff that I don't ever seem to get around to
updating :/

Latest version is here:

  https://lkml.kernel.org/r/20141020215633.717315139@infradead.org

Plenty of bits left to sort with that, but the general idea is to use
the split page-table locks (PTLs) as range lock for the mmap_sem.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>