Re: [RFC 0/4] mm: Introduce lazy exec permission setting on a page

[Anshuman Khandual <anshuman.khandual@arm.com>]

On 02/14/2019 10:25 PM, Dave Hansen wrote:
> On 2/13/19 8:12 PM, Anshuman Khandual wrote:
>> On 02/13/2019 09:14 PM, Dave Hansen wrote:
>>> On 2/13/19 12:06 AM, Anshuman Khandual wrote:
>>>> Setting an exec permission on a page normally triggers I-cache invalidation
>>>> which might be expensive. I-cache invalidation is not mandatory on a given
>>>> page if there is no immediate exec access on it. Non-fault modification of
>>>> user page table from generic memory paths like migration can be improved if
>>>> setting of the exec permission on the page can be deferred till actual use.
>>>> There was a performance report [1] which highlighted the problem.
>>>
>>> How does this happen?  If the page was not executed, then it'll
>>> (presumably) be non-present which won't require icache invalidation.
>>> So, this would only be for pages that have been executed (and won't
>>> again before the next migration), *or* for pages that were mapped
>>> executable but never executed.
>> I-cache invalidation happens while migrating a 'mapped and executable' page
>> irrespective whether that page was really executed for being mapped there
>> in the first place.
> 
> Ahh, got it.  I also assume that the Accessed bit on these platforms is
> also managed similar to how we do it on x86 such that it can't be used
> to drive invalidation decisions?

Drive I-cache invalidation ? Could you please elaborate on this. Is not that
the access bit mechanism is to identify dirty pages after write faults when
it is SW updated or write accesses when HW updated. In SW updated method, given
PTE goes through pte_young() during page fault. Then how to differentiate exec
fault/access from an write fault/access and decide to invalidate the I-cache.
Just being curious.

> 
>>> Any idea which one it is?
>>
>> I am not sure about this particular reported case. But was able to reproduce
>> the problem through a test case where a buffer was mapped with R|W|X, get it
>> faulted/mapped through write, migrate and then execute from it.
> 
> Could you make sure, please?

The test in the report [1] does not create any explicit PROT_EXEC maps and just
attempts to migrate all pages of the process (which has 10 child processes)
including the exec pages. So the only exec mappings would be the primary text
segment and the mapped shared glibc segment. Looks like the shared libraries
have some mapped pages.

$cat /proc/[PID]/numa_maps  | grep libc

ffffaa4c9000 default file=/lib/aarch64-linux-gnu/libc-2.28.so mapped=150 mapmax=57 N0=150 kernelpagesize_kB=4
ffffaa621000 default file=/lib/aarch64-linux-gnu/libc-2.28.so
ffffaa630000 default file=/lib/aarch64-linux-gnu/libc-2.28.so anon=4 dirty=4 mapmax=11 N0=4 kernelpagesize_kB=4
ffffaa634000 default file=/lib/aarch64-linux-gnu/libc-2.28.so anon=2 dirty=2 mapmax=11 N0=2 kernelpagesize_kB=4

Will keep looking into this.

> 
> Write and Execute at the same time are generally a "bad idea".  Given

But wont this be the case for all run-time generate code which gets written to a
buffer before being executed from there.

> the hardware, I'm not surprised that this problem pops up, but it would
> be great to find out if this is a real application, or a "doctor it
> hurts when I do this."

Is not that a problem though :)

> 
>>> If it's pages that got mapped in but were never executed, how did that
>>> happen?  Was it fault-around?  If so, maybe it would just be simpler to
>>> not do fault-around for executable pages on these platforms.
>> Page can get mapped through a different access (write) without being executed.
>> Even if it got mapped through execution and subsequent invalidation, the
>> invalidation does not have to be repeated again after migration without first
>> getting an exec access subsequently. This series just tries to hold off the
>> invalidation after migration till subsequent exec access.
> 
> This set generally seems to be assuming an environment with "lots of
> migration, and not much execution".  That seems like a kinda odd
> situation to me.

Irrespective of the reported problem which is user driven, there are many kernel
triggered migrations which can accumulate I-cache invalidation cost over time on
a memory heavy system with high number of exec enabled user pages. Will that be
such a rare situation !

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2018-December/620357.html