Re: [RFC 0/2] New MAP_PMEM_AWARE mmap flag

[Andy Lutomirski <luto@amacapital.net>]

On Fri, Mar 11, 2016 at 11:07 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Thu, Mar 10, 2016 at 10:44 PM, Andy Lutomirski <luto@kernel.org> wrote:
>> On 02/21/2016 09:03 AM, Boaz Harrosh wrote:
>>>
>>> Hi all
>>>
>>> Recent DAX code fixed the cl_flushing ie durability of mmap access
>>> of direct persistent-memory from applications. It uses the radix-tree
>>> per inode to track the indexes of a file that where page-faulted for
>>> write. Then at m/fsync time it would cl_flush these pages and clean
>>> the radix-tree, for the next round.
>>>
>>> Sigh, that is life, for legacy applications this is the price we must
>>> pay. But for NV aware applications like nvml library, we pay extra extra
>>> price, even if we do not actually call m/fsync eventually. For these
>>> applications these extra resources and especially the extra radix locking
>>> per page-fault, costs a lot, like x3 a lot.
>>>
>>> What we propose here is a way for those applications to enjoy the
>>> boost and still not sacrifice any correctness of legacy applications.
>>> Any concurrent access from legacy apps vs nv-aware apps even to the same
>>> file / same page, will work correctly.
>>>
>>> We do that by defining a new MMAP flag that is set by the nv-aware
>>> app. this flag is carried by the VMA. In the dax code we bypass any
>>> radix handling of the page if this flag is set. Those pages accessed
>>> *without*
>>> this flag will be added to the radix-tree, those with will not.
>>> At m/fsync time if the radix tree is then empty nothing will happen.
>>>
>>
>> I'm a little late to the party, but let me offer a variant that might be
>> considerably safer:
>>
>> Add a flag MAP_DAX_WRITETHROUGH (name could be debated -- MAP_DAX_FASTFLUSH
>> might be more architecture-neutral, but I'm only familiar with the x86
>> semantics).
>>
>> MAP_DAX_WRITETHROUGH does whatever is needed to ensure that writing through
>> the mapping and then calling fsync is both safe and fast.  On x86, it would
>> (surprise, surprise!) map the pages writethrough and skip adding them to the
>> radix tree.  fsync makes sure to do sfence before pcommit.
>>
>> This is totally safe.  You *can't* abuse this to cause fsync to leave
>> non-persistent dirty cached data anywhere.
>>
>> It makes sufficiently DAX-aware applications very fast.  Reads are
>> unaffected, and non-temporal writes should be the same speed as they are
>> under any other circumstances.
>>
>> It makes applications that set it blindly very slow.  Applications that use
>> standard writes (i.e. plain stores that are neither fast string operations
>> nor explicit non-temporal writes) will suffer.  But they'll still work
>> correctly.
>>
>> Applications that want a WB mapping with manually-managed persistence can
>> still do it, but fsync will be slow.  Adding an fmetadatasync() for their
>> benefit might be a decent idea, but it would just be icing on the cake.
>>
>> Unlike with MAP_DAX_AWARE, there's no issue with malicious users who map the
>> thing with the wrong flag, write, call fsync, and snicker because now the
>> other applications might read data and be surprised that the data they just
>> read isn't persistent even if they subsequently call fsync.
>>
>> There would be details to be hashed out in case a page is mapped normally
>> and with MAP_DAX_WRITETHROUGH in separate mappings.
>>
>
> Interesting...
>
> The mixed mapping problem is made slightly more difficult by the fact
> that we add persistent memory to the direct-map when allocating struct
> page, but probably not insurmountable.  Also, this still has the
> syscall overhead that a MAP_SYNC semantic eliminates, but we need to
> collect numbers to see if that matters.
>
> However, chatting with Andy R. about the NVML use case, the library
> alternates between streaming non-temporal writes and byte-accesses +
> clwb().  The byte accesses get slower with a write-through mapping.
> So, performance data is needed all around to see where these options
> land.

When you say  "byte-access + clwb()", do you mean literally write a
byte, clwb, write a byte, clwb... or do you mean lots of byte accesses
and then one clwb?  If the former, I suspect it could be changed to
non-temporal store + sfence and be faster.

My understanding is that non-temporal store + sfence doesn't populate
the cache, though, which is unfortunate for some use cases.

The real solution would be for Intel to add an efficient operation to
force writeback on a large region of physical pages.

--Andy

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>