Re: [LSFMM] RDMA data corruption potential during FS writeback

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, May 18, 2018 at 02:37:52PM +0000, Christopher Lameter wrote:
> There was a session at the Linux Filesystem and Memory Management summit
> on issues that are caused by devices using get_user_pages() or elevated
> refcounts to pin pages and then do I/O on them.
> 
> See https://lwn.net/Articles/753027/
> 
> Basically filesystems need to mark the pages readonly during writeback.
> Concurrent DMA into the page while it is written by a filesystem can cause
> corrupted data being written to the disk, cause incorrect checksums etc
> etc.
> 
> The solution that was proposed at the meeting was that mmu notifiers can
> remedy that situation by allowing callbacks to the RDMA device to ensure
> that the RDMA device and the filesystem do not do concurrent writeback.

This keeps coming up, and I understand why it seems appealing from the
MM side, but the reality is that very little RDMA hardware supports
this, and it carries with it a fairly big performance penalty so many
users don't like using it.

> This issue has been around for a long time and so far not caused too much
> grief it seems. Doing I/O to two devices from the same memory location is
> naturally a bit inconsistent in itself.

Well, I've seen various reports of FS's oopsing and what not, as the
LWN article points out.. So it is breaking stuff for some users.

> But could we do more to prevent issues here? I think what may be useful is
> to not allow the memory registrations of file back writable mappings
> unless the device driver provides mmu callbacks or something like that.

Why does every proposed solution to this involve crippling RDMA? Are
there really no ideas no ideas to allow the FS side to accommodate
this use case??

> There may even be more issues if DAX is being used but the FS writeback
> has the potential of biting anyone at this point it seems.

I think Dan already 'solved' this via get_user_pages_longterm which
just fails for DAX backed pages.

Jason