[LSFMM] RDMA data corruption potential during FS writeback

[Christopher Lameter <cl@linux.com>]

There was a session at the Linux Filesystem and Memory Management summit
on issues that are caused by devices using get_user_pages() or elevated
refcounts to pin pages and then do I/O on them.

See https://lwn.net/Articles/753027/

Basically filesystems need to mark the pages readonly during writeback.
Concurrent DMA into the page while it is written by a filesystem can cause
corrupted data being written to the disk, cause incorrect checksums etc
etc.

The solution that was proposed at the meeting was that mmu notifiers can
remedy that situation by allowing callbacks to the RDMA device to ensure
that the RDMA device and the filesystem do not do concurrent writeback.

This issue has been around for a long time and so far not caused too much
grief it seems. Doing I/O to two devices from the same memory location is
naturally a bit inconsistent in itself.

But could we do more to prevent issues here? I think what may be useful is
to not allow the memory registrations of file back writable mappings
unless the device driver provides mmu callbacks or something like that.

There is also the longstanding issue of the refcounts that are held over
long time periods. If we require mmu notifier callbacks then we may as
well go to on demand paging mode for RDMA memory registrations. This
avoids increasing the refcounts long term and allows easy access control /
page removal for memory management.

There may even be more issues if DAX is being used but the FS writeback
has the potential of biting anyone at this point it seems.

I think we need to put some thought into these issues and we need some
coordination between the RDMA developers and memory management. RDMA seems
to be more and more important and thus its likely that issues like this
will become more important.