Re: [RFC 0/7] Peer-direct memory

[davide rossetti <davide.rossetti@gmail.com>]

resending, sorry

On Tue, Feb 16, 2016 at 10:22 AM, Jason Gunthorpe
<jgunthorpe@obsidianresearch.com> wrote:
>
> On Sun, Feb 14, 2016 at 04:27:20PM +0200, Haggai Eran wrote:
> > [apologies: sending again because linux-mm address was wrong]
> >
> > On 11/02/2016 21:18, Jason Gunthorpe wrote:
> > > Resubmit those parts under the mm subsystem, or another more
> > > appropriate place.
> >
> > We want the feedback from linux-mm, and they are now Cced.
>
> Resubmit to mm means put this stuff someplace outside
> drivers/infiniband in the tree and don't try and inappropriately send
> memory management stuff through Doug's tree.
>

Jason,
I beg to differ.

1) I see mm as appropriate for real memory, i.e. something that
user-space apps can pass around. This is not totally true for BAR
memory, for instance:
 a) as long as CPU initiated atomic ops are not supported on BAR space
of PCIe devices.
 b) OTOT, CPU reading from BAR is awful (BW being abysmal,~10MB/s),
while high BW writing requires use of vector instructions (at least on
x86_64).
Bottom line is, BAR mappings are not like plain memory.

2) Instead, I see appropriate that two sophisticated devices, like an
IB NIC and a storage/accelerator device, can freely target each other
for I/O, i.e. exchanging peer-to-peer PCIe transactions. And as long
as the existing sophisticated initiators are confined to the RDMA
subsystem, that is where this support belongs to.

On a different note, this reminds me that the current patch set may be
missing a way to disable the use of platform PCIe atomics when the
target is the BAR of a peer device.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>