On Tue, Feb 16, 2016 at 10:22 AM, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote:

On Sun, Feb 14, 2016 at 04:27:20PM +0200, Haggai Eran wrote:
> [apologies: sending again because linux-mm address was wrong]
>
> On 11/02/2016 21:18, Jason Gunthorpe wrote:
> > Resubmit those parts under the mm subsystem, or another more
> > appropriate place.
>
> We want the feedback from linux-mm, and they are now Cced.

Resubmit to mm means put this stuff someplace outside
drivers/infiniband in the tree and don't try and inappropriately send
memory management stuff through Doug's tree.

Jason,
I beg to differ.

1) I see mm as appropriate for real memory, i.e. something that user-space apps can pass around.

This is not totally true for BAR memory, for instance as long as CPU initiated atomic ops are not supported on BAR space of PCIe devices.

OTOT, CPU reading from BAR is awful (BW being abysmal,~10MB/s), while high BW writing requires use of vector instructions (at least on x86_64).

2) Instead, I see appropriate that two sophisticated devices, like an IB NIC and a storage/accelerator device, can freely target each other for I/O, i.e. exchanging peer-to-peer PCIe transactions. And as long as the existing sophisticated initiators are confined to the RDMA subsystem, that is where this support belongs to.

On a different note, this reminds me that the current patch set may be missing a way to disable the use of platform PCIe atomics when the target is the BAR of a peer device.

sincerely,

email: davide DOT rossetti AT gmail DOT com
work: drossetti AT nvidia DOT com
facebook: http://www.facebook.com/dado.rossetti
twitter: @dado_rossetti
skype: d.rossetti