Re: Memory management facing a 400Gpbs network link

[Jerome Glisse <jglisse@redhat.com>]

On Tue, Feb 12, 2019 at 06:25:50PM +0000, Christopher Lameter wrote:
> 400G Infiniband will become available this year. This means that the data
> ingest speeds can be higher than the bandwidth of the processor
> interacting with its own memory.
> 
> For example a single hardware thread is limited to 20Gbyte/sec whereas the
> network interface provides 50Gbytes/sec. These rates can only be obtained
> currently with pinned memory.
> 
> How can we evolve the memory management subsystem to operate at higher
> speeds with more the comforts of paging and system calls that we are used
> to?

Couple questions. This is not saturating PCIe ie we are talking 400Gbps so
~40GBytes/s right ? PCIE 4 is ~32GBytes/s with x16 so is this some kind of
weird hardware that have 2 PCIE bridge and can be on 2 different root PCIE
complex ? I heard this idea floating around to get more bandwidth without
having to wait for PCIE 5 ...

Regarding memory management what will be the target memory ? Page cache or
private anonymous ? Or a mix of both ?


More to the point, my feeling is that we want something like page cache for
dma (and for page in page cache we would like to be able to also keep track
of device reference). So when they are no memory pressure we can try to use
as much memory not only for page cache but also for dma cache/pool. This
might mean that we will need to rebalance the page cache and dma cache/pool
depending on knob set by admin.

Obviously when you run out of memory, pressure will degrade the performance.


> 
> It is likely that these speeds with increase further and since the lead
> processor vendor seems to be caught in a management induced corporate
> suicide attempt we will not likely see any process on the processors from
> there. The straightforward solution would be to use the high speed tech
> for fabrics for the internal busses (doh!). Alternate processors are
> likely to show up in 2019 and 2020 but those will take a long time to
> mature.
> 
> So what does the future hold and how do we scale up our HPC systems given
> these problems?

I think peer to peer will also be a big part here, for instance RDMA to/from
GPU memory, which completely bypass the main memory. Some HPC people talks
about even having GPU and CPU run almost unrelated workload and thus trying
to isolate them from one another.

Cheers,
Jérôme