Re: [Ksummit-discuss] [CORE TOPIC] Core Kernel support for Compute-Offload Devices

[Joerg Roedel <joro@8bytes.org>]

On Thu, Jul 30, 2015 at 02:31:38PM +0100, David Woodhouse wrote:
> On Thu, 2015-07-30 at 15:00 +0200, Joerg Roedel wrote:
> > 	    (2.3) How can we attach common state for off-CPU tasks to
> > 	          mm_struct (and what needs to be in there)?
> 
> And how do we handle the assignment of Address Space IDs? The AMD
> implementation currently allows the PASID space to be managed per
> -device, but I understand ARM systems handle the TLB shootdown
> broadcasts in hardware and need the PASID that the device sees to be
> identical to the ASID on the CPU's MMU? And there are reasons why we
> might actually want that model on Intel systems too. I'm working on the
> Intel SVM right now, and looking at a single-PASID-space model (partly
> because the PASID tables have to be physically contiguous, and they can
> be huge!).

True, ASIDs would be one thing that needs to be attached to a mm_struct,
but I am also interested what other platforms might need here. For
example, is there a better way to track these off-cpu users than using
mmu-notifiers.

> > 	(3) Does it make sense to implement automatic migration of
> > 	    system memory to device memory (when available) and vice
> > 	    versa? How do we decide what and when to migrate?
> 
> This is quite a horrid one, but perhaps ties into generic NUMA
> considerations — if a memory page is being frequently accessed by
> something that it's far away from, can we move it to closer memory?

Yeah, conceptually it is NUMA, so it might fit there. But the difference
to the current NUMA handling is that the device memory is not always
completly visible to the CPU, so I think quite some significant changes
are necessary to make this work.

Another idea is to handle migration like swapping. The difference to
real swapping is that it is not relying on the LRU lists but the device
access patterns we measure.

> The question is how we handle that. We do have Extended Accessed bits
> in the Intel implementation of SVM that let us know that a given PTE
> was used from a device. Although not *which* device, in cases where
> there might be more than one.

One way would be to use seperate page-tables for the devices (which, on
the other side, somehow contradicts the design of the hardware, because
its designed to reuse cpu page-tables).

And I don't know which features other devices have (like the CAPI
devices on Power that Paul wrote about) to help in this decission.



	Joerg