Re: [Ksummit-discuss] [CORE TOPIC] Core Kernel support for Compute-Offload Devices

[Arnd Bergmann <arnd@arndb.de>]

On Thursday 30 July 2015 15:00:27 Joerg Roedel wrote:
> [
>  The topic is highly technical and could be a tech topic. But it also
>  touches multiple subsystems, so I decided to submit it as a core
>  topic.
> ]
> 
> Across architectures and vendors there are new devices coming up for
> offloading tasks from the CPUs. Most of these devices are capable to
> operate on user address spaces.
> 
> Besides the commonalities there are important differences in the memory
> model these devices offer. Some work only on system RAM, others come
> with their own memory which may or may not be accessible by the CPU.
> 
> I'd like to discuss what support we need in the core kernel for these
> devices. A probably incomplete list of open questions:
> 
> 	(1) Do we need the concept of an off-CPU task in the kernel
> 	    together with a common interface to create and manage them
> 	    and probably a (collection of) batch scheduler(s) for these
> 	    tasks?

I think we did this part right with the Cell SPUs 10 years ago: A
task is a task, and you just switch between running in user mode and
running on the offload engine through some syscall or ioctl.

The part that got us into endless trouble though was trying to
satisfy two opposite requirements: 

a) having the kernel schedule tasks automatically onto the offload
   engines and take care of context switches and placement, so you
   can do multi-user and multi-tasking processing on them.

b) getting most performance out of the of offload engines, by giving
   a single user total control over the placement and no do any
   scheduling in the kernel at all.

I would strongly recommend now that any new interface tries to do
only one of the two models, but does it right.

> I am particularily interested in this topic because on PCIe newer IOMMUs
> are often an integral part in supporting these devices (ARM-SMMUv3,
> Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also
> touch the IOMMU code.
> 
> Probably (uncomplete list of) interested people:
> 
> 	David Woodhouse
> 	Jesse Barnes
> 	Will Deacon
> 	Paul E. McKenney
> 	Rik van Riel
> 	Mel Gorman
> 	Andrea Arcangeli
> 	Christoph Lameter
> 	Jérôme Glisse

Add me in as well,

	Arnd