Re: [Ksummit-discuss] [CORE TOPIC] Core Kernel support for Compute-Offload Devices

[Catalin Marinas <catalin.marinas@arm.com>]

On Mon, Aug 03, 2015 at 10:12:54PM +0100, David Woodhouse wrote:
> On Mon, 2015-08-03 at 23:10 +0200, Joerg Roedel wrote:
> > On Mon, Aug 03, 2015 at 03:01:45PM -0400, Jerome Glisse wrote:
> > > This is not the case with current AMD hw which IIRC only support 8bits or
> > > 9bits for PASID. Dunno if there next hardware will have more bits or not.
> > > So i need to check PCIE spec but i do not think the 20bits is a mandatory
> > > limit.
> > 
> > AMD hardware currently implements PASIDs with 16 bits. Given that only
> > mm_structs which are used by offload devices get one, this should be
> > enough to put them into a global pool and have one PASID per mm_struct.
> 
> I think there are many ARM systems which need this model because of the
> way TLB shootdowns are handled in hardware, and shared with the IOMMU?
> So we have to use the same ASID for both MMU and IOMMU there, AIUI.
> 
> Not that I claim to be an expert on the ARM IOMMUs.

Neither am I, cc'ing Will.

As it's the case with the ARM architecture (whether CPU or SMMU/IOMMU),
many features are optional. So an implementation may or may not support
handling of TLB invalidation broadcasting from the CPU. Even when it
does, this is configurable, so it is not forced to share the same ASID
space as the CPU.

AFAICT, the ARM SMMU uses StreamID/SubstreamID to map (1:1?) PCIe
Request-ID and PASID (when stage 1 translation is supported). The
StreamID/SubstreamID can then be mapped onto an ASID via stream tables.
Currently the arm-smmu drivers use their own ASID space but, if we are
going to support compute-offload devices, they could be made to share
the same ASID as the corresponding user processes (with an additional
API).

For a more generic compute-offload API, I guess we would need callbacks
into the (IOMMU) drivers for page table management, including TLB
invalidation and CPU ASID management events (like renewing the CPU ASID
for an existing task). An implementation supporting sharing of IOMMU/CPU
page tables would just have a minimal implementation of such API. We
probably need a (void *)iommu_context pointer in mm_struct that drivers
can point to their own context information.

-- 
Catalin