Re: [Ksummit-discuss] [CORE TOPIC] Core Kernel support for Compute-Offload Devices

[Jerome Glisse <j.glisse@gmail.com>]

On Mon, Aug 03, 2015 at 12:07:49PM -0700, Andy Lutomirski wrote:
> On Mon, Aug 3, 2015 at 12:01 PM, Jerome Glisse <j.glisse@gmail.com> wrote:
> > On Mon, Aug 03, 2015 at 07:51:02PM +0100, David Woodhouse wrote:
> >> On Fri, 2015-07-31 at 12:34 -0400, Jerome Glisse wrote:
> >> > No the ASID should not be associated with mm_struct. There is to
> >> > few ASID to have enough of them. I think currently there is only
> >> > 8bits worth of ASID. So what happen is that the GPU device driver
> >> > schedule process and recycle ASID as it does.
> >>
> >> In PCIe we have 20 bits of PASID. And we are going to expect hardware
> >> to implement them all, even if it can only do caching for fewer PASIDs
> >> than that.
> >>
> >
> > This is not the case with current AMD hw which IIRC only support 8bits or
> > 9bits for PASID. Dunno if there next hardware will have more bits or not.
> > So i need to check PCIE spec but i do not think the 20bits is a mandatory
> > limit.
> >
> >> There is also an expectation that a given MM will have the *same* PASID
> >> across all devices.
> >
> > I understand that this would be prefered. But in case of hw that have only
> > limited number of bit for PASID you surely do not want to starve it ie it
> > would be better to have the device recycle PASID to maximize its usage.
> >
> 
> FWIW, x86 PCID has 12 bits, and, if I ever try to implement support
> for it, my thought would be to only use 3 or 4 of those bits and
> aggressively recycle PCIDs.
> 
> I have no idea whether ASIC and PCID are supposed to be related at
> all, given that I don't know anything about how to program these
> unified memory contraptions.

So it is even worse than i tought, PASID spec says that the PASID capability
register have a field indicating the number of bit for PASID the device
supports. Valid value include 1bit which is kind of low, and it does not
seems there is any mandatory lower limit.

So if we want one page table (mm) associated with one PASID this might be
tedious if not impossible. Let say one device support 12bits and process
start using that device, we set PASID to ((1 << 12) - 1), then process
want to use another device that only support 8bits, so we would need to
switch PASID to something that fit. The first device might not allow to
switch PASID without big hammer like fully stoping current job or waiting
for current job to complete with no garanty on how long it could takes,
minutes, hours, days, ...

I agree and wish we could have one mm one PASID but given hardware, i think
we need to do a reality check here :(

Cheers,
Jérôme