Re: IOVA allocation dependency between firmware buffer and remaining buffers

[Robin Murphy <robin.murphy@arm.com>]

On 2020-09-24 11:47, Marek Szyprowski wrote:
> Hi Robin,
> 
> On 24.09.2020 12:40, Robin Murphy wrote:
>> On 2020-09-24 11:16, Thierry Reding wrote:
>>> On Thu, Sep 24, 2020 at 10:46:46AM +0200, Marek Szyprowski wrote:
>>>> On 24.09.2020 10:28, Joerg Roedel wrote:
>>>>> On Wed, Sep 23, 2020 at 08:48:26AM +0200, Marek Szyprowski wrote:
>>>>>> It allows to remap given buffer at the specific IOVA address,
>>>>>> although
>>>>>> it doesn't guarantee that those specific addresses won't be later
>>>>>> used
>>>>>> by the IOVA allocator. Probably it would make sense to add an API for
>>>>>> generic IOMMU-DMA framework to mark the given IOVA range as
>>>>>> reserved/unused to protect them.
>>>>> There is an API for that, the IOMMU driver can return IOVA reserved
>>>>> regions per device and the IOMMU core code will take care of mapping
>>>>> these regions and reserving them in the IOVA allocator, so that
>>>>> DMA-IOMMU code will not use it for allocations.
>>>>>
>>>>> Have a look at the iommu_ops->get_resv_regions() and
>>>>> iommu_ops->put_resv_regions().
>>>>
>>>> I know about the reserved regions IOMMU API, but the main problem here,
>>>> in case of Exynos, is that those reserved regions won't be created by
>>>> the IOMMU driver but by the IOMMU client device. It is just a result
>>>> how
>>>> the media drivers manages their IOVA space. They simply have to load
>>>> firmware at the IOVA address lower than the any address of the used
>>>> buffers.
>>>
>>> I've been working on adding a way to automatically add direct mappings
>>> using reserved-memory regions parsed from device tree, see:
>>>
>>> https://lore.kernel.org/lkml/20200904130000.691933-1-thierry.reding@gmail.com/
>>>
>>> Perhaps this can be of use? With that you should be able to add a
>>> reserved-memory region somewhere in the lower range that you need for
>>> firmware images and have that automatically added as a direct mapping
>>> so that it won't be reused later on for dynamic allocations.
>>
>> It can't easily be a *direct* mapping though - if the driver has to
>> use the DMA masks to ensure that everything stays within the
>> addressable range, then (as far as I'm aware) there's no physical
>> memory that low down to equal the DMA addresses.
>>
>> TBH I'm not convinced that this is a sufficiently common concern to
>> justify new APIs, or even to try to make overly generic. I think just
>> implementing a new DMA attribute to say "please allocate/map this
>> particular request at the lowest DMA address possible" would be good
>> enough. Such a thing could also serve PCI drivers that actually care
>> about SAC/DAC to give us more of a chance of removing the "try a
>> 32-bit mask first" trick from everyone's hotpath...
> 
> Hmm, I like the idea of such DMA attribute! It should make things really
> simple, especially in the drivers. Thanks for the great idea! I will try
> to implement it then instead of the workarounds I've proposed in
> s5p-mfc/exynos4-is drivers.

Right, I think it's fair to draw a line and say that anyone who wants a 
*specific* address needs to manage their own IOMMU domain.

In the backend I suspect it's going to be cleanest to implement a 
dedicated iova_alloc_low() (or similar) function in the IOVA API that 
sidesteps all of the existing allocation paths and goes straight to the 
rbtree.

Robin.