Re: IOVA allocation dependency between firmware buffer and remaining buffers

[Marek Szyprowski <m.szyprowski@samsung.com>]

Hi Robin,

On 24.09.2020 13:06, Robin Murphy wrote:
> On 2020-09-24 11:47, Marek Szyprowski wrote:
>> On 24.09.2020 12:40, Robin Murphy wrote:
>>> On 2020-09-24 11:16, Thierry Reding wrote:
>>>> On Thu, Sep 24, 2020 at 10:46:46AM +0200, Marek Szyprowski wrote:
>>>>> On 24.09.2020 10:28, Joerg Roedel wrote:
>>>>>> On Wed, Sep 23, 2020 at 08:48:26AM +0200, Marek Szyprowski wrote:
>>>>>>> It allows to remap given buffer at the specific IOVA address,
>>>>>>> although
>>>>>>> it doesn't guarantee that those specific addresses won't be later
>>>>>>> used
>>>>>>> by the IOVA allocator. Probably it would make sense to add an 
>>>>>>> API for
>>>>>>> generic IOMMU-DMA framework to mark the given IOVA range as
>>>>>>> reserved/unused to protect them.
>>>>>> There is an API for that, the IOMMU driver can return IOVA reserved
>>>>>> regions per device and the IOMMU core code will take care of mapping
>>>>>> these regions and reserving them in the IOVA allocator, so that
>>>>>> DMA-IOMMU code will not use it for allocations.
>>>>>>
>>>>>> Have a look at the iommu_ops->get_resv_regions() and
>>>>>> iommu_ops->put_resv_regions().
>>>>>
>>>>> I know about the reserved regions IOMMU API, but the main problem 
>>>>> here,
>>>>> in case of Exynos, is that those reserved regions won't be created by
>>>>> the IOMMU driver but by the IOMMU client device. It is just a result
>>>>> how
>>>>> the media drivers manages their IOVA space. They simply have to load
>>>>> firmware at the IOVA address lower than the any address of the used
>>>>> buffers.
>>>>
>>>> I've been working on adding a way to automatically add direct mappings
>>>> using reserved-memory regions parsed from device tree, see:
>>>>
>>>> https://lore.kernel.org/lkml/20200904130000.691933-1-thierry.reding@gmail.com/ 
>>>>
>>>>
>>>> Perhaps this can be of use? With that you should be able to add a
>>>> reserved-memory region somewhere in the lower range that you need for
>>>> firmware images and have that automatically added as a direct mapping
>>>> so that it won't be reused later on for dynamic allocations.
>>>
>>> It can't easily be a *direct* mapping though - if the driver has to
>>> use the DMA masks to ensure that everything stays within the
>>> addressable range, then (as far as I'm aware) there's no physical
>>> memory that low down to equal the DMA addresses.
>>>
>>> TBH I'm not convinced that this is a sufficiently common concern to
>>> justify new APIs, or even to try to make overly generic. I think just
>>> implementing a new DMA attribute to say "please allocate/map this
>>> particular request at the lowest DMA address possible" would be good
>>> enough. Such a thing could also serve PCI drivers that actually care
>>> about SAC/DAC to give us more of a chance of removing the "try a
>>> 32-bit mask first" trick from everyone's hotpath...
>>
>> Hmm, I like the idea of such DMA attribute! It should make things really
>> simple, especially in the drivers. Thanks for the great idea! I will try
>> to implement it then instead of the workarounds I've proposed in
>> s5p-mfc/exynos4-is drivers.
>
> Right, I think it's fair to draw a line and say that anyone who wants 
> a *specific* address needs to manage their own IOMMU domain.
>
> In the backend I suspect it's going to be cleanest to implement a 
> dedicated iova_alloc_low() (or similar) function in the IOVA API that 
> sidesteps all of the existing allocation paths and goes straight to 
> the rbtree.

Just for the record - I've implemented this approach here: 
https://lore.kernel.org/lkml/20200925141218.13550-1-m.szyprowski@samsung.com/

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland