Re: [LSF/MM/BPF TOPIC] CMA reservation optimizations

[Zi Yan <ziy@nvidia.com>]

On 28 Jan 2025, at 13:33, David Hildenbrand wrote:

> On 28.01.25 18:07, Juan Yescas wrote:
>> On Tue, Jan 28, 2025 at 1:58 AM David Hildenbrand <david@redhat.com> wrote:
>>>
>>> On 28.01.25 02:04, Juan Yescas wrote:
>>>> Hi LSF organizers,
>>>>
>>>> I would like to continue discussing this topic with the mm community:
>>>>
>>>> "CMA reservation optimizations"
>>>>
>>>> Note: There is already an email in the linux-mm mailing list that is
>>>> discussing this issue. The title is:
>>>>
>>>> "CMA reservations require 32MiB alignment in 16KiB page size kernels
>>>> instead of 8MiB in 4KiB page size kernel"
>>>>
>>>> Background
>>>>
>>>> When the drivers reserve CMA memory in 16KiB kernels, the minimum
>>>> alignment is 32 MiB as per CMA_MIN_ALIGNMENT_BYTES. However, in 4KiB
>>>> kernels, the CMA alignment is 4MiB.
>>>
>>> I'm curious, here you say 4 MiB, above 8 MiB.
>>>
>>
>> My bad, it is a typo. I meant 4 MiB.
>
> That makes sense.
>
>>
>>> But nowadays it's usually 2 MiB (pageblock size), no?
>>
>> That's right for the case when THPs are enabled in 4KiB page size configs.
>>
>> #define pageblock_order MIN_T(unsigned int, HPAGE_PMD_ORDER, MAX_PAGE_ORDER)
>> https://elixir.bootlin.com/linux/v6.13/source/include/linux/pageblock-flags.h#L50
>>
>> This evals to pageblock_order = min(21 - 12, 10) = 9
>>
>> #define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages
>> #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES)
>> https://elixir.bootlin.com/linux/v6.13/source/include/linux/cma.h#L21
>>
>> CMA_MIN_ALIGNMENT_BYTES = (4096 * 2 ^ 9) = (4096 * 512) = 2097152 = 2 MiB
>>
>> However, when THPs are disabled, we get:
>>
>> #define pageblock_order MAX_PAGE_ORDER // 10
>> https://elixir.bootlin.com/linux/v6.13/source/arch/arm64/Kconfig#L1630
>> https://elixir.bootlin.com/linux/v6.13/source/include/linux/pageblock-flags.h#L55
>>
>> CMA_MIN_ALIGNMENT_BYTES = (4096 * 2 ^ 10) = (4096 * 1024) = 4194304 = 4 MiB
>
> Right, and it can depend on ARCH_FORCE_MAX_ORDER.
>
> I've been wondering for a while if pageblock_order should nowadays default to HPAGE_PMD_ORDER, with the option to make it smaller/larger (likely smaller) -- as discussed.
>
> As discussed, the topic you are touching on is also relevant for virtio-mem, which can add/remove memory currently in pageblock granularity: 512 MiB on arm64 are not particularly helpful. I think we could support adding/removing smaller granularity, but it requires a bit of work, and always isolating 512MiB worth of pages just to effectively allocate e.g., 2 MiB worth of pages is rather suboptimal. Same applies to CMA I assume.
>
> So there is more infrastructure that could benefit from pageblocks to rather be on the smaller side, even when hugetlb+THP might not be around in a config.


It is related to anti-fragmentation mechanism in the kernel (Mel and Vlastima
are cc’d, feel free to add more since I must miss others).

If pageblock size is smaller than a PMD THP size, current compaction
code will not be able to efficiently defragment memory for PMD THP creation,
since compaction code works at pageblock granularity. This means we need to
decouple compaction granularity from pageblock. pageblock uses different
migratetypes (UNMOVABLE, MOVABLE, RECLAIMABLE, ...) to help reduce memory
fragmentation by grouping pages by mobility. When compaction works on multiple
pageblocks to generate a PMD THP, it needs all pageblocks within the range
share the same migratetype, otherwise, compacting memory within a region with
MIGRATE_UNMOVABLE would highly likely result in a waste of time, since unmovable
pages just prevent a big free page from creation. So the key question is
how to prevent MIGRATE_UNMOVABLE fragment a PMD THP size pageblock range,
anti-fragmentation for pageblocks?

Do we want to have super-pageblock for that? Or we allow sub-pageblock for
virtio-mem and CMA reservation? That is probably what we want to discuss
on the infrastructure side.


Best Regards,
Yan, Zi