Re: [LSF/MM/BPF TOPIC] CMA reservation optimizations

[David Hildenbrand <david@redhat.com>]

On 28.01.25 18:07, Juan Yescas wrote:
> On Tue, Jan 28, 2025 at 1:58 AM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 28.01.25 02:04, Juan Yescas wrote:
>>> Hi LSF organizers,
>>>
>>> I would like to continue discussing this topic with the mm community:
>>>
>>> "CMA reservation optimizations"
>>>
>>> Note: There is already an email in the linux-mm mailing list that is
>>> discussing this issue. The title is:
>>>
>>> "CMA reservations require 32MiB alignment in 16KiB page size kernels
>>> instead of 8MiB in 4KiB page size kernel"
>>>
>>> Background
>>>
>>> When the drivers reserve CMA memory in 16KiB kernels, the minimum
>>> alignment is 32 MiB as per CMA_MIN_ALIGNMENT_BYTES. However, in 4KiB
>>> kernels, the CMA alignment is 4MiB.
>>
>> I'm curious, here you say 4 MiB, above 8 MiB.
>>
> 
> My bad, it is a typo. I meant 4 MiB.

That makes sense.

> 
>> But nowadays it's usually 2 MiB (pageblock size), no?
> 
> That's right for the case when THPs are enabled in 4KiB page size configs.
> 
> #define pageblock_order MIN_T(unsigned int, HPAGE_PMD_ORDER, MAX_PAGE_ORDER)
> https://elixir.bootlin.com/linux/v6.13/source/include/linux/pageblock-flags.h#L50
> 
> This evals to pageblock_order = min(21 - 12, 10) = 9
> 
> #define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages
> #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES)
> https://elixir.bootlin.com/linux/v6.13/source/include/linux/cma.h#L21
> 
> CMA_MIN_ALIGNMENT_BYTES = (4096 * 2 ^ 9) = (4096 * 512) = 2097152 = 2 MiB
> 
> However, when THPs are disabled, we get:
> 
> #define pageblock_order MAX_PAGE_ORDER // 10
> https://elixir.bootlin.com/linux/v6.13/source/arch/arm64/Kconfig#L1630
> https://elixir.bootlin.com/linux/v6.13/source/include/linux/pageblock-flags.h#L55
> 
> CMA_MIN_ALIGNMENT_BYTES = (4096 * 2 ^ 10) = (4096 * 1024) = 4194304 = 4 MiB

Right, and it can depend on ARCH_FORCE_MAX_ORDER.

I've been wondering for a while if pageblock_order should nowadays 
default to HPAGE_PMD_ORDER, with the option to make it smaller/larger 
(likely smaller) -- as discussed.

As discussed, the topic you are touching on is also relevant for 
virtio-mem, which can add/remove memory currently in pageblock 
granularity: 512 MiB on arm64 are not particularly helpful. I think we 
could support adding/removing smaller granularity, but it requires a bit 
of work, and always isolating 512MiB worth of pages just to effectively 
allocate e.g., 2 MiB worth of pages is rather suboptimal. Same applies 
to CMA I assume.

So there is more infrastructure that could benefit from pageblocks to 
rather be on the smaller side, even when hugetlb+THP might not be around 
in a config.

-- 
Cheers,

David / dhildenb