Re: mm: CMA reservations require 32MiB alignment in 16KiB page size kernels instead of 8MiB in 4KiB page size kernel.

["Zi Yan" <ziy@nvidia.com>]

On Sun Jan 19, 2025 at 7:38 PM EST, Barry Song wrote:
> On Mon, Jan 20, 2025 at 1:26 PM Zi Yan <ziy@nvidia.com> wrote:
>>
>> On 19 Jan 2025, at 19:17, Barry Song wrote:
>>
>> > On Sat, Jan 18, 2025 at 12:00 PM Juan Yescas <jyescas@google.com> wrote:
>> >>
>> >> + iamjoonsoo.kim@lge.com
>> >> + quic_charante@quicinc.com
>> >>
>> >> On Fri, Jan 17, 2025 at 2:52 PM Juan Yescas <jyescas@google.com> wrote:
>> >>>
>> >>> +Suren Baghdasaryan
>> >>> +Kalesh Singh
>> >>> +T.J. Mercier
>> >>> +Isaac Manjarres
>> >>>
>> >>> On Fri, Jan 17, 2025 at 2:51 PM Juan Yescas <jyescas@google.com> wrote:
>> >>>>
>> >>>> Hi Linux memory team
>> >>>>
>> >>>> When the drivers reserve CMA memory in 16KiB kernels, the minimum
>> >>>> alignment is 32 MiB as per CMA_MIN_ALIGNMENT_BYTES. However, in 4KiB
>> >>>> kernels, the CMA alignment is 4MiB.
>> >>>>
>> >>>> This is forcing the drivers to reserve more memory in 16KiB kernels,
>> >>>> even if they only require 4MiB or 8MiB.
>> >>>>
>> >>>> reserved-memory {
>> >>>>       #address-cells = <2>;
>> >>>>       #size-cells = <2>;
>> >>>>       ranges;
>> >>>>       tpu_cma_reserve: tpu_cma_reserve {
>> >>>>             compatible = "shared-dma-pool";
>> >>>>             reusable;
>> >>>>            size = <0x0 0x2000000>; /* 32 MiB */
>> >>>> }
>> >>>>
>> >>>> One workaround to continue using 4MiB alignment is:
>> >>>>
>> >>>> - Disable CONFIG_TRANSPARENT_HUGEPAGE so the buddy allocator does NOT
>> >>>> have to allocate huge pages (32 MiB in 16KiB page sizes)
>> >>>> - Set ARCH_FORCE_MAX_ORDER for ARM64_16K_PAGES to "8", instead of
>> >>>> "11", so CMA_MIN_ALIGNMENT_BYTES is equals to 4 MiB
>> >>>>
>> >>>>     config ARCH_FORCE_MAX_ORDER
>> >>>>         int
>> >>>>         default "13" if ARM64_64K_PAGES
>> >>>>         default "8" if ARM64_16K_PAGES
>> >>>>        default "10"
>> >>>>
>> >>>> #define MAX_PAGE_ORDER CONFIG_ARCH_FORCE_MAX_ORDER      // 8
>> >>>> #define pageblock_order MAX_PAGE_ORDER              // 8
>> >>>> #define pageblock_nr_pages (1UL << pageblock_order)    // 256
>> >>>> #define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages      // 256
>> >>>> #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES)
>> >>>>   // 16384 * 256 = 4194304 = 4 MiB
>> >>>>
>> >>>> After compiling the kernel with this changes, the kernel boots without
>> >>>> warnings and the memory is reserved:
>> >>>>
>> >>>> [    0.000000] Reserved memory: created CMA memory pool at
>> >>>> 0x000000007f800000, size 8 MiB
>> >>>> [    0.000000] OF: reserved mem: initialized node tpu_cma_reserve,
>> >>>> compatible id shared-dma-pool
>> >>>> [    0.000000] OF: reserved mem:
>> >>>> 0x000000007f800000..0x000000007fffffff (8192 KiB) map reusable
>> >>>> tpu_cma_reserve
>> >>>>
>> >>>> #  uname -a
>> >>>> Linux buildroot 6.12.9-dirty
>> >>>> # zcat /proc/config.gz | grep ARM64_16K
>> >>>> CONFIG_ARM64_16K_PAGES=y
>> >>>> # zcat /proc/config.gz | grep TRANSPARENT_HUGE
>> >>>> CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
>> >>>> # CONFIG_TRANSPARENT_HUGEPAGE is not set
>> >>>> # cat /proc/pagetypeinfo
>> >>>> Page block order: 8
>> >>>> Pages per block:  256
>> >>>>
>> >>>> Free pages count per migrate type at order       0      1      2
>> >>>> 3      4      5      6      7      8
>> >>>> Node    0, zone      DMA, type    Unmovable      1      1     13
>> >>>> 6      5      2      0      0      1
>> >>>> Node    0, zone      DMA, type      Movable      9     16     19
>> >>>> 13     13      5      2      0    182
>> >>>> Node    0, zone      DMA, type  Reclaimable      0      1      0
>> >>>> 1      1      0      0      1      0
>> >>>> Node    0, zone      DMA, type   HighAtomic      0      0      0
>> >>>> 0      0      0      0      0      0
>> >>>> Node    0, zone      DMA, type          CMA      1      0      0
>> >>>> 0      0      0      0      0     49
>> >>>> Node    0, zone      DMA, type      Isolate      0      0      0
>> >>>> 0      0      0      0      0      0
>> >>>> Number of blocks type     Unmovable      Movable  Reclaimable
>> >>>> HighAtomic          CMA      Isolate
>> >>>> Node 0, zone      DMA            6          199            1
>> >>>>  0           50            0
>> >>>>
>> >>>>
>> >>>> However, with this workaround, we can't use transparent huge pages.
>> >
>> > I don’t think this is accurate. You can still use mTHP with a size
>> > equal to or smaller than 4MiB,
>> > right?
>> >
>> > By the way, what specific regression have you observed when reserving
>> > a larger size like
>> > 32MB?
>> > For CMA, the over-reserved memory is still available to the system for
>> > movable folios. 28MiB
>>
>> The fallbacks table does not have MIGRATE_CMA as a fallback for any
>> migratetype. How can it be used for movable folios? Am I missing something?
>
> The whole purpose of CMA is to allow the memory reserved for a
> device's dma_alloc_coherent or other contiguous memory needs to
> be freely used by movable allocations when the device doesn't
> require it. When the device's DMA needs the memory, the movable
> folios can be migrated to make it available for the device.
>
> /* Must be called after current_gfp_context() which can change gfp_mask */
> static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask,
>                                                   unsigned int alloc_flags)
> {
> #ifdef CONFIG_CMA
>         if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE)
>                 alloc_flags |= ALLOC_CMA;
> #endif
>
>         return alloc_flags;
> }
>
> So there’s no waste here. cma can be used by normal buddy.

Ah, you are right. I missed the above code, which adds ALLOC_CMA. I
agree with you that there is no waste unless the system has a heavy use
of unmovable data.

-- 
Best Regards,
Yan, Zi