Re: [LSF/MM/BPF TOPIC] Restricting or migrating unmovable kernel allocations from slow tier

[David Hildenbrand <david@redhat.com>]

On 01.02.25 14:29, Hyeonggon Yoo wrote:
> Hi,
> 
> Byungchul and I would like to suggest a topic about the performance impact of
> kernel allocations on CXL memory.
> 
> As CXL-enabled servers and memory devices are being developed, CXL-supported
> hardware is expected to continue emerging in the coming years.
> 
> The Linux kernel supports hot-plugging CXL memory via dax/kmem functionality.
> The hot-plugged memory allows either unmovable kernel allocations
> (ZONE_NORMAL), or restricts them to movable allocations (ZONE_MOVABLE)
> depending on the hot-plug policy.
> 
> Recently, Byungchul and I observed a measurable performance degradation with
> memhp_default_state=online compared to memhp_default_state=online_movable
> on a server where the ratio of memory capacity between DRAM and CXL is 1:2
> when running the llama.cpp workload with the default mempolicy.
> The workload performs LLM inference and pressures the memory subsystem
> due to its large working set size.
> 
> Obviously, allowing kernel allocations from CXL memory degrades performance
> because kernel memory like page tables, kernel stacks, and slab allocations,
> is accessed frequently and may reside in physical memory with significantly
> higher access latency.
> 
> However, as far as I can tell there are at least two reasons why we need to
> support ZONE_NORMAL for CXL memory (please add if there are more):
>    1. When hot-plugging a huge amount of CXL memory, the size of
>       the struct page array might not fit into DRAM
>       -> This could be relaxed with memmap_on_memory

There are some others, although most are less significant, and I tried 
documenting them here:

https://www.kernel.org/doc/html/latest/admin-guide/mm/memory-hotplug.html#zone-movable-sizing-considerations


E.g., a 4 KiB page requires a single PTE (8 bytes) to be mapped into 
user space, corresponding to 0.2 %. At least for anonymous memory, 
PMD-sized THPs don't help, because we still have to allocate the page 
table to be prepared for a PMD->PTE remapping. In the worst case, the 
directmap requires another 0.2 % (but usually, we rely on PMD mappings). 
So that usage depends on how you are intending to use the CXL memory 
(e.g., pagecache vs. anonymous memory).


 >    2. To hot-unplug CXL memory, pages in CXL memory should be 
migrated to DRAM,
 >       which means sometimes some portion of CXL memory should be 
ZONE_NORMAL.

I don't quite understand that argument for ZONE_NORMAL.

-- 
Cheers,

David / dhildenb