Re: collision between ZONE_MOVABLE and memblock allocations

[David Hildenbrand <david@redhat.com>]

On 19.07.23 10:06, Michal Hocko wrote:
> On Wed 19-07-23 10:59:52, Mike Rapoport wrote:
>> On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
>>> On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
>>> [...]
>>>> I do think that we need to fix this collision between ZONE_MOVABLE and memmap
>>>> allocations, because this issue essentially makes the movablecore= kernel
>>>> command line parameter useless in many cases, as the ZONE_MOVABLE region it
>>>> creates will often actually be unmovable.
>>>
>>> movablecore is kinda hack and I would be more inclined to get rid of it
>>> rather than build more into it. Could you be more specific about your
>>> use case?
>>>
>>>> Here are the options I currently see for resolution:
>>>>
>>>> 1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from
>>>> the beginning of the NUMA node instead of the end. This should fix my use case,
>>>> but again is prone to breakage in other configurations (# of NUMA nodes, other
>>>> architectures) where ZONE_MOVABLE and memblock allocations might overlap.  I
>>>> think that this should be relatively straightforward and low risk, though.
>>>>
>>>> 2. Make the code which processes the movablecore= command line option aware of
>>>> the memblock allocations, and have it choose a region for ZONE_MOVABLE which
>>>> does not have these allocations. This might be done by checking for
>>>> PageReserved() as we do with offlining memory, though that will take some boot
>>>> time reordering, or we'll have to figure out the overlap in another way. This
>>>> may also result in us having two ZONE_NORMAL zones for a given NUMA node, with
>>>> a ZONE_MOVABLE section in between them.  I'm not sure if this is allowed?
>>>
>>> Yes, this is no problem. Zones are allowed to be sparse.
>>
>> The current initialization order is roughly
>>
>> * very early initialization with some memblock allocations
>> * determine zone locations and sizes
>> * initialize memory map	
>>    - memblock_alloc(lots of memory)
>> * lots of unrelated initializations that may allocate memory
>> * release free pages from memblock to the buddy allocator
>>
>> With 2) we can make sure the memory map and early allocations won't be in
>> the ZONE_MOVABLE, but we'll still may have reserved pages there.
> 
> Yes this will always be fragile. If the spefic placement of the movable
> memory is not important and the only thing that matters is the size and
> numa locality then an easier to maintain solution would be to simply
> offline enough memory blocks very early in the userspace bring up and
> online it back as movable. If offlining fails just try another
> memblock. This doesn't require any kernel code change.

As an alternative, we might use the "memmap=nn[KMG]!ss[KMG]" [1] 
parameter to mark some memory as protected.

That memory can then be configured as devdax device and online to 
ZONE_MOVABLE (dev/dax).

[1] 
https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap

-- 
Cheers,

David / dhildenb