Re: FW: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory

[David Hildenbrand <david@redhat.com>]

On 12.04.23 13:10, Kyungsan Kim wrote:
>>> Gregory Price <gregory.price@memverge.com> writes:
>>>
>>>> On Tue, Apr 11, 2023 at 02:37:50PM +0800, Huang, Ying wrote:
>>>>> Gregory Price <gregory.price@memverge.com> writes:
>>>>>
>>>>> [snip]
>>>>>
>>>>>> 2. During the migration process, the memory needs to be forced not to be
>>>>>>      migrated to another node by other means (tiering software, swap,
>>>>>>      etc).  The obvious way of doing this would be to migrate and
>>>>>>      temporarily pin the page... but going back to problem #1 we see that
>>>>>>      ZONE_MOVABLE and Pinning are mutually exclusive.  So that's
>>>>>>      troublesome.
>>>>>
>>>>> Can we use memory policy (cpusets, mbind(), set_mempolicy(), etc.) to
>>>>> avoid move pages out of CXL.mem node?  Now, there are gaps in tiering,
>>>>> but I think it is fixable.
>>>>>
>>>>> Best Regards,
>>>>> Huang, Ying
>>>>>
>>>>> [snip]
>>>>
>>>> That feels like a hack/bodge rather than a proper solution to me.
>>>>
>>>> Maybe this is an affirmative argument for the creation of an EXMEM
>>>> zone.
>>>
>>> Let's start with requirements.  What is the requirements for a new zone
>>> type?
>>
>> I'm stills scratching my head regarding this. I keep hearing all
>> different kind of statements that just add more confusions "we want it
>> to be hotunpluggable" "we want to allow for long-term pinning memory"
>> "but we still want it to be movable" "we want to place some unmovable
>> allocations on it". Huh?
>>
>> Just to clarify: ZONE_MOVABLE allows for pinning. It just doesn't allow
>> for long-term pinning of memory.
>>
>> For good reason, because long-term pinning of memory is just the worst
>> (memory waste, fragmentation, overcommit) and instead of finding new
>> ways to *avoid* long-term pinnings, we're coming up with advanced
>> concepts to work-around the fundamental property of long-term pinnings.
>>
>> We want all memory to be long-term pinnable and we want all memory to be
>> movable/hotunpluggable. That's not going to work.
> 
> Looks there is misunderstanding about ZONE_EXMEM argument.
> Pinning and plubbability is mutual exclusive so it can not happen at the same time.
> What we argue is ZONE_EXMEM does not "confine movability". an allocation context can determine the movability attribute.
> Even one unmovable allocation will make the entire CXL DRAM unpluggable.
> When you see ZONE_EXMEM just on movable/unmoable aspect, we think it is the same with ZONE_NORMAL,
> but ZONE_EXMEM works on an extended memory, as of now CXL DRAM.
> 
> Then why ZONE_EXMEM is, ZONE_EXMEM considers not only the pluggability aspect, but CXL identifier for user/kenelspace API,
> the abstraction of multiple CXL DRAM channels, and zone unit algorithm for CXL HW characteristics.
> The last one is potential at the moment, though.
> 
> As mentioned in ZONE_EXMEM thread, we are preparing slides to explain experiences and proposals.
> It it not final version now[1].
> [1] https://github.com/OpenMPDK/SMDK/wiki/93.-%5BLSF-MM-BPF-TOPIC%5D-SMDK-inspired-MM-changes-for-CXL

Yes, hopefully we can discuss at LSF/MM also the problems we are trying 
to solve instead of focusing on one solution. [did not have the time to 
look at the slides yet, sorry]

-- 
Thanks,

David / dhildenb