RE: FW: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory

[Kyungsan Kim <ks0204.kim@samsung.com>]

>On 12.04.23 13:10, Kyungsan Kim wrote:
>>>> Gregory Price <gregory.price@memverge.com> writes:
>>>>
>>>>> On Tue, Apr 11, 2023 at 02:37:50PM +0800, Huang, Ying wrote:
>>>>>> Gregory Price <gregory.price@memverge.com> writes:
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>> 2. During the migration process, the memory needs to be forced not to be
>>>>>>>      migrated to another node by other means (tiering software, swap,
>>>>>>>      etc).  The obvious way of doing this would be to migrate and
>>>>>>>      temporarily pin the page... but going back to problem #1 we see that
>>>>>>>      ZONE_MOVABLE and Pinning are mutually exclusive.  So that's
>>>>>>>      troublesome.
>>>>>>
>>>>>> Can we use memory policy (cpusets, mbind(), set_mempolicy(), etc.) to
>>>>>> avoid move pages out of CXL.mem node?  Now, there are gaps in tiering,
>>>>>> but I think it is fixable.
>>>>>>
>>>>>> Best Regards,
>>>>>> Huang, Ying
>>>>>>
>>>>>> [snip]
>>>>>
>>>>> That feels like a hack/bodge rather than a proper solution to me.
>>>>>
>>>>> Maybe this is an affirmative argument for the creation of an EXMEM
>>>>> zone.
>>>>
>>>> Let's start with requirements.  What is the requirements for a new zone
>>>> type?
>>>
>>> I'm stills scratching my head regarding this. I keep hearing all
>>> different kind of statements that just add more confusions "we want it
>>> to be hotunpluggable" "we want to allow for long-term pinning memory"
>>> "but we still want it to be movable" "we want to place some unmovable
>>> allocations on it". Huh?
>>>
>>> Just to clarify: ZONE_MOVABLE allows for pinning. It just doesn't allow
>>> for long-term pinning of memory.
>>>
>>> For good reason, because long-term pinning of memory is just the worst
>>> (memory waste, fragmentation, overcommit) and instead of finding new
>>> ways to *avoid* long-term pinnings, we're coming up with advanced
>>> concepts to work-around the fundamental property of long-term pinnings.
>>>
>>> We want all memory to be long-term pinnable and we want all memory to be
>>> movable/hotunpluggable. That's not going to work.
>> 
>> Looks there is misunderstanding about ZONE_EXMEM argument.
>> Pinning and plubbability is mutual exclusive so it can not happen at the same time.
>> What we argue is ZONE_EXMEM does not "confine movability". an allocation context can determine the movability attribute.
>> Even one unmovable allocation will make the entire CXL DRAM unpluggable.
>> When you see ZONE_EXMEM just on movable/unmoable aspect, we think it is the same with ZONE_NORMAL,
>> but ZONE_EXMEM works on an extended memory, as of now CXL DRAM.
>> 
>> Then why ZONE_EXMEM is, ZONE_EXMEM considers not only the pluggability aspect, but CXL identifier for user/kenelspace API,
>> the abstraction of multiple CXL DRAM channels, and zone unit algorithm for CXL HW characteristics.
>> The last one is potential at the moment, though.
>> 
>> As mentioned in ZONE_EXMEM thread, we are preparing slides to explain experiences and proposals.
>> It it not final version now[1].
>> [1] https://protect2.fireeye.com/v1/url?k=265f4f76-47d45a59-265ec439-74fe485cbfe7-1e8ec1d2f0c2fd0a&q=1&e=727e97be-fc78-4fa6-990b-a86c256978d1&u=https%3A%2F%2Fgithub.com%2FOpenMPDK%2FSMDK%2Fwiki%2F93.-%255BLSF-MM-BPF-TOPIC%255D-SMDK-inspired-MM-changes-for-CXL
>
>Yes, hopefully we can discuss at LSF/MM also the problems we are trying 
>to solve instead of focusing on one solution. [did not have the time to 
>look at the slides yet, sorry]

For sure.. The purpose of LSF/MM this year is weighted on sharing experiences and issues as CXL provider for last couple of years.
We don't think our solution is the only way, but propose it.
Hopefully, we gradually figure out the best way with experts here.

>
>-- 
>Thanks,
>
>David / dhildenb