Re: [LSF/MM/BPF TOPIC] Restricting or migrating unmovable kernel allocations from slow tier

[Byungchul Park <byungchul@sk.com>]

On Sat, Feb 01, 2025 at 02:04:17PM +0000, Matthew Wilcox wrote:
> On Sat, Feb 01, 2025 at 10:29:23PM +0900, Hyeonggon Yoo wrote:
> > The Linux kernel supports hot-plugging CXL memory via dax/kmem functionality.
> > The hot-plugged memory allows either unmovable kernel allocations
> > (ZONE_NORMAL), or restricts them to movable allocations (ZONE_MOVABLE)
> > depending on the hot-plug policy.
> 
> This all seems like a grand waste of time.  Don't do that.  Don't allow
> kernel allocations from CXL at all.  Don't build systems that have
> vast quantities of CXL memory (or if you do, expose it as really fast
> swap, not as memory).
> 
> All of the CXL topics I see this year are "It really hurts performance
> when ..." and my reaction is "Yes, I told you it would hurt and you did
> it anyway".  Just stop doing it.  CXL is this decade's Infiniband / ATM
> / (name your favourite misguided dead technology here).  You can't stop
> other people from doing foolish things, but you don't have to join in.
> And we don't have to take stupid patches.

Hyeonggon and I described the topic based on what we observed in CXL
memory environment, but fundamentally it doesn't have to be only CXL
memory issue but also heterogeneous memory or ZONE_NORMAL cost issue as
you and others mentioned.  Lemme clarify it.

<general mm issue>

   1. Allow kernel object to be movable:
      a. ZONE_NORMAL cost will be reduced. (less reclaim and oom)
      b. ZONE_NORMAL covers bigger whole memory.
      c. A smaller ZONE_NORMAL is sufficient.
      d. Need additional consideration about when(or what) to move.

   2. Never allow kernel object to be movable:
      a. ZONE_NORMAL cost keeps high. (premature reclaim and oom)
      b. ZONE_NORMAL covers smaller whole memory.
      c. A bigger ZONE_NORMAL is required.

<heterogeneous memory specific issue>

   3. Allow ZONE_NORMAL in non-DRAM:
      a. Mitigate ZONE_NORMAL cost. (less reclaim and oom)
      b. Followed by e.g. hot-unplug issue.
      c. Option 1: No restricting the ZONE_NORMAL size.
      d. Option 2: Restricting the size as budget to cover its capacity.
      e. Option 3: ?

   4. Never allow ZONE_NORMAL in non-DRAM:
      a. ZONE_NORMAL cost should be low enough to cover non-DRAM too.
      b. Any efforts to reduce ZONE_NORMAL cost should be welcome.
      c. Matthew's work would mitigate the cost.
      d. Allowing kernel object to be movable would work for it too.

Plus, I think Metthew's effort to reduce ZONE_NORMAL cost is amazing and
hope successfully make it.  However, ZONE_NORMAL cost can be reduced in
many ways and all the efforts can be considered meaningful.

We can work with from the easiest object e.g. page table, struct page,
and kernel stack, to harder ones, while struct page cost is getting
reduced by Matthew's work at the same time.

When it comes to this topic, the most important thing is the collected
*direction* from the community so that we can start the work under the
*direction*.

	Byungchul