[LSF/MM/BPF TOPIC][RFC PATCH v4 00/27] Private Memory Nodes (w/ Compressed RAM)

[Gregory Price <gourry@gourry.net>]

Topic type: MM

Presenter: Gregory Price <gourry@gourry.net>

This series introduces N_MEMORY_PRIVATE, a NUMA node state for memory
managed by the buddy allocator but excluded from normal allocations.

I present it with an end-to-end Compressed RAM service (mm/cram.c)
that would otherwise not be possible (or would be considerably more
difficult, be device-specific, and add to the ZONE_DEVICE boondoggle).


TL;DR
===

N_MEMORY_PRIVATE is all about isolating NUMA nodes and then punching
explicit holes in that isolation to do useful things we couldn't do
before without re-implementing entire portions of mm/ in a driver.


/* This is my memory. There are many like it, but this one is mine. */
rc = add_private_memory_driver_managed(nid, start, size, name, flags,
                                       online_type, private_context);

page = alloc_pages_node(nid, __GFP_PRIVATE, 0);

/* Ok but I want to do something useful with it */
static const struct node_private_ops ops = {
        .migrate_to     = my_migrate_to,
        .folio_migrate  = my_folio_migrate,
        .flags = NP_OPS_MIGRATION | NP_OPS_MEMPOLICY,
};
node_private_set_ops(nid, &ops);

/* And now I can use mempolicy with my memory */
buf = mmap(...);
mbind(buf, len, mode, private_node, ...);
buf[0] = 0xdeadbeef;  /* Faults onto private node */

/* And to be clear, no one else gets my memory */
buf2 = malloc(4096);  /* Standard allocation */
buf2[0] = 0xdeadbeef; /* Can never land on private node */

/* But i can choose to migrate it to the private node */
move_pages(0, 1, &buf, &private_node, NULL, ...);

/* And more fun things like this */


Patchwork
===
A fully working branch based on cxl/next can be found here:
https://github.com/gourryinverse/linux/tree/private_compression

A QEMU device which can inject high/low interrupts can be found here:
https://github.com/gourryinverse/qemu/tree/compressed_cxl_clean

The additional patches on these branches are CXL and DAX driver
housecleaning only tangentially relevant to this RFC, so i've
omitted them for the sake of trying to keep it somewhat clean
here.  Those patches should (hopefully) be going upstream anyway.

Patches 1-22: Core Private Node Infrastructure

  Patch  1:      Introduce N_MEMORY_PRIVATE scaffolding
  Patch  2:      Introduce __GFP_PRIVATE
  Patch  3:      Apply allocation isolation mechanisms
  Patch  4:      Add N_MEMORY nodes to private fallback lists
  Patches 5-9:   Filter operations not yet supported
  Patch 10:      free_folio callback
  Patch 11:      split_folio callback
  Patches 12-20: mm/ service opt-ins:
                   Migration, Mempolicy, Demotion, Write Protect,
                   Reclaim, OOM, NUMA Balancing, Compaction,
                   LongTerm Pinning
  Patch 21:      memory_failure callback
  Patch 22:      Memory hotplug plumbing for private nodes

Patch 23: mm/cram -- Compressed RAM Management

Patches 24-27: CXL Driver examples
  Sysram Regions with Private node support
  Basic Driver Example: (MIGRATION | MEMPOLICY)
  Compression Driver Example (Generic)


Background
===

Today, drivers that want mm-like services on non-general-purpose
memory either use ZONE_DEVICE (self-managed memory) or hotplug into
N_MEMORY and accept the risk of uncontrolled allocation.

Neither option provides what we really want - the ability to:
	1) selectively participate in mm/ subsystems, while
	2) isolating that memory from general purpose use.

Some device-attached memory cannot be managed as fully general-purpose
system RAM.  CXL devices with inline compression, for example, may
corrupt data or crash the machine if the compression ratio drops
below a threshold -- we simply run out of physical memory.

This is a hard problem to solve: how does an operating system deal
with a device that basically lies about how much capacity it has?

(We'll discuss that in the CRAM section)


Core Proposal: N_MEMORY_PRIVATE
===

Introduce N_MEMORY_PRIVATE, a NUMA node state for memory managed by
the buddy allocator, but excluded from normal allocation paths.

Private nodes:

  - Are filtered from zonelist fallback: all existing callers to
    get_page_from_freelist cannot reach these nodes through any
    normal fallback mechanism.

  - Filter allocation requests on __GFP_PRIVATE
    	numa_zone_allowed() excludes them otherwise. 

    Applies to systems with and without cpusets.

    GFP_PRIVATE is (__GFP_PRIVATE | __GFP_THISNODE).

    Services use it when they need to allocate specifically from
    a private node (e.g., CRAM allocating a destination folio).

    No existing allocator path sets __GFP_PRIVATE, so private nodes
    are unreachable by default.

  - Use standard struct page / folio.  No ZONE_DEVICE, no pgmap,
    no struct page metadata limitations.

  - Use a node-scoped metadata structure to accomplish filtering
    and callback support.

  - May participate in the buddy allocator, reclaim, compaction,
    and LRU like normal memory, gated by an opt-in set of flags.

The key abstraction is node_private_ops: a per-node callback table
registered by a driver or service.  

Each callback is individually gated by an NP_OPS_* capability flag.

A driver opts in only to the mm/ operations it needs.

It is similar to ZONE_DEVICE's pgmap at a node granularity.

In fact...


Re-use of ZONE_DEVICE Hooks
===

The callback insertion points deliberately mirror existing ZONE_DEVICE
hooks to minimize the surface area of the mechanism.

I believe this could subsume most DEVICE_COHERENT users, and greatly
simplify the device-managed memory development process (no more
per-driver allocator and migration code).

(Also it's just "So Fresh, So Clean").

The base set of callbacks introduced include:

  free_folio           - mirrors ZONE_DEVICE's
                         free_zone_device_page() hook in
                         __folio_put() / folios_put_refs()

  folio_split          - mirrors ZONE_DEVICE's
  			 called when a huge page is split up

  migrate_to           - demote_folio_list() custom demotion (same
                         site as ZONE_DEVICE demotion rejection)

  folio_migrate        - called when private node folio is moved to
                         another location (e.g. compaction)

  handle_fault         - mirrors the ZONE_DEVICE fault dispatch in
                         handle_pte_fault() (do_wp_page path)

  reclaim_policy       - called by reclaim to let a driver own the
                         boost lifecycle (driver can driver node reclaim)

  memory_failure       - parallels memory_failure_dev_pagemap(),
                         but for online pages that enter the normal
                         hwpoison path

At skip sites (mlock, madvise, KSM, user migration), a unified
folio_is_private_managed() predicate covers both ZONE_DEVICE and
N_MEMORY_PRIVATE folios, consolidating existing zone_device checks
with private node checks rather than adding new ones.

  static inline bool folio_is_private_managed(struct folio *folio)
  {
          return folio_is_zone_device(folio) ||
                 folio_is_private_node(folio);
  }

Most integration points become a one-line swap:

  -     if (folio_is_zone_device(folio))
  +     if (unlikely(folio_is_private_managed(folio)))


Where a one-line integration is insufficient, the integration is
kept as clean as possible with zone_device, rather than simply
adding more call-sites on top of it:

static inline bool folio_managed_handle_fault(struct folio *folio,
  struct vm_fault *vmf, vm_fault_t *ret)
{
  /* Zone device pages use swap entries; handled in do_swap_page */
  if (folio_is_zone_device(folio))
    return false;

  if (folio_is_private_node(folio)) {
    const struct node_private_ops *ops = folio_node_private_ops(folio);

    if (ops && ops->handle_fault) {
      *ret = ops->handle_fault(vmf);
      return true;
    }
  }
  return false;
}



Flag-gated behavior (NP_OPS_*) controls:
===

We use OPS flags to denote what mm/ services we want to allow on our
private node.   I've plumbed these through so far:

  NP_OPS_MIGRATION       - Node supports migration
  NP_OPS_MEMPOLICY       - Node supports mempolicy actions
  NP_OPS_DEMOTION        - Node appears in demotion target lists
  NP_OPS_PROTECT_WRITE   - Node memory is read-only (wrprotect)
  NP_OPS_RECLAIM         - Node supports reclaim
  NP_OPS_NUMA_BALANCING  - Node supports numa balancing
  NP_OPS_COMPACTION      - Node supports compaction
  NP_OPS_LONGTERM_PIN    - Node supports longterm pinning
  NP_OPS_OOM_ELIGIBLE	 - (MIGRATION | DEMOTION), node is reachable
                           as normal system ram storage, so it should
			   be considered in OOM pressure calculations.

I wasn't quite sure how to classify ksm, khugepaged, madvise, and
mlock - so i have omitted those for now.

Most hooks are straightforward.

Including a node as a demotion-eligible target was as simple as:

static void establish_demotion_targets(void)
{
  ..... snip .....
  /*
   * Include private nodes that have opted in to demotion
   * via NP_OPS_DEMOTION.  A node might have custom migrate
   */
  all_memory = node_states[N_MEMORY];
  for_each_node_state(node, N_MEMORY_PRIVATE) {
      if (node_private_has_flag(node, NP_OPS_DEMOTION))
      node_set(node, all_memory);
  }
  ..... snip .....
}

The Migration and Mempolicy support are the two most complex pieces,
and most useful things are built on top of Migration (meaning the
remaining implementations are usually simple).


Private Node Hotplug Lifecycle
===

Registration follows a strict order enforced by
add_private_memory_driver_managed():

  1. Driver calls add_private_memory_driver_managed(nid, start,
     size, resource_name, mhp_flags, online_type, &np).

  2. node_private_register(nid, &np) stores the driver's
     node_private in pgdat and sets pgdat->private.  N_MEMORY and
     N_MEMORY_PRIVATE are mutually exclusive -- registration fails
     with -EBUSY if the node already has N_MEMORY set.

     Only one driver may register per private node.

  3. Memory is hotplugged via __add_memory_driver_managed().

     When online_pages() runs, it checks pgdat->private and sets
     N_MEMORY_PRIVATE instead of N_MEMORY.  

     Zonelist construction gives private nodes a self-only NOFALLBACK
     list and an N_MEMORY fallback list (so kernel/slab allocations on
     behalf of private node work can fall back to DRAM).

  4. kswapd and kcompactd are NOT started for private nodes.  The
     owning service is responsible for driving reclaim if needed
     (e.g., CRAM uses watermark_boost to wake kswapd on demand).

Teardown is the reverse:

  1. Driver calls offline_and_remove_private_memory(nid, start,
     size).

  2. offline_pages() offlines the memory.  When the last block is
     offlined, N_MEMORY_PRIVATE is cleared automatically.

  3. node_private_unregister() clears pgdat->node_private and
     drops the refcount.  It refuses to unregister (-EBUSY) if
     N_MEMORY_PRIVATE is still set (other memory ranges remain).

The driver is responsible for ensuring memory is hot-unpluggable
before teardown.  The service must ensure all memory is cleaned
up before hot-unplug - or the service must support migration (so
memory_hotplug.c can evacuate the memory itself).

In the CRAM example, the service supports migration, so memory
hot-unplug can remove memory without any special infrastructure.


Application: Compressed RAM (mm/cram)
===

Compressed RAM has a serious design issue:  Its capacity a lie.

A compression device reports more capacity than it physically has.
If workloads write faster than the OS can reclaim from the device,
we run out of real backing store and corrupt data or crash.

I call this problem: "Trying to Out Run A Bear"

I.e. This is only stable as long as we stay ahead of the pressure.

We don't want to design a system where stability depends on outrunning
a bear - I am slow and do not know where to acquire bear spray.

  Fun fact:   Grizzly bears have a top-speed of 56-64 km/h.
  Unfun Fact: Humans typically top out at ~24 km/h.

This MVP takes a conservative position:

   all compressed memory is mapped read-only.

  - Folios reach the private node only via reclaim (demotion)
  - migrate_to implements custom demotion with backpressure.
  - fixup_migration_pte write-protects PTEs on arrival.
  - wrprotect hooks prevent silent upgrades
  - handle_fault promotes folios back to DRAM on write.
  - free_folio scrubs stale data before buddy free.

Because pages are read-only, writes can never cause runaway
compression ratio loss behind the allocator's back.  Every write
goes through handle_fault, which promotes the folio to DRAM first.

The device only ever sees net compression (demotion in) and explicit
decompression (promotion out via fault or reclaim), and has a much
wider timeframe to respond to poor compression scenarios.

That means there's no bear to out run. The bears are safely asleep in
their bear den, and even if they show up we have a bear-proof cage.

The backpressure system is our bear-proof cage: the driver reports real
device utilization (generalized via watermark_boost on the private
node's zone), and CRAM throttles demotion when capacity is tight.

If compression ratios are bad, we stop demoting pages and start
evicting pages aggressively.

The service as designed is ~350 functional lines of code because it
re-uses mm/ services:

  - Existing reclaim/vmscan code handles demotion.
  - Existing migration code handles migration to/from.
  - Existing page fault handling dispatches faults.

The driver contains all the CXL nastiness core developers don't want
anything to do with - No vendor logic touches mm/ internals.



Future CRAM : Loosening the read-only constraint
===

The read-only model is safe but conservative.  For workloads where
compressed pages are occasionally written, the promotion fault adds
latency.  A future optimization could allow a tunable fraction of
compressed pages to be mapped writable, accepting some risk of
write-driven decompression in exchange for lower overhead.

The private node ops make this straightforward:

  - Adjust fixup_migration_pte to selectively skip
    write-protection.
  - Use the backpressure system to either revoke writable mappings,
    deny additional demotions, or evict when device pressure rises.

This comes at a mild memory overhead: 32MB of DRAM per 1TB of CRAM.
(1 bit per 4KB page).

This is not proposed here, but it should be somewhat trivial.


Discussion Topics
===
0. Obviously I've included the set as an RFC, please rip it apart.

1. Is N_MEMORY_PRIVATE the right isolation abstraction, or should
   this extend ZONE_DEVICE?  Prior feedback pushed away from new
   ZONE logic, but this will likely be debated further.

   My comments on this:

   ZONE_DEVICE requires re-implementing every service you want to
   provide to your device memory, including basic allocation.

   Private nodes use real struct pages with no metadata
   limitations, participate in the buddy allocator, and get NUMA
   topology for free.

2. Can this subsume ZONE_DEVICE COHERENT users?  The architecture
   was designed with this in mind, but it is only a thought experiment.

3. Is a dedicated mm/ service (cram) the right place for compressed
   memory management, or should this be purely driver-side until
   more devices exist?

   I wrote it this way because I forsee more "innovation" in the
   compressed RAM space given current... uh... "Market Conditions".

   I don't see CRAM being CXL-specific, though the only solutions I've
   seen have been CXL.  Nothing is stopping someone from soldering such
   memory directly to a PCB.

5. Where is your hardware-backed data that shows this works?

   I should have some by conference time.

Thanks for reading
Gregory (Gourry)


Gregory Price (27):
  numa: introduce N_MEMORY_PRIVATE node state
  mm,cpuset: gate allocations from N_MEMORY_PRIVATE behind __GFP_PRIVATE
  mm/page_alloc: add numa_zone_allowed() and wire it up
  mm/page_alloc: Add private node handling to build_zonelists
  mm: introduce folio_is_private_managed() unified predicate
  mm/mlock: skip mlock for managed-memory folios
  mm/madvise: skip madvise for managed-memory folios
  mm/ksm: skip KSM for managed-memory folios
  mm/khugepaged: skip private node folios when trying to collapse.
  mm/swap: add free_folio callback for folio release cleanup
  mm/huge_memory.c: add private node folio split notification callback
  mm/migrate: NP_OPS_MIGRATION - support private node user migration
  mm/mempolicy: NP_OPS_MEMPOLICY - support private node mempolicy
  mm/memory-tiers: NP_OPS_DEMOTION - support private node demotion
  mm/mprotect: NP_OPS_PROTECT_WRITE - gate PTE/PMD write-upgrades
  mm: NP_OPS_RECLAIM - private node reclaim participation
  mm/oom: NP_OPS_OOM_ELIGIBLE - private node OOM participation
  mm/memory: NP_OPS_NUMA_BALANCING - private node NUMA balancing
  mm/compaction: NP_OPS_COMPACTION - private node compaction support
  mm/gup: NP_OPS_LONGTERM_PIN - private node longterm pin support
  mm/memory-failure: add memory_failure callback to node_private_ops
  mm/memory_hotplug: add add_private_memory_driver_managed()
  mm/cram: add compressed ram memory management subsystem
  cxl/core: Add cxl_sysram region type
  cxl/core: Add private node support to cxl_sysram
  cxl: add cxl_mempolicy sample PCI driver
  cxl: add cxl_compression PCI driver

 drivers/base/node.c                           |  250 +++-
 drivers/cxl/Kconfig                           |    2 +
 drivers/cxl/Makefile                          |    2 +
 drivers/cxl/core/Makefile                     |    1 +
 drivers/cxl/core/core.h                       |    4 +
 drivers/cxl/core/port.c                       |    2 +
 drivers/cxl/core/region_sysram.c              |  381 ++++++
 drivers/cxl/cxl.h                             |   53 +
 drivers/cxl/type3_drivers/Kconfig             |    3 +
 drivers/cxl/type3_drivers/Makefile            |    3 +
 .../cxl/type3_drivers/cxl_compression/Kconfig |   20 +
 .../type3_drivers/cxl_compression/Makefile    |    4 +
 .../cxl_compression/compression.c             | 1025 +++++++++++++++++
 .../cxl/type3_drivers/cxl_mempolicy/Kconfig   |   16 +
 .../cxl/type3_drivers/cxl_mempolicy/Makefile  |    4 +
 .../type3_drivers/cxl_mempolicy/mempolicy.c   |  297 +++++
 include/linux/cpuset.h                        |    9 -
 include/linux/cram.h                          |   66 ++
 include/linux/gfp_types.h                     |   15 +-
 include/linux/memory-tiers.h                  |    9 +
 include/linux/memory_hotplug.h                |   11 +
 include/linux/migrate.h                       |   17 +-
 include/linux/mm.h                            |   22 +
 include/linux/mmzone.h                        |   16 +
 include/linux/node_private.h                  |  532 +++++++++
 include/linux/nodemask.h                      |    1 +
 include/trace/events/mmflags.h                |    4 +-
 include/uapi/linux/mempolicy.h                |    1 +
 kernel/cgroup/cpuset.c                        |   49 +-
 mm/Kconfig                                    |   10 +
 mm/Makefile                                   |    1 +
 mm/compaction.c                               |   32 +-
 mm/cram.c                                     |  508 ++++++++
 mm/damon/paddr.c                              |    3 +
 mm/huge_memory.c                              |   23 +-
 mm/hugetlb.c                                  |    2 +-
 mm/internal.h                                 |  226 +++-
 mm/khugepaged.c                               |    7 +-
 mm/ksm.c                                      |    9 +-
 mm/madvise.c                                  |    5 +-
 mm/memory-failure.c                           |   15 +
 mm/memory-tiers.c                             |   46 +-
 mm/memory.c                                   |   26 +
 mm/memory_hotplug.c                           |  122 +-
 mm/mempolicy.c                                |   69 +-
 mm/migrate.c                                  |   63 +-
 mm/mlock.c                                    |    5 +-
 mm/mprotect.c                                 |    4 +-
 mm/oom_kill.c                                 |   52 +-
 mm/page_alloc.c                               |   79 +-
 mm/rmap.c                                     |    4 +-
 mm/slub.c                                     |    3 +-
 mm/swap.c                                     |   21 +-
 mm/vmscan.c                                   |   55 +-
 54 files changed, 4057 insertions(+), 152 deletions(-)
 create mode 100644 drivers/cxl/core/region_sysram.c
 create mode 100644 drivers/cxl/type3_drivers/Kconfig
 create mode 100644 drivers/cxl/type3_drivers/Makefile
 create mode 100644 drivers/cxl/type3_drivers/cxl_compression/Kconfig
 create mode 100644 drivers/cxl/type3_drivers/cxl_compression/Makefile
 create mode 100644 drivers/cxl/type3_drivers/cxl_compression/compression.c
 create mode 100644 drivers/cxl/type3_drivers/cxl_mempolicy/Kconfig
 create mode 100644 drivers/cxl/type3_drivers/cxl_mempolicy/Makefile
 create mode 100644 drivers/cxl/type3_drivers/cxl_mempolicy/mempolicy.c
 create mode 100644 include/linux/cram.h
 create mode 100644 include/linux/node_private.h
 create mode 100644 mm/cram.c

-- 
2.53.0