Re: [LSF/MM/BPF TOPIC][RFC PATCH v4 00/27] Private Memory Nodes (w/ Compressed RAM)

["David Hildenbrand (Arm)" <david@kernel.org>]

On 2/22/26 09:48, Gregory Price wrote:
> Topic type: MM
> 
> Presenter: Gregory Price <gourry@gourry.net>
> 
> This series introduces N_MEMORY_PRIVATE, a NUMA node state for memory
> managed by the buddy allocator but excluded from normal allocations.
> 
> I present it with an end-to-end Compressed RAM service (mm/cram.c)
> that would otherwise not be possible (or would be considerably more
> difficult, be device-specific, and add to the ZONE_DEVICE boondoggle).
> 
> 
> TL;DR
> ===
> 
> N_MEMORY_PRIVATE is all about isolating NUMA nodes and then punching
> explicit holes in that isolation to do useful things we couldn't do
> before without re-implementing entire portions of mm/ in a driver.
> 
> 
> /* This is my memory. There are many like it, but this one is mine. */
> rc = add_private_memory_driver_managed(nid, start, size, name, flags,
>                                         online_type, private_context);
> 
> page = alloc_pages_node(nid, __GFP_PRIVATE, 0);
> 
> /* Ok but I want to do something useful with it */
> static const struct node_private_ops ops = {
>          .migrate_to     = my_migrate_to,
>          .folio_migrate  = my_folio_migrate,
>          .flags = NP_OPS_MIGRATION | NP_OPS_MEMPOLICY,
> };
> node_private_set_ops(nid, &ops);
> 
> /* And now I can use mempolicy with my memory */
> buf = mmap(...);
> mbind(buf, len, mode, private_node, ...);
> buf[0] = 0xdeadbeef;  /* Faults onto private node */
> 
> /* And to be clear, no one else gets my memory */
> buf2 = malloc(4096);  /* Standard allocation */
> buf2[0] = 0xdeadbeef; /* Can never land on private node */
> 
> /* But i can choose to migrate it to the private node */
> move_pages(0, 1, &buf, &private_node, NULL, ...);
> 
> /* And more fun things like this */
> 
> 
> Patchwork
> ===
> A fully working branch based on cxl/next can be found here:
> https://github.com/gourryinverse/linux/tree/private_compression
> 
> A QEMU device which can inject high/low interrupts can be found here:
> https://github.com/gourryinverse/qemu/tree/compressed_cxl_clean
> 
> The additional patches on these branches are CXL and DAX driver
> housecleaning only tangentially relevant to this RFC, so i've
> omitted them for the sake of trying to keep it somewhat clean
> here.  Those patches should (hopefully) be going upstream anyway.
> 
> Patches 1-22: Core Private Node Infrastructure
> 
>    Patch  1:      Introduce N_MEMORY_PRIVATE scaffolding
>    Patch  2:      Introduce __GFP_PRIVATE
>    Patch  3:      Apply allocation isolation mechanisms
>    Patch  4:      Add N_MEMORY nodes to private fallback lists
>    Patches 5-9:   Filter operations not yet supported
>    Patch 10:      free_folio callback
>    Patch 11:      split_folio callback
>    Patches 12-20: mm/ service opt-ins:
>                     Migration, Mempolicy, Demotion, Write Protect,
>                     Reclaim, OOM, NUMA Balancing, Compaction,
>                     LongTerm Pinning
>    Patch 21:      memory_failure callback
>    Patch 22:      Memory hotplug plumbing for private nodes
> 
> Patch 23: mm/cram -- Compressed RAM Management
> 
> Patches 24-27: CXL Driver examples
>    Sysram Regions with Private node support
>    Basic Driver Example: (MIGRATION | MEMPOLICY)
>    Compression Driver Example (Generic)
> 
> 
> Background
> ===
> 
> Today, drivers that want mm-like services on non-general-purpose
> memory either use ZONE_DEVICE (self-managed memory) or hotplug into
> N_MEMORY and accept the risk of uncontrolled allocation.
> 
> Neither option provides what we really want - the ability to:
> 	1) selectively participate in mm/ subsystems, while
> 	2) isolating that memory from general purpose use.
> 
> Some device-attached memory cannot be managed as fully general-purpose
> system RAM.  CXL devices with inline compression, for example, may
> corrupt data or crash the machine if the compression ratio drops
> below a threshold -- we simply run out of physical memory.
> 
> This is a hard problem to solve: how does an operating system deal
> with a device that basically lies about how much capacity it has?
> 
> (We'll discuss that in the CRAM section)
> 
> 
> Core Proposal: N_MEMORY_PRIVATE
> ===
> 
> Introduce N_MEMORY_PRIVATE, a NUMA node state for memory managed by
> the buddy allocator, but excluded from normal allocation paths.
> 
> Private nodes:
> 
>    - Are filtered from zonelist fallback: all existing callers to
>      get_page_from_freelist cannot reach these nodes through any
>      normal fallback mechanism.
> 
>    - Filter allocation requests on __GFP_PRIVATE
>      	numa_zone_allowed() excludes them otherwise.
> 
>      Applies to systems with and without cpusets.
> 
>      GFP_PRIVATE is (__GFP_PRIVATE | __GFP_THISNODE).
> 
>      Services use it when they need to allocate specifically from
>      a private node (e.g., CRAM allocating a destination folio).
> 
>      No existing allocator path sets __GFP_PRIVATE, so private nodes
>      are unreachable by default.
> 
>    - Use standard struct page / folio.  No ZONE_DEVICE, no pgmap,
>      no struct page metadata limitations.
> 
>    - Use a node-scoped metadata structure to accomplish filtering
>      and callback support.
> 
>    - May participate in the buddy allocator, reclaim, compaction,
>      and LRU like normal memory, gated by an opt-in set of flags.
> 
> The key abstraction is node_private_ops: a per-node callback table
> registered by a driver or service.
> 
> Each callback is individually gated by an NP_OPS_* capability flag.
> 
> A driver opts in only to the mm/ operations it needs.
> 
> It is similar to ZONE_DEVICE's pgmap at a node granularity.
> 
> In fact...
> 
> 
> Re-use of ZONE_DEVICE Hooks
> ===
> 
> The callback insertion points deliberately mirror existing ZONE_DEVICE
> hooks to minimize the surface area of the mechanism.
> 
> I believe this could subsume most DEVICE_COHERENT users, and greatly
> simplify the device-managed memory development process (no more
> per-driver allocator and migration code).
> 
> (Also it's just "So Fresh, So Clean").
> 
> The base set of callbacks introduced include:
> 
>    free_folio           - mirrors ZONE_DEVICE's
>                           free_zone_device_page() hook in
>                           __folio_put() / folios_put_refs()
> 
>    folio_split          - mirrors ZONE_DEVICE's
>    			 called when a huge page is split up
> 
>    migrate_to           - demote_folio_list() custom demotion (same
>                           site as ZONE_DEVICE demotion rejection)
> 
>    folio_migrate        - called when private node folio is moved to
>                           another location (e.g. compaction)
> 
>    handle_fault         - mirrors the ZONE_DEVICE fault dispatch in
>                           handle_pte_fault() (do_wp_page path)
> 
>    reclaim_policy       - called by reclaim to let a driver own the
>                           boost lifecycle (driver can driver node reclaim)
> 
>    memory_failure       - parallels memory_failure_dev_pagemap(),
>                           but for online pages that enter the normal
>                           hwpoison path
> 
> At skip sites (mlock, madvise, KSM, user migration), a unified
> folio_is_private_managed() predicate covers both ZONE_DEVICE and
> N_MEMORY_PRIVATE folios, consolidating existing zone_device checks
> with private node checks rather than adding new ones.
> 
>    static inline bool folio_is_private_managed(struct folio *folio)
>    {
>            return folio_is_zone_device(folio) ||
>                   folio_is_private_node(folio);
>    }
> 
> Most integration points become a one-line swap:
> 
>    -     if (folio_is_zone_device(folio))
>    +     if (unlikely(folio_is_private_managed(folio)))
> 
> 
> Where a one-line integration is insufficient, the integration is
> kept as clean as possible with zone_device, rather than simply
> adding more call-sites on top of it:
> 
> static inline bool folio_managed_handle_fault(struct folio *folio,
>    struct vm_fault *vmf, vm_fault_t *ret)
> {
>    /* Zone device pages use swap entries; handled in do_swap_page */
>    if (folio_is_zone_device(folio))
>      return false;
> 
>    if (folio_is_private_node(folio)) {
>      const struct node_private_ops *ops = folio_node_private_ops(folio);
> 
>      if (ops && ops->handle_fault) {
>        *ret = ops->handle_fault(vmf);
>        return true;
>      }
>    }
>    return false;
> }
> 
> 
> 
> Flag-gated behavior (NP_OPS_*) controls:
> ===
> 
> We use OPS flags to denote what mm/ services we want to allow on our
> private node.   I've plumbed these through so far:
> 
>    NP_OPS_MIGRATION       - Node supports migration
>    NP_OPS_MEMPOLICY       - Node supports mempolicy actions
>    NP_OPS_DEMOTION        - Node appears in demotion target lists
>    NP_OPS_PROTECT_WRITE   - Node memory is read-only (wrprotect)
>    NP_OPS_RECLAIM         - Node supports reclaim
>    NP_OPS_NUMA_BALANCING  - Node supports numa balancing
>    NP_OPS_COMPACTION      - Node supports compaction
>    NP_OPS_LONGTERM_PIN    - Node supports longterm pinning
>    NP_OPS_OOM_ELIGIBLE	 - (MIGRATION | DEMOTION), node is reachable
>                             as normal system ram storage, so it should
> 			   be considered in OOM pressure calculations.
> 
> I wasn't quite sure how to classify ksm, khugepaged, madvise, and
> mlock - so i have omitted those for now.
> 
> Most hooks are straightforward.
> 
> Including a node as a demotion-eligible target was as simple as:
> 
> static void establish_demotion_targets(void)
> {
>    ..... snip .....
>    /*
>     * Include private nodes that have opted in to demotion
>     * via NP_OPS_DEMOTION.  A node might have custom migrate
>     */
>    all_memory = node_states[N_MEMORY];
>    for_each_node_state(node, N_MEMORY_PRIVATE) {
>        if (node_private_has_flag(node, NP_OPS_DEMOTION))
>        node_set(node, all_memory);
>    }
>    ..... snip .....
> }
> 
> The Migration and Mempolicy support are the two most complex pieces,
> and most useful things are built on top of Migration (meaning the
> remaining implementations are usually simple).
> 
> 
> Private Node Hotplug Lifecycle
> ===
> 
> Registration follows a strict order enforced by
> add_private_memory_driver_managed():
> 
>    1. Driver calls add_private_memory_driver_managed(nid, start,
>       size, resource_name, mhp_flags, online_type, &np).
> 
>    2. node_private_register(nid, &np) stores the driver's
>       node_private in pgdat and sets pgdat->private.  N_MEMORY and
>       N_MEMORY_PRIVATE are mutually exclusive -- registration fails
>       with -EBUSY if the node already has N_MEMORY set.
> 
>       Only one driver may register per private node.
> 
>    3. Memory is hotplugged via __add_memory_driver_managed().
> 
>       When online_pages() runs, it checks pgdat->private and sets
>       N_MEMORY_PRIVATE instead of N_MEMORY.
> 
>       Zonelist construction gives private nodes a self-only NOFALLBACK
>       list and an N_MEMORY fallback list (so kernel/slab allocations on
>       behalf of private node work can fall back to DRAM).
> 
>    4. kswapd and kcompactd are NOT started for private nodes.  The
>       owning service is responsible for driving reclaim if needed
>       (e.g., CRAM uses watermark_boost to wake kswapd on demand).
> 
> Teardown is the reverse:
> 
>    1. Driver calls offline_and_remove_private_memory(nid, start,
>       size).
> 
>    2. offline_pages() offlines the memory.  When the last block is
>       offlined, N_MEMORY_PRIVATE is cleared automatically.
> 
>    3. node_private_unregister() clears pgdat->node_private and
>       drops the refcount.  It refuses to unregister (-EBUSY) if
>       N_MEMORY_PRIVATE is still set (other memory ranges remain).
> 
> The driver is responsible for ensuring memory is hot-unpluggable
> before teardown.  The service must ensure all memory is cleaned
> up before hot-unplug - or the service must support migration (so
> memory_hotplug.c can evacuate the memory itself).
> 
> In the CRAM example, the service supports migration, so memory
> hot-unplug can remove memory without any special infrastructure.
> 
> 
> Application: Compressed RAM (mm/cram)
> ===
> 
> Compressed RAM has a serious design issue:  Its capacity a lie.
> 
> A compression device reports more capacity than it physically has.
> If workloads write faster than the OS can reclaim from the device,
> we run out of real backing store and corrupt data or crash.
> 
> I call this problem: "Trying to Out Run A Bear"
> 
> I.e. This is only stable as long as we stay ahead of the pressure.
> 
> We don't want to design a system where stability depends on outrunning
> a bear - I am slow and do not know where to acquire bear spray.
> 
>    Fun fact:   Grizzly bears have a top-speed of 56-64 km/h.
>    Unfun Fact: Humans typically top out at ~24 km/h.
> 
> This MVP takes a conservative position:
> 
>     all compressed memory is mapped read-only.
> 
>    - Folios reach the private node only via reclaim (demotion)
>    - migrate_to implements custom demotion with backpressure.
>    - fixup_migration_pte write-protects PTEs on arrival.
>    - wrprotect hooks prevent silent upgrades
>    - handle_fault promotes folios back to DRAM on write.
>    - free_folio scrubs stale data before buddy free.
> 
> Because pages are read-only, writes can never cause runaway
> compression ratio loss behind the allocator's back.  Every write
> goes through handle_fault, which promotes the folio to DRAM first.
> 
> The device only ever sees net compression (demotion in) and explicit
> decompression (promotion out via fault or reclaim), and has a much
> wider timeframe to respond to poor compression scenarios.
> 
> That means there's no bear to out run. The bears are safely asleep in
> their bear den, and even if they show up we have a bear-proof cage.
> 
> The backpressure system is our bear-proof cage: the driver reports real
> device utilization (generalized via watermark_boost on the private
> node's zone), and CRAM throttles demotion when capacity is tight.
> 
> If compression ratios are bad, we stop demoting pages and start
> evicting pages aggressively.
> 
> The service as designed is ~350 functional lines of code because it
> re-uses mm/ services:
> 
>    - Existing reclaim/vmscan code handles demotion.
>    - Existing migration code handles migration to/from.
>    - Existing page fault handling dispatches faults.
> 
> The driver contains all the CXL nastiness core developers don't want
> anything to do with - No vendor logic touches mm/ internals.
> 
> 
> 
> Future CRAM : Loosening the read-only constraint
> ===
> 
> The read-only model is safe but conservative.  For workloads where
> compressed pages are occasionally written, the promotion fault adds
> latency.  A future optimization could allow a tunable fraction of
> compressed pages to be mapped writable, accepting some risk of
> write-driven decompression in exchange for lower overhead.
> 
> The private node ops make this straightforward:
> 
>    - Adjust fixup_migration_pte to selectively skip
>      write-protection.
>    - Use the backpressure system to either revoke writable mappings,
>      deny additional demotions, or evict when device pressure rises.
> 
> This comes at a mild memory overhead: 32MB of DRAM per 1TB of CRAM.
> (1 bit per 4KB page).
> 
> This is not proposed here, but it should be somewhat trivial.
> 
> 
> Discussion Topics
> ===
> 0. Obviously I've included the set as an RFC, please rip it apart.
> 
> 1. Is N_MEMORY_PRIVATE the right isolation abstraction, or should
>     this extend ZONE_DEVICE?  Prior feedback pushed away from new
>     ZONE logic, but this will likely be debated further.
> 
>     My comments on this:
> 
>     ZONE_DEVICE requires re-implementing every service you want to
>     provide to your device memory, including basic allocation.
> 
>     Private nodes use real struct pages with no metadata
>     limitations, participate in the buddy allocator, and get NUMA
>     topology for free.
> 
> 2. Can this subsume ZONE_DEVICE COHERENT users?  The architecture
>     was designed with this in mind, but it is only a thought experiment.
> 
> 3. Is a dedicated mm/ service (cram) the right place for compressed
>     memory management, or should this be purely driver-side until
>     more devices exist?
> 
>     I wrote it this way because I forsee more "innovation" in the
>     compressed RAM space given current... uh... "Market Conditions".
> 
>     I don't see CRAM being CXL-specific, though the only solutions I've
>     seen have been CXL.  Nothing is stopping someone from soldering such
>     memory directly to a PCB.
> 
> 5. Where is your hardware-backed data that shows this works?
> 
>     I should have some by conference time.
> 
> Thanks for reading
> Gregory (Gourry)
> 
> 
> Gregory Price (27):
>    numa: introduce N_MEMORY_PRIVATE node state
>    mm,cpuset: gate allocations from N_MEMORY_PRIVATE behind __GFP_PRIVATE
>    mm/page_alloc: add numa_zone_allowed() and wire it up
>    mm/page_alloc: Add private node handling to build_zonelists
>    mm: introduce folio_is_private_managed() unified predicate
>    mm/mlock: skip mlock for managed-memory folios
>    mm/madvise: skip madvise for managed-memory folios
>    mm/ksm: skip KSM for managed-memory folios
>    mm/khugepaged: skip private node folios when trying to collapse.
>    mm/swap: add free_folio callback for folio release cleanup
>    mm/huge_memory.c: add private node folio split notification callback
>    mm/migrate: NP_OPS_MIGRATION - support private node user migration
>    mm/mempolicy: NP_OPS_MEMPOLICY - support private node mempolicy
>    mm/memory-tiers: NP_OPS_DEMOTION - support private node demotion
>    mm/mprotect: NP_OPS_PROTECT_WRITE - gate PTE/PMD write-upgrades

I'm concerned about adding more special-casing (similar to what we 
already added for ZONE_DEVICE) all over the place.

Like the whole folio_managed_() stuff in mprotect.c

Having that said, sounds like a reasonable topic to discuss.

-- 
Cheers,

David