Re: [LSF/MM/BPF TOPIC][RFC PATCH v4 00/27] Private Memory Nodes (w/ Compressed RAM)

David Hildenbrand (Arm)
2026-02-23T13:07:41Z

On 2/22/26 09:48, Gregory Price wrote:
> Topic type: MM
>
> Presenter: Gregory Price <gourry@gourry.net>
>
> This series introduces N_MEMORY_PRIVATE, a NUMA node state for memory
> managed by the buddy allocator but excluded from normal allocations.
>
> I present it with an end-to-end Compressed RAM service (mm/cram.c)
> that would otherwise not be possible (or would be considerably more
> difficult, be device-specific, and add to the ZONE_DEVICE boondoggle).
>
>
> TL;DR
> ===
>
> N_MEMORY_PRIVATE is all about isolating NUMA nodes and then punching
> explicit holes in that isolation to do useful things we couldn't do
> before without re-implementing entire portions of mm/ in a driver.
>
>
> /* This is my memory. There are many like it, but this one is mine. */
> rc = add_private_memory_driver_managed(nid, start, size, name, flags,
> online_type, private_context);
>
> page = alloc_pages_node(nid, __GFP_PRIVATE, 0);
>
> /* Ok but I want to do something useful with it */
> static const struct node_private_ops ops = {
> .migrate_to = my_migrate_to,
> .folio_migrate = my_folio_migrate,
> .flags = NP_OPS_MIGRATION | NP_OPS_MEMPOLICY,
> };
> node_private_set_ops(nid, &ops);
>
> /* And now I can use mempolicy with my memory */
> buf = mmap(...);
> mbind(buf, len, mode, private_node, ...);
> buf[0] = 0xdeadbeef; /* Faults onto private node */
>
> /* And to be clear, no one else gets my memory */
> buf2 = malloc(4096); /* Standard allocation */
> buf2[0] = 0xdeadbeef; /* Can never land on private node */
>
> /* But i can choose to migrate it to the private node */
> move_pages(0, 1, &buf, &private_node, NULL, ...);
>
> /* And more fun things like this */
>
>
> Patchwork
> ===
> A fully working branch based on cxl/next can be found here:
> https://github.com/gourryinverse/linux/tree/private_compression
>
> A QEMU device which can inject high/low interrupts can be found here:
> https://github.com/gourryinverse/qemu/tree/compressed_cxl_clean
>
> The additional patches on these branches are CXL and DAX driver
> housecleaning only tangentially relevant to this RFC, so i've
> omitted them for the sake of trying to keep it somewhat clean
> here. Those patches should (hopefully) be going upstream anyway.
>
> Patches 1-22: Core Private Node Infrastructure
>
> Patch 1: Introduce N_MEMORY_PRIVATE scaffolding
> Patch 2: Introduce __GFP_PRIVATE
> Patch 3: Apply allocation isolation mechanisms
> Patch 4: Add N_MEMORY nodes to private fallback lists
> Patches 5-9: Filter operations not yet supported
> Patch 10: free_folio callback
> Patch 11: split_folio callback
> Patches 12-20: mm/ service opt-ins:
> Migration, Mempolicy, Demotion, Write Protect,
> Reclaim, OOM, NUMA Balancing, Compaction,
> LongTerm Pinning
> Patch 21: memory_failure callback
> Patch 22: Memory hotplug plumbing for private nodes
>
> Patch 23: mm/cram -- Compressed RAM Management
>
> Patches 24-27: CXL Driver examples
> Sysram Regions with Private node support
> Basic Driver Example: (MIGRATION | MEMPOLICY)
> Compression Driver Example (Generic)
>
>
> Background
> ===
>
> Today, drivers that want mm-like services on non-general-purpose
> memory either use ZONE_DEVICE (self-managed memory) or hotplug into
> N_MEMORY and accept the risk of uncontrolled allocation.
>
> Neither option provides what we really want - the ability to:
> 1) selectively participate in mm/ subsystems, while
> 2) isolating that memory from general purpose use.
>
> Some device-attached memory cannot be managed as fully general-purpose
> system RAM. CXL devices with inline compression, for example, may
> corrupt data or crash the machine if the compression ratio drops
> below a threshold -- we simply run out of physical memory.
>
> This is a hard problem to solve: how does an operating system deal
> with a device that basically lies about how much capacity it has?
>
> (We'll discuss that in the CRAM section)
>
>
> Core Proposal: N_MEMORY_PRIVATE
> ===
>
> Introduce N_MEMORY_PRIVATE, a NUMA node state for memory managed by
> the buddy allocator, but excluded from normal allocation paths.
>
> Private nodes:
>
> - Are filtered from zonelist fallback: all existing callers to
> get_page_from_freelist cannot reach these nodes through any
> normal fallback mechanism.
>
> - Filter allocation requests on __GFP_PRIVATE
> numa_zone_allowed() excludes them otherwise.
>
> Applies to systems with and without cpusets.
>
> GFP_PRIVATE is (__GFP_PRIVATE | __GFP_THISNODE).
>
> Services use it when they need to allocate specifically from
> a private node (e.g., CRAM allocating a destination folio).
>
> No existing allocator path sets __GFP_PRIVATE, so private nodes
> are unreachable by default.
>
> - Use standard struct page / folio. No ZONE_DEVICE, no pgmap,
> no struct page metadata limitations.
>
> - Use a node-scoped metadata structure to accomplish filtering
> and callback support.
>
> - May participate in the buddy allocator, reclaim, compaction,
> and LRU like normal memory, gated by an opt-in set of flags.
>
> The key abstraction is node_private_ops: a per-node callback table
> registered by a driver or service.
>
> Each callback is individually gated by an NP_OPS_* capability flag.
>
> A driver opts in only to the mm/ operations it needs.
>
> It is similar to ZONE_DEVICE's pgmap at a node granularity.
>
> In fact...
>
>
> Re-use of ZONE_DEVICE Hooks
> ===
>
> The callback insertion points deliberately mirror existing ZONE_DEVICE
> hooks to minimize the surface area of the mechanism.
>
> I believe this could subsume most DEVICE_COHERENT users, and greatly
> simplify the device-managed memory development process (no more
> per-driver allocator and migration code).
>
> (Also it's just "So Fresh, So Clean").
>
> The base set of callbacks introduced include:
>
> free_folio - mirrors ZONE_DEVICE's
> free_zone_device_page() hook in
> __folio_put() / folios_put_refs()
>
> folio_split - mirrors ZONE_DEVICE's
> called when a huge page is split up
>
> migrate_to - demote_folio_list() custom demotion (same
> site as ZONE_DEVICE demotion rejection)
>
> folio_migrate - called when private node folio is moved to
> another location (e.g. compaction)
>
> handle_fault - mirrors the ZONE_DEVICE fault dispatch in
> handle_pte_fault() (do_wp_page path)
>
> reclaim_policy - called by reclaim to let a driver own the
> boost lifecycle (driver can driver node reclaim)
>
> memory_failure - parallels memory_failure_dev_pagemap(),
> but for online pages that enter the normal
> hwpoison path
>
> At skip sites (mlock, madvise, KSM, user migration), a unified
> folio_is_private_managed() predicate covers both ZONE_DEVICE and
> N_MEMORY_PRIVATE folios, consolidating existing zone_device checks
> with private node checks rather than adding new ones.
>
> static inline bool folio_is_private_managed(struct folio *folio)
> {
> return folio_is_zone_device(folio) ||
> folio_is_private_node(folio);
> }
>
> Most integration points become a one-line swap:
>
> - if (folio_is_zone_device(folio))
> + if (unlikely(folio_is_private_managed(folio)))
>
>
> Where a one-line integration is insufficient, the integration is
> kept as clean as possible with zone_device, rather than simply
> adding more call-sites on top of it:
>
> static inline bool folio_managed_handle_fault(struct folio *folio,
> struct vm_fault *vmf, vm_fault_t *ret)
> {
> /* Zone device pages use swap entries; handled in do_swap_page */
> if (folio_is_zone_device(folio))
> return false;
>
> if (folio_is_private_node(folio)) {
> const struct node_private_ops *ops = folio_node_private_ops(folio);
>
> if (ops && ops->handle_fault) {
> *ret = ops->handle_fault(vmf);
> return true;
> }
> }
> return false;
> }
>
>
>
> Flag-gated behavior (NP_OPS_*) controls:
> ===
>
> We use OPS flags to denote what mm/ services we want to allow on our
> private node. I've plumbed these through so far:
>
> NP_OPS_MIGRATION - Node supports migration
> NP_OPS_MEMPOLICY - Node supports mempolicy actions
> NP_OPS_DEMOTION - Node appears in demotion target lists
> NP_OPS_PROTECT_WRITE - Node memory is read-only (wrprotect)
> NP_OPS_RECLAIM - Node supports reclaim
> NP_OPS_NUMA_BALANCING - Node supports numa balancing
> NP_OPS_COMPACTION - Node supports compaction
> NP_OPS_LONGTERM_PIN - Node supports longterm pinning
> NP_OPS_OOM_ELIGIBLE - (MIGRATION | DEMOTION), node is reachable
> as normal system ram storage, so it should
> be considered in OOM pressure calculations.
>
> I wasn't quite sure how to classify ksm, khugepaged, madvise, and
> mlock - so i have omitted those for now.
>
> Most hooks are straightforward.
>
> Including a node as a demotion-eligible target was as simple as:
>
> static void establish_demotion_targets(void)
> {
> ..... snip .....
> /*
> * Include private nodes that have opted in to demotion
> * via NP_OPS_DEMOTION. A node might have custom migrate
> */
> all_memory = node_states[N_MEMORY];
> for_each_node_state(node, N_MEMORY_PRIVATE) {
> if (node_private_has_flag(node, NP_OPS_DEMOTION))
> node_set(node, all_memory);
> }
> ..... snip .....
> }
>
> The Migration and Mempolicy support are the two most complex pieces,
> and most useful things are built on top of Migration (meaning the
> remaining implementations are usually simple).
>
>
> Private Node Hotplug Lifecycle
> ===
>
> Registration follows a strict order enforced by
> add_private_memory_driver_managed():
>
> 1. Driver calls add_private_memory_driver_managed(nid, start,
> size, resource_name, mhp_flags, online_type, &np).
>
> 2. node_private_register(nid, &np) stores the driver's
> node_private in pgdat and sets pgdat->private. N_MEMORY and
> N_MEMORY_PRIVATE are mutually exclusive -- registration fails
> with -EBUSY if the node already has N_MEMORY set.
>
> Only one driver may register per private node.
>
> 3. Memory is hotplugged via __add_memory_driver_managed().
>
> When online_pages() runs, it checks pgdat->private and sets
> N_MEMORY_PRIVATE instead of N_MEMORY.
>
> Zonelist construction gives private nodes a self-only NOFALLBACK
> list and an N_MEMORY fallback list (so kernel/slab allocations on
> behalf of private node work can fall back to DRAM).
>
> 4. kswapd and kcompactd are NOT started for private nodes. The
> owning service is responsible for driving reclaim if needed
> (e.g., CRAM uses watermark_boost to wake kswapd on demand).
>
> Teardown is the reverse:
>
> 1. Driver calls offline_and_remove_private_memory(nid, start,
> size).
>
> 2. offline_pages() offlines the memory. When the last block is
> offlined, N_MEMORY_PRIVATE is cleared automatically.
>
> 3. node_private_unregister() clears pgdat->node_private and
> drops the refcount. It refuses to unregister (-EBUSY) if
> N_MEMORY_PRIVATE is still set (other memory ranges remain).
>
> The driver is responsible for ensuring memory is hot-unpluggable
> before teardown. The service must ensure all memory is cleaned
> up before hot-unplug - or the service must support migration (so
> memory_hotplug.c can evacuate the memory itself).
>
> In the CRAM example, the service supports migration, so memory
> hot-unplug can remove memory without any special infrastructure.
>
>
> Application: Compressed RAM (mm/cram)
> ===
>
> Compressed RAM has a serious design issue: Its capacity a lie.
>
> A compression device reports more capacity than it physically has.
> If workloads write faster than the OS can reclaim from the device,
> we run out of real backing store and corrupt data or crash.
>
> I call this problem: "Trying to Out Run A Bear"
>
> I.e. This is only stable as long as we stay ahead of the pressure.
>
> We don't want to design a system where stability depends on outrunning
> a bear - I am slow and do not know where to acquire bear spray.
>
> Fun fact: Grizzly bears have a top-speed of 56-64 km/h.
> Unfun Fact: Humans typically top out at ~24 km/h.
>
> This MVP takes a conservative position:
>
> all compressed memory is mapped read-only.
>
> - Folios reach the private node only via reclaim (demotion)
> - migrate_to implements custom demotion with backpressure.
> - fixup_migration_pte write-protects PTEs on arrival.
> - wrprotect hooks prevent silent upgrades
> - handle_fault promotes folios back to DRAM on write.
> - free_folio scrubs stale data before buddy free.
>
> Because pages are read-only, writes can never cause runaway
> compression ratio loss behind the allocator's back. Every write
> goes through handle_fault, which promotes the folio to DRAM first.
>
> The device only ever sees net compression (demotion in) and explicit
> decompression (promotion out via fault or reclaim), and has a much
> wider timeframe to respond to poor compression scenarios.
>
> That means there's no bear to out run. The bears are safely asleep in
> their bear den, and even if they show up we have a bear-proof cage.
>
> The backpressure system is our bear-proof cage: the driver reports real
> device utilization (generalized via watermark_boost on the private
> node's zone), and CRAM throttles demotion when capacity is tight.
>
> If compression ratios are bad, we stop demoting pages and start
> evicting pages aggressively.
>
> The service as designed is ~350 functional lines of code because it
> re-uses mm/ services:
>
> - Existing reclaim/vmscan code handles demotion.
> - Existing migration code handles migration to/from.
> - Existing page fault handling dispatches faults.
>
> The driver contains all the CXL nastiness core developers don't want
> anything to do with - No vendor logic touches mm/ internals.
>
>
>
> Future CRAM : Loosening the read-only constraint
> ===
>
> The read-only model is safe but conservative. For workloads where
> compressed pages are occasionally written, the promotion fault adds
> latency. A future optimization could allow a tunable fraction of
> compressed pages to be mapped writable, accepting some risk of
> write-driven decompression in exchange for lower overhead.
>
> The private node ops make this straightforward:
>
> - Adjust fixup_migration_pte to selectively skip
> write-protection.
> - Use the backpressure system to either revoke writable mappings,
> deny additional demotions, or evict when device pressure rises.
>
> This comes at a mild memory overhead: 32MB of DRAM per 1TB of CRAM.
> (1 bit per 4KB page).
>
> This is not proposed here, but it should be somewhat trivial.
>
>
> Discussion Topics
> ===
> 0. Obviously I've included the set as an RFC, please rip it apart.
>
> 1. Is N_MEMORY_PRIVATE the right isolation abstraction, or should
> this extend ZONE_DEVICE? Prior feedback pushed away from new
> ZONE logic, but this will likely be debated further.
>
> My comments on this:
>
> ZONE_DEVICE requires re-implementing every service you want to
> provide to your device memory, including basic allocation.
>
> Private nodes use real struct pages with no metadata
> limitations, participate in the buddy allocator, and get NUMA
> topology for free.
>
> 2. Can this subsume ZONE_DEVICE COHERENT users? The architecture
> was designed with this in mind, but it is only a thought experiment.
>
> 3. Is a dedicated mm/ service (cram) the right place for compressed
> memory management, or should this be purely driver-side until
> more devices exist?
>
> I wrote it this way because I forsee more "innovation" in the
> compressed RAM space given current... uh... "Market Conditions".
>
> I don't see CRAM being CXL-specific, though the only solutions I've
> seen have been CXL. Nothing is stopping someone from soldering such
> memory directly to a PCB.
>
> 5. Where is your hardware-backed data that shows this works?
>
> I should have some by conference time.
>
> Thanks for reading
> Gregory (Gourry)
>
>
> Gregory Price (27):
> numa: introduce N_MEMORY_PRIVATE node state
> mm,cpuset: gate allocations from N_MEMORY_PRIVATE behind __GFP_PRIVATE
> mm/page_alloc: add numa_zone_allowed() and wire it up
> mm/page_alloc: Add private node handling to build_zonelists
> mm: introduce folio_is_private_managed() unified predicate
> mm/mlock: skip mlock for managed-memory folios
> mm/madvise: skip madvise for managed-memory folios
> mm/ksm: skip KSM for managed-memory folios
> mm/khugepaged: skip private node folios when trying to collapse.
> mm/swap: add free_folio callback for folio release cleanup
> mm/huge_memory.c: add private node folio split notification callback
> mm/migrate: NP_OPS_MIGRATION - support private node user migration
> mm/mempolicy: NP_OPS_MEMPOLICY - support private node mempolicy
> mm/memory-tiers: NP_OPS_DEMOTION - support private node demotion
> mm/mprotect: NP_OPS_PROTECT_WRITE - gate PTE/PMD write-upgrades

I'm concerned about adding more special-casing (similar to what we
already added for ZONE_DEVICE) all over the place.

Like the whole folio_managed_() stuff in mprotect.c

Having that said, sounds like a reasonable topic to discuss.

--
Cheers,

David