Re: [DISCUSSION] Revisiting Slab Movable Objects

[Pedro Falcato <pfalcato@suse.de>]

On Mon, Apr 21, 2025 at 10:47:39PM +0900, Harry Yoo wrote:
> Hi folks,
>

Hi Harry,

Some passing thoughts...

> As a long term project, I'm starting to look into resurrecting
> Slab Movable Objects. The goal is to make certain types of slab memory
> movable and thus enable targeted reclamation, migration, and
> defragmentation.
> 
> The main purpose of this posting is to briefly review what's been tried
> in the past, ask people why prior efforts have stalled (due to lack of
> time or insufficient justification for additional complexity?),
> and discuss what's feasible today.
> 
> Please add anyone I may have missed to Cc. :)
> 
> Previous Work on Slab Movable Objects
> =====================================
> 
> Christoph Lameter, Slab Defragmentation Reduction, 2007-2017 (V16: [2]):
> Christoph Lameter, Slab object migration for xarray, 2017-2018 (V2: [3]):
>   Christoph's long-standing effort (since 2007) aiming to defragment
>   slab memory in cases where sparsely populated slabs occupy excessive
>   amount of memory.
> 
>   Early versions of the work focused on defragmenting slab caches
>   for filesystem data structures such as inode, dentry, and buffer head.
>   updatedb was suggested as the standard way to trigger for generating
>   sparsely populated slabs on file servers.
> 
>   However, defragmenting slabs for filesystem data structures has proven
>   to be very difficult to fully solve, because inodes and dentries are
>   neither reclaimable nor migratable, limiting the effectiveness of
>   defragmentation.
> 
>   In late 2018, the effort was revived with a new focus on migrating
>   XArray nodes. However, it appears the work was discontinued after
>   V2 [3]?
> 
> Tobin C. Harding, Slab Movable Objects, 2019 (First Non-RFC: [5])
> - Tobin C. Harding revived Christoph's earlier work and introduced
>   a few enhancements, including partial shrinking of dentries, moving
>   objects to and from a specific NUMA node, and balancing objects across
>   all NUMA nodes.
> 
>   Also appears to be discontinued after the first non-RFC version [5]? 
> 
> At LSFMM 2017, Andrea Arcangeli suggested [6] virtually mapped slabs,
> which might be useful since migrating them does not require changing the
> address of objects. But as Rik van Riel pointed out at that time, it
> isn't really useful for defragmentation. Andrea Arcangeli responded
> that it can be beneficial for memory hotplug, compaction and out-of-memory
> avoidance.
> 
> The exact mechanism wasn't described in [6], but I assume it'll involve
> 1) unmap a slab (and page faults after unmap need to wait for migration
> to complete), 2) copy objects to a new slab, and 3) map the new slab?
> But the idea hasn't gained enough attention for anyone to actually
> implement it.

I don't think this is a silver bullet. It opens a whole separate can of worms
while maintaining similar issues. But instead of worrying about updating pointers,
you're worrying about locking out _any_ sort of access, which would involve stop_machine().
You can't even atomically replace a PTE without running into issues (arm requires BBM, thus
this doesn't work) so it cannot be applied to anything that can't page fault (xarray and 
the maple tree are used in IRQ paths, if I'm not mistaken).

> 
> Potential Candidates of SMO
> ===========================
> 
> Basic Rules
> -----------
> 
> - Slab memory can only be reclaimed or migrated if the user of the slab
>   provides a way to isolate / migrate objects.
> - If objects can be reclaimed, it makes sense to simply reclaim them
>   instead of migrating them (unless we know it's better to keep that
>   object in memory).

In any case I think you want to give subsystems the power to decide between
{RECLAIMED, MIGRATED, SKIPPED}.

> - Some objects can't be reclaimed, but migrating them is (if possible)
>   still useful for defragmentation and compaction.
>   - However it is not always feasible 
> 
> Potential candidates include (but not limited to):
> --------------------------------------------------
> 
> - XArray nodes can be migrated (can't be reclaimed as they're being used)
>   - Can be reclaimed if it only includes shadow entries.
> - Maple tree nodes (if without external locking) and VMAs can be migrated
>   and obviously can't be reclaimed.
> - Negative dentry should be reclaimed, instead of being migrated.
> - Only unused dentries can be reclaimed without high cost.

Unused dentries can also have a high cost if they're accessed in the future
(and the dentry LRU has some difficulty in... LRU'ing - thus the negative dentry problem).

In any case, it would be interesting to see if the existing shrinker interface
could be used for this stuff. We already best-effort-reclaim objects, maybe we
could best-effort-migrate objects too? The problem of reclaiming is adjacent to
migrating, and we already have infrastructure for it...

-- 
Pedro