Re: [PATCH 03/62] mm: Split slab into its own type

[Johannes Weiner <hannes@cmpxchg.org>]

On Thu, Oct 14, 2021 at 09:22:13AM +0200, David Hildenbrand wrote:
> On 13.10.21 20:31, Matthew Wilcox wrote:
> > On Wed, Oct 13, 2021 at 02:08:57PM -0400, Johannes Weiner wrote:
> >> Btw, I think slab_nid() is an interesting thing when it comes to page
> >> polymorphy. We want to know the nid for all sorts of memory types:
> >> slab, file, anon, buddy etc. In the goal of distilling page down to
> >> the fewest number of bytes, this is probably something that should
> >> remain in the page rather than be replicated in all subtypes.
> > 
> > Oh, this is a really interesting point.
> > 
> > Node ID is typically 10 bits (I checked Debian & Oracle configs for
> > various architectures).  That's far more than we can store in the bottom
> > bits of a single word, and it's even a good chunk of a second word.
> > 
> > I was assuming that, for the page allocator's memory descriptor and for
> > that of many allocators (such as slab), it would be stored *somewhere*
> > in the memory descriptor.  It wouldn't necessarily have to be the same
> > place for all memory descriptors, and maybe (if it's accessed rarely),
> > we delegate finding it to the page allocator's knowledge.
> > 
> > But not all memory descriptors want/need/can know this.  For example,
> > vmalloc() might well spread its memory across multiple nodes.  As long
> > as we can restore the node assignment again once the pages are vfree(),
> > there's no particular need for the vmalloc memory descriptor to know
> > what node an individual page came from (and the concept of asking
> > vmalloc what node a particular allocation came from is potentially
> > nonsense, unless somebody used vmalloc_node() or one of the variants).
> > 
> > Not sure there's an obviously right answer here.  I was assuming that at
> > first we'd enforce memdesc->flags being the first word of every memory
> > descriptor and so we could keep passing page->flags around.  That could
> > then change later, but it'd be a good first step?
> > 
> 
> <offtopic>
> It's really hard to make an educated guess here without having a full
> design proposal of what we actually want to achieve and especially how
> we're going to treat all the corner cases (as raised already in
> different context).
> 
> I'm all for simplifying struct page and *eventually* being able to
> shrink it, even if we end up only shrinking by a little. However, I'm
> not sold on doing that by any means (e.g., I cannot agree to any
> fundamental page allocator rewrite without an idea what it does to
> performance but also complexity). We might always have a space vs.
> performance cost and saving space by sacrificing performance isn't
> necessarily always a good idea. But again, it's really hard to make an
> educated guess.
> 
> Again, I'm all for cleanups and simplifications as long as they really
> make things cleaner. So I'm going to comment on the current state and
> how the cleanups make sense with the current state.
> </offtopic>

Very well put, and not off-topic at all.

A clearer overarching design proposal that exists in more than one
head, that people agree on, and that is concrete enough to allow
others to make educated guesses on how random snippets of code would
or should look like in the new world, would help immensely.

(This applies here, but to a certain degree to folio as well.)

HOWEVER, given the scope of struct page, I also think this is a very
difficult problem. There are a LOT of nooks and crannies that throw
curveballs at these refactors. We're finding new ones daily.

I think smaller iterations, with a manageable amount of novelty, that
can be reviewed and judged by everybody involved - against the current
code base, rather than against diverging future speculation - make
more sense. These can still push the paradigm in the direction we
want, but we can work out kinks in the overarching ideas as we go.

I think what isn't going to work is committing vast amounts of code to
open-ended projects and indefinite transitory chaos that makes the
codebase go through every conceptual dead end along the way. There are
just too many users and too much development work against each release
of the kernel nowadays to operate like this.

> Node/zone is a property of a base page and belongs into struct page OR
> has to be very easily accessible without any kind of heavy locking. The
> node/zone is determined once memory gets exposed to the system (e.g., to
> the buddy during boot or during memory onlining) and is stable until
> memory is offlined again (as of right now, one could imagine changing
> zones at runtime).

Yes, it's a property of the page frame, for which struct page is the
descriptor. Even if we could get away with stuffing them into the
higher-level memory descriptors, it's inevitable that this will lead
to duplication, make adding new types more cumbersome, and get in the
way of writing generic code that works on those shared attributes.

struct page really is the right place for it.

> For example, node/zone information is required for (almost) lockless PFN
> walkers in memory offlining context, to figure out if all pages we're
> dealing with belong to one node/zone, but also to properly shrink
> zones+nodes to eventually be able to offline complete nodes. I recall
> that there are other PFN walkers (page compaction) that need this
> information easily accessible.

kmemleak is another one of them.

page_ext is another lower-level plumbing thing. (Although this one we
*may* be able to incorporate into dynamically allocated higher-level
descriptors. Another future idea that is worth keeping on the map, but
need to be careful not to make assumptions on today.)

Then there is generic code that doesn't care about higher-level types:
vmstats comes to mind, the generic list_lru code, page table walkers
(gup, numa balancing, per-task numa stats), ...