Re: [PATCH RFC 00/19] mm: Add __GFP_UNMAPPED

[Brendan Jackman <jackmanb@google.com>]

On Thu Mar 5, 2026 at 2:51 PM UTC, Kevin Brodsky wrote:
> On 25/02/2026 17:34, Brendan Jackman wrote:
>> .:::: Design: Introducing "freetypes"
>>
>> The biggest challenge for efficiently getting stuff out of the direct
>> map is TLB flushing. Pushing this problem into the page allocator turns
>> out to enable amortising that flush cost into almost nothing. The core
>> idea is to have pools of already-unmapped pages. We'd like those pages
>> to be physically contiguous so they don't unduly fragment the pagetables
>> around them, and we'd like to be able to efficiently look up these
>> already-unmapped pages during allocation. The page allocator already has
>> deeply-ingrained functionality for physically grouping pages by a
>> certain attribute, and then indexing free pages by that attribute, this
>> mechanism is: migratetypes.
>>
>> So basically, this series extends the concepts of migratetypes in the
>> allocator so that as well as just representing mobility, they can
>> represent other properties of the page too. (Actually, migratetypes are
>> already sort of overloaded, but the main extension is to be able to
>> represent _orthogonal_ properties). In order to avoid further
>> overloading the concept of a migratetype, this extension is done by
>> adding a new concept on top of migratetype: the _freetype_. A freetype
>> is basically just a migratetype plus some flags, and it replaces
>> migratetypes wherever the latter is currently used as to index free
>> pages.
>>
>> The first freetype flag is then added, which marks the pages it indexes
>> as being absent from the direct map. This is then used to implement the
>> new __GFP_UNMAPPED flag, which allocates pages from pageblocks that have
>> the new flag, or unmaps pages if no existing ones are already available.
>
> This approach seems very interesting to me, and I wonder if it could be
> applied to another use-case.
>
> I am working on a security feature to protect page table pages (PTPs)
> using pkeys [1]. This relies on all PTPs being mapped with a specific
> pkey (in the direct map). That requires changing a mapping attribute
> rather than making it invalid, but AFAICT this is essentially the same
> problem as the one you're trying to solve.

Yeah, I think so:

1. The fragmentation issues seem exactly the same.

2. The TLB flushing issues are probably also basically the same, I
assume you need to flush the TLB when you convert a page to use for
pagetables, and without allocator integration that can happen pretty
often and in hot paths. Correct?

> There are however extra challenges with mapping PTPs with special
> attributes. The main one, which you mention in patch 17, is that
> splitting the direct map may require allocating PTPs, which may lead to
> recursion.
>
> [1] introduces a dedicated page table allocator on top of the buddy
> allocator, which attempts to cache PMD-sized blocks if possible. It
> ensures that no recursion occurs by using a special flag when allocating
> PTPs while splitting the direct map, and keeping a reserve of pages
> specifically for that situation (patch 15 and 24). 

Right, and actually just today someone pointed out mm/execmem.c to me, I
think execmem_cache_populate() is basically doing the same thing
(although it's also creating a separate virtual mapping).

> There is also special
> handling for early page tables (essentially keeping track of them and
> setting their pkey once we can split the direct map).
>
> Do you think that this freetype infrastructure could be used for that
> purpose, instead of introducing a layer on top of the buddy allocator? 

Yes!!! 100% definitely, my code certainly solves all your problems...

> I
> expect that much of the special handling for allocating PTPs can be kept
> separate. Ensuring that protected pages are always available to split
> the direct map may be difficult though... This is deeply embedded in the
> allocator I proposed.

...Oh, hm, well, um, good point. Thinking aloud a bit...

The way this series dodges the question is (copying from the code
comments in patch 17 for convenient reading):

1) - The direct map starts out fully mapped at boot. (This is not really
 *   an assumption" as its in direct control of page_alloc.c).
 *
2) - Once pages in the direct map are broken down, they are not
 *   re-aggregated into larger pages again.
 *
3) - Pagetables are never allocated with __GFP_UNMAPPED.
 *
 * Under these assumptions, a pagetable might need to be allocated while
 * _unmapping_ stuff from the direct map during a __GFP_UNMAPPED
 * allocation. But, the allocation of that pagetable never requires
 * allocating a further pagetable.

In other words, we might need to allocate while we allocate (which is
fine because I have to do locking shenanigans anyway due to x86 TLB
shootdown requirements), but there's no further recursion after that.

Can we come up with an analogue for protected PTPs? Point 3) is
the inflexible one, and we obviously can't say "PTPs are never allocated
as PTPs". But if we invert it and _also_ invert point 1) I think we get
something that works in principle:

1) The direct map starts out _fully protected_ (i.e. we treat everything
   as if it's a pagetable at first).

2) We assume the direct map doesn't get reaggregated once we've broken
   things down to serve PTP allocations

3) PTPs are always PTPs...

But... this is a bit silly, since what it means is we'll then go through
~all the pagetblocks in the system (except the ones that _are_ actually
used for PTPs) and flip their pkey, breaking down the physmap to
pageblock granularity as we go. And... if we're gonna do that, we might
as well just say the physmap has to be at pageblock granularity to begin
with.

(Could we do that? Maybe - Mike Rapoport has previously argued that
physmap fragmentation is not a very big deal, so I guess the question
is whether we're ready to really lean into that analysis, it would be
quite painful if it turned out to be wrong).

Another potential "dodge": Is it really important that the PTPs are
always protected from the very moment they are created?
Coz this feature still seems pretty useful even if there's an awkward
fallback case where, under specific memory pressure patterns, we
temporarily use unprotected pagetables to set up protected pagetables.
That still makes exploiting a pagetable overwrite an order of magnitude
harder than before, right? Similar to how there's probably ways to
exploit bugs if you can get them to race with the intended pagetable
update paths that flip the pkey register, or if you can get a ROP chain
to flip that register for you or whatever.