[LSF/MM BPF Topic] Warming up to frozen pages

[LSF/MM BPF Topic] Warming up to frozen pages

[Hannes Reinecke]

Hey all,

(Thanks, Jon, for the title :-)
the recent discussion around frozen pages and when to do a 
get_page()/put_page() and when not resulted in quite some unresolved issues.
So I would like to propose a session at LSF/MM:

'Warming up to frozen pages'
With the frozen pages patchset from Willy slab pages don't need
(and, in fact, can have) a page reference anymore. While this easy
to state, and to implement when using iov iterators, problems
arise when these iov iterators get mangled eg when being passed
via the various layers in the kernel.
Case in point: 'recvmsg()', when called from userspace, is being
passed an iov, and the iterator type defines if a page reference
need to be taken. However, when called from other kernel subsystems
(eg from nvme-tcp or iscsi), the iov is filled from a bvec which
in itself is filled from an iov iter from userspace, so the iov
iterator will assume it's a 'normal' bvec, and get a reference for
all entries as it wouldn't know which entry is a 'normal' and which
is a 'slab' page.
As Christoph indicated this is _not_ how things should be, so
a discussion on how to disentangle this would be good.
Maybe we even manage to lay down some rules when a page reference
should be taken and when not.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [LSF/MM BPF Topic] Warming up to frozen pages

[Matthew Wilcox]

On Tue, Mar 18, 2025 at 03:47:31PM +0100, Hannes Reinecke wrote:
> 'Warming up to frozen pages'
> With the frozen pages patchset from Willy slab pages don't need
> (and, in fact, can have) a page reference anymore. While this easy
> to state, and to implement when using iov iterators, problems
> arise when these iov iterators get mangled eg when being passed
> via the various layers in the kernel.
> Case in point: 'recvmsg()', when called from userspace, is being
> passed an iov, and the iterator type defines if a page reference
> need to be taken. However, when called from other kernel subsystems
> (eg from nvme-tcp or iscsi), the iov is filled from a bvec which
> in itself is filled from an iov iter from userspace, so the iov
> iterator will assume it's a 'normal' bvec, and get a reference for
> all entries as it wouldn't know which entry is a 'normal' and which
> is a 'slab' page.
> As Christoph indicated this is _not_ how things should be, so
> a discussion on how to disentangle this would be good.
> Maybe we even manage to lay down some rules when a page reference
> should be taken and when not.

My only concern is that we might not have anybody from networking to talk
about their side of all this.  We need Dave Howells for this as one of
the network filesystem people, he probably understands this fairly well.
Anna might have some network stack knowledge too.  Maybe we can get some
of the BPF people to join in; although their track looks very dense,
so we'll have to try hard to find a time when there's a topic that the
networking-BPF people aren't so interested in.

Re: [LSF/MM BPF Topic] Warming up to frozen pages

[Vlastimil Babka]

On 3/18/25 15:47, Hannes Reinecke wrote:
> Hey all,
> 
> (Thanks, Jon, for the title :-)
> the recent discussion around frozen pages and when to do a 
> get_page()/put_page() and when not resulted in quite some unresolved issues.
> So I would like to propose a session at LSF/MM:
> 
> 'Warming up to frozen pages'
> With the frozen pages patchset from Willy slab pages don't need
> (and, in fact, can have) a page reference anymore. While this easy

BTW, my hope is that large kmalloc folios would also drop the refcount. That
means anything obtained by kmalloc(N) where N is over 8k (order-1 folio on
4k PAGE_SIZE). That 8k is an implementation detail of SLUB (SLAB used to
have more) and thus all kmalloc() buffers should better behave the same to
avoid surprises when some particular allocation size changes, or the thing
becomes used on an architecture with larger than 4k page size and thus
becomes a "small" kmalloc() there.

Willy already added a page type for large kmalloc. Given we can expect
possible troubles after the small kmalloc() ones already hit us, I'd first
just add get_page()/put_page() warnings for that large kmalloc page type in
addition to the slab page type ones and expose that to -next (after the 6.15
merge window) to see if something needs fixing.

> to state, and to implement when using iov iterators, problems
> arise when these iov iterators get mangled eg when being passed
> via the various layers in the kernel.
> Case in point: 'recvmsg()', when called from userspace, is being
> passed an iov, and the iterator type defines if a page reference
> need to be taken. However, when called from other kernel subsystems
> (eg from nvme-tcp or iscsi), the iov is filled from a bvec which
> in itself is filled from an iov iter from userspace, so the iov
> iterator will assume it's a 'normal' bvec, and get a reference for
> all entries as it wouldn't know which entry is a 'normal' and which
> is a 'slab' page.
> As Christoph indicated this is _not_ how things should be, so
> a discussion on how to disentangle this would be good.
> Maybe we even manage to lay down some rules when a page reference
> should be taken and when not.
> 
> Cheers,
> 
> Hannes

Re: [LSF/MM BPF Topic] Warming up to frozen pages

[Christoph Hellwig]

On Tue, Mar 18, 2025 at 03:10:01PM +0000, Matthew Wilcox wrote:
> My only concern is that we might not have anybody from networking to talk
> about their side of all this.  We need Dave Howells for this as one of
> the network filesystem people, he probably understands this fairly well.
> Anna might have some network stack knowledge too.  Maybe we can get some
> of the BPF people to join in; although their track looks very dense,
> so we'll have to try hard to find a time when there's a topic that the
> networking-BPF people aren't so interested in.

The problem isn't the network file systems, they'd do much better
by not having the network stack play silly refcount games.  This
needs the core networking folks to agree to this and then someone
to do the work, which might or might not reach deep down in all
the various network protocols supported.