Re: [LSF/MM/BPF TOPIC] Migrating the un-migratable

[David Hildenbrand <david@redhat.com>]

On 30.01.25 22:48, Matthew Wilcox wrote:
> On Wed, Jan 29, 2025 at 05:10:03PM +0100, David Hildenbrand wrote:
>> As one example, in context of FUSE we recently discovered that folios that
>> are under writeback cannot be migrated, and user space in control of when
>> writeback will end. Something similar can happen ->readahead() where user
>> space is in charge of supplying page content. Networking filesystems in
>> general seem to be prone to this as well.
> 
> You're not wrong.  The question is whether we're willing to put the
> asterisk on "In the presence of a misbehaving server (network or fuse),
> our usual guarantees do not apply".  I'm not sure it's a soluble
> problem, though.  Normally writeback (or, as you observed, readahead)
> completes just fine and we don't need to use non-movable pages for them.

Yes, we discussed a lot of that, and where it could be handled, and 
where it simply cannot be handled. I also don't believe that we can -- 
or even should try to -- be perfect.

There are certainly cases that simply cannot be handled, or only very 
very painfully. Always falling back to allocating from use non-movable 
memory "simply because it could happen" (e.g., someone could trip over 
the network cable) is stupid.

I think it's all a matter of seeing how far we can get with reasonable 
effort, and which cases are really problematic -- e.g., untrusted fuse 
-- and how they could be better handled.

The discussion so far already revealed a bunch of interesting 
approaches, but also limitations (e.g., fuse with splice).

> 
> But if someone trips over the network cable, anything in flight becomes
> unmovable until someone plugs it back in.  We've given the DMA address
> of the memory to a network adapter, and that's generally a non-revokable
> step (maybe the iommu could save us, but at what cost?)

Right, and as we discussed as part of the FUSE discussion.

It will be very interesting to hear into which problems others (e.g., 
Frank) ran into and how they could be mitigated/solved.

> 
>> As another example, failing to split large folios can prevent migration if
>> memory is fragmented. XFS (IOMAP in general) refuses to split folios that
>> are dirty [3]. Splitting of folios and page migration have a lot in common.
> 
> Welll ... yes and no.  iomap refuses to split a dirty folio because it
> has a per-folio data structure which tells us which blocks in the folio
> are dirty.  If we split the folio, we have to allocate an extra data
> structure for each new folio that we create.  It's not impossible, but
> it's a big ask for slab.  It'll be a lot better once Zi Yan's patch is
> in to only split folios as needed rather than all the way.
> 
> That problem doesn't arise for migration.  filemap_release_folio() is
> only called by fallback_migrate_folio(), which is only called if the
> filesystem doesn't provide a ->migrate_folio callback.  All iomap
> users should use filemap_migrate_folio() which just has to move the
> data structure from the old folio to the new.

Right, that's why I said: "if memory is fragmented".

Try migrating a 512 MiB folio (arm64 64k ..) when you cannot split 
(dirty) and memory is all fragmented such that you cannot easily grab a 
free 512 MiB one. Of course, that's an extreme example, but it can also 
easily happen on systems with smaller folios ...

... and then, we now have folios that cannot be split below a certain 
min order ... (well, we kind-of had that in an extreme form with hugetlb 
that cannot be slit at all)

-- 
Cheers,

David / dhildenb