Re: [PATCH v2 1/1] mm/memory_hotplug: PG_offline_skippable for offlining memory blocks with PageOffline pages

[Oscar Salvador <osalvador@suse.de>]

On Tue, May 20, 2025 at 06:42:11PM +0200, David Hildenbrand wrote:
> A long-term goal is supporting frozen PageOffline pages, and later
> PageOffline pages that don't have a refcount at all. Some more work for
> that is needed -- in particular around non-folio page migration and
> memory ballooning drivers -- but let's start by handling PageOffline pages
> that can be skipped during memory offlining differently.
> 
> Note that PageOffline is used to mark pages that are logically offline
> in an otherwise online memory block (e.g., 128 MiB). If a memory
> block is offline, the memmap is considered compeltely uninitialized
> and stale (see pfn_to_online_page()).
> 
> Let's introduce a PageOffline specific page flag (PG_offline_skippable)
> that for now reuses PG_owner_2. In the memdesc future, it will be one of
> a small number of per-memdesc flags stored alongside the type.
> 
> By setting PG_offline_skippable, a driver indicates that it can
> restore the PageOffline state of these specific pages when re-onlining a
> memory block: it knows that these pages are supposed to be PageOffline()
> without the information in the vmemmap, so it can filter them out and
> not expose them to the buddy -> they stay PageOffline().
> 
> While PG_offline_offlineable might be clearer, it is also super
> confusing. Alternatives (PG_offline_sticky?) also don't quite feel right.
> So let's use "skippable" for now.
> 
> The flag is not supposed to be used for movable PageOffline pages as
> used for balloon compaction; movable PageOffline() pages can simply be
> migrated during the memory offlining stage, turning the migration
> destination page PageOffline() and turning the migration source page
> into a free buddy page.
> 
> Let's convert the single user from our MEM_GOING_OFFLINE approach
> to the new PG_offline_skippable approach: virtio-mem. Fortunately,
> this simplifies the code quite a lot. The only corner case we have to
> take care of is when force-unloading the virtio-mem driver: we have to
> prevent partially-plugged memory blocks from getting offlined by
> clearing PG_offline_skippable again.
> 
> What if someone decides to grab a reference on these pages although they
> really shouldn't? After all, we'll now keep the refcount at 1 (until we
> can properly stop using the refcount completely).
> 
> Well, less worse things will happen than would currently: currently,
> if someone would grab a reference to these pages, in MEM_GOING_OFFLINE
> we would run into the
> 		if (WARN_ON(!page_ref_dec_and_test(page)))
> 			dump_page(page, "fake-offline page referenced");
> 
> And once that unexpected reference would get dropped, we would end up
> freeing that page to the buddy: ouch.
> 
> Now, we'll allow for offlining that memory, and when that unexpected
> reference would get dropped, we would not end up freeing that page to
> the buddy. Once we have frozen PageOffline() pages, it will all get a
> lot cleaner.
> 
> Note that we didn't see the existing WARN_ON so far, because nobody
> should ever be referencing such pages.
> 
> An alternative might be to have another callback chain from memory hotplug
> code, where a driver that owns that page could agree to skip the
> PageOffline() page. However, we would have to repeatedly issue these
> callbacks for individual PageOffline() pages, which does not sound
> compelling. As we have spare bits, let's use this simpler approach for
> now.
> 
> Acked-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>

Hi David, sorry for jumping in late

> @@ -1157,6 +1083,7 @@ static void virtio_mem_set_fake_offline(unsigned long pfn,
>  			SetPageDirty(page);
>  		else
>  			__SetPageOffline(page);
> +		__SetPageOfflineSkippable(page);
>  		VM_WARN_ON_ONCE(!PageOffline(page));

I think I am having some issues understanding this, let me see if I get
it.

- virtio-mem defines PageOffline pages, which are logically-offlined
  pages within an onlined memory-block
- PageOffline pages have a refcount of '0' once they are properly
  initialized, meaning that refcount > 0 implies somebody is holding
  a refcount and that should not really happen
- logically-offline pages belonging to onlined memory-blocks are marked PageDirty,
  while logically-offlined pages we allocated via alloc_contig_range are marked
  PageOffline (I am getting a bit lost between fake-online, fake-offline, my fault)
- If we want to release logically-offline pages belonging to an onlined memory-block,
  we ClearDirty them and be done
- If we want to release logically-offlined pages belonging we allocated
  via alloc_contig_range, we clear PageOffline and be done
- PageOfflineSkipabble are unmovable PageOffline pages, which cannot be
  migrated? 
- So for a PageOffline to be able to be migrated away must be Movable or
  marked PageOfflineSkipabble, making do_migrate_range ignore it
- PageOfflineSkipabble will be marked PageOffline upon re-onlining? Will
  still be marked as PageOfflineSkipabble?

> +
> +		/*
> +		 * Only PageOffline() pages that are marked "skippable" cannot
> +		 * be migrated but can be skipped when offlining. See

It is probably me, and nevermind the comment but I somehow find
"PageOfflineSkipabble are not migrated but skipped when offlining" a bit
easier.

 

-- 
Oscar Salvador
SUSE Labs