Question about virtio_mm: why plug/unplug with 4MB granularity？

Question about virtio_mm: why plug/unplug with 4MB granularity？

[Eric Ren]

Hi David,

As the subject, I'm wondering if we can make virtio_mm plug/unplug with 
futher more fine granularity like

2MB?


I gave it try as below, and it works.

1. Revert this patch:   aac65321ba69("mm/memory_hotplug: simplify page 
onlining");

2. Tell mm hotplug core invoke online page callback with order 9 with 
changes as below;


```

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 5873563a3518..22a9636402b1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -622,7 +622,9 @@ static int online_pages_range(unsigned long 
start_pfn, unsigned long nr_pages,
          * them as being online/belonging to this zone ("present").
          */
         for (pfn = start_pfn; pfn < end_pfn; pfn += 1ul << order) {
-               order = min(MAX_ORDER - 1, get_order(PFN_PHYS(end_pfn - 
pfn)));
+               /* 2MB callback */
+               order = min(MAX_ORDER - 2, get_order(PFN_PHYS(end_pfn - 
pfn)));
                 /* __free_pages_core() wants pfns to be aligned to the 
order */
```

3. in virtio_mem driver, change subblock size to 2MB.


Thought the hack testing works, I'm feeling not safe.


Could you please help give some insights on these questions?

1. Is 4M a hard limitation?

2. What troubles we can foresee if using 2MB as granularity?


Best regards,

Eric

Re: Question about virtio_mm: why plug/unplug with 4MB granularity？

[David Hildenbrand]

On 25.11.21 12:27, Eric Ren wrote:
> Hi David,

Hi Eric,

> 
> As the subject, I'm wondering if we can make virtio_mm plug/unplug with 
> futher more fine granularity like
> 
> 2MB?


That's actually very high on my todo list. See "Hot(un)plug Granularity"
under:

https://virtio-mem.gitlab.io/developer-guide.html

> 
> 
> I gave it try as below, and it works.
> 
> 1. Revert this patch:   aac65321ba69("mm/memory_hotplug: simplify page 
> onlining");
> 
> 2. Tell mm hotplug core invoke online page callback with order 9 with 
> changes as below;
> 
> 
> ```
> 
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 5873563a3518..22a9636402b1 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -622,7 +622,9 @@ static int online_pages_range(unsigned long 
> start_pfn, unsigned long nr_pages,
>           * them as being online/belonging to this zone ("present").
>           */
>          for (pfn = start_pfn; pfn < end_pfn; pfn += 1ul << order) {
> -               order = min(MAX_ORDER - 1, get_order(PFN_PHYS(end_pfn - 
> pfn)));
> +               /* 2MB callback */
> +               order = min(MAX_ORDER - 2, get_order(PFN_PHYS(end_pfn - 
> pfn)));
>                  /* __free_pages_core() wants pfns to be aligned to the 
> order */
> ```
> 
> 3. in virtio_mem driver, change subblock size to 2MB.
> 
> 
> Thought the hack testing works, I'm feeling not safe.

Yes, it's not safe in general when unplugging memory. When plugging
memory, some changes to virtio_mem_online_page_cb() are required. It has
to traverse all pageblocks part of the passed range -- so that avoids
the mm/memory_hotplug.c change above.

But for memory unplug, alloc_contig_range() and pageblock isolation
needs changes (below).

> 
> 
> Could you please help give some insights on these questions?
> 
> 1. Is 4M a hard limitation?

No, the target is 2M. It will improve virtio-mem capability on
ZONE_NORMAL significantly.

> 
> 2. What troubles we can foresee if using 2MB as granularity?

alloc_contig_range() needs to be taught to handle pageblock granularity
(2M) correctly. Otherwise unplug on ZONE_NORMAL will be mostly broken.

Fortunately, this is WIP, but still needs to fix some things:

https://lkml.kernel.org/r/20211115193725.737539-1-zi.yan@sent.com


-- 
Thanks,

David / dhildenb