Re: [PATCH v1 3/3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

[Tyler Sanderson <tysand@google.com>]

[-- Attachment #1: Type: text/plain, Size: 10633 bytes --]

On Wed, Feb 5, 2020 at 2:52 PM David Hildenbrand <david@redhat.com> wrote:

>
>
> Am 05.02.2020 um 23:37 schrieb Tyler Sanderson <tysand@google.com>:
>
> 
>
>
> On Wed, Feb 5, 2020 at 8:34 AM David Hildenbrand <david@redhat.com> wrote:
>
>> Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker")
>> changed the behavior when deflation happens automatically. Instead of
>> deflating when called by the OOM handler, the shrinker is used.
>>
>> However, the balloon is not simply some slab cache that should be
>> shrunk when under memory pressure. The shrinker does not have a concept of
>> priorities, so this behavior cannot be configured.
>>
>> There was a report that this results in undesired side effects when
>> inflating the balloon to shrink the page cache. [1]
>>         "When inflating the balloon against page cache (i.e. no free
>> memory
>>          remains) vmscan.c will both shrink page cache, but also invoke
>> the
>>          shrinkers -- including the balloon's shrinker. So the balloon
>>          driver allocates memory which requires reclaim, vmscan gets this
>>          memory by shrinking the balloon, and then the driver adds the
>>          memory back to the balloon. Basically a busy no-op."
>>
>> The name "deflate on OOM" makes it pretty clear when deflation should
>> happen - after other approaches to reclaim memory failed, not while
>> reclaiming. This allows to minimize the footprint of a guest - memory
>> will only be taken out of the balloon when really needed.
>>
>> Especially, a drop_slab() will result in the whole balloon getting
>> deflated - undesired. While handling it via the OOM handler might not be
>> perfect, it keeps existing behavior. If we want a different behavior, then
>> we need a new feature bit and document it properly (although, there should
>> be a clear use case and the intended effects should be well described).
>>
>> Keep using the shrinker for VIRTIO_BALLOON_F_FREE_PAGE_HINT, because
>> this has no such side effects. Always register the shrinker with
>> VIRTIO_BALLOON_F_FREE_PAGE_HINT now. We are always allowed to reuse free
>> pages that are still to be processed by the guest. The hypervisor takes
>> care of identifying and resolving possible races between processing a
>> hinting request and the guest reusing a page.
>>
>> In contrast to pre commit 71994620bb25 ("virtio_balloon: replace oom
>> notifier with shrinker"), don't add a moodule parameter to configure the
>> number of pages to deflate on OOM. Can be re-added if really needed.
>> Also, pay attention that leak_balloon() returns the number of 4k pages -
>> convert it properly in virtio_balloon_oom_notify().
>>
>> Note1: using the OOM handler is frowned upon, but it really is what we
>>        need for this feature.
>>
>> Note2: without VIRTIO_BALLOON_F_MUST_TELL_HOST (iow, always with QEMU) we
>>        could actually skip sending deflation requests to our hypervisor,
>>        making the OOM path *very* simple. Besically freeing pages and
>>        updating the balloon. If the communication with the host ever
>>        becomes a problem on this call path.
>>
>> [1] https://www.spinics.net/lists/linux-virtualization/msg40863.html
>>
>> Reported-by: Tyler Sanderson <tysand@google.com>
>> Cc: Michael S. Tsirkin <mst@redhat.com>
>> Cc: Wei Wang <wei.w.wang@intel.com>
>> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
>> Cc: David Rientjes <rientjes@google.com>
>> Cc: Nadav Amit <namit@vmware.com>
>> Cc: Michal Hocko <mhocko@kernel.org>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
>>  drivers/virtio/virtio_balloon.c | 107 +++++++++++++-------------------
>>  1 file changed, 44 insertions(+), 63 deletions(-)
>>
>> diff --git a/drivers/virtio/virtio_balloon.c
>> b/drivers/virtio/virtio_balloon.c
>> index 7e5d84caeb94..e7b18f556c5e 100644
>> --- a/drivers/virtio/virtio_balloon.c
>> +++ b/drivers/virtio/virtio_balloon.c
>> @@ -14,6 +14,7 @@
>>  #include <linux/slab.h>
>>  #include <linux/module.h>
>>  #include <linux/balloon_compaction.h>
>> +#include <linux/oom.h>
>>  #include <linux/wait.h>
>>  #include <linux/mm.h>
>>  #include <linux/mount.h>
>> @@ -27,7 +28,9 @@
>>   */
>>  #define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned)(PAGE_SIZE >>
>> VIRTIO_BALLOON_PFN_SHIFT)
>>  #define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256
>> -#define VIRTBALLOON_OOM_NOTIFY_PRIORITY 80
>> +/* Maximum number of (4k) pages to deflate on OOM notifications. */
>> +#define VIRTIO_BALLOON_OOM_NR_PAGES 256
>> +#define VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY 80
>>
>>  #define VIRTIO_BALLOON_FREE_PAGE_ALLOC_FLAG (__GFP_NORETRY |
>> __GFP_NOWARN | \
>>                                              __GFP_NOMEMALLOC)
>> @@ -112,8 +115,11 @@ struct virtio_balloon {
>>         /* Memory statistics */
>>         struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR];
>>
>> -       /* To register a shrinker to shrink memory upon memory pressure */
>> +       /* Shrinker to return free pages -
>> VIRTIO_BALLOON_F_FREE_PAGE_HINT */
>>         struct shrinker shrinker;
>> +
>> +       /* OOM notifier to deflate on OOM -
>> VIRTIO_BALLOON_F_DEFLATE_ON_OOM */
>> +       struct notifier_block oom_nb;
>>  };
>>
>>  static struct virtio_device_id id_table[] = {
>> @@ -786,50 +792,13 @@ static unsigned long shrink_free_pages(struct
>> virtio_balloon *vb,
>>         return blocks_freed * VIRTIO_BALLOON_HINT_BLOCK_PAGES;
>>  }
>>
>> -static unsigned long leak_balloon_pages(struct virtio_balloon *vb,
>> -                                          unsigned long pages_to_free)
>> -{
>> -       return leak_balloon(vb, pages_to_free *
>> VIRTIO_BALLOON_PAGES_PER_PAGE) /
>> -               VIRTIO_BALLOON_PAGES_PER_PAGE;
>> -}
>> -
>> -static unsigned long shrink_balloon_pages(struct virtio_balloon *vb,
>> -                                         unsigned long pages_to_free)
>> -{
>> -       unsigned long pages_freed = 0;
>> -
>> -       /*
>> -        * One invocation of leak_balloon can deflate at most
>> -        * VIRTIO_BALLOON_ARRAY_PFNS_MAX balloon pages, so we call it
>> -        * multiple times to deflate pages till reaching pages_to_free.
>> -        */
>> -       while (vb->num_pages && pages_freed < pages_to_free)
>> -               pages_freed += leak_balloon_pages(vb,
>> -                                                 pages_to_free -
>> pages_freed);
>> -
>> -       update_balloon_size(vb);
>> -
>> -       return pages_freed;
>> -}
>> -
>>  static unsigned long virtio_balloon_shrinker_scan(struct shrinker
>> *shrinker,
>>                                                   struct shrink_control
>> *sc)
>>  {
>> -       unsigned long pages_to_free, pages_freed = 0;
>>         struct virtio_balloon *vb = container_of(shrinker,
>>                                         struct virtio_balloon, shrinker);
>>
>> -       pages_to_free = sc->nr_to_scan;
>> -
>> -       if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
>> -               pages_freed = shrink_free_pages(vb, pages_to_free);
>> -
>> -       if (pages_freed >= pages_to_free)
>> -               return pages_freed;
>> -
>> -       pages_freed += shrink_balloon_pages(vb, pages_to_free -
>> pages_freed);
>> -
>> -       return pages_freed;
>> +       return shrink_free_pages(vb, sc->nr_to_scan);
>>  }
>>
>>  static unsigned long virtio_balloon_shrinker_count(struct shrinker
>> *shrinker,
>> @@ -837,26 +806,22 @@ static unsigned long
>> virtio_balloon_shrinker_count(struct shrinker *shrinker,
>>  {
>>         struct virtio_balloon *vb = container_of(shrinker,
>>                                         struct virtio_balloon, shrinker);
>> -       unsigned long count;
>> -
>> -       count = vb->num_pages / VIRTIO_BALLOON_PAGES_PER_PAGE;
>> -       count += vb->num_free_page_blocks *
>> VIRTIO_BALLOON_HINT_BLOCK_PAGES;
>>
>> -       return count;
>> +       return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES;
>>  }
>>
>> -static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb)
>> +static int virtio_balloon_oom_notify(struct notifier_block *nb,
>> +                                    unsigned long dummy, void *parm)
>>  {
>> -       unregister_shrinker(&vb->shrinker);
>> -}
>> +       struct virtio_balloon *vb = container_of(nb,
>> +                                                struct virtio_balloon,
>> oom_nb);
>> +       unsigned long *freed = parm;
>>
>> -static int virtio_balloon_register_shrinker(struct virtio_balloon *vb)
>> -{
>> -       vb->shrinker.scan_objects = virtio_balloon_shrinker_scan;
>> -       vb->shrinker.count_objects = virtio_balloon_shrinker_count;
>> -       vb->shrinker.seeks = DEFAULT_SEEKS;
>> +       *freed += leak_balloon(vb, VIRTIO_BALLOON_OOM_NR_PAGES) /
>> +                 VIRTIO_BALLOON_PAGES_PER_PAGE;
>> +       update_balloon_size(vb);
>>
>> -       return register_shrinker(&vb->shrinker);
>> +       return NOTIFY_OK;
>>  }
>>
>>  static int virtballoon_probe(struct virtio_device *vdev)
>> @@ -933,22 +898,35 @@ static int virtballoon_probe(struct virtio_device
>> *vdev)
>>                         virtio_cwrite(vb->vdev, struct
>> virtio_balloon_config,
>>                                       poison_val, &poison_val);
>>                 }
>> -       }
>> -       /*
>> -        * We continue to use VIRTIO_BALLOON_F_DEFLATE_ON_OOM to decide
>> if a
>> -        * shrinker needs to be registered to relieve memory pressure.
>> -        */
>> -       if (virtio_has_feature(vb->vdev,
>> VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) {
>> -               err = virtio_balloon_register_shrinker(vb);
>> +
>> +               /*
>> +                * We're allowed to reuse any free pages, even if they are
>> +                * still to be processed by the host.
>>
> It is important to clarify that pages that are on the inflate queue but
> not ACKed by the host (the queue entry has not been returned) are _not_
> okay to reuse.
> If the host is going to do something destructive to the page (like deback
> it) then that needs to happen before the entry is returned.
>
>
> While you are correct, this comment is in the „free page hinting“
> section/if (not obvious by looking at the diff only), so it does not apply
> to inflate/deflate queues - but only free pages that are getting hinted. Or
> am I misreading your suggestion/missing something?
>
Ah you are right. Thanks!

>
> Thanks!
>
>
>

[-- Attachment #2: Type: text/html, Size: 13158 bytes --]