From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 052A3C33CAC for ; Thu, 6 Feb 2020 09:11:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A24A4206CC for ; Thu, 6 Feb 2020 09:11:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TSPuwI/b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A24A4206CC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4EF176B0003; Thu, 6 Feb 2020 04:11:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A0186B0006; Thu, 6 Feb 2020 04:11:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B5D76B0007; Thu, 6 Feb 2020 04:11:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0200.hostedemail.com [216.40.44.200]) by kanga.kvack.org (Postfix) with ESMTP id 1F3966B0003 for ; Thu, 6 Feb 2020 04:11:23 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C4BD98E5C for ; Thu, 6 Feb 2020 09:11:22 +0000 (UTC) X-FDA: 76459133604.02.train69_290696d807903 X-HE-Tag: train69_290696d807903 X-Filterd-Recvd-Size: 14797 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Thu, 6 Feb 2020 09:11:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580980281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=szMWp9dCf657bVR24xnlhqIStiGlTnAs3pmffl1FCyQ=; b=TSPuwI/bbvheb1R6hbVyjUQR8WGtJManDpM56psMFS1rf3h8m/0fzTk9qflF5ihis26LOC wuby9GnudOX+wnEMQ3q40Ur5BOdtws5CW8XtElXLz962nOoTpimM6ppgiGRpJcXVycnu4j QiR73DeXSviBPkojdXvlA3Vf80vT604= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-269-5I8KQ_0kNrW7-KXnczC0gA-1; Thu, 06 Feb 2020 04:11:16 -0500 Received: by mail-qv1-f70.google.com with SMTP id g15so3267390qvq.20 for ; Thu, 06 Feb 2020 01:11:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=/t9maecIs5tki36A7uRhOFb40wkNs+40rTQgn09/WK8=; b=iqcAhA57QVknxczDcxfSnTnQazLENuIVJHefmzViXjto20i8bMSqhzCev7tCErU+bC AU1sN+ba75tZH3IvOzVSfzRav0Uy0P8YBlWRiEQ7cSI79vEXcvwaGvRaHFCDt+ZVIw1n iSHYmpu8OHMz+3Q6JsO/MjUV8wjNkNIEhOM/BvAQ3IPEWz5gpjYNRONGTfps4HhoHEsc 4Txv71ho8z2DtdK1EVhq1wxuw6lmwW88laUOa1aFCMgDYFCkm2aM6aApzbtv+jlnWq23 3vY5x0tGdGIWyvJcK9AjV8zyoWv8a3jGA+hN7nMZQJEerg+aNkB6GlP6ZWalVLND0XP7 mL3g== X-Gm-Message-State: APjAAAWDMrLfPMLOEhA4lIrwkgub/IGde71DA7NrhCkNLfqrs4JS8vko qi50owc+TwvhCBHFJ4axoOdEsrYIWmn55iQJpkRKMmzSEwF6G43BI+xE/3yMD09ng8PNRZXhIQX Tqb0IzakRzco= X-Received: by 2002:a05:620a:9cc:: with SMTP id y12mr1588237qky.446.1580980275517; Thu, 06 Feb 2020 01:11:15 -0800 (PST) X-Google-Smtp-Source: APXvYqxH9AALMiv9HPt4nTzXlyGpnRQpNx3LPKyvEl2gt+gFQ8/4ngbLEjEieYdPDlm7tmznn9TLTg== X-Received: by 2002:a05:620a:9cc:: with SMTP id y12mr1588220qky.446.1580980275214; Thu, 06 Feb 2020 01:11:15 -0800 (PST) Received: from redhat.com (bzq-79-176-41-183.red.bezeqint.net. [79.176.41.183]) by smtp.gmail.com with ESMTPSA id y194sm1123257qkb.113.2020.02.06.01.11.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Feb 2020 01:11:14 -0800 (PST) Date: Thu, 6 Feb 2020 04:11:10 -0500 From: "Michael S. Tsirkin" To: David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org, Tyler Sanderson , Wei Wang , Alexander Duyck , David Rientjes , Nadav Amit , Michal Hocko Subject: Re: [PATCH v1 3/3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM Message-ID: <20200206040958-mutt-send-email-mst@kernel.org> References: <20200205163402.42627-1-david@redhat.com> <20200205163402.42627-4-david@redhat.com> MIME-Version: 1.0 In-Reply-To: <20200205163402.42627-4-david@redhat.com> X-MC-Unique: 5I8KQ_0kNrW7-KXnczC0gA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 05, 2020 at 05:34:02PM +0100, David Hildenbrand wrote: > Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker"= ) > changed the behavior when deflation happens automatically. Instead of > deflating when called by the OOM handler, the shrinker is used. >=20 > However, the balloon is not simply some slab cache that should be > shrunk when under memory pressure. The shrinker does not have a concept o= f > priorities, so this behavior cannot be configured. >=20 > There was a report that this results in undesired side effects when > inflating the balloon to shrink the page cache. [1] > =09"When inflating the balloon against page cache (i.e. no free memory > =09 remains) vmscan.c will both shrink page cache, but also invoke the > =09 shrinkers -- including the balloon's shrinker. So the balloon > =09 driver allocates memory which requires reclaim, vmscan gets this > =09 memory by shrinking the balloon, and then the driver adds the > =09 memory back to the balloon. Basically a busy no-op." >=20 > The name "deflate on OOM" makes it pretty clear when deflation should > happen - after other approaches to reclaim memory failed, not while > reclaiming. This allows to minimize the footprint of a guest - memory > will only be taken out of the balloon when really needed. >=20 > Especially, a drop_slab() will result in the whole balloon getting > deflated - undesired. While handling it via the OOM handler might not be > perfect, it keeps existing behavior. If we want a different behavior, the= n > we need a new feature bit and document it properly (although, there shoul= d > be a clear use case and the intended effects should be well described). >=20 > Keep using the shrinker for VIRTIO_BALLOON_F_FREE_PAGE_HINT, because > this has no such side effects. Always register the shrinker with > VIRTIO_BALLOON_F_FREE_PAGE_HINT now. We are always allowed to reuse free > pages that are still to be processed by the guest. The hypervisor takes > care of identifying and resolving possible races between processing a > hinting request and the guest reusing a page. >=20 > In contrast to pre commit 71994620bb25 ("virtio_balloon: replace oom > notifier with shrinker"), don't add a moodule parameter to configure the > number of pages to deflate on OOM. Can be re-added if really needed. > Also, pay attention that leak_balloon() returns the number of 4k pages - > convert it properly in virtio_balloon_oom_notify(). >=20 > Note1: using the OOM handler is frowned upon, but it really is what we > need for this feature. >=20 > Note2: without VIRTIO_BALLOON_F_MUST_TELL_HOST (iow, always with QEMU) we > could actually skip sending deflation requests to our hypervisor, > making the OOM path *very* simple. Besically freeing pages and > updating the balloon. If the communication with the host ever > becomes a problem on this call path. >=20 > [1] https://www.spinics.net/lists/linux-virtualization/msg40863.html >=20 > Reported-by: Tyler Sanderson > Cc: Michael S. Tsirkin > Cc: Wei Wang > Cc: Alexander Duyck > Cc: David Rientjes > Cc: Nadav Amit > Cc: Michal Hocko > Signed-off-by: David Hildenbrand So the revert looks ok, from that POV and with commit log changes Acked-by: Michael S. Tsirkin however, let's see what do others think, and whether Wei can come up with a fixup for the shrinker. > --- > drivers/virtio/virtio_balloon.c | 107 +++++++++++++------------------- > 1 file changed, 44 insertions(+), 63 deletions(-) >=20 > diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_ball= oon.c > index 7e5d84caeb94..e7b18f556c5e 100644 > --- a/drivers/virtio/virtio_balloon.c > +++ b/drivers/virtio/virtio_balloon.c > @@ -14,6 +14,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -27,7 +28,9 @@ > */ > #define VIRTIO_BALLOON_PAGES_PER_PAGE (unsigned)(PAGE_SIZE >> VIRTIO_BAL= LOON_PFN_SHIFT) > #define VIRTIO_BALLOON_ARRAY_PFNS_MAX 256 > -#define VIRTBALLOON_OOM_NOTIFY_PRIORITY 80 > +/* Maximum number of (4k) pages to deflate on OOM notifications. */ > +#define VIRTIO_BALLOON_OOM_NR_PAGES 256 > +#define VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY 80 > =20 > #define VIRTIO_BALLOON_FREE_PAGE_ALLOC_FLAG (__GFP_NORETRY | __GFP_NOWAR= N | \ > =09=09=09=09=09 __GFP_NOMEMALLOC) > @@ -112,8 +115,11 @@ struct virtio_balloon { > =09/* Memory statistics */ > =09struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; > =20 > -=09/* To register a shrinker to shrink memory upon memory pressure */ > +=09/* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */ > =09struct shrinker shrinker; > + > +=09/* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM *= / > +=09struct notifier_block oom_nb; > }; > =20 > static struct virtio_device_id id_table[] =3D { > @@ -786,50 +792,13 @@ static unsigned long shrink_free_pages(struct virti= o_balloon *vb, > =09return blocks_freed * VIRTIO_BALLOON_HINT_BLOCK_PAGES; > } > =20 > -static unsigned long leak_balloon_pages(struct virtio_balloon *vb, > - unsigned long pages_to_free) > -{ > -=09return leak_balloon(vb, pages_to_free * VIRTIO_BALLOON_PAGES_PER_PAGE= ) / > -=09=09VIRTIO_BALLOON_PAGES_PER_PAGE; > -} > - > -static unsigned long shrink_balloon_pages(struct virtio_balloon *vb, > -=09=09=09=09=09 unsigned long pages_to_free) > -{ > -=09unsigned long pages_freed =3D 0; > - > -=09/* > -=09 * One invocation of leak_balloon can deflate at most > -=09 * VIRTIO_BALLOON_ARRAY_PFNS_MAX balloon pages, so we call it > -=09 * multiple times to deflate pages till reaching pages_to_free. > -=09 */ > -=09while (vb->num_pages && pages_freed < pages_to_free) > -=09=09pages_freed +=3D leak_balloon_pages(vb, > -=09=09=09=09=09=09 pages_to_free - pages_freed); > - > -=09update_balloon_size(vb); > - > -=09return pages_freed; > -} > - > static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrin= ker, > =09=09=09=09=09=09 struct shrink_control *sc) > { > -=09unsigned long pages_to_free, pages_freed =3D 0; > =09struct virtio_balloon *vb =3D container_of(shrinker, > =09=09=09=09=09struct virtio_balloon, shrinker); > =20 > -=09pages_to_free =3D sc->nr_to_scan; > - > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > -=09=09pages_freed =3D shrink_free_pages(vb, pages_to_free); > - > -=09if (pages_freed >=3D pages_to_free) > -=09=09return pages_freed; > - > -=09pages_freed +=3D shrink_balloon_pages(vb, pages_to_free - pages_freed= ); > - > -=09return pages_freed; > +=09return shrink_free_pages(vb, sc->nr_to_scan); > } > =20 > static unsigned long virtio_balloon_shrinker_count(struct shrinker *shri= nker, > @@ -837,26 +806,22 @@ static unsigned long virtio_balloon_shrinker_count(= struct shrinker *shrinker, > { > =09struct virtio_balloon *vb =3D container_of(shrinker, > =09=09=09=09=09struct virtio_balloon, shrinker); > -=09unsigned long count; > - > -=09count =3D vb->num_pages / VIRTIO_BALLOON_PAGES_PER_PAGE; > -=09count +=3D vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES= ; > =20 > -=09return count; > +=09return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES; > } > =20 > -static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb= ) > +static int virtio_balloon_oom_notify(struct notifier_block *nb, > +=09=09=09=09 unsigned long dummy, void *parm) > { > -=09unregister_shrinker(&vb->shrinker); > -} > +=09struct virtio_balloon *vb =3D container_of(nb, > +=09=09=09=09=09=09 struct virtio_balloon, oom_nb); > +=09unsigned long *freed =3D parm; > =20 > -static int virtio_balloon_register_shrinker(struct virtio_balloon *vb) > -{ > -=09vb->shrinker.scan_objects =3D virtio_balloon_shrinker_scan; > -=09vb->shrinker.count_objects =3D virtio_balloon_shrinker_count; > -=09vb->shrinker.seeks =3D DEFAULT_SEEKS; > +=09*freed +=3D leak_balloon(vb, VIRTIO_BALLOON_OOM_NR_PAGES) / > +=09=09 VIRTIO_BALLOON_PAGES_PER_PAGE; > +=09update_balloon_size(vb); > =20 > -=09return register_shrinker(&vb->shrinker); > +=09return NOTIFY_OK; > } > =20 > static int virtballoon_probe(struct virtio_device *vdev) > @@ -933,22 +898,35 @@ static int virtballoon_probe(struct virtio_device *= vdev) > =09=09=09virtio_cwrite(vb->vdev, struct virtio_balloon_config, > =09=09=09=09 poison_val, &poison_val); > =09=09} > -=09} > -=09/* > -=09 * We continue to use VIRTIO_BALLOON_F_DEFLATE_ON_OOM to decide if a > -=09 * shrinker needs to be registered to relieve memory pressure. > -=09 */ > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) { > -=09=09err =3D virtio_balloon_register_shrinker(vb); > + > +=09=09/* > +=09=09 * We're allowed to reuse any free pages, even if they are > +=09=09 * still to be processed by the host. > +=09=09 */ > +=09=09vb->shrinker.scan_objects =3D virtio_balloon_shrinker_scan; > +=09=09vb->shrinker.count_objects =3D virtio_balloon_shrinker_count; > +=09=09vb->shrinker.seeks =3D DEFAULT_SEEKS; > +=09=09err =3D register_shrinker(&vb->shrinker); > =09=09if (err) > =09=09=09goto out_del_balloon_wq; > =09} > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) { > +=09=09vb->oom_nb.notifier_call =3D virtio_balloon_oom_notify; > +=09=09vb->oom_nb.priority =3D VIRTIO_BALLOON_OOM_NOTIFY_PRIORITY; > +=09=09err =3D register_oom_notifier(&vb->oom_nb); > +=09=09if (err < 0) > +=09=09=09goto out_unregister_shrinker; > +=09} > + > =09virtio_device_ready(vdev); > =20 > =09if (towards_target(vb)) > =09=09virtballoon_changed(vdev); > =09return 0; > =20 > +out_unregister_shrinker: > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > +=09=09unregister_shrinker(&vb->shrinker); > out_del_balloon_wq: > =09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > =09=09destroy_workqueue(vb->balloon_wq); > @@ -987,8 +965,11 @@ static void virtballoon_remove(struct virtio_device = *vdev) > { > =09struct virtio_balloon *vb =3D vdev->priv; > =20 > -=09if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) > -=09=09virtio_balloon_unregister_shrinker(vb); > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) > +=09=09unregister_oom_notifier(&vb->oom_nb); > +=09if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) > +=09=09unregister_shrinker(&vb->shrinker); > + > =09spin_lock_irq(&vb->stop_update_lock); > =09vb->stop_update =3D true; > =09spin_unlock_irq(&vb->stop_update_lock); > --=20 > 2.24.1