From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49DB0C7EE23 for ; Fri, 26 May 2023 10:18:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AC9E6B0074; Fri, 26 May 2023 06:18:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 534AE900003; Fri, 26 May 2023 06:18:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FCD9900002; Fri, 26 May 2023 06:18:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 29B196B0074 for ; Fri, 26 May 2023 06:18:37 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CC93280B69 for ; Fri, 26 May 2023 10:18:36 +0000 (UTC) X-FDA: 80832007032.12.4D677E7 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf07.hostedemail.com (Postfix) with ESMTP id BC77C40015 for ; Fri, 26 May 2023 10:18:33 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=konsulko.com header.s=google header.b=McV4Spin; dmarc=pass (policy=none) header.from=konsulko.com; spf=pass (imf07.hostedemail.com: domain of vitaly.wool@konsulko.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=vitaly.wool@konsulko.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685096313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RlqAXejfirMJTTl5iB2+w+HwiJoIEIC89FrsVWdTKhE=; b=KRWjP8ODlUQzlezd7LnDob8voYj78lga5ZNHZSw3avpeigXWtyWPLprUIaBGT5cQ5WGNry pExqHXp0B1Ttf1pKRI/Sq4VBij/VyZy66GRBGRynhbuv/xZdMFJ2cErHVGubEYfZgACWrJ WMJBP6qztBM5v2/d3T8SFor4PKCATKE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=konsulko.com header.s=google header.b=McV4Spin; dmarc=pass (policy=none) header.from=konsulko.com; spf=pass (imf07.hostedemail.com: domain of vitaly.wool@konsulko.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=vitaly.wool@konsulko.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685096313; a=rsa-sha256; cv=none; b=GGNqoBz37heUd110IMropyky+1wIQC4nbDIQcxZJShai6stRMV9TBKANDOzjV0YgKG5ZM/ 6/yjILMuHzqyLMcs3lGjlbOVNpFsyNYsMXg4oiVUtmUHL+Mf7kU1z2k+od4EJiSzrhVfYw KcpA6CeN++E4RbItcp3eRA7Y/MKCK8M= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1ae763f9a94so3954495ad.3 for ; Fri, 26 May 2023 03:18:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; t=1685096312; x=1687688312; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=RlqAXejfirMJTTl5iB2+w+HwiJoIEIC89FrsVWdTKhE=; b=McV4SpinmODlndelrGvXMBwr77mQpGgQFCx/TjpCAOYysCLo1uBms1yuLxjx/a2atW 5s3QigePCD3UMiryzMkJPVGoGZT/icdDftgsqOTxRMCI0Pm9MjZXl1Mt+WPODPnjuTAH PGX0MM28eYoVY7w+DAEn9gP8Q9zXCwpGHAEh8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685096312; x=1687688312; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RlqAXejfirMJTTl5iB2+w+HwiJoIEIC89FrsVWdTKhE=; b=e0j+5X8unhneDVrJy2V8DrFkAg7buvdPeie7p6mv6DZyYObpKeSeDVuzg5JAkGJV0h 9FTGcERppGCj4iwnevaDyV3sTTRth5yH8nSuSfmsXhZLaHpWbqUcfTfPVa7qCil3rEu+ sDG7iczhOyV8MAJxj+0MxwFnWGVahVmyp9G15qkKZOx3fpsXxqo7qxF3WFZKXHfQlaSD 20yxBF46xB2jCPglwvYLy87oHZMbFagHoscprp4lVCRTW9uWkEErBtoyrLlKFyw2GHit l7UOA3UWpwzbrtRZr1gmsHrc2A5tw+BGaAmZJV+Fy43ogluRzhXoC+DyFsegkIsK0Ldb VJ+A== X-Gm-Message-State: AC+VfDxNZbXUlBDov/Br6pQcLKj86WqR+98WzYb13UDAVIf5HvV3tZg5 oa3YD9CtiaSDG5/0dZqpfzD1gK6jI3FWfX5Y7pBKzg== X-Google-Smtp-Source: ACHHUZ6hqAqRVb3mcOCYrOmRkmTU3PSJToOeDiWChZ+adogXMRZswpaYWn8rr61PmwxJ5R/+n16pDDyMzdVn1fTb12c= X-Received: by 2002:a17:903:1209:b0:1b0:6c3:e851 with SMTP id l9-20020a170903120900b001b006c3e851mr2368882plh.18.1685096312458; Fri, 26 May 2023 03:18:32 -0700 (PDT) MIME-Version: 1.0 References: <20230524065051.6328-1-cerasuolodomenico@gmail.com> In-Reply-To: <20230524065051.6328-1-cerasuolodomenico@gmail.com> From: Vitaly Wool Date: Fri, 26 May 2023 12:18:21 +0200 Message-ID: Subject: Re: [PATCH] mm: zswap: shrink until can accept To: Domenico Cerasuolo Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, sjenning@redhat.com, ddstreet@ieee.org, yosryahmed@google.com, hannes@cmpxchg.org, kernel-team@fb.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BC77C40015 X-Stat-Signature: x83xwox1stuuwuk4khtqdjztq71jpxab X-Rspam-User: X-HE-Tag: 1685096313-464145 X-HE-Meta: U2FsdGVkX1+iUnkrhzW6XC1Pg0WsCvwKyHgOMBHbaFlH2pYlyYVV5dguBNe/xEHyiH6l6dtODG3PR5D0SAkyigbv9zjOFdc9tg39aoWFhAqtIweNP9er/XaArFdOfc9qYiWQMePuKE9o1X1uZID6vCFV/U/U4z67a2UFozzayf4l2XjLcwSZWn1M9ANXbt6qx3EaGQBQI03e+YUGeDovtGoGZ+iZKQl8xs19RxIrvbTgWHLUeTSuAmcj6AK4IzlsN/PcKOLHFQeXdYw4TqB+5ilOEIMoSkaAUKr9COdXapohfRa0nbpVtkwnEL7Vkz9CpGx0aP5pak7DSwvD6cykvKmi5ucNR7ddOPphpm+nuRWmx07dej52VZ1xtYC/MxSbc2+jC6FNNYxgw0BWAN4qvZ2Eo+PHestwQc4qLU2k8kbH7xMIMtil+1WEEAECff0fyoB1evIkHgR0aE9nTMKYQdL7Jte+xgeY4BOJxNyXVY/rzdHQ9UWn2HAzNcLMn5dXa47ZNv751Pt+WdtK6SRGRpt694kpxZtITYIJpZeR1TdVPfpBx9Fl9r0UNJcfPLqlwGfvsB4f/TKaLtk4Ph5cOXsW6hh41aEQAzp92f/4svFDRA3sZkgZkZDNW6138T6BUZ+HLSUeImB4n6l8u2Zt5JW2S/0eLfuW8gt07PhE2LW9qHRoMghtbpyJZOemS8/YpTk2P0XAbw8YnBxo8zzAtq6TMefpajGal/tqEoV7TwHqbgREp5hOu22/RJUWvIeaVS03430R/I25mzHXk0wml/UCpp9Q5tkKlB4FcWOedfc1XvK8cbk6xhlF0tB7WUVG6klfZgnVGcrx+tU5CUXY3aZyLkie8A/sg327rhBRXLg1S1fNLjqJnZrb9+VkyH0n86uTOGYTlKT6iE8FIU/tsZC+RLwzsQzXDuSP/GPIbuOsNvsTAIbzQ/zaOkX4fITnkLQZTtBlFD93tYS2K3/ PnRPJigU T5B5SGKh0nKshQtD6gIuk3lyrl3s0VeSrWsmQEauGnvzFM5bb4UCwZuShg1i4CdnjJzygcBHXIS/Ek4xPXaNkxM9fJnPvwxKiiUPGG2JJSgsi/nFFwIg/PIxVG4SChyMP+dReMWebDXK+RtgOvaHca2ruaWkfadAYI339oD4xK/DPveyv1ohnDAg13bFDADo9vVlWZVy4GEDZcMOAlgbJMCBR8Dt2WW6bk5ib2WfPXhwAhYREAnZhzk/uX9gEkUIA4dJ/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Domenico, On Wed, May 24, 2023 at 8:50=E2=80=AFAM Domenico Cerasuolo wrote: > > This update addresses an issue with the zswap reclaim mechanism, which > hinders the efficient offloading of cold pages to disk, thereby > compromising the preservation of the LRU order and consequently > diminishing, if not inverting, its performance benefits. > > The functioning of the zswap shrink worker was found to be inadequate, > as shown by basic benchmark test. For the test, a kernel build was > utilized as a reference, with its memory confined to 1G via a cgroup and > a 5G swap file provided. The results are presented below, these are > averages of three runs without the use of zswap: > > real 46m26s > user 35m4s > sys 7m37s > > With zswap (zbud) enabled and max_pool_percent set to 1 (in a 32G > system), the results changed to: > > real 56m4s > user 35m13s > sys 8m43s > > written_back_pages: 18 > reject_reclaim_fail: 0 > pool_limit_hit:1478 > > Besides the evident regression, one thing to notice from this data is > the extremely low number of written_back_pages and pool_limit_hit. > > The pool_limit_hit counter, which is increased in zswap_frontswap_store > when zswap is completely full, doesn't account for a particular > scenario: once zswap hits his limit, zswap_pool_reached_full is set to > true; with this flag on, zswap_frontswap_store rejects pages if zswap is > still above the acceptance threshold. Once we include the rejections due > to zswap_pool_reached_full && !zswap_can_accept(), the number goes from > 1478 to a significant 21578266. > > Zswap is stuck in an undesirable state where it rejects pages because > it's above the acceptance threshold, yet fails to attempt memory > reclaimation. This happens because the shrink work is only queued when > zswap_frontswap_store detects that it's full and the work itself only > reclaims one page per run. > > This state results in hot pages getting written directly to disk, > while cold ones remain memory, waiting only to be invalidated. The LRU > order is completely broken and zswap ends up being just an overhead > without providing any benefits. > > This commit applies 2 changes: a) the shrink worker is set to reclaim > pages until the acceptance threshold is met and b) the task is also > enqueued when zswap is not full but still above the threshold. > > Testing this suggested update showed much better numbers: > > real 36m37s > user 35m8s > sys 9m32s > > written_back_pages: 10459423 > reject_reclaim_fail: 12896 > pool_limit_hit: 75653 > > Fixes: 45190f01dd40 ("mm/zswap.c: add allocation hysteresis if pool limit= is hit") > Signed-off-by: Domenico Cerasuolo > --- > mm/zswap.c | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) > > diff --git a/mm/zswap.c b/mm/zswap.c > index 59da2a415fbb..2ee0775d8213 100644 > --- a/mm/zswap.c > +++ b/mm/zswap.c > @@ -587,9 +587,13 @@ static void shrink_worker(struct work_struct *w) > { > struct zswap_pool *pool =3D container_of(w, typeof(*pool), > shrink_work); > + int ret; > > - if (zpool_shrink(pool->zpool, 1, NULL)) > - zswap_reject_reclaim_fail++; > + do { > + ret =3D zpool_shrink(pool->zpool, 1, NULL); > + if (ret) > + zswap_reject_reclaim_fail++; > + } while (!zswap_can_accept() && ret !=3D -EINVAL); > zswap_pool_put(pool); > } while I do agree with your points, I have a concern about this shrinker logic change. The reason for not doing this as you do was possible real time/responsiveness characteristics degrade. Have you checked that it's not really the case? Thanks, Vitaly > @@ -1188,7 +1192,7 @@ static int zswap_frontswap_store(unsigned type, pgo= ff_t offset, > if (zswap_pool_reached_full) { > if (!zswap_can_accept()) { > ret =3D -ENOMEM; > - goto reject; > + goto shrink; > } else > zswap_pool_reached_full =3D false; > } > -- > 2.34.1 >