From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDDCEC47DAF for ; Thu, 18 Jan 2024 16:16:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23D266B007E; Thu, 18 Jan 2024 11:16:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1ECE76B0080; Thu, 18 Jan 2024 11:16:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B5C06B0081; Thu, 18 Jan 2024 11:16:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EF95E6B007E for ; Thu, 18 Jan 2024 11:16:11 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C201C1C1350 for ; Thu, 18 Jan 2024 16:16:11 +0000 (UTC) X-FDA: 81692933742.08.F7D6C30 Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) by imf02.hostedemail.com (Postfix) with ESMTP id 9373280011 for ; Thu, 18 Jan 2024 16:16:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=xi3KHTkH; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.167.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705594569; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MRkAAxwE545hdA1CHQLahd76L0FTgjCjQ9oVd5rONMw=; b=PF+T0w4Y1OissAwOpsnCryLGMaXpegUqgKqq5E5KK2Q45VGIKzC5iZaO7D9kJLn3dW4PzD Foc8sjOTh+IWmqhjz1C7oUI9a8EDcygCHJC6i2KRZApKas5I2jGdaIaYVyLROpzvnCEdki AYzgWO3VFWeSVE9b1YVK8Wi5SFTNKfg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=xi3KHTkH; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.167.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705594569; a=rsa-sha256; cv=none; b=1ExiMTFOltErQzudiJJrR6+5J9ppoM/ZwjGxEfwhJ7hZKVnHmCcjYh0jBYcTE4qT0EvQ5J qBuAKPEQvNJ04WGsFJ6owwA2Co/+NYCKl4uRXjJS4EIHQM+vH1EpXclPCG3cIcz9KdZ6YL 8PfHwdrMNRPER2KxMeNAmM0EsQh4QXc= Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3bd884146e9so2248396b6e.0 for ; Thu, 18 Jan 2024 08:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1705594568; x=1706199368; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=MRkAAxwE545hdA1CHQLahd76L0FTgjCjQ9oVd5rONMw=; b=xi3KHTkH9/51alDYJ36WVP1pS2m+kpuOT2X4oNq4rcn5lEbjNCdpJUA1nnKzDN5v0k WsSLVcdkiQ+VxlMB2sNA/1zhnwhNkEpzKmo1IolxF7CFW6UnKU2XkmhQZgRTmlfM2i04 CE2Ya3gCBZEa4kQ7RbmFopwM3oPDtLp/7yua2AMkpo0Ggv/Vk2W1JkRe11WML0qi36/P Ah/Io0cIyofR9aLxrS9r0Rv73TxW2JQv+7aqriPvucEXjt54QyI82DOGodhcDhEmHkzs 3yVIG6XWEN2rEhdDfh8BXUeFouyuOVDxdZd2Cn0l+qzWHBdWk5j+zNzONFtKuSdvA+2j ZRTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705594568; x=1706199368; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MRkAAxwE545hdA1CHQLahd76L0FTgjCjQ9oVd5rONMw=; b=SHu2S1kxI4cUwMuCCBULGjES/MV6vjiK3eyV4xJpTdpA0gP1/dX1qkDmMKpOuMSJVV ZNHR8qCRGrZdWB8W+7451sk3OGRxwijkXdG97RHw+SKp7G/ouEOySoJoqCE3uK9puYmb LAbEAaYatdegwRFoJaveiJn7OX/Ag4ymLGT8f1uTfgs5PiUDYE485yWvj4nM/0Pu46Ef qpjqxi4JGOutW9mnYvEbFDi6oYnEZXHMxX+cHKxvXlJtAhcjFv7p/P7FBsMRl23LyW++ zwa52yScgaXt5fXiuVzt1ysQnQOPPOHozC55BZplvoNsUavbKKB6EbKmuNTGCHfNOKaP azzA== X-Gm-Message-State: AOJu0Yynexp4fWY8P/DhUkW9tnxvuD0ogRi2mAmEg9Pc2D32q4s3ObUP KuY/g87ey6DV6X20Q9i1CNxm7QaxdDdc4Hq3Y/kGfcRsi6auHM/E6KpJG8RtoT8= X-Google-Smtp-Source: AGHT+IG+S1DdeH3g6eDWfi0nVMqIeHOEMSECiM1LqNNddjUCuRix8OSGJiLYFXD+hc+8VolIcbx43A== X-Received: by 2002:a05:6808:d50:b0:3bd:a103:490b with SMTP id w16-20020a0568080d5000b003bda103490bmr65459oik.104.1705594568587; Thu, 18 Jan 2024 08:16:08 -0800 (PST) Received: from localhost ([2620:10d:c091:400::5:2fe6]) by smtp.gmail.com with ESMTPSA id bc14-20020a05622a1cce00b00429bdddfb49sm6891681qtb.44.2024.01.18.08.16.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 08:16:07 -0800 (PST) Date: Thu, 18 Jan 2024 11:16:01 -0500 From: Johannes Weiner To: Yosry Ahmed Cc: Nhat Pham , Ronald Monthero , sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, akpm@linux-foundation.org, chrisl@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] mm/zswap: Improve with alloc_workqueue() call Message-ID: <20240118161601.GJ939255@cmpxchg.org> References: <20240116133145.12454-1-debug.penguin32@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9373280011 X-Stat-Signature: ifwh13gyfe6fah9qy5ab8yu6n6p4drzr X-Rspam-User: X-HE-Tag: 1705594569-992434 X-HE-Meta: U2FsdGVkX1+3XboClTkWH/ixMGIoey0An63u1HIgp53t1u7GqA8KkQs4BTPux5QyhbRA4VhRcI8ACS4kIYqN2Tnl0ABvKaLYpkcxOgd+vFgrZ3h7kjs+UdLRhKv55d8mTpOKBxN7tSZo3JqcjhY5GzWAaaIVM16AxrboDIEukCVNAKQpJjp/t9dpi1lGBxrDjrMxSyQrBzu4W5T9PuaEMQZg7/nwYhnNR3VyFy4zbJad0+/DHkDgFbnv+dYbizrnqa6fwU+LBeeaFbArcgj3H3X8M4uJiQPIqzDJEO1uko8CjCbMpXa6AxMTG8B4al/wnatoqRGrwIBjGALFIH8T3DMHV5aIKZXcGaPcHtDW6ol5qY97rPp5tC2zUqihvYCgdYT8jCZuztgXEdFpRuuNP4wKCSYnA57K0v6veijjocvwwJOC+Fowvc3DaSPrTfGYtDMxTxoQKS3Mrlr4JzbP222dtm+sCNt1dkJUbw4uuVuUjpizGmUbF+EV4fYcWqkMh3oIP9LRuxHP+1NPoYtHs/MRil4a0SUEr/zke1P/EK6+W/tlM3QUjQmeHa+hnmYl4JvcCtwt1wmQdLS0GwjAcJQFoP7pFdQjMjf2agj5rzs14eT3ct/J55NkCet7kW96a568CxHcsjKGWes4/5ktzrA4KuPeFccr1IZK7sPlt1kwCGhILM0vc+lrvMFlAIJmwW9Ii3gAH8KKqAyqQTSck2TFFNlWEz263lnuCFcWqB0YrojdnNr4lWBAr31tWcLA2RfCL0tNoKXITFUI65UMYNmg0fCyTdS+MAHXB5hHT5fI0HLzYqH64Rsvdof5h7/XURxQSkMHNTx/VWD0MuDS5w8G8DxQi6y0YwhnGgsqwj2kbfkIniDCSTbAXTrSLQjpq51r4wiRDeJ75ATlrbk+81YXRGPF5qsXIkgOEKvFD6A8qEJ6DSe+WLvuqUNa+WpxnLTRgG7D9c/iDGcSeAq Pof8PMd+ 8K+7C7lw5R7V1A7sCUq7Tyk8O5h43uVbIIg0QK8rpi6uU9vEhdUZq4JKfdLVrljK9+dVjWcL+c5UoxqD10/dQLySxpwz9s3pGKQtd50NN7GhqiEpa/qvywX94V1DYRw0rVh1GjviiQbPuJQ1Eu0NObNPgGIeSzbSETWYF4sEO81cQqNkIgzrmpWGK9rKEWAhrAwCNpgMEM7ii1dMj6zKiObCTF+fvfVL+2IXsuGDqHE8qfUV8Xphu5mTm22eI7FbVPQXO6zl/DttbDpWlJU5EQEDLAiFAiJmvjEVBtvSxCK9prfF8oPVjeSx+tFDj3dgtDy48tS11o6hbTZoVWlCN7s+KyAfMOV+x/DLDuMziB7TIyXlyvI9NouXeZ3zYSJB2PtA0+b25D/4Yv+gDqt/xaOaz6ZruYVjF0DiuN9eQR3J6eWArAIaPEBIlJsp4816fqpjHOZdC35knDECG1W89r0C3h38SMAH5HXhda9JqJ9N4Jgb1f+ZnisYE4NZhOxhtMVn7H+Rr2uQpsv5J+ACIGqoPMiEXJfzJ2mvyv0vIf0RDPmNZg4pcTnV13X83JgKrLs9daYFDfWVvETkQ5dyJ0A28zrTLqHCQ1F/zhWcc+cuJwcdbfkr0YnJLd3dBbVW+OVTfXPGRPotBbN4f9cQxgTejCrPljXWGp4MM6yUgBm4ozj95NUlEy/gOxZ9x0Gaul2ENvPWB15ocGsaGHdvYFWqX+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 17, 2024 at 11:30:50AM -0800, Yosry Ahmed wrote: > On Wed, Jan 17, 2024 at 11:14 AM Nhat Pham wrote: > > > > On Tue, Jan 16, 2024 at 5:32 AM Ronald Monthero > > wrote: > > > > + Johannes and Yosry > > > > > > > > The core-api create_workqueue is deprecated, this patch replaces > > > the create_workqueue with alloc_workqueue. The previous > > > implementation workqueue of zswap was a bounded workqueue, this > > > patch uses alloc_workqueue() to create an unbounded workqueue. > > > The WQ_UNBOUND attribute is desirable making the workqueue > > > not localized to a specific cpu so that the scheduler is free > > > to exercise improvisations in any demanding scenarios for > > > offloading cpu time slices for workqueues. > > > > nit: extra space between paragraph would be nice. > > > > > For example if any other workqueues of the same primary cpu > > > had to be served which are WQ_HIGHPRI and WQ_CPU_INTENSIVE. > > > Also Unbound workqueue happens to be more efficient > > > in a system during memory pressure scenarios in comparison > > > to a bounded workqueue. > > > > > > shrink_wq = alloc_workqueue("zswap-shrink", > > > WQ_UNBOUND|WQ_MEM_RECLAIM, 1); > > > > > > Overall the change suggested in this patch should be > > > seamless and does not alter the existing behavior, > > > other than the improvisation to be an unbounded workqueue. > > > > > > Signed-off-by: Ronald Monthero > > > --- > > > mm/zswap.c | 3 ++- > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > diff --git a/mm/zswap.c b/mm/zswap.c > > > index 74411dfdad92..64dbe3e944a2 100644 > > > --- a/mm/zswap.c > > > +++ b/mm/zswap.c > > > @@ -1620,7 +1620,8 @@ static int zswap_setup(void) > > > zswap_enabled = false; > > > } > > > > > > - shrink_wq = create_workqueue("zswap-shrink"); > > > + shrink_wq = alloc_workqueue("zswap-shrink", > > > + WQ_UNBOUND|WQ_MEM_RECLAIM, 1); > > > > Have you benchmarked this to check if there is any regression, just to > > be safe? With an unbounded workqueue, you're gaining scheduling > > flexibility at the cost of cache locality. My intuition is that it > > doesn't matter too much here, but you should probably double check by > > stress testing - run some workload with a relatively small zswap pool > > limit (i.e heavy global writeback), and see if there is any difference > > in performance. > > I also think this shouldn't make a large difference. The global > shrinking work is already expensive, and I imagine that it exhausts > the caches anyway by iterating memcgs. A performance smoketest would > be reassuring for sure, but I believe it won't make a difference. The LRU inherently makes the shrinker work on the oldest and coldest entries, so I doubt we benefit a lot from cache locality there. What could make a difference though is the increased concurrency by switching max_active from 1 to 0. This could cause a higher rate of shrinker runs, which might increase lock contention and reclaim volume. That part would be good to double check with the shrinker benchmarks. > > On a different note, I wonder if it would help to perform synchronous > > reclaim here instead. With our current design, the zswap store failure > > (due to global limit hit) would leave the incoming page going to swap > > instead, creating an LRU inversion. Not sure if that's ideal. > > The global shrink path keeps reclaiming until zswap can accept again > (by default, that means reclaiming 10% of the total limit). I think > this is too expensive to be done synchronously. That thresholding code is a bit weird right now. It wakes the shrinker and rejects at the same time. We're guaranteed to see rejections, even if the shrinker has no trouble flushing some entries a split second later. It would make more sense to wake the shrinker at e.g. 95% full and have it run until 90%. But with that in place we also *should* do synchronous reclaim once we hit 100%. Just enough to make room for the store. This is important to catch the case where reclaim rate exceeds swapout rate. Rejecting and going to swap means the reclaimer will be throttled down to IO rate anyway, and the app latency isn't any worse. But this way we keep the pipeline alive, and keep swapping out the oldest zswap entries, instead of rejecting and swapping what would be the hottest ones.