From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E870CCDB474 for ; Tue, 17 Oct 2023 04:47:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DA9F8D00E6; Tue, 17 Oct 2023 00:47:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 18AB68D00DE; Tue, 17 Oct 2023 00:47:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0524D8D00E6; Tue, 17 Oct 2023 00:47:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EA2D38D00DE for ; Tue, 17 Oct 2023 00:47:49 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B6E4B140DDB for ; Tue, 17 Oct 2023 04:47:49 +0000 (UTC) X-FDA: 81353720658.19.C45E12D Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by imf14.hostedemail.com (Postfix) with ESMTP id 7FB9910000F for ; Tue, 17 Oct 2023 04:47:47 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ekBbQV7p; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.52 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697518067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/PhEfIj7SJbCh9k3JCM4ljo+v9VHnZesYcsA6g8vYnM=; b=GvZ5mYuPGDIRg5S1tnNL+v3PQSvIveF547CTdAjKKmkTSc8bdVNaNNcwSxI3FuIHKUR3Tg OrMO0H0HUqhtjI6VAzCD8wMkXMCvFE3SOIhAv5MjolMgSsOK3S7/4KHwX75go1TQB+w4wP vpnDakerRxReH0ksY2kkOtTFQEH6Zb0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ekBbQV7p; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.52 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697518067; a=rsa-sha256; cv=none; b=2JJ5swNht8XKDyJ/kfD3FksqJCTuQdB1Aw+Ao34cMXUUC0+U777iKSwuhavOZVMfrY2nDT tTX6F/rDoUM53gZjlmlFtiCen65kfitT/wJzenlZPsB7eo6hXmlpjtER6SjzfdG0ZsHrAK TkJon9ofSSQeRM0LJS7nxsStPXoPf2o= Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-66d134a019cso34012586d6.3 for ; Mon, 16 Oct 2023 21:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1697518066; x=1698122866; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=/PhEfIj7SJbCh9k3JCM4ljo+v9VHnZesYcsA6g8vYnM=; b=ekBbQV7pxbEg3kQDqqwjnFMLQ1LE1CUKJJpj/S6SneLbZ5Hh0IWZze3wMAPZnahbbD 4Py82sZL4oi+SiFF/Ufjc1EcHwSQYgdjhv6LAP/deYjYS43kYbifgbMHvd15MX4+1aNq jP3ikmnVOAeWqrovx61hOIWDErkcf7wgBKQS46UdQ7jgmoQLGbsu9iLeksiq5aSAQcsb weCWBKkBkAFsGgLWcWO1oQL0qzIaXC61vkYtD8a1SXktEecgR6zAWQK2e8aNuFCx7nLp wUL2Y7J6PpI1jVskjIahpT9Zu/Weq+vcJtcDj+yiYygjsZbr2wwLA156b5qwg7Heb+/9 y4SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697518066; x=1698122866; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/PhEfIj7SJbCh9k3JCM4ljo+v9VHnZesYcsA6g8vYnM=; b=azeQdRMT+TqBJD3ngN3YYR9GIkYPC4kWQR39C0mzXE86paS95RJtGcCGyWEIRIHEX0 3MRnLJwKoJRGqtcapFkm/j8qNCbmrB7pJu12WbMaZ/aTVMxWR1Ze/t21n59aR4ntjWhK 3NAO+YLvVzwlsb4qejzLP734+THS42taB8EI5FLfKXuaGi4VVr/pwHWEmjWGUG3S7PDq 92Qt2tMSJBNK6/GaHN9Xo6h0jKKavVzh+S4rzTm4gRt0ahudmvCPSTWSWZFH05G+hiww ReBMnVb75N7MVUE+15q8UWbBax8Z0xxv0E6/mhzI/z+xBEUoVJtYWqEChkJ02SbEs0c+ 6xQw== X-Gm-Message-State: AOJu0YxNV1w0Erlk0U06nNmXbyUbzQa+sF21Q9p+28GTBGmnWgdqFOEd lbyZf+BeLMl0MC7jybSZtG5Yyg== X-Google-Smtp-Source: AGHT+IHefjwi1GPQUVjs5er2OOSVNscN4hSxv/7E2+1W71PV0h0lh3xE8FyABUAGwi0zU6OEDp1BGA== X-Received: by 2002:ad4:576e:0:b0:66d:3636:8a9 with SMTP id r14-20020ad4576e000000b0066d363608a9mr1498132qvx.63.1697518066456; Mon, 16 Oct 2023 21:47:46 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-3012-16a2-6bc2-2937.res6.spectrum.com. [2603:7000:c01:2716:3012:16a2:6bc2:2937]) by smtp.gmail.com with ESMTPSA id o5-20020ac86985000000b00419ab6ffedasm328432qtq.29.2023.10.16.21.47.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 21:47:46 -0700 (PDT) Date: Tue, 17 Oct 2023 00:47:45 -0400 From: Johannes Weiner To: Yosry Ahmed Cc: Nhat Pham , akpm@linux-foundation.org, cerasuolodomenico@gmail.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, hughd@google.com, corbet@lwn.net, konrad.wilk@oracle.com, senozhatsky@chromium.org, rppt@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, david@ixit.cz Subject: Re: [PATCH 0/2] minimize swapping on zswap store failure Message-ID: <20231017044745.GC1042487@cmpxchg.org> References: <20231017003519.1426574-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7FB9910000F X-Stat-Signature: qab1aimkh7cct7m9q5cdkxeg9iummt7w X-HE-Tag: 1697518067-180938 X-HE-Meta: U2FsdGVkX18m7HWmb8RBJLuiY60rYB9MMhmciyxPv59quuTY4rPZyZspziUPG5AZ1wwPkIAIsQ4N22/0/oStLGdhsBtCYHcY6bcxrKLd22YOknRnNebXThMWkn0ebk2ZIVabd3SHoAnn+uUYTRMjIAHwILxNWIcQp5MOxYbv/aHGpO3c08hr3ZEsq+/ECFDtLSojJR9OYVm6/FU2gDvig0OKJR1GtRSDVAaKwyNBxmxSJN8Ljw1ZyJlA/vwTHgMEH29ki+v6PxXH20LaypeAi1mENcGiXAz44geOv5W3pvdLmZt6wlV2gWZqpPhwrwXDjSATpurUcYmT4Ru0s7uVL5kHXG2peemc4YsbBU3u+P4Bs9CuBzyKT/sYuW95FwaXAOXzJafjBkXUUoTXCAw57JIUnDmoXIHsAvgmHcxY0/9hYNA0ItTVjVnAqQEzxT4yUAgnx3tayAbTSUQl5IyUxLRNHBvWnU63SsawqTz85XuSuD5A7Rp2ZeHRqltGCvMAsTxgm7t1tCSvjcooKCKwfIvDC0zV+vh2KSf55YcdSKTRlh6ZIYeXTX2aBqsmB73DlYkeZeKKqKD14nhNHjlk3/+Bc+tw8N6BkZv5OFUl2GdEU/XrTwOkdCZr/WtCbPWNIn4xpA6Ef5Xpu0v17qpMPko5ydRS9GpsV15mHzhhl4yu84T+TG0kyIsqcoE5AcJjvn8tnrgSHU5qpZg+kRnOwM7yUlw/ai0wHchP1OhcJNWcJ+AMJcnVp7fc5rhVsTCSc8K1aA9P/wTydhnEZRx809A53EmPfIVIjkXgjnn/00tdOMfa5uYzeLPunNaKGOvnKu77JIE7EUfyNFfIQgJ82H/UU6y9Ylx8B4C7X/sOgLYC33mnadtjuxiU8L89GDp3wv83WOAXvV74nqorwO+a/9Vnrie5Cqad5LhC9XRuLrhMR79FGaJkKo8cNeMh4YGHEk6pYmoAMMPiHEVUmn0 siOLc28O Xp2L8D1uIl+DXA2VHwxEc9CLp/if6VvhhWloAW8WbhXct/LTMDIIUNgtSEyMLnPowI8SE2nIYAuPNafz7tDrqsOx8zhRMZ6bEFfpTLH5dxxzpAry70FnqrGv1a+8THJDzAk4SSij75375WjoQxJfPHnT6TsDiA+TQLLa4/Q3GLy7GpCI3qG+hRm5AP6FMq9w8C2DYDoQBAHte4xmDoaXzXYW8MjVhmBdGBEviUwGM9lDAjcVpovtduiq/laxAKAMTI8Ss4PFRS4RuhHswJCOjOde3Q1EAXbdGpk9lRuCPSTX3cKS6QjcBjemhlyzG3Ir0RXT6Q7SQlk/Eyi1iyxSCi+w+dALVhIHclcMbyxUHv8DODkHizFS5t+qG6zoUbpx5Q84nF1DPT96k+bAJg9kaLCp2WJPEqf4YZcFM+JaJJaVU66c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000133, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 16, 2023 at 05:57:31PM -0700, Yosry Ahmed wrote: > On Mon, Oct 16, 2023 at 5:35 PM Nhat Pham wrote: > > > > Currently, when a zswap store attempt fails, the page is immediately > > swapped out. This could happen for a variety of reasons. For instance, > > the compression algorithm could fail (such as when the data is not > > compressible), or the backend allocator might not be able to find a > > suitable slot for the compressed page. If these pages are needed > > later on, users will incur IOs from swapins. > > > > This issue prevents the adoption of zswap for potential users who > > cannot tolerate the latency associated with swapping. In many cases, > > these IOs are avoidable if we just keep in memory the pages that zswap > > fail to store. > > > > This patch series add two new features for zswap that will alleviate > > the risk of swapping: > > > > a) When a store attempt fail, keep the page untouched in memory > > instead of swapping it out. > > What about writeback when the zswap limit is hit? I understand the > problem, but I am wondering if this is the correct way of fixing it. > We really need to make zswap work without a backing swapfile, which I > think is the correct way to fix all these problems. I was working on > that, but unfortunately I had to pivot to something else before I had > something that was working. > > At Google, we have "ghost" swapfiles that we use just to use zswap > without a swapfile. They are sparse files, and we have internal kernel > patches to flag them and never try to actually write to them. > > I am not sure how many bandaids we can afford before doing the right > thing. I understand it's a much larger surgery, perhaps there is a way > to get a short-term fix that is also a step towards the final state we > want to reach instead? I agree it should also stop writeback due to the limit. Note that a knob like this is still useful even when zswap space is decoupled from disk swap slots. We still are using disk swap broadly in the fleet as well, so a static ghost file setup wouldn't be a good solution for us. Swapoff with common swapfile sizes is often not an option during runtime, due to how slow it is, and the destabilizing effect it can have on the system unless that's basically completely idle. As such, we expect to continue deploying swap files for physical use, and switch the zswap-is-terminal knob depending on the workload. The other aspect to this is that workloads that do not want the swapout/swapin overhead for themselves are usually co-located with system management software, and/or can share the host with less latency sensitive workloads, that should continue to use disk swap. Due to the latter case, I wonder if a global knob is actually enough. More likely we'd need per-cgroup control over this. [ It's at this point where the historic coupling of zswap to disk space is especially unfortunate. Because of it, zswap usage counts toward the memory.swap allowance. If these were separate, we could have easily set memory.zswap.max=max, memory.swap.max=0 to achieve the desired effect. Alas, that ship has sailed. Even after a decoupling down the line it would be difficult to change established memory.swap semantics. ] So I obviously agree that we still need to invest in decoupling zswap space from physical disk slots. It's insanely wasteful, especially with larger memory capacities. But while it would be a fantastic optimization, I don't see how it would be an automatic solution to the problem that inspired this proposal. We still need some way to reasonably express desired workload policy in a setup that supports multiple, simultaneous modes of operation. > > b) If the store attempt fails at the compression step, allow the page > > to be stored in its uncompressed form in the zswap pool. This maintains > > the LRU ordering of pages, which will be helpful for accurate > > memory reclaim (zswap writeback in particular). > > This is dangerous. Johannes and I discussed this before. This means > that reclaim can end up allocating more memory instead of freeing. > Allocations made in the reclaim path are made under the assumption > that we will eventually free memory. In this case, we won't. In the > worst case scenario, reclaim can leave the system/memcg in a worse > state than before it started. Yeah, this is a concern. It's not such a big deal if it's only a few pages, and we're still shrinking the footprint on aggregate. But it's conceivable this can happen systematically with some datasets, in which case reclaim will drive up the memory consumption and cause OOMs, or potentially deplete the reserves with PF_MEMALLOC and cause memory deadlocks. This isn't something we can reasonably accept as worst-case behavior. > Perhaps there is a way we can do this without allocating a zswap entry? > > I thought before about having a special list_head that allows us to > use the lower bits of the pointers as markers, similar to the xarray. > The markers can be used to place different objects on the same list. > We can have a list that is a mixture of struct page and struct > zswap_entry. I never pursued this idea, and I am sure someone will > scream at me for suggesting it. Maybe there is a less convoluted way > to keep the LRU ordering intact without allocating memory on the > reclaim path. That should work. Once zswap has exclusive control over the page, it is free to muck with its lru linkage. A lower bit tag on the next or prev pointer should suffice to distinguish between struct page and struct zswap_entry when pulling stuff from the list. We'd also have to teach vmscan.c to hand off the page. It currently expects that it either frees the page back to the allocator, or puts it back on the LRU. We'd need a compromise where it continues to tear down the page and remove the mapping, but then leaves it to zswap. Neither of those sound impossible. But since it's a bigger complication than this proposal, it probably needs a new cost/benefit analysis, with potentially more data on the problem of LRU inversions. Thanks for your insightful feedback, Yosry.