From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED5CFC27C75 for ; Fri, 14 Jun 2024 09:40:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20DC26B00B3; Fri, 14 Jun 2024 05:29:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BD026B00B6; Fri, 14 Jun 2024 05:29:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05DAB6B00BA; Fri, 14 Jun 2024 05:29:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D86026B00B3 for ; Fri, 14 Jun 2024 05:29:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A03861A05D0 for ; Fri, 14 Jun 2024 09:29:29 +0000 (UTC) X-FDA: 82228971258.07.BA595C7 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf14.hostedemail.com (Postfix) with ESMTP id C2CD910000A for ; Fri, 14 Jun 2024 09:29:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Af/lEQwW"; spf=pass (imf14.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718357366; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2EFgdU2m8MZMkfNPC/OdfW+RXLLf0iLElY/mcH8eE44=; b=eq/XNkTrZqrkPlxZyOL3zSzt45AxHwmBsMroo93xGmZFpvLnWAuUyHa/0h/0DpOzPX7kTF UOGvSmvU26DIAuAjDwMQusMirgrwVQFQeVVSevEF6kjr0sdRLahK9WyQm0OpboTtnLURpI yaRLbsm+/XR5prPxXLkmLt58BPo/Yro= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Af/lEQwW"; spf=pass (imf14.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718357366; a=rsa-sha256; cv=none; b=22g9ki2kKPs8VBO3WVdTMj3wyvWdDQJqTEhRG/x9txVrVXNuJx8cf74qHz7bv3hkAamAhX FL+bOvI1PMXDgv2dRzyE3BNeaLatqsOYNcLayinHe5Vf9XqUihViof39BBJXKP4hxCfH6b AL3CEEPR38YjEPsvIjhMmn2wAzrQYFE= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-57c61165af6so2314532a12.2 for ; Fri, 14 Jun 2024 02:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718357366; x=1718962166; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2EFgdU2m8MZMkfNPC/OdfW+RXLLf0iLElY/mcH8eE44=; b=Af/lEQwWgb9+OFnKUJKkiFG4lU30ywaDb+Vwwr8uoN4SsDbjxe3EmY11RVKHjBZ5/O 9wUt8eNiETFZwsdfOQlz0guVTh4/DvhlXAv9WN8tBduT99sbXavYI1BNQ7EEfolYyw/3 mykcBwh+VreVYMvHF8PPjNsjNr8+JBZU6ep4jZ41gaHzX1cT+fUPAG/DDSyvGAUjbhsy szkL7p67/HMuSWGz7nOwO6yN3ayWRFx3rr186fLbtM8V8086qzNHyO1tkn5/F8BTnuU1 d6VOoGFKuRtPwIuB5xhfy97LapERTQ/q948xEWFVVvy9gAd5G7T4pmEZizjgyvcle1iX u8XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718357366; x=1718962166; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2EFgdU2m8MZMkfNPC/OdfW+RXLLf0iLElY/mcH8eE44=; b=n2zclLVd7t1IOO7eOM04wD7+H/TPhNciF7uc4uaaD7/zRn+dkf5fQYgGvEAI+zMvjS /Si4XYapwefvtuNnF+uJAyzFnyHcBIpqvYeg8Svb9ohNV9ltIAesQ8aNVmmD9cHFl+ae pIUXBTjR0HHUtp/1wLr8as4fSwWLpiJIG8BEXPrV57gqBo3l5lGSrKHfaljTRz2hsdmD eJvRY4KTyvrVUkRTNJhT57hW9pGWWMIMjTzu7sQmFXwvWBeLZ3CBOtnJdW+zgdMBbZ9C BLLTlpW0m0ZV02GgcTyTvP3EVNW6jPjZdgoY1OxsKyUpw5yuCrTyCqFiYsF4yNFc5dBt 5vmA== X-Forwarded-Encrypted: i=1; AJvYcCV5nQlx5xAOFTlk8M/tNKbJypzUrivTKlVo3pcNvITrAjO4YdbXzV4wzOkvqBPbb/iq7VHdKV4wOw84uyXpqM/ofG0= X-Gm-Message-State: AOJu0YwrEkoYikYJcXTojTTr4ljO9cJI95nXrloAZCYWGXYoU0rabkyF QRvfbGLvEjEYTD/iA1gyHT0BPgzUAdKvVy+zzVaefN20MzUFtKkbKM61DFlGqheB7eZsENY4a3z XebqJK/jBRO2CaM1DsOlVvnWr0V+mgfrimrK9 X-Google-Smtp-Source: AGHT+IE/Dsm5sUO90WDhySqhQle8Wcf1bswme3pUqrR0JWcYyB5h/4IPxDUk/BYo+n4AtSKF3b8oVx+F2L267FDc7+s= X-Received: by 2002:a17:906:b814:b0:a6f:5609:9552 with SMTP id a640c23a62f3a-a6f60cee4ebmr125971166b.10.1718357365840; Fri, 14 Jun 2024 02:29:25 -0700 (PDT) MIME-Version: 1.0 References: <20240610121820.328876-1-usamaarif642@gmail.com> <52df55b7-2e1d-4a95-85bf-19f6680e3fec@gmail.com> In-Reply-To: <52df55b7-2e1d-4a95-85bf-19f6680e3fec@gmail.com> From: Yosry Ahmed Date: Fri, 14 Jun 2024 02:28:47 -0700 Message-ID: Subject: Re: [PATCH v3 0/2] mm: store zero pages to be swapped out in a bitmap To: Usama Arif Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, david@redhat.com, ying.huang@intel.com, hughd@google.com, willy@infradead.org, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: yeyax3c58fgmq1ueurbth7i3g5b5w5y3 X-Rspam-User: X-Rspamd-Queue-Id: C2CD910000A X-Rspamd-Server: rspam02 X-HE-Tag: 1718357367-525286 X-HE-Meta: U2FsdGVkX1//oTXDB7n+hz87YKsl+OzA8QovsnfHRDtXJov7Bu5TD727V3uUNefU0a9HHDxx3gu52YG7FRxAUSj9UY8k499VAqgNumDicgHTUcZcH07i6BedWqg/KsMrGlK4e0GkEKUbh9ODjtaYXgTqAy5mee5NAba+SkdlCuyHdhHkmFFOPUdytpsiF4gUrvI8oKYWt8vtkLPcuBJFIrGqeLR11fivDcWWY2KIriM/gTIc01IKXYptcx2nTYAdxLCzr+uoGpQjB0mdW+eMG9vVGMsbCGEx4sCau/UajIqW3KQaiqMhh9uQlX3/4BoVeDG25fe23PnyGMFY8svFxsXU5W3WnaCO6gAuNa5h7WYOF0WCyAMKXkfrU3Op/nY1oCz7pB8U1eEhH59QhQ8UiYYbDECnteLGQrEWDVj0ftT28TYe/OGoFrQIi2S8iLGzvfefqk4OlKlkpaCSrCSvn8DVQOY3XY0p0rKOi++mnSs6T5OumqBzx5kNxcG1Ii6hbOnsvPJENOuWvSpDAZc5Et6CM0Dd4Qzpvw2JP7AEQkhhqwNeDNpp57GnbIuB+xchp3ks4kw+FauFb1kbGXz19YuonSRKCm85CpQa5Ti3vjhdiGUj/IpegBTeiDZ1AUIQs2JH08XdMocclqevOWoqKzToJ1SJzkbVm9b/R3sqWN8qaGp03bLFnKVTAh3kUWMQ2l11gRQW7WstcPWW8oCXAXbHgpFko7IP/7kaGVpBP8caVQ899jTAMqo8Bk6gcmfPzxI2BRUa/kscU0X565BDwnP/C2XS4u/tuKGTKdDL4XQ8kr/aU7cQx+VaIgO1wuxQeKZTvYTsyHDp83LISYD5RRenMDpzdsoRlQTJMAhTqUA91ibQ4bCY3qPmKPmk/ljWTEbVwwmqZIRXW7WotSLAJG0SPh4vYgPxx9dPNL+yecN/GRhvLRRsK4hVdZlFZsRYfwc23WGkpUBrJBjes3H OyNhMS83 SvZthB36rVV6kvLYlFNb0c29vMBJUh6h7O4n/KRQwnunnwtLzjQTC9KBqN5uBnH0h7SzFmyBKzhv25V20Xz8XxL4XS9zzK1nXzxkw1S2qdc9REwT0YYkan44R43xpVgrta6JgLdiDRUEeRaL+XJAWgRFRDF9e8VyYL4F+5S6cdLMbEkRimEWg/sJIlHdtAqE45Zkv X-Bogosity: Ham, tests=bogofilter, spamicity=0.001622, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jun 14, 2024 at 2:22=E2=80=AFAM Usama Arif = wrote: > > > On 13/06/2024 22:21, Yosry Ahmed wrote: > > On Mon, Jun 10, 2024 at 5:18=E2=80=AFAM Usama Arif wrote: > >> Going back to the v1 implementation of the patchseries. The main reaso= n > >> is that a correct version of v2 implementation requires another rmap > >> walk in shrink_folio_list to change the ptes from swap entry to zero p= ages to > >> work (i.e. more CPU used) [1], is more complex to implement compared t= o v1 > >> and is harder to verify correctness compared to v1, where everything i= s > >> handled by swap. > >> > >> --- > >> As shown in the patchseries that introduced the zswap same-filled > >> optimization [2], 10-20% of the pages stored in zswap are same-filled. > >> This is also observed across Meta's server fleet. > >> By using VM counters in swap_writepage (not included in this > >> patchseries) it was found that less than 1% of the same-filled > >> pages to be swapped out are non-zero pages. > >> > >> For conventional swap setup (without zswap), rather than reading/writi= ng > >> these pages to flash resulting in increased I/O and flash wear, a bitm= ap > >> can be used to mark these pages as zero at write time, and the pages c= an > >> be filled at read time if the bit corresponding to the page is set. > >> > >> When using zswap with swap, this also means that a zswap_entry does no= t > >> need to be allocated for zero filled pages resulting in memory savings > >> which would offset the memory used for the bitmap. > >> > >> A similar attempt was made earlier in [3] where zswap would only track > >> zero-filled pages instead of same-filled. > >> This patchseries adds zero-filled pages optimization to swap > >> (hence it can be used even if zswap is disabled) and removes the > >> same-filled code from zswap (as only 1% of the same-filled pages are > >> non-zero), simplifying code. > >> > >> This patchseries is based on mm-unstable. > > Aside from saving swap/zswap space and simplifying the zswap code > > (thanks for that!), did you observe any performance benefits from not > > having to go into zswap code for zero-filled pages? > > > > In [3], I observed ~1.5% improvement in kernbench just by optimizing > > zswap's handling of zero-filled pages, and that benchmark only > > produced around 1.5% zero-filled pages. I imagine avoiding the zswap > > code entirely, and for workloads that have 10-20% zero-filled pages, > > the performance improvement should be more pronounced. > > > > When zswap is not being used and all swap activity translates to IO, I > > imagine the benefits will be much more significant. > > > > I am curious if you have any numbers with or without zswap :) > > Apart from tracking zero-filled pages (using inaccurate counters not in > this series) which had the same pattern to zswap_same_filled_pages, the > nvme writes went down around 5-10% during stable points in the > production experiment. The performance improved by 2-3% at some points, > but this is comparing 2 sets of machines running production workloads > (which can vary between machine sets), so I would take those numbers > cautiously and which is why I didnt include them in the cover letter. > Yeah this makes sense, thanks. It would have been great if we had comparable numbers with and without this series. But this shouldn't be a big deal, the advantage of the series should be self-explanatory. It's just a shame you don't get to brag about it :)