From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AA10CFA45C for ; Wed, 23 Oct 2024 18:16:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D38796B0092; Wed, 23 Oct 2024 14:16:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE92D6B0096; Wed, 23 Oct 2024 14:16:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B889B6B0098; Wed, 23 Oct 2024 14:16:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9B8E56B0092 for ; Wed, 23 Oct 2024 14:16:30 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3C66C160826 for ; Wed, 23 Oct 2024 18:16:10 +0000 (UTC) X-FDA: 82705671804.04.BDA8225 Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) by imf21.hostedemail.com (Postfix) with ESMTP id DBEAE1C001A for ; Wed, 23 Oct 2024 18:15:54 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NRjMpVuU; spf=pass (imf21.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729707237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SwCuA4yImROJvhJywn+YK8+uWaKia67TwRqA6S/vp1A=; b=WdnRrwCHiBtpAU/EWkdZRSwehzXgl3I+g7pZqFj/pI7mXVf4RwTsXBG/DupdhEOVXQuz81 vXbzGlV2p4lswTxpN2XqUoGucLbwAskJDqA/Dv3HcdbWezjYESnS1Y41qguXBrfq/BU6aG ReAczflRN/kc2yFjESLD4vUjALAdDgY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729707237; a=rsa-sha256; cv=none; b=vSmQWikiQsNju6W+HODflxHgCF/a8gu6XPWSe1RqVihmT4tfCGxHGlffsqIu07sI1fpXkR MkhLZGY61QvAD9tDDJrqEPKBlnLe8HqIMF2mpBxfRo7/M58yMoYPBekUL67Uwr9+Wpoh7D 3pZVrctJT6eSk1wmvRfhrUvwHcI9PlE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=NRjMpVuU; spf=pass (imf21.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-5c948c41edeso62986a12.1 for ; Wed, 23 Oct 2024 11:16:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729707387; x=1730312187; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=SwCuA4yImROJvhJywn+YK8+uWaKia67TwRqA6S/vp1A=; b=NRjMpVuUrwu7XqBpn95dItFNuWRJEpBVh4TWMWCJYTrLZFlvttr9R15tvz9+yX4W96 0A+50XcW2wQgrdJQ7iHDyewZm+jQkVTPhdU3vXk5bugy+eAMtwgzmHDx4laq189xlKeR xUb0u1MQANHRed5pom9loXga7DRGpVyAihqmDO8DlZYf8Fs0CNcKKHeSlRQarq2/503S kZtY9ijN/Puel6VlRXcXU/WzUJ5/uJQ1yjHTmEYvr0RKbCc4uBmiglhZQ4wSr/dpiMvI 0I8kM2PxU8zOpLEaGX6elEEXMh5bQCa8xtPJT5o5BfxM/HC5WCcdmInhVNS0XnJJCLq/ 51UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729707387; x=1730312187; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SwCuA4yImROJvhJywn+YK8+uWaKia67TwRqA6S/vp1A=; b=vevb1zRG9OhjaE8v+ipTYjCMg0F5UmALep5pJzFzZugt2y31ne9USV50KQbIRnhoQI o/lntd6c37pTyd+r8o0SaYdZH9bqIPDBNt0F5R4Q2+9OklRpMuZoqx3k4o+VpdeC+hDy SEkSkhUDX9yxLMbzTKcvKO/Heg5pCpj5Lcsir58gS/X8FhoKwkrYIKHHuLxslVo6AL/n dJ8qIchasemmGkNLzX8f1724CnyeXeqiaetLzIfnjn5v2uMlsP2CAcmt6ccXGtT95VZ7 WuThHRwciKQ09qZUVQ9MI48HvIr+hdwQptJgESiRCgh7ryDV7vo8kWARY0xFottwZk6O sMTw== X-Forwarded-Encrypted: i=1; AJvYcCWwuFPLPLSCCWku2W0Y/AIm8CZIQLeJk2jZIGb53wOaUP0P3qPo8ELCE67IMZUz+cF9P5+kccvCmA==@kvack.org X-Gm-Message-State: AOJu0Yy7ZgKIl7XfaOPTPIZYcjHO/ofFsPknVCDaMzr7ePX6WaXM9OVY SkEl9ByhkXci8n6tkRkIXh9zoGUeAVy7d+HxPTCFfc31zDH1T6HQM/aVT/wExtqmN3PfROiG26m L6sGBKKJOBDgqvdn/Xv+6RHsVlCiAeCWjVAk2 X-Google-Smtp-Source: AGHT+IEt31RESgiqF7AyjCPe7bxP29/f2XcyC3yNA9DrNXUcssSqGbRlVaHXeYozRvGDtobwboAYgwdDefa4Xch1ZNQ= X-Received: by 2002:a17:907:3e0a:b0:a9a:6ab:c93b with SMTP id a640c23a62f3a-a9abf9b5984mr302142766b.62.1729707386560; Wed, 23 Oct 2024 11:16:26 -0700 (PDT) MIME-Version: 1.0 References: <20241018064101.336232-1-kanchana.p.sridhar@intel.com> In-Reply-To: From: Yosry Ahmed Date: Wed, 23 Oct 2024 11:15:50 -0700 Message-ID: Subject: Re: [RFC PATCH v1 00/13] zswap IAA compress batching To: "Sridhar, Kanchana P" Cc: "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , "hannes@cmpxchg.org" , "nphamcs@gmail.com" , "chengming.zhou@linux.dev" , "usamaarif642@gmail.com" , "ryan.roberts@arm.com" , "Huang, Ying" , "21cnbao@gmail.com" <21cnbao@gmail.com>, "akpm@linux-foundation.org" , "linux-crypto@vger.kernel.org" , "herbert@gondor.apana.org.au" , "davem@davemloft.net" , "clabbe@baylibre.com" , "ardb@kernel.org" , "ebiggers@google.com" , "surenb@google.com" , "Accardi, Kristen C" , "zanussi@kernel.org" , "viro@zeniv.linux.org.uk" , "brauner@kernel.org" , "jack@suse.cz" , "mcgrof@kernel.org" , "kees@kernel.org" , "joel.granados@kernel.org" , "bfoster@redhat.com" , "willy@infradead.org" , "linux-fsdevel@vger.kernel.org" , "Feghali, Wajdi K" , "Gopal, Vinodh" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: DBEAE1C001A X-Stat-Signature: 7po7iqza4hz9onypgmii3e6fa49ew5qf X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1729707354-241233 X-HE-Meta: U2FsdGVkX1/eVXaC9L4XrT1oerWiPMMv2Kb71PkZmERBdKPEaJJcVBnXiUjWsZlr+KgJGWIWtNBUqtUYGk0PpW+tVkngEKKn4lZGav91fDbwwUslVQtTlUYWn1hPDG4sDWnWBhe8LnlUmwuzHxoQO2S0j4io7WkxrHRAiirzlB+6irk2sS1eYYh7ZkemxraT4vtsLlCFU4XHDavf0IfAN61s/lj6dmpUxVecF0Qg+/TqRpV1CFlr5jNrSfgrPvgVu7XI2QbqygLBWaTabSxSM2nlbJZt+5VWMwm4H8ppPie6/Hydm37VOhcGq1Pu+DjUZijh+3UoVsmLhGrtmXQfqUaOb9rUmqgdmpo0ddHtgw7qDJ2YXIYK99jcLZHlC+azMuo1BDDuZb1oUY9Abckr/ci5Ke2Mjb163nScxWwBa5ddZ2mTdUeYMW4ikYgaWe1HyNAWU3b8TLfIVKcMIBtEoyL0D+ftjKAKm2kjYtjG1gz+GR1Yprqnpog+pRBRZDykDmvsGgSKRe2tghHqJxMnOXZd9kBXo6cVppPPu2GgQOOhknhzBxtL1x0pJcTgviicViQiTnDsH/OFksuEFghz7JN+klTyCC37phhG52R9/1sg/bqTZKIkGfoG6xZ/G1CJYxr8PpYFx//MZQNu6195ZSZLl45pe6mILOk0hiLploimBkSe8miFYeOb18NrVpY4YgQXqNl2/mm9R33LLgtfHOI+n2+7DdiodAMVQ71/C6jET6q1Ja08o4kQokMiq3fdlpBz6hbrHW+otY+6Jgj2AukUy9ichFwn16N0cRyFMZLNiEfA5Ffz8Ux4px/VDQirIV1goCat4A59oCp1S00yfZVJdKKP+/ZizRu+DfbuCbbummvENC8IhfbOI6397Ie54DIlh8UixaThG8UDo1xLVf3MMBJfPDYOckpyNuqmWI1TXVlFkzaBIOliObLfT2kIVWZAYd+XuSyqgaCUE29 Xca7iKRN hjyTyOSwnijbaEih/0GeR++lMi8xMGu1UAw7sThjTk0IZldOox9KdLNce/a4H38MWxBy9G09dSFkSRshxcbUX7+77KqZrVlfNEki5eL1vBWBk52Jf+LuhM3259KCeSuon4FGxjF7+J/fU1tPHKmdFz/nJ1S8wCrK47zdZgXRRsyilPQVsKD2Pq753EyqEQG9w5zSuD91O1omQJTBjOxFbAZlCRt+JOa5J8f1F2BXlYby+zvKvNOi4/k/10Xn//TO6Srid7Nj2oIBzDCdTqs/+mSlu6HiO72eE8oiyj/N7oP4EtyBxLMQw5KUvcyRoJCm1coM2zcD2c9hH2q5Sp2sdI14dkg7quPClM8WoN0crALQLUXgw8RmGSYTdka6RYH2/ykegxUDc3DhFly7jGQ401/MbSBPyyBOpbDkunfpaLt7RCxUJfST4cVw0hVyR2EdWq04ZFToqg7hAyfC0qX1NLiWDmFz1P0FtgNGNoqfeL/x/ieUlovLoE+K+xphAcpbi3riFN8Yb9CG6PFn7UFvU8cxBwApRS7hikH7c3YvUJ03a9Uxh4Xd9QjybZLpgT9ihqQP+Q8lTKWuS6sqCpq4UGUAgXP/d+BokTWUrMCwBRaAIR2ac8P5vPhUXcXGY/6OaoJ5pGkC3/jBVwWbz02cCuXu1WApujUnYWOXQwZV7psDfxG9Ayqob/OpM3wiJ1SoTNPLLx9hdOkzF02kWJit/vASIrr8pkd9cN9Q9EBRuj4nbDaE0bjP9LL8OcwtX5KETlYppnw/DP+7i7kwqi70ZjhfxTsHQqYcFjU1M7tQGrQ3fVHeGD6oJuZ5u9yZ96LG0o/d67ZTpBkvVEi04hh5ey7dKCXlxgJdMES+fSEyW8Q1mNnbcUHRUpR8OmeFkFT2bdw3f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Oct 22, 2024 at 7:53=E2=80=AFPM Sridhar, Kanchana P wrote: > > Hi Yosry, > > > -----Original Message----- > > From: Yosry Ahmed > > Sent: Tuesday, October 22, 2024 5:57 PM > > To: Sridhar, Kanchana P > > Cc: linux-kernel@vger.kernel.org; linux-mm@kvack.org; > > hannes@cmpxchg.org; nphamcs@gmail.com; chengming.zhou@linux.dev; > > usamaarif642@gmail.com; ryan.roberts@arm.com; Huang, Ying > > ; 21cnbao@gmail.com; akpm@linux-foundation.org; > > linux-crypto@vger.kernel.org; herbert@gondor.apana.org.au; > > davem@davemloft.net; clabbe@baylibre.com; ardb@kernel.org; > > ebiggers@google.com; surenb@google.com; Accardi, Kristen C > > ; zanussi@kernel.org; viro@zeniv.linux.org= .uk; > > brauner@kernel.org; jack@suse.cz; mcgrof@kernel.org; kees@kernel.org; > > joel.granados@kernel.org; bfoster@redhat.com; willy@infradead.org; linu= x- > > fsdevel@vger.kernel.org; Feghali, Wajdi K ; = Gopal, > > Vinodh > > Subject: Re: [RFC PATCH v1 00/13] zswap IAA compress batching > > > > On Thu, Oct 17, 2024 at 11:41=E2=80=AFPM Kanchana P Sridhar > > wrote: > > > > > > > > > IAA Compression Batching: > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > > > > > This RFC patch-series introduces the use of the Intel Analytics Accel= erator > > > (IAA) for parallel compression of pages in a folio, and for batched r= eclaim > > > of hybrid any-order batches of folios in shrink_folio_list(). > > > > > > The patch-series is organized as follows: > > > > > > 1) iaa_crypto driver enablers for batching: Relevant patches are tag= ged > > > with "crypto:" in the subject: > > > > > > a) async poll crypto_acomp interface without interrupts. > > > b) crypto testmgr acomp poll support. > > > c) Modifying the default sync_mode to "async" and disabling > > > verify_compress by default, to facilitate users to run IAA eas= ily for > > > comparison with software compressors. > > > d) Changing the cpu-to-iaa mappings to more evenly balance cores = to IAA > > > devices. > > > e) Addition of a "global_wq" per IAA, which can be used as a glob= al > > > resource for the socket. If the user configures 2WQs per IAA d= evice, > > > the driver will distribute compress jobs from all cores on the > > > socket to the "global_wqs" of all the IAA devices on that sock= et, in > > > a round-robin manner. This can be used to improve compression > > > throughput for workloads that see a lot of swapout activity. > > > > > > 2) Migrating zswap to use async poll in zswap_compress()/decompress(= ). > > > 3) A centralized batch compression API that can be used by swap modu= les. > > > 4) IAA compress batching within large folio zswap stores. > > > 5) IAA compress batching of any-order hybrid folios in > > > shrink_folio_list(). The newly added "sysctl vm.compress-batchsiz= e" > > > parameter can be used to configure the number of folios in [1, 32= ] to > > > be reclaimed using compress batching. > > > > I am still digesting this series but I have some high level questions > > that I left on some patches. My intuition though is that we should > > drop (5) from the initial proposal as it's most controversial. > > Batching reclaim of unrelated folios through zswap *might* make sense, > > but it needs a broader conversation and it needs justification on its > > own merit, without the rest of the series. > > Thanks for these suggestions! Sure, I can drop (5) from the initial patc= h-set. > Agree also, this needs a broader discussion. > > I believe the 4K folios usemem30 data in this patchset does bring across > the batching reclaim benefits to provide justification on its own merit. = I added > the data on batching reclaim with kernel compilation as part of the 4K fo= lios > experiments in the IAA decompression batching patch-series [1]. > Listing it here as well. I will make sure to add this data in subsequent = revs. > > -------------------------------------------------------------------------= - > Kernel compilation in tmpfs/allmodconfig, 2G max memory: > > No large folios mm-unstable-10-16-2024 shrink_folio_list(= ) > batching of folios > ------------------------------------------------------------------------= -- > zswap compressor zstd deflate-iaa deflate-iaa > vm.compress-batchsize n/a n/a 32 > vm.page-cluster 3 3 3 > ------------------------------------------------------------------------= -- > real_sec 783.87 761.69 747.32 > user_sec 15,750.07 15,716.69 15,728.39 > sys_sec 6,522.32 5,725.28 5,399.44 > Max_RSS_KB 1,872,640 1,870,848 1,874,432 > > zswpout 82,364,991 97,739,600 102,780,612 > zswpin 21,303,393 27,684,166 29,016,252 > pswpout 13 222 213 > pswpin 12 209 202 > pgmajfault 17,114,339 22,421,211 23,378,161 > swap_ra 4,596,035 5,840,082 6,231,646 > swap_ra_hit 2,903,249 3,682,444 3,940,420 > ------------------------------------------------------------------------= -- > > The performance improvements seen does depend on compression batching in > the swap modules (zswap). The implementation in patch 12 in the compress > batching series sets up this zswap compression pipeline, that takes an ar= ray of > folios and processes them in batches of 8 pages compressed in parallel in= hardware. > That being said, we do see latency improvements even with reclaim batchin= g > combined with zswap compress batching with zstd/lzo-rle/etc. I haven't do= ne a > lot of analysis of this, but I am guessing fewer calls from the swap laye= r > (swap_writepage()) into zswap could have something to do with this. If we= believe > that batching can be the right thing to do even for the software compress= ors, > I can gather batching data with zstd for v2. Thanks for sharing the data. What I meant is, I think we should focus on supporting large folio compression batching for this series, and only present figures for this support to avoid confusion. Once this lands, we can discuss support for batching the compression of different unrelated folios separately, as it spans areas beyond just zswap and will need broader discussion.