From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12CD6CEB2CD for ; Mon, 30 Sep 2024 23:09:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6246428003A; Mon, 30 Sep 2024 19:09:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AD3A280036; Mon, 30 Sep 2024 19:09:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44E3828003A; Mon, 30 Sep 2024 19:09:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1E01C280036 for ; Mon, 30 Sep 2024 19:09:54 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CB567C0BC1 for ; Mon, 30 Sep 2024 23:09:53 +0000 (UTC) X-FDA: 82622949066.04.58FDC06 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf27.hostedemail.com (Postfix) with ESMTP id D456340019 for ; Mon, 30 Sep 2024 23:09:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eUi3eNVs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727737689; a=rsa-sha256; cv=none; b=N2bEUqHW6m2xzwj+MGs4mRts1p8Dhh76M0D6AsbJyWAnLFA0h1Rb2U0Ye3C8fpfdoDapr2 8Q6vGn63+ET8m2hqDDUB7NkwoZ63zmCVAcFKCveklN9y1ZZbabM+7iQZ21H1TEQ6oL0tFW TTngS1GjZ1j/p6LADReZYkl+Lu6lYgI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eUi3eNVs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727737689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5RNDaXl/U/fth0NO73obA3tDz7SjnrhgWka0eqUJ5Ks=; b=kVYtcTvG9AB1TuVRyvaCArN3EU5NE0WsC3nD3NTlANdSakVivqF9IlGi5q+dxZRvmchlOJ Q1vJOmzA3uJpVxVMfSFcY48sQ/5mkbzOatiNNUvNZcdFCa7fMm7Xye4Umofi1KA4LqXWxn PMBxfMBNPieXYMqz2yA/p5GRBYFG/5k= Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a8d0d82e76aso796397766b.3 for ; Mon, 30 Sep 2024 16:09:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1727737790; x=1728342590; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=5RNDaXl/U/fth0NO73obA3tDz7SjnrhgWka0eqUJ5Ks=; b=eUi3eNVsBeOJNHLeLvfZVMJCewqqpzJh0cAJeAQ5O5LG6tJMjTQMvdPqBLSVoyk6ku ULbHWWm3eKx1lVVWmFJpM1qkB7nQIuqeKOaJQT7Z05DYXiMK8pbrG8Hs+e0dsXTkGO5t M+3vLgMTIlaa3rucc0AMDZiWkYEgQKyb1pbL+1Y7iwnrnTLx+S7qOFP2nXJA6Xd8epsX PdET6d21COKJ7muhDycRB/AA+EQ25DWAa4L7rxNZTS73HhuzfXKmdU5xNrlLlxfpWWkI z1dMX++vq0Sfx++Wygxk5X5CYyVU0Iw5cGiLXYDnpugab+uSOsY6vgYQ8jG1Fu/SETAn Ys7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727737790; x=1728342590; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5RNDaXl/U/fth0NO73obA3tDz7SjnrhgWka0eqUJ5Ks=; b=sqjq+NmtitkVH52qs4i82I7t8eBfjXucktG2FTPXI7GWUjJrzyMDvNKEMSjL+yLwuO H+3WIQVlhyc+F1CpZro9o80h5Ak9HreSO88h3bMBjaf77gN+Fgwm579s4cq7IXxczP6y t5fCGW6IC423uirwj00h/hsv8AMlfqEwX/P6uRuvAPw1kHMtbD8RYVXmRbKbeMxxOEVI JGA/WtBZP/N5WZzBSIPRli2o45QOlXP9QbSIWeaJA0ZGVOQbQ3otKVpTWqsd3rHUUUlh /A7a9MXpQcSQ92WH3TJPzTThWQJL9mSmH0tPeOr20ak4EwHFhd+h92CzmXzMm5zqZ+J7 tJRA== X-Forwarded-Encrypted: i=1; AJvYcCVv+fLdFNp9E5pTJwiDHcjBQv36+y9x8H7Mz/7E3Ow8h4+O5pzXzK8AWfM6XGqBIPoozvZMBY5/Tw==@kvack.org X-Gm-Message-State: AOJu0Yzaf7MzeM3TVJNFW1DWAATwZN6roe/q8/sc0nAWvXOYbllXE5bD o+gI1+tj4OeNbJ6wyxb73rfuAgZyHwbSqfsylsgqy83z9v+0dc/819KQQABHOOCYQ/ns82/sdzn ugu7xvbv0zwwY++81SxqhdzzoX3/g5rv/+nk5 X-Google-Smtp-Source: AGHT+IFmGMs5P/NFTNZMELBvIdBZGXL8578rLT7kSMFHWQufBjUr5NlJVgq9zDezW+iL/NmC6FPSEhJRaIGle6WaSk8= X-Received: by 2002:a17:907:1ca2:b0:a90:3498:93b2 with SMTP id a640c23a62f3a-a93c48e7facmr1717083166b.1.1727737789867; Mon, 30 Sep 2024 16:09:49 -0700 (PDT) MIME-Version: 1.0 References: <20240930221221.6981-1-kanchana.p.sridhar@intel.com> <20240930221221.6981-7-kanchana.p.sridhar@intel.com> In-Reply-To: <20240930221221.6981-7-kanchana.p.sridhar@intel.com> From: Yosry Ahmed Date: Mon, 30 Sep 2024 16:09:12 -0700 Message-ID: Subject: Re: [PATCH v9 6/7] mm: zswap: Support large folios in zswap_store(). To: Kanchana P Sridhar Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org, nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D456340019 X-Stat-Signature: d7q8t6ufans34ymhf86ew6bb5pc9b68s X-Rspam-User: X-HE-Tag: 1727737791-477490 X-HE-Meta: U2FsdGVkX1/WsiD8pTgP2rXzbKVvb9feiU2FDbDNLZdSuCYJDaH3Du1CNFFmJPsOxIMzceKLXX4isRqSXLTwCPV2tSxTPG+RJ+IhKyAw6VmSmBCUc4jzbi4FGPwfMzGCe4A57P4OVlb5ZFgzAmpZKC1C71NUtud1kqL6er5vG9Q+DlUgktH5zoAcAkngE8UUG+pgU99BpSFEYickmhoWpGTP8QJCl0+YhP0KiZXGyTdl6lzVo/ClJzLLY+FG4q6qhy0F5KVL1KHW0xcB78dwxzrfS/hAN4H26dnkQ8Wf/f1SYcQczINJSVv0mRzSBb1rHaZvtX6wuV78nWsm42fUzxKsdd7z5BESNG8Z/SzW5wGtfPBud0spEBPm6R9snAlZ849pbb+k6/Fw/ajH/XTokJLxkTrKGFB8d82wxJ5KfChhK/ZOGOIDHYc4cG1Ds9weaVA0uwpAAtkf4eWyy+pS+Xss8qdByfpibn3xpiIa2w+yHSo0HNo8HX83Qk1Q7oYHHgXEzbXSHca7Lr0xr7nalLtBw2SGKL2V++tdtpuZ65/EI/q3q6anUzFVnjjVUBvmdc3KIJxsxH4M2Xtic6SUWTdHBsaTXddwcjxUsMuKrj+HFQkSc0UXillCfPZM1rxErVKUSalmXgeS3+oEJwxOhWaxZ+6IzMPS0KxEXwdybPQzCCLcXpJhyYoo9KHdOUCzZCWBk0pb42tcCmkWMbPMcfOspcLB210Kj7Qlrn8Oa5R6QCcTZ2l4uor56YvXOyXGatocqLKz++yEKzO0h5RQ9N4uF3bsF0saaQY9HTZT2vvittjY1xrzwi0KhrVqBYxNWMijBQ77tQBzyHmbvwTWlvy2RmyurnDd8sQ/61DC4h0vHq0Hwtlwh03MnQOMvDQvC2hZ5BaEMf2mvHswbBAbepEyHJU9ufxJPAi6peVEV+DIBcpvOfge5maYj59kaJGgmQE5ESohR3LJ1VJQHgp RjOEw0bt cobr5BfO6tTCZEtsf8mZzvHggfn57II4Asysz1JbKoR9gPyEj4ODZ3p8AKXfggqBNQ8k7tZDgfkj8jNfyT1Z4FeZROE4+79IJCcgaqAECylHcIvZE/tPgpL/SzDhHfnjKE2dwyEOSWdSLNquHtyXSROUNLXxwDwDhssn2CULV1gl3mDetaVUH64y+2QBXF54w7lVBHh3Zs2bPCu9U7g99X/zmmwoTwD/HEUsbYIn/np60XheOb4UwiMkeaQRJMJ0Jm3+Q8JrpDvloUvnNNREpicvcB5IMpJ0l6mVeNU7oRBSdsDDG70yX4H13uOmziVM2SvHiT3RRfMxtjOg1VtXw3nQxBbfCeokge5lk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Sep 30, 2024 at 3:12=E2=80=AFPM Kanchana P Sridhar wrote: > > zswap_store() will store large folios by compressing them page by page. > > This patch provides a sequential implementation of storing a large folio > in zswap_store() by iterating through each page in the folio to compress > and store it in the zswap zpool. > > zswap_store() calls the newly added zswap_store_page() function for each > page in the folio. zswap_store_page() handles compressing and storing eac= h > page. > > We check the global and per-cgroup limits once at the beginning of > zswap_store(), and only check that the limit is not reached yet. This is > racy and inaccurate, but it should be sufficient for now. We also obtain > initial references to the relevant objcg and pool to guarantee that > subsequent references can be acquired by zswap_store_page(). A new functi= on > zswap_pool_get() is added to facilitate this. > > If these one-time checks pass, we compress the pages of the folio, while > maintaining a running count of compressed bytes for all the folio's pages= . > If all pages are successfully compressed and stored, we do the cgroup > zswap charging with the total compressed bytes, and batch update the > zswap_stored_pages atomic/zswpout event stats with folio_nr_pages() once, > before returning from zswap_store(). > > If an error is encountered during the store of any page in the folio, > all pages in that folio currently stored in zswap will be invalidated. > Thus, a folio is either entirely stored in zswap, or entirely not stored > in zswap. > > The most important value provided by this patch is it enables swapping ou= t > large folios to zswap without splitting them. Furthermore, it batches som= e > operations while doing so (cgroup charging, stats updates). > > This patch also forms the basis for building compress batching of pages i= n > a large folio in zswap_store() by compressing up to say, 8 pages of the > folio in parallel in hardware using the Intel In-Memory Analytics > Accelerator (Intel IAA). > > This change reuses and adapts the functionality in Ryan Roberts' RFC > patch [1]: > > "[RFC,v1] mm: zswap: Store large folios without splitting" > > [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.robe= rts@arm.com/T/#u > > Also, addressed some of the RFC comments from the discussion in [1]. > > Co-developed-by: Ryan Roberts > Signed-off-by: We haven't been able to get a signoff from Ryan. Andrew, what's the policy = here? > Signed-off-by: Kanchana P Sridhar > --- > mm/zswap.c | 220 +++++++++++++++++++++++++++++++++++++---------------- > 1 file changed, 153 insertions(+), 67 deletions(-) > > diff --git a/mm/zswap.c b/mm/zswap.c > index 2b8da50f6322..b74c8de99646 100644 > --- a/mm/zswap.c > +++ b/mm/zswap.c > @@ -411,6 +411,12 @@ static int __must_check zswap_pool_tryget(struct zsw= ap_pool *pool) > return percpu_ref_tryget(&pool->ref); > } > > +/* The caller must already have a reference. */ > +static void zswap_pool_get(struct zswap_pool *pool) > +{ > + percpu_ref_get(&pool->ref); > +} > + > static void zswap_pool_put(struct zswap_pool *pool) > { > percpu_ref_put(&pool->ref); > @@ -1402,68 +1408,52 @@ static void shrink_worker(struct work_struct *w) > /********************************* > * main API > **********************************/ > -bool zswap_store(struct folio *folio) > + > +/* > + * Stores the page at specified "index" in a folio. > + * > + * @page: The page to store in zswap. > + * @objcg: The folio's objcg. Caller has a reference. > + * @pool: The zswap_pool to store the compressed data for the page. > + * The caller should have obtained a reference to a valid > + * zswap_pool by calling zswap_pool_tryget(), to pass as this > + * argument. > + * @tree: The xarray for the @page's folio's swap. > + * @compressed_bytes: The compressed entry->length value is added > + * to this, so that the caller can get the total > + * compressed lengths of all sub-pages in a folio. > + */ > +static bool zswap_store_page(struct page *page, > + struct obj_cgroup *objcg, > + struct zswap_pool *pool, > + struct xarray *tree, > + size_t *compressed_bytes) > { > - swp_entry_t swp =3D folio->swap; > - pgoff_t offset =3D swp_offset(swp); > - struct xarray *tree =3D swap_zswap_tree(swp); > struct zswap_entry *entry, *old; > - struct obj_cgroup *objcg =3D NULL; > - struct mem_cgroup *memcg =3D NULL; > - > - VM_WARN_ON_ONCE(!folio_test_locked(folio)); > - VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); > - > - /* Large folios aren't supported */ > - if (folio_test_large(folio)) > - return false; > - > - if (!zswap_enabled) > - goto check_old; > - > - /* Check cgroup limits */ > - objcg =3D get_obj_cgroup_from_folio(folio); > - if (objcg && !obj_cgroup_may_zswap(objcg)) { > - memcg =3D get_mem_cgroup_from_objcg(objcg); > - if (shrink_memcg(memcg)) { > - mem_cgroup_put(memcg); > - goto reject; > - } > - mem_cgroup_put(memcg); > - } > - > - if (zswap_check_limits()) > - goto reject; > > /* allocate entry */ > - entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); > + entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(page_foli= o(page))); Can we just use page_to_nid() here? I think the node info exists even for tail pages, right? > if (!entry) { > zswap_reject_kmemcache_fail++; > goto reject; > } > > - /* if entry is successfully added, it keeps the reference */ > - entry->pool =3D zswap_pool_current_get(); > - if (!entry->pool) > - goto freepage; > + /* zswap_store() already holds a ref on 'objcg' and 'pool' */ > + if (objcg) > + obj_cgroup_get(objcg); > + zswap_pool_get(pool); > > - if (objcg) { > - memcg =3D get_mem_cgroup_from_objcg(objcg); > - if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERN= EL)) { > - mem_cgroup_put(memcg); > - goto put_pool; > - } > - mem_cgroup_put(memcg); > - } > + /* if entry is successfully added, it keeps the reference */ > + entry->pool =3D pool; > > - if (!zswap_compress(&folio->page, entry)) > - goto put_pool; > + if (!zswap_compress(page, entry)) > + goto put_pool_objcg; > > - entry->swpentry =3D swp; > + entry->swpentry =3D page_swap_entry(page); > entry->objcg =3D objcg; > entry->referenced =3D true; > > - old =3D xa_store(tree, offset, entry, GFP_KERNEL); > + old =3D xa_store(tree, swp_offset(entry->swpentry), entry, GFP_KE= RNEL); > if (xa_is_err(old)) { > int err =3D xa_err(old); > > @@ -1480,11 +1470,6 @@ bool zswap_store(struct folio *folio) > if (old) > zswap_entry_free(old); > > - if (objcg) { > - obj_cgroup_charge_zswap(objcg, entry->length); > - count_objcg_events(objcg, ZSWPOUT, 1); > - } > - > /* > * We finish initializing the entry while it's already in xarray. > * This is safe because: > @@ -1496,36 +1481,137 @@ bool zswap_store(struct folio *folio) > * an incoherent entry. > */ > if (entry->length) { > + *compressed_bytes +=3D entry->length; > INIT_LIST_HEAD(&entry->lru); > zswap_lru_add(&zswap_list_lru, entry); > } > > - /* update stats */ > - atomic_long_inc(&zswap_stored_pages); > - count_vm_event(ZSWPOUT); > - > + /* > + * We shouldn't have any possibility of failure after the entry i= s > + * added in the xarray. The pool/objcg refs obtained here will on= ly > + * be dropped if/when zswap_entry_free() gets called. > + */ > return true; > > store_failed: > zpool_free(entry->pool->zpool, entry->handle); > -put_pool: > - zswap_pool_put(entry->pool); > -freepage: > +put_pool_objcg: > + zswap_pool_put(pool); > + obj_cgroup_put(objcg); I think if we reorder the function we can drop these calls, make the comments positioned a bit better, and centralize the entry initializations. I am also not a fan of passing a semi-initialized entry to zswap_compress() to get the pool pointer. Does the following diff improve things or did I miss something? diff --git a/mm/zswap.c b/mm/zswap.c index b74c8de996468..eac1f846886a6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -881,7 +881,8 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } -static bool zswap_compress(struct page *page, struct zswap_entry *entry) +static bool zswap_compress(struct page *page, struct zswap_entry *entry, + struct zswap_pool *pool) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -893,7 +894,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry) gfp_t gfp; u8 *dst; - acomp_ctx =3D raw_cpu_ptr(entry->pool->acomp_ctx); + acomp_ctx =3D raw_cpu_ptr(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); @@ -926,7 +927,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry) if (comp_ret) goto unlock; - zpool =3D entry->pool->zpool; + zpool =3D pool->zpool; gfp =3D __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; if (zpool_malloc_support_movable(zpool)) gfp |=3D __GFP_HIGHMEM | __GFP_MOVABLE; @@ -1435,23 +1436,11 @@ static bool zswap_store_page(struct page *page, entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(page_folio(page))); if (!entry) { zswap_reject_kmemcache_fail++; - goto reject; + return false; } - /* zswap_store() already holds a ref on 'objcg' and 'pool' */ - if (objcg) - obj_cgroup_get(objcg); - zswap_pool_get(pool); - - /* if entry is successfully added, it keeps the reference */ - entry->pool =3D pool; - - if (!zswap_compress(page, entry)) - goto put_pool_objcg; - - entry->swpentry =3D page_swap_entry(page); - entry->objcg =3D objcg; - entry->referenced =3D true; + if (!zswap_compress(page, entry, pool)) + goto compress_failed; old =3D xa_store(tree, swp_offset(entry->swpentry), entry, GFP_KERN= EL); if (xa_is_err(old)) { @@ -1470,6 +1459,16 @@ static bool zswap_store_page(struct page *page, if (old) zswap_entry_free(old); + /* + * The entry is successfully compressed and stored in the tree, the= re is + * no further possibility of failure. Grab refs to the pool and obj= cg. + * These refs will be dropped by zswap_entry_free() when the entry = is + * removed from the tree. + */ + zswap_pool_get(pool); + if (objcg) + obj_cgroup_get(objcg); + /* * We finish initializing the entry while it's already in xarray. * This is safe because: @@ -1480,26 +1479,22 @@ static bool zswap_store_page(struct page *page, * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ + entry->pool =3D pool; + entry->swpentry =3D page_swap_entry(page); + entry->objcg =3D objcg; + entry->referenced =3D true; if (entry->length) { *compressed_bytes +=3D entry->length; INIT_LIST_HEAD(&entry->lru); zswap_lru_add(&zswap_list_lru, entry); } - /* - * We shouldn't have any possibility of failure after the entry is - * added in the xarray. The pool/objcg refs obtained here will only - * be dropped if/when zswap_entry_free() gets called. - */ return true; store_failed: - zpool_free(entry->pool->zpool, entry->handle); -put_pool_objcg: - zswap_pool_put(pool); - obj_cgroup_put(objcg); + zpool_free(pool->zpool, entry->handle); +compress_failed: zswap_entry_cache_free(entry); -reject: return false; } > zswap_entry_cache_free(entry); > reject: > + return false; > +} > + > +bool zswap_store(struct folio *folio) > +{ > + long nr_pages =3D folio_nr_pages(folio); > + swp_entry_t swp =3D folio->swap; > + struct xarray *tree =3D swap_zswap_tree(swp); > + struct obj_cgroup *objcg =3D NULL; > + struct mem_cgroup *memcg =3D NULL; > + struct zswap_pool *pool; > + size_t compressed_bytes =3D 0; > + bool ret =3D false; > + long index; > + > + VM_WARN_ON_ONCE(!folio_test_locked(folio)); > + VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); > + > + if (!zswap_enabled) > + goto check_old; > + > + /* > + * Check cgroup zswap limits: > + * > + * The cgroup zswap limit check is done once at the beginning of > + * zswap_store(). The cgroup charging is done once, at the end > + * of a successful folio store. What this means is, if the cgroup > + * was within the zswap_max limit at the beginning of a large fol= io > + * store, it could go over the limit by at most (HPAGE_PMD_NR - 1= ) > + * pages due to this store. > + */ > + objcg =3D get_obj_cgroup_from_folio(folio); > + if (objcg && !obj_cgroup_may_zswap(objcg)) { > + memcg =3D get_mem_cgroup_from_objcg(objcg); > + if (shrink_memcg(memcg)) { > + mem_cgroup_put(memcg); > + goto put_objcg; > + } > + mem_cgroup_put(memcg); > + } > + > + /* > + * Check zpool utilization against zswap limits: > + * > + * The zswap zpool utilization is also checked against the limits > + * just once, at the start of zswap_store(). If the check passes, > + * any breaches of the limits set by zswap_max_pages() or > + * zswap_accept_thr_pages() that may happen while storing this > + * folio, will only be detected during the next call to > + * zswap_store() by any process. > + */ This is essentially a rephrased repetition of the last comment, just refer to it instead. Something like: /* * Check zpool utilization against zswap limits. The possibility = of * going overlimit is the same as the cgroup limit check. */ > + if (zswap_check_limits()) > + goto put_objcg; > + > + pool =3D zswap_pool_current_get(); > + if (!pool) > + goto put_objcg; > + > + if (objcg) { > + memcg =3D get_mem_cgroup_from_objcg(objcg); > + if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERN= EL)) { > + mem_cgroup_put(memcg); > + goto put_pool; > + } > + mem_cgroup_put(memcg); > + } > + > + /* > + * Store each page of the folio as a separate entry. If we fail t= o > + * store a page, unwind by deleting all the pages for this folio > + * currently in zswap. > + */ > + for (index =3D 0; index < nr_pages; ++index) { > + if (!zswap_store_page(folio_page(folio, index), objcg, po= ol, tree, &compressed_bytes)) > + goto put_pool; > + } > + > + if (objcg) { > + obj_cgroup_charge_zswap(objcg, compressed_bytes); > + count_objcg_events(objcg, ZSWPOUT, nr_pages); > + } > + > + atomic_long_add(nr_pages, &zswap_stored_pages); > + count_vm_events(ZSWPOUT, nr_pages); > + > + ret =3D true; > + > +put_pool: > + zswap_pool_put(pool); > +put_objcg: > obj_cgroup_put(objcg); > - if (zswap_pool_reached_full) > + if (!ret && zswap_pool_reached_full) > queue_work(shrink_wq, &zswap_shrink_work); > check_old: > /* > - * If the zswap store fails or zswap is disabled, we must invalid= ate the > - * possibly stale entry which was previously stored at this offse= t. > - * Otherwise, writeback could overwrite the new data in the swapf= ile. > + * If the zswap store fails or zswap is disabled, we must invalid= ate > + * the possibly stale entries which were previously stored at the > + * offsets corresponding to each page of the folio. Otherwise, > + * writeback could overwrite the new data in the swapfile. > */ > - entry =3D xa_erase(tree, offset); > - if (entry) > - zswap_entry_free(entry); > - return false; > + if (!ret) { > + pgoff_t offset =3D swp_offset(swp); > + struct zswap_entry *entry; > + > + for (index =3D 0; index < nr_pages; ++index) { > + entry =3D xa_erase(tree, offset + index); > + if (entry) > + zswap_entry_free(entry); > + } > + } > + > + return ret; > } > > bool zswap_load(struct folio *folio) > -- > 2.27.0 >