From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7C6BC02198 for ; Wed, 12 Feb 2025 16:02:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C40C6B008A; Wed, 12 Feb 2025 11:02:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3739A6B008C; Wed, 12 Feb 2025 11:02:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23BE56B0092; Wed, 12 Feb 2025 11:02:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 05E526B008A for ; Wed, 12 Feb 2025 11:02:08 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 57AAD16214C for ; Wed, 12 Feb 2025 16:02:07 +0000 (UTC) X-FDA: 83111759094.18.B02D933 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by imf12.hostedemail.com (Postfix) with ESMTP id ADCFA4001C for ; Wed, 12 Feb 2025 16:01:50 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kBFyOMRB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf12.hostedemail.com: domain of yosry.ahmed@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=yosry.ahmed@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739376112; a=rsa-sha256; cv=none; b=vloh9Q86r158Hs4/zWtBu4I4AgIlARYZz8EE5MztDGdeGQND0Y/F4FhOJrtxj53JewIv6v yLQKMW98j5o3Sz8Ls3qVbwfTgrpHEuGb+bdRk4guGRKf0YM+k2jPG6tZDJ50FND2K/9J2V E/9L3wYahPIOkoDELLecIOekijqUeXs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=kBFyOMRB; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf12.hostedemail.com: domain of yosry.ahmed@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=yosry.ahmed@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739376112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v4Nj4mEHCXAl1eKShLrxkBx5V4PjtNl3cv+kh46Bl/g=; b=izAzjOEiNCUj4HG5EHS9Ijho82q31c2AaRgNZgFUTZjPqMe2JFr+kH6TVCHR+6nwLvadxK V6BWjzU56Jlug/qK2XHM7pjSNzQADvdCFfvL5vVoQCJpI9XjDCwwW35HKHxUp9kFeex1uz EL+MZogzsWR9Sw7B/d9pqUI8g7pSBAU= Date: Wed, 12 Feb 2025 16:01:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1739376106; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=v4Nj4mEHCXAl1eKShLrxkBx5V4PjtNl3cv+kh46Bl/g=; b=kBFyOMRBYL+85+B/9XOMAECNpb9WXs453vaiFN50Gwpx/7q1XBlM188t7RXg0zP5QzTEa9 E+rSCavAHMpFGgnjQBFUbfEuzWGOzDpFdjyatMYA8cw62QtqKlLtSMNNZcqZHXSYldWtcV KGGco0GAPlpcANByEU76P7o8ieZ/w5c= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Yosry Ahmed To: Sergey Senozhatsky Cc: Andrew Morton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 02/18] zram: permit preemption with active compression stream Message-ID: References: <20250212063153.179231-1-senozhatsky@chromium.org> <20250212063153.179231-3-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250212063153.179231-3-senozhatsky@chromium.org> X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ADCFA4001C X-Stat-Signature: ohi9gdrut13yuunbjmp6ymqxiotwfkxy X-Rspam-User: X-HE-Tag: 1739376110-5194 X-HE-Meta: U2FsdGVkX18I3taD5oZ+l2/FGPmCYNw+NcNu8DTfVoK4t8AA6eG7O16fKUjsPKbjLXBg6ipMcH5+LEpvdKIRUMkeDzG5TAWJOHjGfjKr/ViLMA0zW/Cdz3BnCmTWPBmuPftzQxV/r0mQw5XJUALFz0VJfhV9UNZpVNhSTHA8Xo5ZOrUasPgM/ATwTSenZzUXqesYIZoISmcJzcxfWk2zrFSayq8vdM2m0FrDl36mSL44D/yWEU3KSAibXFfyIkRfVsrGD+W4RErtrELLVEueJ2AlNKN7oPCFlc5Nf9b9cQra5xVvfkc2mOuyEKG7zdU3jNhxd7F2MBUPW9pmFmYQM74NIrEI4BdptCJXY7isDpiw4qNWbtbnmfTRAbe98j24zGRTYepVXdFGnclOwfem8C9SYv05mnDtLjaKMeCfPkUJ6AhWXPviFx8/Pk3I8YiblsQtzalTCdKpImfktkTESg6F9mx9zfRc95RmSIrFpFOOxR391Zac/E4+4BWVWNNqs+WNTG8MZ3MaQTJpAajKxOb9ch6k63c4xSrDfjCJKRrjQEWVvIqVAQ4XWvrygCNnwv9gPsNOJ855AvSxcazvHnU6V4eBGT+CrOd61qTM4meP7epzOc0s95OJ/yhhuKlOE5uX69Pe4GomM79PcS1E95ocqdpn7lQnrtuT0XGBxV54pLh3QlfvHWHTcR52b0SZd3j78jcI2fRAgxjG0alpOG75zj1/mKE9goVgQKMdyBzfriQxYey0pk7r/QcbcLryy/ntF0qrflGW1M88QKeunHPccJHMNwq+23zbXS1AfbDNrLAkeVf74LP45KmWHWnezmXtmedmsuYMLCm7C5162BsDYU+uCJ7yiESKmzrQWwE7uPF0G5+Pq0+/YKEU+Z//KiB38i/m9ZcXIa0mj3DHcx3o+gzeLpu4W1Em8sbPTMl9ugjTyd+InQ6gCeNrYOWtkbkKPcbTobCfH+L6Ckp DfvO7Zjy 7j/JjRuuYSGNczzzUcDAKyz51eyxEq294oCRokMtxzoNhYTjIHRSl/9oJAfme/i3PlMLuD7gtc1Tqkw1w/hbeR9Q9TBgywFMVXAeGK2FEwNtFrWZQs3WZP6S7Y3h9OmLxFVB4WCpE2cjt1OWOEluO2mMVaYvzgbmoBKSIL0oPnjJIYEuUTINXMdnX9uRIbg6YB19MHnrY059Xt6JWvonQr9GlP9JFOjymnbVp95P4tF1jj9A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Feb 12, 2025 at 03:27:00PM +0900, Sergey Senozhatsky wrote: > Currently, per-CPU stream access is done from a non-preemptible > (atomic) section, which imposes the same atomicity requirements on > compression backends as entry spin-lock, and makes it impossible > to use algorithms that can schedule/wait/sleep during compression > and decompression. > > Switch to preemptible per-CPU model, similar to the one used > in zswap. Instead of a per-CPU local lock, each stream carries > a mutex which is locked throughout entire time zram uses it > for compression or decompression, so that cpu-dead event waits > for zram to stop using a particular per-CPU stream and release > it. > > Suggested-by: Yosry Ahmed > Signed-off-by: Sergey Senozhatsky > --- > drivers/block/zram/zcomp.c | 36 +++++++++++++++++++++++++---------- > drivers/block/zram/zcomp.h | 6 +++--- > drivers/block/zram/zram_drv.c | 20 +++++++++---------- > 3 files changed, 39 insertions(+), 23 deletions(-) > > diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c > index bb514403e305..e83dd9a80a81 100644 > --- a/drivers/block/zram/zcomp.c > +++ b/drivers/block/zram/zcomp.c > @@ -7,6 +7,7 @@ > #include > #include > #include > +#include What code changes prompt this? > #include > #include > > @@ -54,6 +55,7 @@ static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) > { > int ret; > > + mutex_init(&zstrm->lock); I don't think we can initialize the mutex in the hotplug callback. I think the following scenario is possible: CPU #1 CPU #2 zcomp_stream_get() zstrm = raw_cpu_ptr() /* task migrated to CPU 2 */ CPU goes offline zcomp_cpu_dead() mutex_lock() .. mutex_unlock() /* migrated task continues */ zcomp_stream_get() mutex_lock() CPU goes online mutex_init() mutex_unlock() /* problem */ In this case we'll end up initializing the mutex on CPU #1 while CPU #2 has it locked. When we unlocked it on CPU #2 we will corrupt it AFAICT. This is why I moved the mutex initialization out of the hotplug callback in zswap. I suspect to do something similar for zram we'd need to do it in zcomp_init()? > ret = comp->ops->create_ctx(comp->params, &zstrm->ctx); > if (ret) > return ret; > @@ -109,13 +111,29 @@ ssize_t zcomp_available_show(const char *comp, char *buf) > > struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) > { > - local_lock(&comp->stream->lock); > - return this_cpu_ptr(comp->stream); > + for (;;) { > + struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream); > + > + /* > + * Inspired by zswap > + * > + * stream is returned with ->mutex locked which prevents > + * cpu_dead() from releasing this stream under us, however > + * there is still a race window between raw_cpu_ptr() and > + * mutex_lock(), during which we could have been migrated > + * to a CPU that has already destroyed its stream. If so "we could have been migrated from** a CPU that has already destroyed its stream"? Right? > + * then unlock and re-try on the current CPU. > + */ > + mutex_lock(&zstrm->lock); > + if (likely(zstrm->buffer)) > + return zstrm; > + mutex_unlock(&zstrm->lock); > + } > } > > -void zcomp_stream_put(struct zcomp *comp) > +void zcomp_stream_put(struct zcomp_strm *zstrm) > { > - local_unlock(&comp->stream->lock); > + mutex_unlock(&zstrm->lock); > } > > int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, > @@ -151,12 +169,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, > int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) > { > struct zcomp *comp = hlist_entry(node, struct zcomp, node); > - struct zcomp_strm *zstrm; > + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); > int ret; > > - zstrm = per_cpu_ptr(comp->stream, cpu); > - local_lock_init(&zstrm->lock); > - > ret = zcomp_strm_init(comp, zstrm); > if (ret) > pr_err("Can't allocate a compression stream\n"); > @@ -166,10 +181,11 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) > int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) > { > struct zcomp *comp = hlist_entry(node, struct zcomp, node); > - struct zcomp_strm *zstrm; > + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); > > - zstrm = per_cpu_ptr(comp->stream, cpu); > + mutex_lock(&zstrm->lock); > zcomp_strm_free(comp, zstrm); > + mutex_unlock(&zstrm->lock); > return 0; > } > > diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h > index ad5762813842..23b8236b9090 100644 > --- a/drivers/block/zram/zcomp.h > +++ b/drivers/block/zram/zcomp.h > @@ -3,7 +3,7 @@ > #ifndef _ZCOMP_H_ > #define _ZCOMP_H_ > > -#include > +#include > > #define ZCOMP_PARAM_NO_LEVEL INT_MIN > > @@ -31,7 +31,7 @@ struct zcomp_ctx { > }; > > struct zcomp_strm { > - local_lock_t lock; > + struct mutex lock; > /* compression buffer */ > void *buffer; > struct zcomp_ctx ctx; > @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); > void zcomp_destroy(struct zcomp *comp); > > struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); > -void zcomp_stream_put(struct zcomp *comp); > +void zcomp_stream_put(struct zcomp_strm *zstrm); > > int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, > const void *src, unsigned int *dst_len); > diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c > index 3708436f1d1f..43f460a45e3e 100644 > --- a/drivers/block/zram/zram_drv.c > +++ b/drivers/block/zram/zram_drv.c > @@ -1608,7 +1608,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) > ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); > kunmap_local(dst); > zs_unmap_object(zram->mem_pool, handle); > - zcomp_stream_put(zram->comps[prio]); > + zcomp_stream_put(zstrm); > > return ret; > } > @@ -1769,14 +1769,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) > kunmap_local(mem); > > if (unlikely(ret)) { > - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); > + zcomp_stream_put(zstrm); > pr_err("Compression failed! err=%d\n", ret); > zs_free(zram->mem_pool, handle); > return ret; > } > > if (comp_len >= huge_class_size) { > - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); > + zcomp_stream_put(zstrm); > return write_incompressible_page(zram, page, index); > } > > @@ -1800,7 +1800,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) > __GFP_HIGHMEM | > __GFP_MOVABLE); > if (IS_ERR_VALUE(handle)) { > - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); > + zcomp_stream_put(zstrm); > atomic64_inc(&zram->stats.writestall); > handle = zs_malloc(zram->mem_pool, comp_len, > GFP_NOIO | __GFP_HIGHMEM | > @@ -1812,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) > } > > if (!zram_can_store_page(zram)) { > - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); > + zcomp_stream_put(zstrm); > zs_free(zram->mem_pool, handle); > return -ENOMEM; > } > @@ -1820,7 +1820,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) > dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); > > memcpy(dst, zstrm->buffer, comp_len); > - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); > + zcomp_stream_put(zstrm); > zs_unmap_object(zram->mem_pool, handle); > > zram_slot_lock(zram, index); > @@ -1979,7 +1979,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, > kunmap_local(src); > > if (ret) { > - zcomp_stream_put(zram->comps[prio]); > + zcomp_stream_put(zstrm); > return ret; > } > > @@ -1989,7 +1989,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, > /* Continue until we make progress */ > if (class_index_new >= class_index_old || > (threshold && comp_len_new >= threshold)) { > - zcomp_stream_put(zram->comps[prio]); > + zcomp_stream_put(zstrm); > continue; > } > > @@ -2047,13 +2047,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, > __GFP_HIGHMEM | > __GFP_MOVABLE); > if (IS_ERR_VALUE(handle_new)) { > - zcomp_stream_put(zram->comps[prio]); > + zcomp_stream_put(zstrm); > return PTR_ERR((void *)handle_new); > } > > dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); > memcpy(dst, zstrm->buffer, comp_len_new); > - zcomp_stream_put(zram->comps[prio]); > + zcomp_stream_put(zstrm); > > zs_unmap_object(zram->mem_pool, handle_new); > > -- > 2.48.1.502.g6dc24dfdaf-goog >