From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9214C0218D for ; Fri, 31 Jan 2025 09:07:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 495BB2800E9; Fri, 31 Jan 2025 04:07:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41D972800E2; Fri, 31 Jan 2025 04:07:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 248AB2800E9; Fri, 31 Jan 2025 04:07:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F0A182800E2 for ; Fri, 31 Jan 2025 04:07:15 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AC11E8098C for ; Fri, 31 Jan 2025 09:07:15 +0000 (UTC) X-FDA: 83067168030.18.EBE0401 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf12.hostedemail.com (Postfix) with ESMTP id C24B040003 for ; Fri, 31 Jan 2025 09:07:13 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=QnLHBJPp; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=cAkPdSPYV9/GNkha/vCd3P4wq3cPjT2DtM7cpgFz91XHC4U5U79PwpxXhzk03/pAgk+2yv ybsFK19+BHz0nPcnMBVkmkTldCdnfkUWHOVRfBW4XJryJqEeOp+YCLQ/V2F+kWTKGOoVqW 2b1ROfcsL6TxziUAow3SSQokHy4IXUE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=QnLHBJPp; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314433; a=rsa-sha256; cv=none; b=R7xRk2gr/QFm+R5YBUmK/Y1vqz3brnjt2XeawlWqCgOQ9iAyXj77979AYJ0wvLOHWrpM2i lMR5ez78fesBRMXppr8j0XfGzr60+gcX+NiStnA3OyoX7YSVzYxDiXPfkP4njMN433csTy rm/rUdV3YujFYwMKCQoN+7tcEmi4qDU= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21670dce0a7so36068235ad.1 for ; Fri, 31 Jan 2025 01:07:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314432; x=1738919232; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=QnLHBJPpmjZPUJD5pAnoSnABbLQyT37cwiJt3ypZ20d60u3+sbESjbfK4Un2rqand2 pZ561008bdmo5jaO6x1NMBows1yQbjmAognZYYN1rltTW/eUSSVjCW9ShtM2EAwH2OKV 9uYbcYZeQp2ZWfEg73uushqjnfh1nxebjVrZk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314432; x=1738919232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=Ljg2u8eZO/xm22MPAhvnkSz1ve++8oZyqiCf9mCUF4lDtJU8sMwRO7isEHYCDtRV4x 0smxxx7PsQkNZvZV/+Zdgq/MoBGQgV2GpSYX8Gq6x6pUSKkOFYwoWTkLX4nUqV72Kd0e g1v38hpn7Ql+uSDR83Qb+6ZXaqY1c5WNQvXohHW8mVzXatOcn5hXTHNptrc/85bgDNxz 1h8qp1yDoWZQonKjxmYCP1p+pBEzKFJAbqey4f3VVuVOcFxu6hD9iRuFvIRiX6qM/dnE FgOksoXVW5IyiVDIt6giX6DR5+VTcJkoJzgJ3JEhe+zzDoQR3y7Aqk/odiqQEmQyPvcg 1hag== X-Forwarded-Encrypted: i=1; AJvYcCWkCCcBST2OveVPn/dJkPg5jFXy+3cn3VyYyRcDFme4wOTTj/7aE5S2HVPI8KT2sCcqMmnJa3BKWA==@kvack.org X-Gm-Message-State: AOJu0YwvVBpfPH6Sf6vxqmaU6nsCH5kGoXH3Rvs3eTwu9leBnvDukISJ bb+aIsiTAMOp/EMHO0Yuk1rVGwPSU8tyClu/kMYzj4Bc1h15fT+ezrdiwYqACg== X-Gm-Gg: ASbGncsr20b5jAbnzSKg9SWarj7KeICUfCR1jjlDfdWwp2Tv6IToj9e59LPFGC3xKpi e9YzxYczAGS/RiLPoDLRFmd21gm5vF4Mu4m9lnjPif8ObBepvsrUq7VsZiGN/9zpFDCs546xqO4 rbwNMZmBAnrAlxUkQl3L0bcJbmRIpS4UEAHpyMrPz3JKx84pGiHw3ZYFyD+KcosuZKRYTesQPkk Jsu/AZxavttkmOKzwhaQp9EKM0lDXZiPAyo5gY+4QRyslpnFpvNOcZBQUIVuSi8OLuvW/1V7W7t kswBksBbkTM+bQjz+w== X-Google-Smtp-Source: AGHT+IHUYlW/s/BROjpWDLIYuBNea13AbLtp5vL/iIvdkHnTpWdyOu8kZuE13OYSyQwa0btUHKfWmA== X-Received: by 2002:a05:6a21:9102:b0:1e1:faa:d8cf with SMTP id adf61e73a8af0-1ed7a6e1efcmr19676241637.40.1738314432483; Fri, 31 Jan 2025 01:07:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebddbb0besm2609414a12.16.2025.01.31.01.07.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:12 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 02/17] zram: do not use per-CPU compression streams Date: Fri, 31 Jan 2025 18:06:01 +0900 Message-ID: <20250131090658.3386285-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C24B040003 X-Stat-Signature: xz9a4mfkxynmqk7nno3onbuqce16x1m6 X-Rspam-User: X-HE-Tag: 1738314433-280625 X-HE-Meta: U2FsdGVkX18/MBHDYp4d9L+2q0D3F4glBCEUlwixG9Xtqsg+3oU/wNjVDBJ23xAYot6mNvXYWKzT9XrfvIMsZSM/Bzm1g6VEEh4l3gpPYVegdaR1x08rR+wB+hM7GWM57wIpyfGaj0CAD3gV7WX0h4SVn6OMUSQQefFrmpZNmchnZB0lR6HCLRSUXIgOKnG8geM1O1j9yG8BnNuOhaQa4DjW/GNM165qOupp0/D8CDoduqOYhtm/AXh5AeAuoYl3cUUFHvlP4MJRk/VcFJK3MjOa+h5XVjCn3FcAQL9Xi1vNqjHf5WpiGpzZ/DoZPKpN7KsIz0RZnddRxZ0RXD74OLgq/AhiZK/d6uHHPBigUxJwStcyCFW0BdH8qy1kUJAPOrr8LTGaqpBW1I4zN0rWhc5lOp4WxPVQaMLEliFwsXnkWQg/9RdxA9YPWQ+BYYXjW7HtNdFaxxLIpMIc1IAf6/TZGk5UAcZYlf15uPsGZsCsMY7P1oUO5RPioKMWRCQ17SVJjLeESZ+wtXQRR3+4hqJBxfl3Q1pg6IRKNrOqRzQVHtsAilWZq1GkliGKabeUm+hNzqkjVs+7VBmCI1CH6VPpZqd0Xrk2Nwm/g/Npygr1fKPW6zChkSjGLjt2dv5ZXv0yC+cn6K/uB2c6oMsnN01bE1jT4rvX6rtFPxJBuzX0mr/CPBmJfhMI95kJOWRHot9ehZVe4Yrd70kpC1Y+OpQ577iqlZsEYHTt3td7dcNDW5CiVXJOjdk0Ootc6fRLlWsjsF3TBqaBBEOlJ4Wcsh6IqbRk85Rk0i6mhv0sZC3tDIdzmoryK06ksKnVXA/b1HV6RAeoMNzYre+ARjulibUCpMEX+LbD4TwFjJLFmx4O88mMt9zOUkIExsMOK6bX+TxoLm457GgjEkmCikeAaqo+DKdDS9mO3YPAyRLq9VfLO8Iv/ueMS1O4iLaOimrYJtE64Z37J89WfPj5OoL /V9iw4Fa RkgbCsWKwsEzCmYF+H394uSh95cIv9/iICEfu5cl0NYhHibby6e5ZP++YRijkBdmUWTEfqBhS0D4JnAcn+lwtGE3AArnHpwFQl+iRuju2/fFU4Oi1rIV1B9iDne9nRwq6lESnDzHTTkWhCriAU2DW3l9D75vkRZaL4r8Wl7o6GoJSOimHPNLiQD1i6Gou95VJ6ilK4JM6rpUhqhBQU+njJsWszrZPSoUntyGqq+/LAYPabXyERL6Y8XehN6Ms9tw5evA5JvcidYSAfkeSRc7eKryZ1pp+oqHyoLbT0t6axd4vpZtZ+FvJ5wm9Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to per-entry spin-lock per-CPU compression streams also have a number of shortcoming. First, per-CPU stream access has to be done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock do and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Second, per-CPU streams noticeably increase memory usage (actually more like wastage) of secondary compression streams. The problem is that secondary compression streams are allocated per-CPU, just like the primary streams are. Yet we never use more that one secondary stream at a time, because recompression is a single threaded action. Which means that remaining num_online_cpu() - 1 streams are allocated for nothing, and this is per-priority list (we can have several secondary compression algorithms). Depending on the algorithm this may lead to a significant memory wastage, in addition each stream also carries a workmem buffer (2 physical pages). Instead of per-CPU streams, maintain a list of idle compression streams and allocate new streams on-demand (something that we used to do many years ago). So that zram read() and write() become non-atomic and ease requirements on the compression algorithm implementation. This also means that we now should have only one secondary stream per-priority list. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 164 +++++++++++++++++++--------------- drivers/block/zram/zcomp.h | 17 ++-- drivers/block/zram/zram_drv.c | 29 +++--- include/linux/cpuhotplug.h | 1 - 4 files changed, 109 insertions(+), 102 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..982c769d5831 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include @@ -43,31 +43,40 @@ static const struct zcomp_ops *backends[] = { NULL }; -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { - comp->ops->destroy_ctx(&zstrm->ctx); - vfree(zstrm->buffer); - zstrm->buffer = NULL; + comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->buffer); + kfree(strm); } -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) { + struct zcomp_strm *strm; int ret; - ret = comp->ops->create_ctx(comp->params, &zstrm->ctx); - if (ret) - return ret; + strm = kzalloc(sizeof(*strm), GFP_KERNEL); + if (!strm) + return NULL; + + INIT_LIST_HEAD(&strm->entry); + + ret = comp->ops->create_ctx(comp->params, &strm->ctx); + if (ret) { + kfree(strm); + return NULL; + } /* - * allocate 2 pages. 1 for compressed data, plus 1 extra for the - * case when compressed size is larger than the original one + * allocate 2 pages. 1 for compressed data, plus 1 extra in case if + * compressed data is larger than the original one. */ - zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { - zcomp_strm_free(comp, zstrm); - return -ENOMEM; + strm->buffer = vzalloc(2 * PAGE_SIZE); + if (!strm->buffer) { + zcomp_strm_free(comp, strm); + return NULL; } - return 0; + return strm; } static const struct zcomp_ops *lookup_backend_ops(const char *comp) @@ -109,13 +118,59 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + struct zcomp_strm *strm; + + might_sleep(); + + while (1) { + spin_lock(&comp->strm_lock); + if (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + spin_unlock(&comp->strm_lock); + return strm; + } + + /* cannot allocate new stream, wait for an idle one */ + if (comp->avail_strm >= num_online_cpus()) { + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, + !list_empty(&comp->idle_strm)); + continue; + } + + /* allocate new stream */ + comp->avail_strm++; + spin_unlock(&comp->strm_lock); + + strm = zcomp_strm_alloc(comp); + if (strm) + break; + + spin_lock(&comp->strm_lock); + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, !list_empty(&comp->idle_strm)); + } + + return strm; } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm) { - local_unlock(&comp->stream->lock); + spin_lock(&comp->strm_lock); + if (comp->avail_strm <= num_online_cpus()) { + list_add(&strm->entry, &comp->idle_strm); + spin_unlock(&comp->strm_lock); + wake_up(&comp->strm_wait); + return; + } + + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + zcomp_strm_free(comp, strm); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -148,61 +203,19 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - int ret; - - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - - ret = zcomp_strm_init(comp, zstrm); - if (ret) - pr_err("Can't allocate a compression stream\n"); - return ret; -} - -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - - zstrm = per_cpu_ptr(comp->stream, cpu); - zcomp_strm_free(comp, zstrm); - return 0; -} - -static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) -{ - int ret; - - comp->stream = alloc_percpu(struct zcomp_strm); - if (!comp->stream) - return -ENOMEM; - - comp->params = params; - ret = comp->ops->setup_params(comp->params); - if (ret) - goto cleanup; - - ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); - if (ret < 0) - goto cleanup; - - return 0; - -cleanup: - comp->ops->release_params(comp->params); - free_percpu(comp->stream); - return ret; -} - void zcomp_destroy(struct zcomp *comp) { - cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + struct zcomp_strm *strm; + + while (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + zcomp_strm_free(comp, strm); + } + comp->ops->release_params(comp->params); - free_percpu(comp->stream); kfree(comp); } @@ -229,7 +242,12 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params) return ERR_PTR(-EINVAL); } - error = zcomp_init(comp, params); + INIT_LIST_HEAD(&comp->idle_strm); + init_waitqueue_head(&comp->strm_wait); + spin_lock_init(&comp->strm_lock); + + comp->params = params; + error = comp->ops->setup_params(comp->params); if (error) { kfree(comp); return ERR_PTR(error); diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..62330829db3f 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,10 +3,10 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include - #define ZCOMP_PARAM_NO_LEVEL INT_MIN +#include + /* * Immutable driver (backend) parameters. The driver may attach private * data to it (e.g. driver representation of the dictionary, etc.). @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct list_head entry; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -60,16 +60,15 @@ struct zcomp_ops { const char *name; }; -/* dynamic per-device compression frontend */ struct zcomp { - struct zcomp_strm __percpu *stream; + struct list_head idle_strm; + spinlock_t strm_lock; + u32 avail_strm; + wait_queue_head_t strm_wait; const struct zcomp_ops *ops; struct zcomp_params *params; - struct hlist_node node; }; -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); ssize_t zcomp_available_show(const char *comp, char *buf); bool zcomp_available_algorithm(const char *comp); @@ -77,7 +76,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1c2df2341704..8d5974ea8ff8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include @@ -1601,7 +1600,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1762,14 +1761,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); return write_incompressible_page(zram, page, index); } @@ -1793,7 +1792,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1805,7 +1804,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1813,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); @@ -1972,7 +1971,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1982,7 +1981,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); continue; } @@ -2040,13 +2039,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); zs_unmap_object(zram->mem_pool, handle_new); @@ -2794,7 +2793,6 @@ static void destroy_devices(void) zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); } static int __init zram_init(void) @@ -2804,15 +2802,9 @@ static int __init zram_init(void) BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); - ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", - zcomp_cpu_up_prepare, zcomp_cpu_dead); - if (ret < 0) - return ret; - ret = class_register(&zram_control_class); if (ret) { pr_err("Unable to register zram-control class\n"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return ret; } @@ -2821,7 +2813,6 @@ static int __init zram_init(void) if (zram_major <= 0) { pr_err("Unable to get major number\n"); class_unregister(&zram_control_class); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return -EBUSY; } diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6cc5e484547c..092ace7db8ee 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -119,7 +119,6 @@ enum cpuhp_state { CPUHP_MM_ZS_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, - CPUHP_ZCOMP_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_TMIGR_PREPARE, CPUHP_MIPS_SOC_PREPARE, -- 2.48.1.362.g079036d154-goog