From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1202DC021B8 for ; Tue, 4 Mar 2025 06:10:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77E386B0082; Tue, 4 Mar 2025 01:10:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 706566B0083; Tue, 4 Mar 2025 01:10:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 580896B0085; Tue, 4 Mar 2025 01:10:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2F6856B0082 for ; Tue, 4 Mar 2025 01:10:32 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CB1D512160B for ; Tue, 4 Mar 2025 06:10:31 +0000 (UTC) X-FDA: 83182844262.24.ACD8A96 Received: from abb.hmeau.com (abb.hmeau.com [144.6.53.87]) by imf27.hostedemail.com (Postfix) with ESMTP id E5D4540005 for ; Tue, 4 Mar 2025 06:10:28 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=hmeau.com header.s=formenos header.b=Kfx0CiKN; dmarc=pass (policy=quarantine) header.from=apana.org.au; spf=pass (imf27.hostedemail.com: domain of herbert@gondor.apana.org.au designates 144.6.53.87 as permitted sender) smtp.mailfrom=herbert@gondor.apana.org.au ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741068630; a=rsa-sha256; cv=none; b=5uO94vG1SjlkU/JPOs2EBRR8/MQPuhEPtLX+QuUqKnFtpNGcGL3dGr6XxYF0146f98cCe+ HQGYLVv+vxwADYPJt6wD5y0rhoXrVwjHVu/3gkkYHugojHGa92D+HLdcX1XXUz+SDmumN5 sCDeuBOnjmT41s7mF8bK6OSwHCzMP0Q= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=hmeau.com header.s=formenos header.b=Kfx0CiKN; dmarc=pass (policy=quarantine) header.from=apana.org.au; spf=pass (imf27.hostedemail.com: domain of herbert@gondor.apana.org.au designates 144.6.53.87 as permitted sender) smtp.mailfrom=herbert@gondor.apana.org.au ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741068630; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jYFV4rSSTCccUrPh12MJ7dlU6kdUs0ysVOn6Eyqvajg=; b=fU8QbovI/vTXgqabvCfkQTClD1+TqU6h06X2NSCr8Qo5sZN3sVOWH5vD84NaFAIzEnij5n bbKauwo+SdiOWYyCFgnb1lTWmSQoOoilgnp4deiicW4+Ac+Ob7GRhR6uBTWHrFXHJQscRT aG4aJgKUMIxOZ+Aq13hzQDLbi1YC4Kw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=hmeau.com; s=formenos; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=jYFV4rSSTCccUrPh12MJ7dlU6kdUs0ysVOn6Eyqvajg=; b=Kfx0CiKNCq46lFMMo1CsYVk8Vv +FJbt8AIvRVirJUlXzvQU9WAS0uG2wMnCEkKA/9qJaaz/1izeoM7bwmwAC0mp9tLnDM4vmnchKNz4 HoG6F2G7iubKMuCrQt9GwBVQ69xghJ1lPXZ91l+AIUACmzQSKRwG2jkO+2uzFMPEMBPDwFCkGsUhN fwK/UeTLAqnErdQfRe3RVC3BAI/HG+khjihHKxCozfxxZ4eVoJjioHSodByIdzxFRjalh/kbaR9dj XxNW7OzkWN2uRGqe8IzGLxSROSYHwHnufe0B/mCySJElfVUcV/d44Mb3g+ulokjnQpu0U8qA98LMz rMd7fK/w==; Received: from loth.rohan.me.apana.org.au ([192.168.167.2]) by formenos.hmeau.com with smtp (Exim 4.96 #2 (Debian)) id 1tpLTh-003WcQ-2s; Tue, 04 Mar 2025 14:10:06 +0800 Received: by loth.rohan.me.apana.org.au (sSMTP sendmail emulation); Tue, 04 Mar 2025 14:10:05 +0800 Date: Tue, 4 Mar 2025 14:10:05 +0800 From: Herbert Xu To: Yosry Ahmed Cc: Eric Biggers , Sergey Senozhatsky , Linux Crypto Mailing List , linux-mm@kvack.org Subject: Re: [RFC PATCH 7/7] mm: zswap: Use acomp virtual address interface Message-ID: References: <20250227183847.GB1613@sol.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: E5D4540005 X-Stat-Signature: 3s85ru4epfqkr1udmcku8f4ekzikws6q X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1741068628-874974 X-HE-Meta: U2FsdGVkX19JrSGdN+Fhr6ZQr8dO2A749XjSYxj/nL5+1B5fwOJAE5T0ygLCHuRBJcIaRduAg1xrYlBq/c2hiEQo2wny9A7to8aDlRPHCpfU/KP+qy7nblD17LbMkOwcIcXBusmNq7sOryqVMaGxXyKT+aNrVgYb8mnaDA1k4KtTvHd64BzqWayHBo/kMGTGkPnvhBb21LqteoArCBNcGGAQGc/V+S8N7J6x3+s89G5l2fTpSQ4y9lKA4HzRoAw1o1zPiNF5py0+UTaxgQtzcOBSJteX1vc5VSrXIBiSrRv+KXa8rpXm1fMoUvgImI5y+0VYJksaRwM5PZkXkdzRVgZ29kY1lCy/8sVjW8wyla4IvN4369lwzKcrv6jcurGi7XIsX1X1RlDijvIMHnMly3eKB6gz6uUYerpKFZke3COg4TGgYo8dNP4ypSQuKBXitthudgNOWrxJlPwE8m8KBZRx/z2UuxJzgLqeE+CpC6QBm7KY/V1BFWm61nu0cPwtCHVwtcUWpU3Vhf65Nn03I/lgVjEkxW2y/lfi50OcwHZ0L4FUzucH6OoYxJM4akoWP3vmwUVCbKL2VQ1r1L8+EW0MpHhUDpWFk+xQo1bBLkCEayq/xvAWwfE3XLDJJsZ8pU5GaIX58vzxi9dp3D22uhNnERFpwg0VVSMKRyE1/WnXu0MCWhP1X0hu5lGmBXoKVkb/bYOH81NNpAxPiyqslABDlxkhO+oOTmzPdbbJxskrelhqMsguFSvvqAD/k94Vg6pYkr8OJveHdYvswkrn767nQ+eXgcgNNxyEEw4DNw7jlvWxLvCtU4w2LQPDEnbAz13GU6lYe5GgfV5FdUwuZWnhl5cxPcp2IAzbLQeyyk2Xc+cuJOq0pZIqu6++86R1x8N3B45EASPdnMzBES0SS+SG//JALkQndfD2DleMkWBMcI4Kr6dOO/OzrbnQ2pFmaETGAdP/jWy1OcA8F23 ZJl8+xID Y9DUDyQMVFMoWdsDxRBHAw/rm4L+UUlrFls90cJx+3Jvm0M1sXvGcSKp8ND/0vZwrIqExT4KmaGTHz+Pyy8njbzj5rUPqSTdl1BrrqogC6/L2/f0ZRWo9Hrpcz+wbn0Mv5nK/Tz7s8zjiEK/ocFMVeb3Nocuv0wF1/lJ6o1frC+zRxdoDV1xfq8AbHFAbAdIR6eyVe5zLNrx6rw75l4YgGiefo6oxYGuhRPVy+4oD8yT270nRTa1gebNp80WWBau2X7bjsJgorRcS/Ww= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Mar 04, 2025 at 04:30:01AM +0000, Yosry Ahmed wrote: > > Looking forward to this :) Here is something that is entirely untested. It doesn't depend on my acomp patch-set and should apply on mainline. commit 71955ecc7d6680386bb685f7f7f3951b5da9da9a Author: Herbert Xu Date: Thu Feb 27 18:10:32 2025 +0800 mm: zswap: Give non-linear objects to Crypto API Instead of copying non-linear objects into a buffer, use the scatterlist to give them directly to the Crypto API. Signed-off-by: Herbert Xu diff --git a/include/linux/zpool.h b/include/linux/zpool.h index a67d62b79698..0250a4f0760d 100644 --- a/include/linux/zpool.h +++ b/include/linux/zpool.h @@ -12,6 +12,7 @@ #ifndef _ZPOOL_H_ #define _ZPOOL_H_ +struct scatterlist; struct zpool; /* @@ -53,6 +54,11 @@ void *zpool_map_handle(struct zpool *pool, unsigned long handle, void zpool_unmap_handle(struct zpool *pool, unsigned long handle); +void zpool_map_sg(struct zpool *pool, unsigned long handle, + enum zpool_mapmode mm, struct scatterlist *sg); + +void zpool_unmap_sg(struct zpool *pool, unsigned long handle); + u64 zpool_get_total_pages(struct zpool *pool); @@ -66,7 +72,9 @@ u64 zpool_get_total_pages(struct zpool *pool); * @free: free mem from a pool. * @sleep_mapped: whether zpool driver can sleep during map. * @map: map a handle. + * @map_sg: map a handle into a two-entry SG list * @unmap: unmap a handle. + * @unmap: unmap a handle given to map_sg. * @total_size: get total size of a pool. * * This is created by a zpool implementation and registered @@ -89,7 +97,11 @@ struct zpool_driver { bool sleep_mapped; void *(*map)(void *pool, unsigned long handle, enum zpool_mapmode mm); + void (*map_sg)(void *pool, unsigned long handle, + enum zpool_mapmode mm, + struct scatterlist *sg); void (*unmap)(void *pool, unsigned long handle); + void (*unmap_sg)(void *pool, unsigned long handle); u64 (*total_pages)(void *pool); }; diff --git a/mm/z3fold.c b/mm/z3fold.c index 379d24b4fef9..7be8e3b3ba13 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -1402,6 +1403,15 @@ static void z3fold_zpool_unmap(void *pool, unsigned long handle) z3fold_unmap(pool, handle); } +static void z3fold_zpool_map_sg(void *pool, unsigned long handle, + enum zpool_mapmode mm, + struct scatterlist sg[2]) +{ + void *buf = z3fold_map(pool, handle); + + sg_init_one(sg, buf, PAGE_SIZE - offset_in_page(buf)); +} + static u64 z3fold_zpool_total_pages(void *pool) { return z3fold_get_pool_pages(pool); @@ -1416,6 +1426,7 @@ static struct zpool_driver z3fold_zpool_driver = { .malloc = z3fold_zpool_malloc, .free = z3fold_zpool_free, .map = z3fold_zpool_map, + .map_sg = z3fold_zpool_map_sg, .unmap = z3fold_zpool_unmap, .total_pages = z3fold_zpool_total_pages, }; diff --git a/mm/zbud.c b/mm/zbud.c index e9836fff9438..f6a4da93c985 100644 --- a/mm/zbud.c +++ b/mm/zbud.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -410,6 +411,14 @@ static void zbud_zpool_unmap(void *pool, unsigned long handle) zbud_unmap(pool, handle); } +static void zbud_zpool_map_sg(void *pool, unsigned long handle, + enum zpool_mapmode mm, struct scatterlist sg[2]) +{ + void *buf = (void *)handle; + + sg_init_one(sg, buf, PAGE_SIZE - offset_in_page(buf)); +} + static u64 zbud_zpool_total_pages(void *pool) { return zbud_get_pool_pages(pool); @@ -424,7 +433,9 @@ static struct zpool_driver zbud_zpool_driver = { .malloc = zbud_zpool_malloc, .free = zbud_zpool_free, .map = zbud_zpool_map, + .map_sg = zbud_zpool_map_sg, .unmap = zbud_zpool_unmap, + .unmap_sg = zbud_zpool_unmap, .total_pages = zbud_zpool_total_pages, }; diff --git a/mm/zpool.c b/mm/zpool.c index b9fda1fa857d..120dbca8ca6e 100644 --- a/mm/zpool.c +++ b/mm/zpool.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -305,6 +306,33 @@ void *zpool_map_handle(struct zpool *zpool, unsigned long handle, return zpool->driver->map(zpool->pool, handle, mapmode); } +/** + * zpool_map_handle() - Map a previously allocated handle into an SG list + * @zpool: The zpool that the handle was allocated from + * @handle: The handle to map + * @mapmode: How the memory should be mapped + * @sg: 2-entry SG list to store the mapping + * + * This maps a previously allocated handle into an SG list. The + * @mapmode param indicates to the implementation how the memory + * will be * used, i.e. read-only, write-only, read-write. If the + * implementation does not support it, the memory will be treated + * as read-write. + * + * This may hold locks, disable interrupts, and/or preemption, + * and the zpool_unmap_handle() must be called to undo those + * actions. The code that uses the mapped handle should complete + * its operations on the mapped handle memory quickly and unmap + * as soon as possible. As the implementation may use per-cpu + * data, multiple handles should not be mapped concurrently on + * any cpu. + */ +void zpool_map_sg(struct zpool *zpool, unsigned long handle, + enum zpool_mapmode mapmode, struct scatterlist *sg) +{ + zpool->driver->map_sg(zpool->pool, handle, mapmode, sg); +} + /** * zpool_unmap_handle() - Unmap a previously mapped handle * @zpool: The zpool that the handle was allocated from @@ -320,6 +348,21 @@ void zpool_unmap_handle(struct zpool *zpool, unsigned long handle) zpool->driver->unmap(zpool->pool, handle); } +/** + * zpool_unmap_sg() - Unmap a previously SG-mapped handle + * @zpool: The zpool that the handle was allocated from + * @handle: The handle to unmap + * + * This unmaps a previously mapped handle. Any locks or other + * actions that the implementation took in zpool_map_handle() + * will be undone here. The memory area returned from + * zpool_map_handle() should no longer be used after this. + */ +void zpool_unmap_sg(struct zpool *zpool, unsigned long handle) +{ + zpool->driver->unmap_sg(zpool->pool, handle); +} + /** * zpool_get_total_pages() - The total size of the pool * @zpool: The zpool to check diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..122294dd4105 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -49,6 +49,7 @@ #include #include #include +#include #include "zpdesc.h" #define ZSPAGE_MAGIC 0x58 @@ -306,6 +307,10 @@ static void init_deferred_free(struct zs_pool *pool) {} static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage) {} #endif +static void zs_map_object_sg(struct zs_pool *pool, unsigned long handle, + enum zs_mapmode mm, struct scatterlist sg[2]); +static void zs_unmap_object_sg(struct zs_pool *pool, unsigned long handle); + static int create_cache(struct zs_pool *pool) { char *name; @@ -426,6 +431,32 @@ static void zs_zpool_unmap(void *pool, unsigned long handle) zs_unmap_object(pool, handle); } +static void zs_zpool_map_sg(void *pool, unsigned long handle, + enum zpool_mapmode mm, struct scatterlist sg[2]) +{ + enum zs_mapmode zs_mm; + + switch (mm) { + case ZPOOL_MM_RO: + zs_mm = ZS_MM_RO; + break; + case ZPOOL_MM_WO: + zs_mm = ZS_MM_WO; + break; + case ZPOOL_MM_RW: + default: + zs_mm = ZS_MM_RW; + break; + } + + zs_map_object_sg(pool, handle, zs_mm, sg); +} + +static void zs_zpool_unmap_sg(void *pool, unsigned long handle) +{ + zs_unmap_object_sg(pool, handle); +} + static u64 zs_zpool_total_pages(void *pool) { return zs_get_total_pages(pool); @@ -440,7 +471,9 @@ static struct zpool_driver zs_zpool_driver = { .malloc = zs_zpool_malloc, .free = zs_zpool_free, .map = zs_zpool_map, + .map_sg = zs_zpool_map_sg, .unmap = zs_zpool_unmap, + .unmap_sg = zs_zpool_unmap_sg, .total_pages = zs_zpool_total_pages, }; @@ -1281,6 +1314,72 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +static void zs_map_object_sg(struct zs_pool *pool, unsigned long handle, + enum zs_mapmode mm, struct scatterlist sg[2]) +{ + int handle_size = ZS_HANDLE_SIZE; + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + + struct size_class *class; + struct zpdesc *zpdescs[2]; + + /* It guarantees it can get zspage from handle safely */ + read_lock(&pool->migrate_lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* + * migration cannot move any zpages in this zspage. Here, class->lock + * is too heavy since callers would take some time until they calls + * zs_unmap_object API so delegate the locking from class to zspage + * which is smaller granularity. + */ + migrate_read_lock(zspage); + read_unlock(&pool->migrate_lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (unlikely(ZsHugePage(zspage))) + handle_size = 0; + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + sg_init_table(sg, 1); + sg_set_page(sg, zpdesc_page(zpdesc), class->size - handle_size, + off + handle_size); + return; + } + + /* this object spans two pages */ + zpdescs[0] = zpdesc; + zpdescs[1] = get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); + + sg_init_table(sg, 2); + sg_set_page(sg, zpdesc_page(zpdescs[0]), + PAGE_SIZE - off - handle_size, off + handle_size); + sg_set_page(&sg[1], zpdesc_page(zpdescs[1]), + class->size - (PAGE_SIZE - off - handle_size), 0); +} + +static void zs_unmap_object_sg(struct zs_pool *pool, unsigned long handle) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned int obj_idx; + unsigned long obj; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + migrate_read_unlock(zspage); +} + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. diff --git a/mm/zswap.c b/mm/zswap.c index 6504174fbc6a..004fdf26da61 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -13,6 +13,8 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include +#include #include #include #include @@ -26,7 +28,6 @@ #include #include #include -#include #include #include #include @@ -928,9 +929,9 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct scatterlist input, output; int comp_ret = 0, alloc_ret = 0; unsigned int dlen = PAGE_SIZE; + struct scatterlist sg[2]; unsigned long handle; struct zpool *zpool; - char *buf; gfp_t gfp; u8 *dst; @@ -972,9 +973,9 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, if (alloc_ret) goto unlock; - buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); - memcpy(buf, dst, dlen); - zpool_unmap_handle(zpool, handle); + zpool_map_sg(zpool, handle, ZPOOL_MM_WO, sg); + memcpy_to_sglist(sg, 0, dst, dlen); + zpool_unmap_sg(zpool, handle); entry->handle = handle; entry->length = dlen; @@ -994,37 +995,19 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) { struct zpool *zpool = entry->pool->zpool; - struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; - u8 *src; + struct scatterlist input[2]; + struct scatterlist output; acomp_ctx = acomp_ctx_get_cpu_lock(entry->pool); - src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - /* - * If zpool_map_handle is atomic, we cannot reliably utilize its mapped buffer - * to do crypto_acomp_decompress() which might sleep. In such cases, we must - * resort to copying the buffer to a temporary one. - * Meanwhile, zpool_map_handle() might return a non-linearly mapped buffer, - * such as a kmap address of high memory or even ever a vmap address. - * However, sg_init_one is only equipped to handle linearly mapped low memory. - * In such cases, we also must copy the buffer to a temporary and lowmem one. - */ - if ((acomp_ctx->is_sleepable && !zpool_can_sleep_mapped(zpool)) || - !virt_addr_valid(src)) { - memcpy(acomp_ctx->buffer, src, entry->length); - src = acomp_ctx->buffer; - zpool_unmap_handle(zpool, entry->handle); - } - - sg_init_one(&input, src, entry->length); + zpool_map_sg(zpool, entry->handle, ZPOOL_MM_RO, input); sg_init_table(&output, 1); sg_set_folio(&output, folio, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); + acomp_request_set_params(acomp_ctx->req, input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - if (src != acomp_ctx->buffer) - zpool_unmap_handle(zpool, entry->handle); + zpool_unmap_sg(zpool, entry->handle); acomp_ctx_put_unlock(acomp_ctx); } -- Email: Herbert Xu Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt