From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8207EC3814E for ; Sat, 6 Jul 2024 02:25:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC4396B009C; Fri, 5 Jul 2024 22:25:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C734E6B009E; Fri, 5 Jul 2024 22:25:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEAEC6B009F; Fri, 5 Jul 2024 22:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8ED1C6B009C for ; Fri, 5 Jul 2024 22:25:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2F0D41604C9 for ; Sat, 6 Jul 2024 02:25:51 +0000 (UTC) X-FDA: 82307737302.06.EA07D99 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) by imf01.hostedemail.com (Postfix) with ESMTP id 5B7444000C for ; Sat, 6 Jul 2024 02:25:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=etfKjXvK; spf=pass (imf01.hostedemail.com: domain of flintglass@gmail.com designates 209.85.166.42 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720232736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xRoXdXns+v24A0oUujywQmDw48klACW/EhvT639iYLM=; b=m272w3zgoH8BF0PBCXBCOLzXqTZmLjmw7ctpyZhOge2+VruETzE7AlagoebHXxcZJBjRZz 7K7ZCweCJkEYtCZ1VxnNllmHEXZNuf8zUkRrkslpmvUxXa1C4BqG49TAK7dgAf/Ebgot2M mNwmk3RZXYaif8Zj/S9yxw6v+ZL70Cs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=etfKjXvK; spf=pass (imf01.hostedemail.com: domain of flintglass@gmail.com designates 209.85.166.42 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720232736; a=rsa-sha256; cv=none; b=1l90X99ljwOq/mYOIafiK/nysXg/Nqr0iPUEny88I+YcBqKC1FancIGn/kSrV+GGTdFlF2 3IUcCQxAloSFFc6cOdcT4ClmP+eRL6Lk1zYcJSIIJvkf37fXLpQi1DvBttMuurPeYeWcoY kVxxLzE9TxNUEtTcMpUyrMBhtm7ZbLA= Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7f38da3800aso94086139f.0 for ; Fri, 05 Jul 2024 19:25:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720232748; x=1720837548; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xRoXdXns+v24A0oUujywQmDw48klACW/EhvT639iYLM=; b=etfKjXvKZjP66jBgxGP9mZAqT2/mObdoGJDKGRd6qMKtRzXe+BCOAOhL2Zu7eg5YDN lYaTGB1RVqi2PtEsBNBv+TcG1RWWA7dt1FftMBkCDhPsOiOJ7vDAgboH7uw6JYD8ljsy Os9TqjOLgLbGNs1pmQlWagjOL7fTh9MTxfyR6NUfE4WJAWF2X2Yu9jIT8doxBfrkZZYK MYWW7vJaRWtLOicO0Q9EA3jXe/vnOHzcp5IsEWaqddNrZ0lU9Q/L1juNJtmxCJuicG3B RKlaOrvxRWc6QBHB1mEzmYHvMaqhfYOF+3xQrqf+4aHpn2CiPgKi6QxZ5X/FkAgUNlo4 z6Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720232748; x=1720837548; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xRoXdXns+v24A0oUujywQmDw48klACW/EhvT639iYLM=; b=sUZHPcToAlg9RWdXANQS/qelRt1f4lz/ISdjdqWZAqpEgp6ypUSraOLNmdJ/xlGXA0 LjLoQIabgL//XHFXuRVB0rZua7gDutN4KoLXpxo9xPffHcG8ReUadO8KemR5UsP0hZrH 6qO89l4xAOG1eAPs5V0nfn8cgVP8HKdV4pgaMaJ99iF3JuWOGqckWBNaLhGK/CjeXvxF ny1wHFqedLNQct11iDk+CaWQCK/FuWWJRbOZzJ5+7zRyv6S9bG140eIt6FbhQDXEp651 XpX7c5eIhU4/15cRkCBACztmwvlrXm2yUCqR3t7rrIBnlUPrqK9kjc4MSDvRbs7RNUkL bGzg== X-Forwarded-Encrypted: i=1; AJvYcCWFVfo0dcSxiFyw0AFRua+tTx/GXlxYsY3bjvGI8j/KFKM7U9+kjpdzB8ON3ZHFP2G00XDFlGfhwxwi7OkLzC63A2M= X-Gm-Message-State: AOJu0YzsYfCtHgkW6sGOj5QAPpoLQ1EsHLr3XkyK2l2orBvkDRF3WuZI EErqN7KJ9Z+0DfbEF4LCdB8Gb9MY1rubQc+zFRVS8S8gEtmcTrNg X-Google-Smtp-Source: AGHT+IFFVMQYWzHMHiBIAC3NAK4i3uZWoGO71TLphZOCqV9oHXY+GLtfEill58Xv8awN8i0NB1YT5Q== X-Received: by 2002:a05:6e02:20c2:b0:375:8a71:4cc1 with SMTP id e9e14a558f8ab-3839b285940mr70623885ab.32.1720232748356; Fri, 05 Jul 2024 19:25:48 -0700 (PDT) Received: from cbuild.incus (h101-111-009-128.hikari.itscom.jp. [101.111.9.128]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70b15417a7bsm971274b3a.205.2024.07.05.19.25.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Jul 2024 19:25:47 -0700 (PDT) From: Takero Funaki To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Jonathan Corbet , Andrew Morton , Domenico Cerasuolo Cc: Takero Funaki , linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/6] mm: zswap: store incompressible page as-is Date: Sat, 6 Jul 2024 02:25:21 +0000 Message-ID: <20240706022523.1104080-6-flintglass@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240706022523.1104080-1-flintglass@gmail.com> References: <20240706022523.1104080-1-flintglass@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5B7444000C X-Stat-Signature: k3cgzthswrumrt3tza9hwo3b6yo4gtun X-HE-Tag: 1720232749-137219 X-HE-Meta: U2FsdGVkX18JxSz1+CJjCUZP7JiiTIj7S0mlsdYqJIbCYD19nHkVLPlU46dzwLbrg7ZhG8B1jrxJEHrsNTw7Mza8oC5i+x78aodJgMW12/OmxqAm3CI2zxfB8u9S0rh94cI3Z236dCjJFX8+0IA0lplhy/Ft/85MvbC4kG7KGkpemFLLi+T075BuKRAlFRPYc555hFvHLrBWTfA8VzXAkdER5WK/oCgJTBkK5g99c05prYJ1LEzQyRn7vecjssiZFj4lwDyejT/F4t+h1UHEpq0k91/5+aif5SV0f12BihYxihQ/pfFrTw5YWu79TFQAIMnwWkU6JvD6SdqL4P0JwjHcpljHgrqCfHGqoykv2XrbcWvHyCWCFN4HplMjoL77l4Z93fc00gv4Xqfmu/0sO2K12Jh+xP1ELEem+WsRS22J1Nsm1kDz2/e9knIpXVBsDmTBUZM3MrfWANINwpJ2qaHRdCvBhoUW6iM36XiXU4IgEQtvCOt5mB1NVVJlGGcetntqfyDVO9Aw/0jY/fKkzfH7OGRh9O8JcK/ypbFvonlUq1KJG3M2gQBz9GJThnemmC4qtYPjkn4nCYIuYwSJErsItpd90b+nwJ7LVaVMtDJUBQwkNsZegwNthE5qI1Ydr9611GTSGujQWG2Ma6d2i7UDrPdAf/bom+ilQjG2Oqh093Y89CcP3q5KA61cuDRaKIwfzBe/JDGOjVzfeQIqN3koKSm1n7yiZq43ThcKtoqBSeAXbRIwr3/VQPXMzfoCfx+YsSHnTIqFwMdyxN6CKnmsu/s2WvUKhfCuYoG4OsFGG2GNBZ8IXfCl+xpNTtpZ5o2QXdHVutRySLcBXlTpKaqzzHq+4XrizhvZMh1pbFoy6rmW+NgIGq95A0cpwQ2hmr113hkXSlvQDKOd2QA3dAA38M9gBAdfcGU4D9EDNuIagI62dJHAqi+E93gORYfzuTOLbNcGd2su/1OkVvF D2TzTi66 C/stPGDLQp+0dWENyJyiEf10LfdncR+yN7F1aBMuicrji4hSmjXDoj+ptAcAJiYzH0maWUyNW1BKIcgYfZkkxBs2V2P2oBUvbKv8WVqSc+Hh6kA3Ncy8OZa1r9WfnjBvrDp2a6kHuv0auCeURx8pDktqV4Ea+PP/OoIolcCXcDsug1COrM+OZQOgcl6JZG4dtcLvo3NQouUc/J8IQgTYnwI//uj2bYuobtavz5XNr4kbJtTvPf4bQRge6OVJFDTg4CMI5/lII6w/zQBDh2wOdTxCam5XK++M9239d4sZ6nxPcvdhnuYQHnrpmcx6sd5P2jMVq/m5Wbe8Z1MubLfPk8hJLYDN5hUWbTQZ4R/RVHRpm55wS+9yFBq4QW3gdP3ZrzVZb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch allows zswap to accept incompressible pages and store them into zpool if possible. This change is required to achieve zero rejection on zswap_store(). With proper amount of proactive shrinking, swapout can be buffered by zswap without IO latency. Storing incompressible pages may seem costly, but it can reduce latency. A rare incompressible page in a large batch of compressive pages can delay the entire batch during swapping. The memory overhead is negligible because the underlying zsmalloc already accepts nearly incompressible pages. zsmalloc stores data close to PAGE_SIZE to a dedicated page. Thus storing as-is saves decompression cycles without allocation overhead. zswap itself has not rejected pages in these cases. To store the page as-is, use the compressed data size field `length` in struct `zswap_entry`. The length == PAGE_SIZE indicates incompressible data. If a zpool backend does not support allocating PAGE_SIZE (zbud), the behavior remains unchanged. The allocation failure reported by the zpool blocks accepting the page as before. Signed-off-by: Takero Funaki --- mm/zswap.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 76691ca7b6a7..def0f948a4ab 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -186,6 +186,8 @@ static struct shrinker *zswap_shrinker; * length - the length in bytes of the compressed page data. Needed during * decompression. For a same value filled page length is 0, and both * pool and lru are invalid and must be ignored. + * If length is equal to PAGE_SIZE, the data stored in handle is + * not compressed. The data must be copied to page as-is. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data * value - value of the same-value filled pages which have same content @@ -969,9 +971,23 @@ static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) */ comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); dlen = acomp_ctx->req->dlen; - if (comp_ret) + + /* coa_compress returns -EINVAL for errors including insufficient dlen */ + if (comp_ret && comp_ret != -EINVAL) goto unlock; + /* + * If the data cannot be compressed well, store the data as-is. + * Switching by a threshold at + * PAGE_SIZE - (allocation granularity) + * zbud and z3fold use 64B granularity. + * zsmalloc stores >3632B in one page for 4K page arch. + */ + if (comp_ret || dlen > PAGE_SIZE - 64) { + /* we do not use compressed result anymore */ + comp_ret = 0; + dlen = PAGE_SIZE; + } zpool = zswap_find_zpool(entry); gfp = __GFP_NORETRY | __GFP_NOWARN | __GFP_KSWAPD_RECLAIM; if (zpool_malloc_support_movable(zpool)) @@ -981,14 +997,20 @@ static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) goto unlock; buf = zpool_map_handle(zpool, handle, ZPOOL_MM_WO); - memcpy(buf, dst, dlen); + + /* PAGE_SIZE indicates not compressed. */ + if (dlen == PAGE_SIZE) + memcpy_from_folio(buf, folio, 0, PAGE_SIZE); + else + memcpy(buf, dst, dlen); + zpool_unmap_handle(zpool, handle); entry->handle = handle; entry->length = dlen; unlock: - if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC) + if (alloc_ret == -ENOSPC) zswap_reject_compress_poor++; else if (comp_ret) zswap_reject_compress_fail++; @@ -1006,6 +1028,14 @@ static void zswap_decompress(struct zswap_entry *entry, struct page *page) struct crypto_acomp_ctx *acomp_ctx; u8 *src; + if (entry->length == PAGE_SIZE) { + /* the content is not compressed. copy back as-is. */ + src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); + memcpy_to_page(page, 0, src, entry->length); + zpool_unmap_handle(zpool, entry->handle); + return; + } + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); -- 2.43.0