From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53BD1C3ABBE for ; Thu, 8 May 2025 19:42:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8113B6B00B1; Thu, 8 May 2025 15:41:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B57F6B00B3; Thu, 8 May 2025 15:41:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 573456B00B2; Thu, 8 May 2025 15:41:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 262056B00AF for ; Thu, 8 May 2025 15:41:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3BD3DC1051 for ; Thu, 8 May 2025 19:41:49 +0000 (UTC) X-FDA: 83420760738.05.B383E1C Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by imf21.hostedemail.com (Postfix) with ESMTP id CABDD1C0003 for ; Thu, 8 May 2025 19:41:46 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VkBAoceq; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.17 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746733307; a=rsa-sha256; cv=none; b=GfJSET0KKm4hZM0izK0HOYtzmDHQG4lfqh3BvCrigzM93nZ5DWEZE4U4r3PfeRJ5AsuhMU R3lZM4sCdU0ypmaxvIYS5UVb1GjaNxkOsO1S+8qu2ik/zmEr7s80spbU2H0QuMjTml+IRY 2hRn+4VKezuMA2RPdoeTFsIW989IKIo= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VkBAoceq; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.17 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746733307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=26qAnZqm0go3iyZAf7ulRyjPocEjzHGzO1QTq5KZnTE=; b=wRfKLZFhAdYkva4skaH75vPvClCLcirIzktdA11b9BJlNyZN+bQQ27rxLyYUB0Yf+UIAZ8 5PRTN7qMHGGYy8VikzLur1ND+EA/+d2+lTrLrQMkaziP7EKPuJfGhk3QGmwp0s6cPppYeY QJ/VViOCkJ+m6p6Jjp3/5QFCX3eZ+xM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1746733307; x=1778269307; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UyA9sOUzeFWM/IfsI80doZFP3YbMuXrREZWQXOgHADM=; b=VkBAoceqk6eiXqT3b3mjp3YsDZsIjJnFGn3w3nolBLItyGo0EV+2qU12 oUgifmLW1sU05nXYFrvWqkAjE6P9q1NxD5ZLTZhsL4nL6aZeaDC3/eeBs 2V1T8Fuve92TO7u8qa4HA03Ge0RJBq+oTMKXlmFxOgs3PzfPsE8AYsr0U WLZkIgnxOnD28ld7LqXwuc1oFf8gI09HsiZWpko1I6Of/oMxzrj6ZRcc5 qApuJs0AOU1vXZDECLPCx1RUWsBqFNTkBwJfc/x5kzH617pVD3zTNEMJk a9zM6m1LBY/298IvYDDSPrSPfUSQMJrCBJ35ZYvqVdCWsC1CE69k71Nm8 g==; X-CSE-ConnectionGUID: LW8dzkzKTvmX3ME7ltns+g== X-CSE-MsgGUID: H55uMIPGS/i6fevEzvKgIA== X-IronPort-AV: E=McAfee;i="6700,10204,11427"; a="48454716" X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="48454716" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2025 12:41:42 -0700 X-CSE-ConnectionGUID: K4lxUqTMSzCylnQWzosqTA== X-CSE-MsgGUID: eUm8qbXhSCC1RvKiO9HcpQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.15,273,1739865600"; d="scan'208";a="136880866" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by fmviesa010.fm.intel.com with ESMTP; 08 May 2025 12:41:41 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, ying.huang@linux.alibaba.com, akpm@linux-foundation.org, senozhatsky@chromium.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, vinicius.gomes@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v9 11/19] crypto: iaa - Implement crypto_acomp batching interfaces for Intel IAA. Date: Thu, 8 May 2025 12:41:26 -0700 Message-Id: <20250508194134.28392-12-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> References: <20250508194134.28392-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: CABDD1C0003 X-Rspamd-Server: rspam04 X-Stat-Signature: b35rheja1bxqpytqpdhgew5usfzetb7m X-HE-Tag: 1746733306-590946 X-HE-Meta: U2FsdGVkX1/FxUSUpMCxVJlGnkfHUHtfEfVDa9JPBY4nWM+puA1NSufJi6McS60yYHPDq44I2dDFEoE0kwc/5cy9vgK/QsftMc36rYKNzlsQeuDl1OwWPNGDWroDobnfVfn+TAQG4XRh4dauSSjfuWc2+DluCW+WgvPlySlFIL4GF+BaCo1HpLovBI/StW8iDkWeZltCG3XZJ+N/xcan84YSRu1gJ3SgTC1ZEdlIo4NGOqURHumjUGwJ9nXaBUeUVGtXsDVK+EZHnOCm4qxBmNk78ADFbT7YfCIr9r2Lzj0/WSH6ZKHQ/PC5zg7EBVZA51mi6KSV7j2rj7h9sG/5hJ5DwjhbqM6/bGOrF0K5M8NfNHcKCaJQhxRfgEzutv7pluiLbJWdWaqabUHaGrRomNp3KpHUSnI7GQ7m82Ky8woiP1xzzZzQsPBvhfo7Gt22MsKTNafadDxb264aC8CJiJTgA53TjsrI12/xH16GfhMpxtM8/WLvD39RuKzgwnf0AeHzDnK7k3rDrwy60AZUcD4s68WjdZzigMcLn32rr4XO/zJLFSbQa2OuWYPJRR9zdOcDPl7Zs8+85T00OPCxerRUX3egEzZv48MW52g9BqaqSrl+harbwr/TAOb+gI2N3TW7RHf1X4rzGMtThg61enWA4kiJXUDQ7klNGM3+soVs0GVJFnAUi1Hl/kU/JvQiHRT1/iFywnK+Fbx91u88wUtEgcBtOSyf514CX7f3cMFI6UIImnxE2K2gHVasWzQGDY8Aa4U3YhW7wgTedYTX86y7dYQhlio+Ooi7U12jln1n2a81E+b0GB76FdXcOEFrc8tDgGENXtlaeHMik6E6kWJQ+MRvqTPOGTsq/SZxwB8By5RokN9tmVh95hs/1TZgx6S1vFEkNUTU+Y13Eejn5FqSoROCAEnsb84GV9w01byP4RCtUt8fpBNn9sksJIABDnGciWEYa80Y2JEygUp KTQyFxIu 1aWHVFTSRBTuVCxhLELC9khTiFaZDMoTwyLpMjf/J8N5xXlfOqGzsyAe0ufzP+cGUclHs0UQp7UimmUHFe7Z9AmkkkX9p+V3aaC9bAj1hcz+9lZNHA5PcF0QWCK6d4FZrxr/dr72JF5HdsqWI6Ia9JWixH2a4Q1eJLM5vB58l5a3uM2zQ2v3AImH30aeY/s8VI6Ck2niQZkxoMljJuwpe4UBtPyCuu9Ke0XorPCLV9twH2zo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: iaa_crypto implements the new crypto_acomp get_batch_size() interface that returns an iaa_driver specific constant, IAA_CRYPTO_MAX_BATCH_SIZE (set to 8U currently). This patch also provides the iaa_crypto driver implementations for the newly added crypto_acomp batch_compress() and batch_decompress() interfaces. This allows swap modules such as zswap to allocate required batching resources and then invoke fully asynchronous batch parallel compression/decompression of pages on systems with Intel IAA, by invoking these crypto API, respectively: crypto_acomp_batch_size(...); crypto_acomp_batch_compress(...); crypto_acomp_batch_decompress(...); This enables zswap compress batching code to be developed in a manner similar to the current single-page synchronous calls to: crypto_acomp_compress(...); thereby, facilitating encapsulated and modular hand-off between the kernel mm/zswap code and the crypto_acomp layer. Suggested-by: Yosry Ahmed Signed-off-by: Kanchana P Sridhar --- drivers/crypto/intel/iaa/iaa_crypto.h | 9 + drivers/crypto/intel/iaa/iaa_crypto_main.c | 288 +++++++++++++++++++++ 2 files changed, 297 insertions(+) diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h index b4a94da2c315..90ce336879f1 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto.h +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -42,6 +42,15 @@ IAA_DECOMP_CHECK_FOR_EOB | \ IAA_DECOMP_STOP_ON_EOB) +/* + * The maximum compress/decompress batch size for IAA's implementation of + * the crypto_acomp batch_compress() and batch_decompress() interfaces. + * The IAA compression algorithms should provide the crypto_acomp + * get_batch_size() interface through a function that returns this + * constant. + */ +#define IAA_CRYPTO_MAX_BATCH_SIZE 8U + /* Representation of IAA workqueue */ struct iaa_wq { struct list_head list; diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index 52fe68606f4d..d577f555d6ab 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -2149,6 +2149,291 @@ static void compression_ctx_init(struct iaa_compression_ctx *ctx) ctx->use_irq = use_irq; } +static __always_inline unsigned int iaa_comp_get_batch_size(void) +{ + return IAA_CRYPTO_MAX_BATCH_SIZE; +} + +static int iaa_comp_poll(struct acomp_req *req) +{ + struct idxd_desc *idxd_desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + struct idxd_wq *wq; + bool compress_op; + int ret; + + idxd_desc = req->data; + if (!idxd_desc) + return -EAGAIN; + + compress_op = (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS); + wq = idxd_desc->wq; + iaa_wq = idxd_wq_get_private(wq); + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + ret = check_completion(dev, idxd_desc->iax_completion, compress_op, true); + if (ret == -EAGAIN) + return ret; + if (ret) + goto out; + + req->dlen = idxd_desc->iax_completion->output_size; + + /* Update stats */ + if (compress_op) { + update_total_comp_bytes_out(req->dlen); + update_wq_comp_bytes(wq, req->dlen); + } else { + update_total_decomp_bytes_in(req->slen); + update_wq_decomp_bytes(wq, req->slen); + } + + if (iaa_verify_compress && (idxd_desc->iax_hw->opcode == IAX_OPCODE_COMPRESS)) { + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + + src_addr = sg_dma_address(req->src); + dst_addr = sg_dma_address(req->dst); + + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } +out: + /* caller doesn't call crypto_wait_req, so no acomp_request_complete() */ + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + + idxd_free_desc(idxd_desc->wq, idxd_desc); + + dev_dbg(dev, "%s: returning ret=%d\n", __func__, ret); + + return ret; +} + +static __always_inline void iaa_set_req_poll( + struct acomp_req *reqs[], + int nr_reqs, + bool set_flag) +{ + int i; + + for (i = 0; i < nr_reqs; ++i) { + set_flag ? (reqs[i]->base.flags |= CRYPTO_ACOMP_REQ_POLL) : + (reqs[i]->base.flags &= ~CRYPTO_ACOMP_REQ_POLL); + } +} + +/** + * This API provides IAA compress batching functionality for use by swap + * modules. + * + * @reqs: @nr_reqs asynchronous compress requests. + * @pages: Pages to be compressed by IAA. + * @dsts: Pre-allocated destination buffers to store results of IAA + * compression. Each element of @dsts must be of size "PAGE_SIZE * 2". + * @dlens: Will contain the compressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_reqs: The number of requests, up to IAA_CRYPTO_MAX_BATCH_SIZE, + * to be compressed. + * + * Returns true if all compress requests in the batch complete successfully, + * false otherwise. + */ +static bool iaa_comp_acompress_batch( + struct acomp_req *reqs[], + struct page *pages[], + u8 *dsts[], + unsigned int dlens[], + int errors[], + int nr_reqs) +{ + struct scatterlist inputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + struct scatterlist outputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + bool compressions_done = false; + int i, err = 0; + + BUG_ON(nr_reqs > IAA_CRYPTO_MAX_BATCH_SIZE); + + iaa_set_req_poll(reqs, nr_reqs, true); + + /* + * Prepare and submit the batch of acomp_reqs to IAA. IAA will process + * these compress jobs in parallel. + */ + for (i = 0; i < nr_reqs; ++i) { + sg_init_table(&inputs[i], 1); + sg_set_page(&inputs[i], pages[i], PAGE_SIZE, 0); + + /* + * We need PAGE_SIZE * 2 here since there maybe over-compression case, + * and hardware-accelerators may won't check the dst buffer size, so + * giving the dst buffer with enough length to avoid buffer overflow. + */ + sg_init_one(&outputs[i], dsts[i], PAGE_SIZE * 2); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], PAGE_SIZE, PAGE_SIZE); + + errors[i] = iaa_comp_acompress(reqs[i]); + + if (errors[i] != -EINPROGRESS) { + errors[i] = -EINVAL; + err = -EINVAL; + } else { + errors[i] = -EAGAIN; + } + } + + /* + * Asynchronously poll for and process IAA compress job completions. + */ + while (!compressions_done) { + compressions_done = true; + + for (i = 0; i < nr_reqs; ++i) { + /* + * Skip, if the compression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + compressions_done = false; + else + err = -EINVAL; + } else { + dlens[i] = reqs[i]->dlen; + } + } + } + + /* + * For the same 'reqs[]' to be usable by + * iaa_comp_acompress()/iaa_comp_adecompress(), + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs. + */ + iaa_set_req_poll(reqs, nr_reqs, false); + + return !err; +} + +/** + * This API provides IAA decompress batching functionality for use by swap + * modules. + * + * @reqs: @nr_reqs asynchronous decompress requests. + * @srcs: The src buffers to be decompressed by IAA. + * @pages: The pages to store the decompressed buffers. + * @slens: Compressed lengths of @srcs. + * @dlens: Will contain the decompressed lengths. + * @errors: zero on successful compression of the corresponding + * req, or error code in case of error. + * @nr_reqs: The number of pages, up to IAA_CRYPTO_MAX_BATCH_SIZE, + * to be decompressed. + * + * Returns true if all decompress requests complete successfully, + * false otherwise. + */ +static bool iaa_comp_adecompress_batch( + struct acomp_req *reqs[], + u8 *srcs[], + struct page *pages[], + unsigned int slens[], + unsigned int dlens[], + int errors[], + int nr_reqs) +{ + struct scatterlist inputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + struct scatterlist outputs[IAA_CRYPTO_MAX_BATCH_SIZE]; + bool decompressions_done = false; + int i, err = 0; + + BUG_ON(nr_reqs > IAA_CRYPTO_MAX_BATCH_SIZE); + + iaa_set_req_poll(reqs, nr_reqs, true); + + /* + * Prepare and submit the batch of acomp_reqs to IAA. IAA will process + * these decompress jobs in parallel. + */ + for (i = 0; i < nr_reqs; ++i) { + sg_init_one(&inputs[i], srcs[i], slens[i]); + sg_init_table(&outputs[i], 1); + sg_set_page(&outputs[i], pages[i], PAGE_SIZE, 0); + acomp_request_set_params(reqs[i], &inputs[i], + &outputs[i], slens[i], PAGE_SIZE); + + errors[i] = iaa_comp_adecompress(reqs[i]); + + /* + * If it failed desc allocation/submission, errors[i] can + * be 0 or error value from software decompress. + */ + if (errors[i] != -EINPROGRESS) { + errors[i] = -EINVAL; + err = -EINVAL; + } else { + errors[i] = -EAGAIN; + } + } + + /* + * Asynchronously poll for and process IAA decompress job completions. + */ + while (!decompressions_done) { + decompressions_done = true; + + for (i = 0; i < nr_reqs; ++i) { + /* + * Skip, if the decompression has already completed + * successfully or with an error. + */ + if (errors[i] != -EAGAIN) + continue; + + errors[i] = iaa_comp_poll(reqs[i]); + + if (errors[i]) { + if (errors[i] == -EAGAIN) + decompressions_done = false; + else + err = -EINVAL; + } else { + /* + * For batch decompressions, the caller should + * check @errors and handle dlens[i] != PAGE_SIZE. + */ + dlens[i] = reqs[i]->dlen; + } + } + } + + /* + * For the same 'reqs[]' to be usable by + * iaa_comp_acompress()/iaa_comp_adecompress(), + * clear the CRYPTO_ACOMP_REQ_POLL bit on all acomp_reqs. + */ + iaa_set_req_poll(reqs, nr_reqs, false); + + return !err; +} + /********************************************* * Interfaces to crypto_alg and crypto_acomp. *********************************************/ @@ -2169,6 +2454,9 @@ static struct acomp_alg iaa_acomp_fixed_deflate = { .init = iaa_comp_init_fixed, .compress = iaa_comp_acompress, .decompress = iaa_comp_adecompress, + .get_batch_size = iaa_comp_get_batch_size, + .batch_compress = iaa_comp_acompress_batch, + .batch_decompress = iaa_comp_adecompress_batch, .base = { .cra_name = "deflate", .cra_driver_name = "deflate-iaa", -- 2.27.0