From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04950C369CB for ; Wed, 23 Apr 2025 08:15:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B44656B009F; Wed, 23 Apr 2025 04:15:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF3C56B00A0; Wed, 23 Apr 2025 04:15:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96B6E6B00A1; Wed, 23 Apr 2025 04:15:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 788CD6B009F for ; Wed, 23 Apr 2025 04:15:38 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0873981792 for ; Wed, 23 Apr 2025 08:15:39 +0000 (UTC) X-FDA: 83364599598.08.2AA17BA Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf12.hostedemail.com (Postfix) with ESMTP id 557B940005 for ; Wed, 23 Apr 2025 08:15:37 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L5gnr8EL; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745396137; a=rsa-sha256; cv=none; b=SMfrHiBUODV7CXIL0um7x24T7hwnkcFIJJoGmlwHvMnwros57dLZ7xjlrHiKPNw1ZYzb1F q8u5smYXU7RtDb5RbfXJA7ZQVI8ssT9rxVAuhELTgsRcM6On0zqhVZytKJZHrY3UnrwNMK p2tI8RszmtrxobepoUbiqG6aQnetbno= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745396137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=43blswvOBPU4BMfUWYNBiOR2a0ASqQkiCnkibSf2hHQ=; b=Vg3JJeQ5uEN1IXOt02FbXqQQuaOJm0vO41sb63csvLS1JnqqTv7wo3KsewFOUhUWC7g90O SS/h0vXHfWLfiHI8F9kKWhwot80EANx/XFaKhSzIcnSY+fgHNJvyg1hEs19LcLLcy3Z5Ez NwaPKIIXvbOeIZQiPCz2MCT15uBc79o= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L5gnr8EL; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id DC37E4A43E; Wed, 23 Apr 2025 08:15:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 34D3CC4CEE2; Wed, 23 Apr 2025 08:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745396136; bh=ZE/m4wI1ph0h7rG40pku32JCR3EN9MIKEzOdQT6q1LI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L5gnr8ELzuRM0jLcY/k1+4/4J4woQsbBEVR8YeNu7Jv/Hoa0yPtwdBQr/pH/SBVFH 9OQ6obw1D7NZco5Sk6xNnQ998+8LG7U2rVEMfZC9rnmgOA08PUWIYN4nBN5Owx9VO4 xhXvFYMmALmTArNT8NQMftlISubSM/SEdZabrouhUDEE+GzsX+x28U77UvmHvSuc7T 1JJ0HsZ2TWujQHNRYvfeKd6d+GjCkg05ILaF+N0aSgDSNIey7yM/Ifaf1YHCCYfDux lVwYj/idlOQadwNvXBrwYAiuaVNkxuDQTv8MT5CfozqnvElf3nj/PC1xu7BNVbcQm0 4IghMky2KwcCQ== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v9 22/24] nvme-pci: use a better encoding for small prp pool allocations Date: Wed, 23 Apr 2025 11:13:13 +0300 Message-ID: <973ed41249e12766383b3cedac799692f9bda3b8.1745394536.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 557B940005 X-Stat-Signature: pqhhjuwo8iiae8zgxc53pkkix3zkd69t X-Rspam-User: X-HE-Tag: 1745396137-106264 X-HE-Meta: U2FsdGVkX18pGpINYsrwWePexzpK4lK+z1UMPm23UMgdHqq14JIuE6MLH/4PmEBr+zYS5OpcaoCeY36p9cH/Zu4TVMzYNFAKXUxniidEwF2SVZ0gXCp6+sT4YRajWX7fl1ejUCtAiZQrX/vZ6SkaZY+45CQZvwyU1eF3DKvoAar2V7fkzAKOIWS+Wx4HaKFYxZrMkxPNN1kvr9mDYAhOU9W0BZdyMR96EWYYqsJTV0B3DHXEl9USjj4nMLkq4AwJJuaa6cmxa9XK3CVjWXdxq68lhoFfowsmqZADffBsZZgVo0D+3wrpFuRonwc9C78ld5LarCmzFM472J5vXZFfOOO/wAYe053kYZKLA0oJFHtuov4chxPfyNHTQAmBaKageWK9uGVM9zgU9ah33N11sPzE2rrXHm8pq8tFnOQaWrOl0E59guTUeBl5bdIV0lzOaQRnb9SkipS0SO7f36WjX1GaeY+dPE8N/yRz6s2DGJMC6wqczt+8zoirWPIQJKxJ4cjw32/bHzYc8njIYFrVIHBAquNsw4b02pR0pqNJByvqwTlfUc2pPscTcW15mFqxl+kUK8O1eazHww6OQ14DTLwIfhtqSQkE3f4RTWcAuCh7rIljUucSBHRgQ7xR080LuzD0hJf67RKgUZe93ORNY4BW8m9iwJMom9c+D0iW/0bswM/0YI5J2f9k5njIcwh6jz/T2MrStOB+yAo1+myHlUpGswkMr78oac798LOC4vynagYb7ZRdObxJ3eKJpGU5Sq88vjpUTQf+cqFxV+MpHVJiV3fUv5oA34Xd3wPOtDRyXIdAScULrIgUAQK2GOL4zydjHoOs2tg96uT4CCItDPF4xGaXqWr45Zcmi5PeFUNOADC543LGsB65wTZT6RjbW3l28gEoZ8GypqyjVN9u48PP2ZOoZvq9+I/U617ahVRGC8Qj3X+f6ExbDFM85Yn+7ZEpJ36ePvCFAJjDktV gAV03vdE NCnmAdirhJ5+mFa0D3zDxKqleFnBr7bAayPb3v++sbDbrVaWxqDn13lXejSgz7c5shJIgrViJPc39DlVGJkxAtuTTBBqhlaCjD1TqcyGt5NkwDp9uJa4Es5mGq3j2gSZIAGm9nhKjrv2GfNmO5ktEZIu6pp6T9XeRzdYYIEA9TJO5e+AVv3fJas81vLjHDxakgVNxeIaoFuuvezb3e4hw1P+TOp8FvoqLqx3/WN65Ps6WWpYFzRufboWajK8Bj2Tq6B8H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig There is plenty of unused space in the iod next to nr_descriptors. Add a separate flag to encode that the transfer is using the full page sized pool, and use a normal 0..n count for the number of descriptors. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe [ Leon: changed original bool variable to be flag as was proposed by Kanchan ] Signed-off-by: Kanchan Joshi Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 93 ++++++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 47 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 638e759b29ad..7e93536d01cb 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -44,6 +44,7 @@ #define NVME_MAX_SEGS 128 #define NVME_MAX_META_SEGS 15 #define NVME_MAX_NR_DESCRIPTORS 5 +#define NVME_SMALL_DESCRIPTOR_SIZE 256 static int use_threaded_interrupts; module_param(use_threaded_interrupts, int, 0444); @@ -219,6 +220,10 @@ struct nvme_queue { struct completion delete_done; }; +enum { + IOD_LARGE_DESCRIPTORS = 1, /* uses the full page sized descriptor pool */ +}; + /* * The nvme_iod describes the data in an I/O. */ @@ -226,8 +231,8 @@ struct nvme_iod { struct nvme_request req; struct nvme_command cmd; bool aborted; - /* # of PRP/SGL descriptors: (0 for small pool) */ - s8 nr_descriptors; + u8 nr_descriptors; /* # of PRP/SGL descriptors */ + unsigned int flags; unsigned int dma_len; /* length of single DMA segment mapping */ dma_addr_t first_dma; dma_addr_t meta_dma; @@ -529,13 +534,27 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, return true; } -static void nvme_free_prps(struct nvme_dev *dev, struct request *req) +static inline struct dma_pool *nvme_dma_pool(struct nvme_dev *dev, + struct nvme_iod *iod) +{ + if (iod->flags & IOD_LARGE_DESCRIPTORS) + return dev->prp_page_pool; + return dev->prp_small_pool; +} + +static void nvme_free_descriptors(struct nvme_dev *dev, struct request *req) { const int last_prp = NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; struct nvme_iod *iod = blk_mq_rq_to_pdu(req); dma_addr_t dma_addr = iod->first_dma; int i; + if (iod->nr_descriptors == 1) { + dma_pool_free(nvme_dma_pool(dev, iod), iod->descriptors[0], + dma_addr); + return; + } + for (i = 0; i < iod->nr_descriptors; i++) { __le64 *prp_list = iod->descriptors[i]; dma_addr_t next_dma_addr = le64_to_cpu(prp_list[last_prp]); @@ -558,15 +577,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) WARN_ON_ONCE(!iod->sgt.nents); dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); - - if (iod->nr_descriptors == 0) - dma_pool_free(dev->prp_small_pool, iod->descriptors[0], - iod->first_dma); - else if (iod->nr_descriptors == 1) - dma_pool_free(dev->prp_page_pool, iod->descriptors[0], - iod->first_dma); - else - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); mempool_free(iod->sgt.sgl, dev->iod_mempool); } @@ -588,7 +599,6 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, struct request *req, struct nvme_rw_command *cmnd) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - struct dma_pool *pool; int length = blk_rq_payload_bytes(req); struct scatterlist *sg = iod->sgt.sgl; int dma_len = sg_dma_len(sg); @@ -596,7 +606,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, int offset = dma_addr & (NVME_CTRL_PAGE_SIZE - 1); __le64 *prp_list; dma_addr_t prp_dma; - int nprps, i; + int i; length -= (NVME_CTRL_PAGE_SIZE - offset); if (length <= 0) { @@ -618,27 +628,23 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, goto done; } - nprps = DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE); - if (nprps <= (256 / 8)) { - pool = dev->prp_small_pool; - iod->nr_descriptors = 0; - } else { - pool = dev->prp_page_pool; - iod->nr_descriptors = 1; - } + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) > + NVME_SMALL_DESCRIPTOR_SIZE / sizeof(__le64)) + iod->flags |= IOD_LARGE_DESCRIPTORS; - prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); - if (!prp_list) { - iod->nr_descriptors = -1; + prp_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) return BLK_STS_RESOURCE; - } - iod->descriptors[0] = prp_list; + iod->descriptors[iod->nr_descriptors++] = prp_list; iod->first_dma = prp_dma; i = 0; for (;;) { if (i == NVME_CTRL_PAGE_SIZE >> 3) { __le64 *old_prp_list = prp_list; - prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); + + prp_list = dma_pool_alloc(dev->prp_page_pool, + GFP_ATOMIC, &prp_dma); if (!prp_list) goto free_prps; iod->descriptors[iod->nr_descriptors++] = prp_list; @@ -665,7 +671,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma); return BLK_STS_OK; free_prps: - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); return BLK_STS_RESOURCE; bad_sgl: WARN(DO_ONCE(nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents), @@ -694,7 +700,6 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, struct request *req, struct nvme_rw_command *cmd) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - struct dma_pool *pool; struct nvme_sgl_desc *sg_list; struct scatterlist *sg = iod->sgt.sgl; unsigned int entries = iod->sgt.nents; @@ -709,21 +714,13 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, return BLK_STS_OK; } - if (entries <= (256 / sizeof(struct nvme_sgl_desc))) { - pool = dev->prp_small_pool; - iod->nr_descriptors = 0; - } else { - pool = dev->prp_page_pool; - iod->nr_descriptors = 1; - } + if (entries > NVME_SMALL_DESCRIPTOR_SIZE / sizeof(*sg_list)) + iod->flags |= IOD_LARGE_DESCRIPTORS; - sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); - if (!sg_list) { - iod->nr_descriptors = -1; + sg_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, &sgl_dma); + if (!sg_list) return BLK_STS_RESOURCE; - } - - iod->descriptors[0] = sg_list; + iod->descriptors[iod->nr_descriptors++] = sg_list; iod->first_dma = sgl_dma; nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries); @@ -915,7 +912,8 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) blk_status_t ret; iod->aborted = false; - iod->nr_descriptors = -1; + iod->nr_descriptors = 0; + iod->flags = 0; iod->sgt.nents = 0; iod->meta_sgt.nents = 0; @@ -2833,7 +2831,7 @@ static int nvme_disable_prepare_reset(struct nvme_dev *dev, bool shutdown) static int nvme_setup_prp_pools(struct nvme_dev *dev) { - size_t small_align = 256; + size_t small_align = NVME_SMALL_DESCRIPTOR_SIZE; dev->prp_page_pool = dma_pool_create("prp list page", dev->dev, NVME_CTRL_PAGE_SIZE, @@ -2841,12 +2839,13 @@ static int nvme_setup_prp_pools(struct nvme_dev *dev) if (!dev->prp_page_pool) return -ENOMEM; + BUILD_BUG_ON(NVME_SMALL_DESCRIPTOR_SIZE != 256); if (dev->ctrl.quirks & NVME_QUIRK_DMAPOOL_ALIGN_512) - small_align = 512; + small_align *= 2; /* Optimisation for I/Os between 4k and 128k */ dev->prp_small_pool = dma_pool_create("prp list 256", dev->dev, - 256, small_align, 0); + NVME_SMALL_DESCRIPTOR_SIZE, small_align, 0); if (!dev->prp_small_pool) { dma_pool_destroy(dev->prp_page_pool); return -ENOMEM; -- 2.49.0