From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BB43C369AB for ; Fri, 18 Apr 2025 06:49:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B489B28011D; Fri, 18 Apr 2025 02:49:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF248280118; Fri, 18 Apr 2025 02:49:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D1E328011D; Fri, 18 Apr 2025 02:49:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6401F280118 for ; Fri, 18 Apr 2025 02:49:29 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5D9D014126F for ; Fri, 18 Apr 2025 06:49:30 +0000 (UTC) X-FDA: 83346238500.03.B106E72 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf02.hostedemail.com (Postfix) with ESMTP id F3A0580003 for ; Fri, 18 Apr 2025 06:49:28 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NyRjrrJC; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744958969; a=rsa-sha256; cv=none; b=f7DKIdQWumGvnDkQGcveBh9Gxj+dJK5YEeAT+cr3BXgGUiubHrajbGP7uCw8O04YrHKn7y w9Zjrq9OFJeHv5JIQMbxSI44AdEQGl5K3jUc953cvZpimIL2g/VeNOTfZUrGanceUliA0B mbJeSWMuU1Wgr15eBXhNyQ1ACs2nmt4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NyRjrrJC; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744958969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=58X7drC/7y3+nIM+KuDwoMupmKxw6e2kM6B18XE2K2Y=; b=w46EBOiON31vhEhjK8gjkOeYCAWk+Ed5bdVRCnGq3pAUogjf7HYsWeFV+AmIzxu11Mh+KJ 6IwVzc6HAH751ut/iixq411922YRmzdI/0g9/e1R9cZu0bJiHBdqUnhfxrnLWVOl474tTc /+aVWYlkM8w6vZI+Uik9SdhqYgdfjxM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 3D3C5A4B29C; Fri, 18 Apr 2025 06:44:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83A19C4CEED; Fri, 18 Apr 2025 06:49:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744958968; bh=294NxRUIohhQBJNMY7N2MX6lkvZ3TDh+LJnGbG4ssHk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NyRjrrJCYDcV4Q6X5lDPSqfKb5XXiaedFINEKeEaKj5c2isWltJhfsALSM3VgwvZl lS53PtklWfaycHbYjw4cvysee218GZYrOc7qZGi7BYsVIazNbU66R9j5fFmm2Ltt4n j8BjJ7+Ln/k8ahdAGCb7OnK13g4GjmTZLqhl9ArxHySQocsDC80u3s611uawF8YuFQ gy+bzOE2yHfYfcXxHSWpHrWT2xPfE0wadBFYg/6L2E0vNcNLVPrgL4DIN8o9MMHN1F 7PwwigxiizqoPXaEoeYzngVz1ND13xNGyXE0zjw8hkyHjlOvk92XVBi5cwbqAu6RPZ OlpBHekA1a6zA== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Kanchan Joshi , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Chaitanya Kulkarni , Nitesh Shetty , Leon Romanovsky Subject: [PATCH v8 24/24] nvme-pci: optimize single-segment handling Date: Fri, 18 Apr 2025 09:47:54 +0300 Message-ID: <670389227a033bd5b7c5aa55191aac9943244028.1744825142.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: F3A0580003 X-Rspamd-Server: rspam04 X-Stat-Signature: 9gcuj17xzee6okjkgjp7ey9swraxb6ae X-HE-Tag: 1744958968-278142 X-HE-Meta: U2FsdGVkX1/WwZUZMHBeDqiXCL7fOqDbw7FE11bg4vN6FqEH33d2hYxuUTFLxAu9fdOdx+PJmEhXCZU+ewebs38UZrxE53AGY2/rINEFtwhneLkpEbwpjF4udhvZk/1eOb8OZr//Y7sL5Kunzn/YFDJLB3o/OiyVGXTAF9qTtzeQ9WQlLA9gBqCXLayI7C01Oyl2tUHtUJTLulaY2nP6k1R1btwpsbfYk/qgtWrHouNMWhmutdgJqpK8Jc0lfGbc9V/PNiYiRS6v+vOgawzeiGqtdeBr0KBMOP6iYA3I6rGcuGibUk4YXtzBKGBVFxtAALk8sqcIGohzQOGquTy/HdFQEqf2dufyvFncBExh17Kp+uN5/k2D90q1zpMeVbHdfIfBszxDdBe89u4Go53VednPEIiCqpgRJOUxZG2SlQr2rRst9jwxIFrFQ6/rCm+NN8ZFhaNfWvQNOml+6ZHRqymoc1Na3928wSZKgMN84w7YbekDt/0WxXO+HDty6YKY602XIHjlgjrTbeA0TNNeGPWoUzB/A2oo+0MnjvUVCc0lkD3PdDnfrA/7+ys8vejkVCNP1MRHaIp5r9y7Vz+JmjP9wRSQbcTAtzYr/RLCEwuHKNYXLKcPNle06HF6a3O0X0XaqdYU1Y4hgzuaX18kD3hobQnbLOf+QoEmnJVC6q6dxxPVK0DNApjRSJwj6srg4ymbh63rpVxVIwtwfFEc1HRvPHYQ9SVUUN6ZFnOPEr35uUzPjw80bgLyzrygcf1Pl5oOrJ2sGhCbQpQ1PoXbjvfSGhXJMGq7Cf3AzQYhnlQL45GZNPkw0L7gIGbnX1Kq3x/hhyc05hWjM8FHzHLWHds5vdcg1DLKXYpXXWDC7P+TEd8VbcZ7Pdamj7aaSd9rT2ZKdPfGGqUsAh7rUtjjlaUQC9cadDuddBaAyxHEcCxNxmYBYzStNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kanchan Joshi blk_rq_dma_map API is costly for single-segment requests. Avoid using it and map the bio_vec directly. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 65 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 8d99a8f871ea..cf020de82962 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -216,6 +216,11 @@ struct nvme_queue { struct completion delete_done; }; +enum { + IOD_LARGE_DESCRIPTORS = 1, /* uses the full page sized descriptor pool */ + IOD_SINGLE_SEGMENT = 2, /* single segment dma mapping */ +}; + /* * The nvme_iod describes the data in an I/O. */ @@ -224,7 +229,7 @@ struct nvme_iod { struct nvme_command cmd; bool aborted; u8 nr_descriptors; /* # of PRP/SGL descriptors */ - bool large_descriptors; /* uses the full page sized descriptor pool */ + unsigned int flags; unsigned int total_len; /* length of the entire transfer */ unsigned int total_meta_len; /* length of the entire metadata transfer */ dma_addr_t meta_dma; @@ -529,7 +534,7 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, static inline struct dma_pool *nvme_dma_pool(struct nvme_dev *dev, struct nvme_iod *iod) { - if (iod->large_descriptors) + if (iod->flags & IOD_LARGE_DESCRIPTORS) return dev->prp_page_pool; return dev->prp_small_pool; } @@ -630,6 +635,15 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + unsigned int nr_segments = blk_rq_nr_phys_segments(req); + dma_addr_t dma_addr; + + if (nr_segments == 1 && (iod->flags & IOD_SINGLE_SEGMENT)) { + dma_addr = le64_to_cpu(iod->cmd.common.dptr.prp1); + dma_unmap_page(dev->dev, dma_addr, iod->total_len, + rq_dma_dir(req)); + return; + } if (!blk_rq_dma_unmap(req, dev->dev, &iod->dma_state, iod->total_len)) { if (iod->cmd.common.flags & NVME_CMD_SGL_METABUF) @@ -642,6 +656,41 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) nvme_free_descriptors(dev, req); } +static bool nvme_try_setup_prp_simple(struct nvme_dev *dev, struct request *req, + struct nvme_rw_command *cmnd, + struct blk_dma_iter *iter) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + struct bio_vec bv = req_bvec(req); + unsigned int first_prp_len; + + if (is_pci_p2pdma_page(bv.bv_page)) + return false; + if ((bv.bv_offset & (NVME_CTRL_PAGE_SIZE - 1)) + bv.bv_len > + NVME_CTRL_PAGE_SIZE * 2) + return false; + + iter->addr = dma_map_bvec(dev->dev, &bv, rq_dma_dir(req), 0); + if (dma_mapping_error(dev->dev, iter->addr)) { + iter->status = BLK_STS_RESOURCE; + goto out; + } + iod->total_len = bv.bv_len; + cmnd->dptr.prp1 = cpu_to_le64(iter->addr); + + first_prp_len = NVME_CTRL_PAGE_SIZE - + (bv.bv_offset & (NVME_CTRL_PAGE_SIZE - 1)); + if (bv.bv_len > first_prp_len) + cmnd->dptr.prp2 = cpu_to_le64(iter->addr + first_prp_len); + else + cmnd->dptr.prp2 = 0; + + iter->status = BLK_STS_OK; + iod->flags |= IOD_SINGLE_SEGMENT; +out: + return true; +} + static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, struct request *req) { @@ -652,6 +701,12 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, dma_addr_t prp1_dma, prp2_dma = 0; unsigned int prp_len, i; __le64 *prp_list; + unsigned int nr_segments = blk_rq_nr_phys_segments(req); + + if (nr_segments == 1) { + if (nvme_try_setup_prp_simple(dev, req, cmnd, &iter)) + return iter.status; + } if (!blk_rq_dma_map_iter_start(req, dev->dev, &iod->dma_state, &iter)) return iter.status; @@ -693,7 +748,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) > NVME_SMALL_DESCRIPTOR_SIZE / sizeof(__le64)) - iod->large_descriptors = true; + iod->flags |= IOD_LARGE_DESCRIPTORS; prp_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, &prp2_dma); @@ -808,7 +863,7 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, } if (entries > NVME_SMALL_DESCRIPTOR_SIZE / sizeof(*sg_list)) - iod->large_descriptors = true; + iod->flags |= IOD_LARGE_DESCRIPTORS; sg_list = dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, &sgl_dma); if (!sg_list) @@ -932,7 +987,7 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) iod->aborted = false; iod->nr_descriptors = 0; - iod->large_descriptors = false; + iod->flags = 0; iod->total_len = 0; iod->total_meta_len = 0; -- 2.49.0