From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 837C5C369D3 for ; Wed, 23 Apr 2025 08:14:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D7E96B0022; Wed, 23 Apr 2025 04:14:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 585976B0023; Wed, 23 Apr 2025 04:14:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 427F76B0024; Wed, 23 Apr 2025 04:14:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 21BEB6B0022 for ; Wed, 23 Apr 2025 04:14:13 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6D90A14170E for ; Wed, 23 Apr 2025 08:14:13 +0000 (UTC) X-FDA: 83364595986.29.55614DA Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf20.hostedemail.com (Postfix) with ESMTP id A57271C0002 for ; Wed, 23 Apr 2025 08:14:11 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kfahehoe; spf=pass (imf20.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745396051; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jQ4XDcIlr/fCxipJxNB2V/E0w+IYBz8W3yNE5hu+0VE=; b=LfXWcEOgmnxUd1TxeKf+OSdqgIqxMwyySylK4NLnqrSR5F77AlbYCyJu2N8fTl8SMq1rBb PuiFTfV1DNooSmwIjjd2CrqYZ+GLLKg4V49jlcswGwZyjU1SJDglsTl0kRi0SD4uJlq2iA Z1sVmBygCnTl+TWUPLdtfroPkLJ50Ao= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kfahehoe; spf=pass (imf20.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745396051; a=rsa-sha256; cv=none; b=Vz9eLwbYH0aFq9bX7xu1YrELKUf63c4W1N1WQx5vmX53bh6BjRfXaYaOsZbgd+d0DFIwO9 rXLHNzSDkYYqtxfF4MH/nQ/J6xwb1dcfBq8bwe6LlldBScIRy7SCCKfJG5yKVvO4JE+i1H LHmUDIB9sY2tFJykBfHrsnCF0ixrnUc= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 41FD44A47F; Wed, 23 Apr 2025 08:14:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3390C4CEE2; Wed, 23 Apr 2025 08:14:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745396050; bh=stuDwq6h9t+5JhBkVAbX8zdJRr9cymhHPPBPnybsHRc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kfahehoeylx4NChOIat5N4W7SquodKQT1FM5PxvEhBzqRj2AioM2Ls1LcWMna5R4n PDM8+UmZyCdpRW4mv1jzFDdPGJPENZUQtd7kfJxEHMrds+2fA6S/rU1qDqs0MAWohT QreF3nFBf3bR9WRIrWZEiPncq2HBbg6bKIAWMK3JiKunu6QY9xy7BR81GIus4W4Hnh dl/nZdbF+sNJjI+Zpz6itukQnQ2SFjhI83wNUtIYQqDj7EIJj9zcf5nqJbJj4+2Z/1 53RNNmWhhzybpzGDXJbE9vCt/eAYLxYj+F/hVSCvCJRjyCip+Ig7gm16EnWLQQq/gr s96NTCZObaJ6g== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: [PATCH v9 05/24] dma-mapping: Provide an interface to allow allocate IOVA Date: Wed, 23 Apr 2025 11:12:56 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 3uig5ser9raway3pkx4gc8bsjukityho X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A57271C0002 X-Rspam-User: X-HE-Tag: 1745396051-634199 X-HE-Meta: U2FsdGVkX1+9Wtg3WGk06wlCAxmWziUwcl/uINtHeoz2yDkq/XJItA1Wi6YrEkDgho9g9+iIbZVrAYg+YamVb9+YI8nMUy/D4Ei0R7A88P5/kbvKO6Uu8wC7KbxKeklyHz2QNsyfCDq19LnmLNDntJh85cY6f6I/ckH5SWie+O99o32YsDu0+d2lwH79le8cSxHH35TkCy5v282YhqC86UwqJN5CLcd88pog47mb8cYg6waw+5apAMKq6yzWr8NFfbpWXtXMilT9DvXusH3wTvdxvwXAhp4obU43tVfTYt+2nP1gZxbwHfFmCVahglTa8xrhz/D7TSU5p1zQ3CaLUnMPGkwMoAb9Dk1dLuyWwxz61z0y/EUTGfCmxKIH0A6zhPYEpYN2bOt+mDEhqQNP+8TB+4W2OjQu+fMlv58dyr9tt2XesB9dHt683jymIT8UQXqjBN4JbiAyZ9lcMAT3lPmmoM+0prbIGmC4RsGY+dwMw8AYqTRFMecG1Fly5Cld+cpDqXS9r5YYiGycrXytkzoRed9vTrvlEtpY920hxJcbs+nSSxLzwGSDc9brLfk5oohjgVLYjb0AicdmfkooI/q1Y2oNSnf/xV4W3XF2DBys7tJQhgEo/zKgDFtRJ9vYsdgWAAjXlUsjSUm84o6dc0yjfM6hmfEaBdqGo/XB+a0uUrrENawMiO5jcpoPyC/Lxxsh9Hp5lzS2K3qUPx03NgO7S99AXKMhVBysoILyoRuRqgj+TdALVCqI5KLyxdDCU4EqVk7M0FzL0bZu2DZDXLXjoMBNa3rogix05l1767tY2wC3FwGqHHqK0tkOWNXL149mNh9I5Cy7z2O7KGSocUG/dNH+y/a1KjPzRZDOejX9juHT/62MDQi4/H3X64oeQmw9ggm0ktIIj/FjyyNrS8NrFzeEXfDSAVF+s0/BOSNH2v4NfDcmNmFlvnqiFvPdmjj7FpMV8g11NYOrqB3 DnAodGbe Z2hwKbI6E/CzpNbbE5/1JNsRX61h0vwQ+AUvdDcU18CdifsCjHmTt+k/grHp18bQjoT+SvU24b0F3noQAPh4FLoNqosBxOXT+x4kaCedCrGDcvT131aD4zpXdP3x7vZyaGABd3wj0vSh0a7PSXGhLHx3NG31MibA1OgWtQGGXhVYa1auQOJVlUokpb4xJpaJJwO+Qn4fJiHHjS6GdGvFu9ernOFr/armhjwWO4lUAreE1GNM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky The existing .map_page() callback provides both allocating of IOVA and linking DMA pages. That combination works great for most of the callers who use it in control paths, but is less effective in fast paths where there may be multiple calls to map_page(). These advanced callers already manage their data in some sort of database and can perform IOVA allocation in advance, leaving range linkage operation to be in fast path. Provide an interface to allocate/deallocate IOVA and next patch link/unlink DMA ranges to that specific IOVA. In the new API a DMA mapping transaction is identified by a struct dma_iova_state, which holds some recomputed information for the transaction which does not change for each page being mapped, so add a check if IOVA can be used for the specific transaction. The API is exported from dma-iommu as it is the only implementation supported, the namespace is clearly different from iommu_* functions which are not allowed to be used. This code layout allows us to save function call per API call used in datapath as well as a lot of boilerplate code. Reviewed-by: Christoph Hellwig Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 86 +++++++++++++++++++++++++++++++++++++ include/linux/dma-mapping.h | 48 +++++++++++++++++++++ 2 files changed, 134 insertions(+) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 145606498b4c..6ca9305a26cc 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1723,6 +1723,92 @@ size_t iommu_dma_max_mapping_size(struct device *dev) return SIZE_MAX; } +/** + * dma_iova_try_alloc - Try to allocate an IOVA space + * @dev: Device to allocate the IOVA space for + * @state: IOVA state + * @phys: physical address + * @size: IOVA size + * + * Check if @dev supports the IOVA-based DMA API, and if yes allocate IOVA space + * for the given base address and size. + * + * Note: @phys is only used to calculate the IOVA alignment. Callers that always + * do PAGE_SIZE aligned transfers can safely pass 0 here. + * + * Returns %true if the IOVA-based DMA API can be used and IOVA space has been + * allocated, or %false if the regular DMA API should be used. + */ +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size) +{ + struct iommu_dma_cookie *cookie; + struct iommu_domain *domain; + struct iova_domain *iovad; + size_t iova_off; + dma_addr_t addr; + + memset(state, 0, sizeof(*state)); + if (!use_dma_iommu(dev)) + return false; + + domain = iommu_get_dma_domain(dev); + cookie = domain->iova_cookie; + iovad = &cookie->iovad; + iova_off = iova_offset(iovad, phys); + + if (static_branch_unlikely(&iommu_deferred_attach_enabled) && + iommu_deferred_attach(dev, iommu_get_domain_for_dev(dev))) + return false; + + if (WARN_ON_ONCE(!size)) + return false; + + /* + * DMA_IOVA_USE_SWIOTLB is flag which is set by dma-iommu + * internals, make sure that caller didn't set it and/or + * didn't use this interface to map SIZE_MAX. + */ + if (WARN_ON_ONCE((u64)size & DMA_IOVA_USE_SWIOTLB)) + return false; + + addr = iommu_dma_alloc_iova(domain, + iova_align(iovad, size + iova_off), + dma_get_mask(dev), dev); + if (!addr) + return false; + + state->addr = addr + iova_off; + state->__size = size; + return true; +} +EXPORT_SYMBOL_GPL(dma_iova_try_alloc); + +/** + * dma_iova_free - Free an IOVA space + * @dev: Device to free the IOVA space for + * @state: IOVA state + * + * Undoes a successful dma_try_iova_alloc(). + * + * Note that all dma_iova_link() calls need to be undone first. For callers + * that never call dma_iova_unlink(), dma_iova_destroy() can be used instead + * which unlinks all ranges and frees the IOVA space in a single efficient + * operation. + */ +void dma_iova_free(struct device *dev, struct dma_iova_state *state) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_start_pad = iova_offset(iovad, state->addr); + size_t size = dma_iova_size(state); + + iommu_dma_free_iova(domain, state->addr - iova_start_pad, + iova_align(iovad, size + iova_start_pad), NULL); +} +EXPORT_SYMBOL_GPL(dma_iova_free); + void iommu_setup_dma_ops(struct device *dev) { struct iommu_domain *domain = iommu_get_domain_for_dev(dev); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index b79925b1c433..de7f73810d54 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -72,6 +72,22 @@ #define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1)) +struct dma_iova_state { + dma_addr_t addr; + u64 __size; +}; + +/* + * Use the high bit to mark if we used swiotlb for one or more ranges. + */ +#define DMA_IOVA_USE_SWIOTLB (1ULL << 63) + +static inline size_t dma_iova_size(struct dma_iova_state *state) +{ + /* Casting is needed for 32-bits systems */ + return (size_t)(state->__size & ~DMA_IOVA_USE_SWIOTLB); +} + #ifdef CONFIG_DMA_API_DEBUG void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr); void debug_dma_map_single(struct device *dev, const void *addr, @@ -277,6 +293,38 @@ static inline int dma_mmap_noncontiguous(struct device *dev, } #endif /* CONFIG_HAS_DMA */ +#ifdef CONFIG_IOMMU_DMA +/** + * dma_use_iova - check if the IOVA API is used for this state + * @state: IOVA state + * + * Return %true if the DMA transfers uses the dma_iova_*() calls or %false if + * they can't be used. + */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return state->__size != 0; +} + +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size); +void dma_iova_free(struct device *dev, struct dma_iova_state *state); +#else /* CONFIG_IOMMU_DMA */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return false; +} +static inline bool dma_iova_try_alloc(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t size) +{ + return false; +} +static inline void dma_iova_free(struct device *dev, + struct dma_iova_state *state) +{ +} +#endif /* CONFIG_IOMMU_DMA */ + #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir); -- 2.49.0