From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B6B87CF34B0 for ; Wed, 19 Nov 2025 13:42:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 258176B0005; Wed, 19 Nov 2025 08:42:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22F046B000E; Wed, 19 Nov 2025 08:42:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16C336B00A8; Wed, 19 Nov 2025 08:42:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0659F6B0005 for ; Wed, 19 Nov 2025 08:42:56 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BB05858F9C for ; Wed, 19 Nov 2025 13:42:55 +0000 (UTC) X-FDA: 84127472310.16.C875449 Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf06.hostedemail.com (Postfix) with ESMTP id F3226180002 for ; Wed, 19 Nov 2025 13:42:53 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LZPWhNTp; spf=pass (imf06.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1763559774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A9gOBVQm3SjNxY9RdaAh39uUQQ3mUpsdkBwekLQKP9Y=; b=3oQ/qhYKaF0PgUT4YbEfkFpTZLgu7Q8ruXo42XOTY7whd1Aa1mZbJckYDKcytMxSxsNUcO YACxUolDreGD066ZCKkVGML9J4FZwgoCby8dXJN/dHrg4ie41/x3864BBHeNP4SlOQ0NGl msdc1weXjLDFDcSMEDNVkn3tGLRuudk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LZPWhNTp; spf=pass (imf06.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1763559774; a=rsa-sha256; cv=none; b=kcJywHqXCxNTV05z4lZAp6E3ITLpU7DaenGR0zcbGMi6ofn4J5dQNb8BdNYdBgUPTgR1q7 lbSPcP8S7lkAjmc6Nvi/U9zyFFTWBSIMg4YTgBWwYWkGUfBK3aI4DjWPC3hJND3nMcZk48 5v9X1R85kfzS9nuJJqYpWP/ThNfGJJE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id AAFE3445CC; Wed, 19 Nov 2025 13:42:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0206DC4AF10; Wed, 19 Nov 2025 13:42:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1763559772; bh=P/9pmsmDZmkNlTyjbeOkKXUp9EgbKAdqcxRg8cTn9uY=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=LZPWhNTptIctgTzaFpTd1ekG9qrGfohBqAXAHzzaU/HV67myANBywQIS/3x/phUjm DYDIhDYRi2Ivwc7B5q7hVUqiI/vIfjCdDNCVRado+aVdcD0tmv7IO4i0D/u40bFrWT CjvOkJ2Z7lepRX0ifqfXVjCrMPqnStFWbLVY19g24sOYIuWTbQo0y1QAm4DprRGO1i aboRJ2HuKLHhg3MYwK/LN0Tzpuf27/kObDzlA9GsZ9k+CZgy9xrXEQJk0bKopSx1Dd Bv28ygxN1NRHIbPAah5Y1t2rJIGCMFf0WAQN8PfHquHssFUz8z2mldqA3t1Hi8XD3S jDXI73w0cwhrg== Date: Wed, 19 Nov 2025 15:42:45 +0200 From: Leon Romanovsky To: Christian =?iso-8859-1?Q?K=F6nig?= Cc: Bjorn Helgaas , Logan Gunthorpe , Jens Axboe , Robin Murphy , Joerg Roedel , Will Deacon , Marek Szyprowski , Jason Gunthorpe , Andrew Morton , Jonathan Corbet , Sumit Semwal , Kees Cook , "Gustavo A. R. Silva" , Ankit Agrawal , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Krishnakant Jaju , Matt Ochs , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, iommu@lists.linux.dev, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, kvm@vger.kernel.org, linux-hardening@vger.kernel.org, Alex Mastro , Nicolin Chen Subject: Re: [Linaro-mm-sig] [PATCH v8 06/11] dma-buf: provide phys_vec to scatter-gather mapping routine Message-ID: <20251119134245.GD18335@unreal> References: <20251111-dmabuf-vfio-v8-0-fd9aa5df478f@nvidia.com> <20251111-dmabuf-vfio-v8-6-fd9aa5df478f@nvidia.com> <8a11b605-6ac7-48ac-8f27-22df7072e4ad@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8a11b605-6ac7-48ac-8f27-22df7072e4ad@amd.com> X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: F3226180002 X-Stat-Signature: 1zfw6wizeb1r6wh8qjmysido6bedc1zj X-Rspam-User: X-HE-Tag: 1763559773-168988 X-HE-Meta: U2FsdGVkX196fMsygI9IAkhnsLzYPgUe+inn3QJWEsgLBPCvwReV7wH3tWQGiyMgvsqpjz50YPGc/d5J+wIYEATC0Gi9JeTSpCTBpg5BIrxUeTIzkBVDuk7FaudomSGzPqeOViSoOfRmAlxJtCbfk1v5cBlgUYCqY2UJtJujbdwOk9Y4tMI2O81tzJC0xZrI8bpiBOduUMy5GK8h5FFZtVNHatJSeAZhCic7Pf8Im4uL+jnVbLd59fmrB3870VFqBuliEOAytomy1uzRE+Fenrn7WAsY5cnibk0NDwwkXvBdvoX1K62rMAB8XSJtFpSHbVRV5OuIuXsOc5/64ANuDH5kW9HYOonnOXMF4jW4HfVYVIwP4E0oeX8qjbxoi3VscblKDVszzVsrRmw240b6QVv7lPUkOB32y6qLdL0CfRPcH9XiJimH86C7wiFcWTdObwydmFlcvtbnTLz6oSWQf8Qdf+WXbZzxhnw7+1WRYYgSNCbW9CZzALjrrbjfUqkqsqdV7tXBOD3wtnsQQWedbYdnzno5wGoY1NyF1rVYkYfP1hlnWPgiArq5ErT6BzvLaaIUY422EIKljia6Byc3DkAQ+NTFgpAHIOhDa6VfmokPJqxIM0oGTNMVg+mTZCytIKhuYb5Q1RbUWz0+diemfTUMLrTRfzOao69lCq8vUgUZZz1CynIPZ43xdqpCu/nafWDnd8obFaPi8aOZGK/WDSeLjQKbzqL8Po6FjREjShG2pZJZPmE3mK7p3WXzoAUfirp1XiP+HCbNT5oJo6TT3sCnxzZZb8inqulzw/Tc4mq9FfPwfAzaDbmiw/kAw5lAlc6CQzGhXlkhe7aOsVkSTbOX4VfqVGq31SGWPAfC0aHc9RIHyaCLhZ1ifPMsmfq/yye4SMFouyQnjx0H8dMbPP1zlJvm7JC1uFwtLqcJDryIy/NtBBnon45o5RZdjg9XYdXXOb5xsGt61M+w5GL NYr/CiOA SfDi5L6h9qTMPZsn9SIF2LeiXjmIBtphnRt2TlVaGOPG4/srgmin16YeHvxQycgi7m/7Cn9KA1AZTRR/cNAaxT4Clb2z+YPRbSZyiDbExnkI/OPp0T6Airn5pcywRZwmgcQQUx+ky5uPEuiEwQ+wXQ1hV6SwRh7DhqvFRri11/k1s0QpGNMYIqrqKPTyiBVXDUsmUFY6F+AjrRBT0W2b0kt9WmX1Twn+Yyc27oF4M35kzBsN0HkMhZZiJWS65lECw5fONKHuyz/EEki/o19jQ4zMZgRc72N39ySWHEvEV9uKM68Nyg274CmuFry43rTDHIC2NvKkUbKFgzHbQPtxo8UvQbszIbhocb6xyyhBlOEXZPGD25MG8maCeDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Nov 19, 2025 at 02:16:57PM +0100, Christian König wrote: > > > On 11/11/25 10:57, Leon Romanovsky wrote: > > From: Leon Romanovsky > > > > Add dma_buf_map() and dma_buf_unmap() helpers to convert an array of > > MMIO physical address ranges into scatter-gather tables with proper > > DMA mapping. > > > > These common functions are a starting point and support any PCI > > drivers creating mappings from their BAR's MMIO addresses. VFIO is one > > case, as shortly will be RDMA. We can review existing DRM drivers to > > refactor them separately. We hope this will evolve into routines to > > help common DRM that include mixed CPU and MMIO mappings. > > > > Compared to the dma_map_resource() abuse this implementation handles > > the complicated PCI P2P scenarios properly, especially when an IOMMU > > is enabled: > > > > - Direct bus address mapping without IOVA allocation for > > PCI_P2PDMA_MAP_BUS_ADDR, using pci_p2pdma_bus_addr_map(). This > > happens if the IOMMU is enabled but the PCIe switch ACS flags allow > > transactions to avoid the host bridge. > > > > Further, this handles the slightly obscure, case of MMIO with a > > phys_addr_t that is different from the physical BAR programming > > (bus offset). The phys_addr_t is converted to a dma_addr_t and > > accommodates this effect. This enables certain real systems to > > work, especially on ARM platforms. > > > > - Mapping through host bridge with IOVA allocation and DMA_ATTR_MMIO > > attribute for MMIO memory regions (PCI_P2PDMA_MAP_THRU_HOST_BRIDGE). > > This happens when the IOMMU is enabled and the ACS flags are forcing > > all traffic to the IOMMU - ie for virtualization systems. > > > > - Cases where P2P is not supported through the host bridge/CPU. The > > P2P subsystem is the proper place to detect this and block it. > > > > Helper functions fill_sg_entry() and calc_sg_nents() handle the > > scatter-gather table construction, splitting large regions into > > UINT_MAX-sized chunks to fit within sg->length field limits. > > > > Since the physical address based DMA API forbids use of the CPU list > > of the scatterlist this will produce a mangled scatterlist that has > > a fully zero-length and NULL'd CPU list. The list is 0 length, > > all the struct page pointers are NULL and zero sized. This is stronger > > and more robust than the existing mangle_sg_table() technique. It is > > a future project to migrate DMABUF as a subsystem away from using > > scatterlist for this data structure. > > > > Tested-by: Alex Mastro > > Tested-by: Nicolin Chen > > Signed-off-by: Leon Romanovsky > > --- > > drivers/dma-buf/dma-buf.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++ > > include/linux/dma-buf.h | 18 ++++ > > 2 files changed, 253 insertions(+) > > > > diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c > > index 2bcf9ceca997..cb55dff1dad5 100644 > > --- a/drivers/dma-buf/dma-buf.c > > +++ b/drivers/dma-buf/dma-buf.c > > @@ -1254,6 +1254,241 @@ void dma_buf_unmap_attachment_unlocked(struct dma_buf_attachment *attach, > > } > > EXPORT_SYMBOL_NS_GPL(dma_buf_unmap_attachment_unlocked, "DMA_BUF"); > > > > +static struct scatterlist *fill_sg_entry(struct scatterlist *sgl, size_t length, > > + dma_addr_t addr) > > +{ > > + unsigned int len, nents; > > + int i; > > + > > + nents = DIV_ROUND_UP(length, UINT_MAX); > > + for (i = 0; i < nents; i++) { > > + len = min_t(size_t, length, UINT_MAX); > > + length -= len; > > + /* > > + * DMABUF abuses scatterlist to create a scatterlist > > + * that does not have any CPU list, only the DMA list. > > + * Always set the page related values to NULL to ensure > > + * importers can't use it. The phys_addr based DMA API > > + * does not require the CPU list for mapping or unmapping. > > + */ > > + sg_set_page(sgl, NULL, 0, 0); > > + sg_dma_address(sgl) = addr + i * UINT_MAX; > > + sg_dma_len(sgl) = len; > > + sgl = sg_next(sgl); > > + } > > + > > + return sgl; > > +} > > + > > +static unsigned int calc_sg_nents(struct dma_iova_state *state, > > + struct dma_buf_phys_vec *phys_vec, > > + size_t nr_ranges, size_t size) > > +{ > > + unsigned int nents = 0; > > + size_t i; > > + > > + if (!state || !dma_use_iova(state)) { > > + for (i = 0; i < nr_ranges; i++) > > + nents += DIV_ROUND_UP(phys_vec[i].len, UINT_MAX); > > + } else { > > + /* > > + * In IOVA case, there is only one SG entry which spans > > + * for whole IOVA address space, but we need to make sure > > + * that it fits sg->length, maybe we need more. > > + */ > > + nents = DIV_ROUND_UP(size, UINT_MAX); > > + } > > + > > + return nents; > > +} > > + > > +/** > > + * struct dma_buf_dma - holds DMA mapping information > > + * @sgt: Scatter-gather table > > + * @state: DMA IOVA state relevant in IOMMU-based DMA > > + * @size: Total size of DMA transfer > > + */ > > +struct dma_buf_dma { > > + struct sg_table sgt; > > + struct dma_iova_state *state; > > + size_t size; > > +}; > > + > > +/** > > + * dma_buf_map - Returns the scatterlist table of the attachment from arrays > > + * of physical vectors. This funciton is intended for MMIO memory only. > > + * @attach: [in] attachment whose scatterlist is to be returned > > + * @provider: [in] p2pdma provider > > + * @phys_vec: [in] array of physical vectors > > + * @nr_ranges: [in] number of entries in phys_vec array > > + * @size: [in] total size of phys_vec > > + * @dir: [in] direction of DMA transfer > > + * > > + * Returns sg_table containing the scatterlist to be returned; returns ERR_PTR > > + * on error. May return -EINTR if it is interrupted by a signal. > > + * > > + * On success, the DMA addresses and lengths in the returned scatterlist are > > + * PAGE_SIZE aligned. > > + * > > + * A mapping must be unmapped by using dma_buf_unmap(). > > + */ > > +struct sg_table *dma_buf_map(struct dma_buf_attachment *attach, > > That is clearly not a good name for this function. We already have overloaded the term *mapping* with something completely different. This function performs DMA mapping, so what name do you suggest instead of dma_buf_map()? > > > + struct p2pdma_provider *provider, > > + struct dma_buf_phys_vec *phys_vec, > > + size_t nr_ranges, size_t size, > > + enum dma_data_direction dir) > > +{ > > + unsigned int nents, mapped_len = 0; > > + struct dma_buf_dma *dma; > > + struct scatterlist *sgl; > > + dma_addr_t addr; > > + size_t i; > > + int ret; > > + > > + dma_resv_assert_held(attach->dmabuf->resv); > > + > > + if (WARN_ON(!attach || !attach->dmabuf || !provider)) > > + /* This function is supposed to work on MMIO memory only */ > > + return ERR_PTR(-EINVAL); > > + > > + dma = kzalloc(sizeof(*dma), GFP_KERNEL); > > + if (!dma) > > + return ERR_PTR(-ENOMEM); > > + > > + switch (pci_p2pdma_map_type(provider, attach->dev)) { > > + case PCI_P2PDMA_MAP_BUS_ADDR: > > + /* > > + * There is no need in IOVA at all for this flow. > > + */ > > + break; > > + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: > > + dma->state = kzalloc(sizeof(*dma->state), GFP_KERNEL); > > + if (!dma->state) { > > + ret = -ENOMEM; > > + goto err_free_dma; > > + } > > + > > + dma_iova_try_alloc(attach->dev, dma->state, 0, size); > > Oh, that is a clear no-go for the core DMA-buf code. > > It's intentionally up to the exporter how to create the DMA addresses the importer can work with. I didn't fully understand the email either. The importer needs to configure DMA and it supports only MMIO addresses. Exporter controls it by asking for peer2peer. Thanks