From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DE58C35FF1 for ; Fri, 14 Mar 2025 18:49:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B34C5280004; Fri, 14 Mar 2025 14:49:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABDC6280001; Fri, 14 Mar 2025 14:49:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95F14280004; Fri, 14 Mar 2025 14:49:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7837C280001 for ; Fri, 14 Mar 2025 14:49:18 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4EEF056D76 for ; Fri, 14 Mar 2025 18:49:19 +0000 (UTC) X-FDA: 83221044438.26.A540FBB Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 90E26100006 for ; Fri, 14 Mar 2025 18:49:17 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=rQVmjcjZ; spf=pass (imf05.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741978157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=meG9ZA3MJCgBc8xoMq4fmTlRmYixqZc1cv6C0aMOIH4=; b=eIaMKKAByrPd0wBpHFHkXPn2zj5jBhep5ZlLjTn9w1LkS4MksPBP6j1NdvE9l6NsJZ3lTH ui7XCOFGlZn0UEsswMYttbSLQdu5/6MZSATeL1ixCMSnGQ7rDoa+oAOpLOFWixnOwpntRy tVj77cj6fvBsh1cpE2Gopo+Zx3Gn66g= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=rQVmjcjZ; spf=pass (imf05.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741978157; a=rsa-sha256; cv=none; b=h07OZHKc+Njilwd8T9Bj+1NoIcsRStZ7P5M7sI7Tp6e5kEBftZOqeY+a5f+OrpnOjRZcII 9N78EKSeVX95ulYS/sXjHYNGDvwqCbLq1xkWgV5jm8WoYGKqEqMFJoe0x035n/pThc8Gsg Jp9CRKyZr6uR8DTnxo0h81UOoOPkZ/s= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id A08A05C4C71; Fri, 14 Mar 2025 18:46:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C4A3C4CEE9; Fri, 14 Mar 2025 18:49:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741978156; bh=O6TQHcNZOIDORUQUex99gbjkbXRSAsx+LziCN6QYCcQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=rQVmjcjZhe/sj5yONxV2qVxL98CsQ/XLSI8I1aZI9aDjryLW+sVGMqcfI9Bca9bOS BjeRuzyi8Km8hWOU7pWxLh0zSj3Q5xmQc8mdYFqkiDMoJN2jPM4GlDtxa49auZ9+wv o4Atu8WInpcl8PAepC/fXyU6iiNJ15uue0LdQRbYK1nJzWh83j90x9tnoa0sRasVK1 38iJcCj/HUFJNu/wqR3UGB0O36uPS2R51CHZ5YoiakTWuHP1tXd0Pvob4wtZn293CI sVep3kKdHjc3A4MK0/PKcdqXaMCX1hnu3s63bSmSw7dabMqNw6WwX2YJoxC1T+nsVF 6r+AVkQEbO0Yg== Date: Fri, 14 Mar 2025 20:49:11 +0200 From: Leon Romanovsky To: Marek Szyprowski Cc: Robin Murphy , Christoph Hellwig , Jason Gunthorpe , Jens Axboe , Joerg Roedel , Will Deacon , Sagi Grimberg , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Randy Dunlap Subject: Re: [PATCH v7 00/17] Provide a new two step DMA mapping API Message-ID: <20250314184911.GR1322339@unreal> References: <20250220124827.GR53094@unreal> <1166a5f5-23cc-4cce-ba40-5e10ad2606de@arm.com> <20250312193249.GI1322339@unreal> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Stat-Signature: s3htq8r3fx8ahjmnorui3t7s5y1mgn8z X-Rspamd-Queue-Id: 90E26100006 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1741978157-675942 X-HE-Meta: U2FsdGVkX1+0Y3ab5yBl7cUtFFqIvIlOuBilGoB5e+50v0QYF6knQ36biW6HFnju0FU9IT+4woNb2S7QhkX4TiVVeEinbAGtuk7ajGNHhJDhzNHco39eX+HRktRBa0kEhcTWH+IWZnTBGXM976xvC5T20sTvrL0mk3nz7Pmy2F8+DZx1RoVUjNt7mXJpNqItHbQrA/Kz3em1o0WMdUciDCtRPYk4QxSmr8aXn4maWEZNa/mpu41Nf248q6qD4Y0eiPj4RB9rm7oApg1pPiU3zmlJn/NQQDojV6GKVDjn+e+eHRRckKby/j2CYkiG7+f86lmzevmfJytHS+wWR1AV7ENPDhFXo0CvAmcIMgR60C20W27V0NzLPkCQO6pii18MSYS3KVN4UVeGTlCKV/cpJOALQUusP7BE6UUQyvgT0jLetwfTp3FlDORpgDlVZCiSPHyJGc2Jyye3dacP2X9QQET3tiMu7PfSA+Jn//hztwIGJkl82oix9BR3nM8nNftwR1/GI6jfI6jbDfVToVDxK3nMzxZt6jHju7a67TRnC24NY0iYkbOSEh7TE8nG6U6kF7hV408qx/ZZXg/D3dVgne4MPk2D+x9JN73hWdOeCaENd+nrXhf0cIuKTIXDiCTjpJ+CRZn0VNgGhr1J6yOaS7ycc/6qzKkxeKsgMJg/olH3YAgklpymmDOOaUx4raCu7Rf7B9inyhakkugJ4hvM28jMumcFcVfxTVuUVLc51v9qUTWwOETyveWmgHOR7L2M6yVDECJE6TOo5zOYzUxNxNtnJha1W/Nfv2Z5+6HI6d/l+iW8jYpfgjmuP0Rj6BWv6Gf4uhW93ysKMD7X9IZ0qcJKW+IuIkiR1couMXrssItGKesYwJyqVG0E49C6RTM6L9ZoCdglmakb2XYOCo2iz8frGj+Kn+485yUQ2N8JfFrs4JWoWazE5yYOxn6LgcKlheiUl8XdVs486IB5wgG 7ykYsfrN v0/ITL+pNmSe0xnIcSaiM9NHM+qGvSf8Bv3CEzFTIStc8leJAto5W3cQt+NXhlNj6e4+EP33JfCqlzM3NlNWfFW9Jk3fj1J6dWYTt//Fvmey3mKC3QjAJ1rWlqSQBjkn5/5AkRx+8gsaRQr/JJnj0s9sjgx9hwz9GwGVlm6SuO8d9u8j01QEAUINXCmp4+KLh2rerQic0k9XffJzW5PjAH+a1cWxOmyf91CWkjTO80zV8ZuKtf7V0d/aKe0kXdzQ3+JgHqS794z9MsFwjSOlALzzHFAt1m5cgVFTaJNHH96gyO5PqXn2SwPSgzNVcz8nGvB+SZQdE34CKq/Vl+biPvjhRhZhDznI3GZEL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Mar 14, 2025 at 11:52:58AM +0100, Marek Szyprowski wrote: > On 12.03.2025 20:32, Leon Romanovsky wrote: > > On Wed, Mar 12, 2025 at 10:28:32AM +0100, Marek Szyprowski wrote: > >> Hi Robin > >> > >> On 28.02.2025 20:54, Robin Murphy wrote: > >>> On 20/02/2025 12:48 pm, Leon Romanovsky wrote: > >>>> On Wed, Feb 05, 2025 at 04:40:20PM +0200, Leon Romanovsky wrote: > >>>>> From: Leon Romanovsky > >>>>> > >>>>> Changelog: > >>>>> v7: > >>>>>   * Rebased to v6.14-rc1 > >>>> <...> > >>>> > >>>>> Christoph Hellwig (6): > >>>>>    PCI/P2PDMA: Refactor the p2pdma mapping helpers > >>>>>    dma-mapping: move the PCI P2PDMA mapping helpers to pci-p2pdma.h > >>>>>    iommu: generalize the batched sync after map interface > >>>>>    iommu/dma: Factor out a iommu_dma_map_swiotlb helper > >>>>>    dma-mapping: add a dma_need_unmap helper > >>>>>    docs: core-api: document the IOVA-based API > >>>>> > >>>>> Leon Romanovsky (11): > >>>>>    iommu: add kernel-doc for iommu_unmap and iommu_unmap_fast > >>>>>    dma-mapping: Provide an interface to allow allocate IOVA > >>>>>    dma-mapping: Implement link/unlink ranges API > >>>>>    mm/hmm: let users to tag specific PFN with DMA mapped bit > >>>>>    mm/hmm: provide generic DMA managing logic > >>>>>    RDMA/umem: Store ODP access mask information in PFN > >>>>>    RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page > >>>>>      linkage > >>>>>    RDMA/umem: Separate implicit ODP initialization from explicit ODP > >>>>>    vfio/mlx5: Explicitly use number of pages instead of allocated > >>>>> length > >>>>>    vfio/mlx5: Rewrite create mkey flow to allow better code reuse > >>>>>    vfio/mlx5: Enable the DMA link API > >>>>> > >>>>>   Documentation/core-api/dma-api.rst   |  70 ++++ > >>>>   drivers/infiniband/core/umem_odp.c   | 250 +++++--------- > >>>>>   drivers/infiniband/hw/mlx5/mlx5_ib.h |  12 +- > >>>>>   drivers/infiniband/hw/mlx5/odp.c     |  65 ++-- > >>>>>   drivers/infiniband/hw/mlx5/umr.c     |  12 +- > >>>>>   drivers/iommu/dma-iommu.c            | 468 > >>>>> +++++++++++++++++++++++---- > >>>>>   drivers/iommu/iommu.c                |  84 ++--- > >>>>>   drivers/pci/p2pdma.c                 |  38 +-- > >>>>>   drivers/vfio/pci/mlx5/cmd.c          | 375 +++++++++++---------- > >>>>>   drivers/vfio/pci/mlx5/cmd.h          |  35 +- > >>>>>   drivers/vfio/pci/mlx5/main.c         |  87 +++-- > >>>>>   include/linux/dma-map-ops.h          |  54 ---- > >>>>>   include/linux/dma-mapping.h          |  85 +++++ > >>>>>   include/linux/hmm-dma.h              |  33 ++ > >>>>>   include/linux/hmm.h                  |  21 ++ > >>>>>   include/linux/iommu.h                |   4 + > >>>>>   include/linux/pci-p2pdma.h           |  84 +++++ > >>>>>   include/rdma/ib_umem_odp.h           |  25 +- > >>>>>   kernel/dma/direct.c                  |  44 +-- > >>>>>   kernel/dma/mapping.c                 |  18 ++ > >>>>>   mm/hmm.c                             | 264 +++++++++++++-- > >>>>>   21 files changed, 1435 insertions(+), 693 deletions(-) > >>>>>   create mode 100644 include/linux/hmm-dma.h > >>>> Kind reminder. > > <...> > > > >> Removing the need for scatterlists was advertised as the main goal of > >> this new API, but it looks that similar effects can be achieved with > >> just iterating over the pages and calling page-based DMA API directly. > > Such iteration can't be enough because P2P pages don't have struct pages, > > so you can't use reliably and efficiently dma_map_page_attrs() call. > > > > The only way to do so is to use dma_map_sg_attrs(), which relies on SG > > (the one that we want to remove) to map P2P pages. > > That's something I don't get yet. How P2P pages can be used with > dma_map_sg_attrs(), but not with dma_map_page_attrs()? Both operate > internally on struct page pointer. Yes, and no. See users of is_pci_p2pdma_page(...) function. In dma_*_sg() APIs, there is a real check and support for p2p. In dma_map_page_attrs() variants, this support is missing (ignored, or error is returned). > > >> Maybe I missed something. I still see some advantages in this DMA API > >> extension, but I would also like to see the clear benefits from > >> introducing it, like perf logs or other benchmark summary. > > We didn't focus yet on performance, however Christoph mentioned in his > > block RFC [1] that even simple conversion should improve performance as > > we are performing one P2P lookup per-bio and not per-SG entry as was > > before [2]. In addition it decreases memory [3] too. > > > > [1] https://lore.kernel.org/all/cover.1730037261.git.leon@kernel.org/ > > [2] https://lore.kernel.org/all/34d44537a65aba6ede215a8ad882aeee028b423a.1730037261.git.leon@kernel.org/ > > [3] https://lore.kernel.org/all/383557d0fa1aa393dbab4e1daec94b6cced384ab.1730037261.git.leon@kernel.org/ > > > > So clear benefits are: > > 1. Ability to use native for subsystem structure, e.g. bio for block, > > umem for RDMA, dmabuf for DRM, e.t.c. It removes current wasteful > > conversions from and to SG in order to work with DMA API. > > 2. Batched request and iotlb sync optimizations (perform only once). > > 3. Avoid very expensive call to pgmap pointer. > > 4. Expose MMIO over VFIO without hacks (PCI BAR doesn't have struct pages). > > See this series for such a hack > > https://lore.kernel.org/all/20250307052248.405803-1-vivek.kasireddy@intel.com/ > > I see those benefits and I admit that for typical DMA-with-IOMMU case it > would improve some things. I think that main concern from Robin was how > to handle it for the cases without an IOMMU. In such case, we fallback to non-IOMMU flow (old, well-established one). See this HMM patch as an example https://lore.kernel.org/all/a796da065fa8a9cb35d591ce6930400619572dcc.1738765879.git.leonro@nvidia.com/ +dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, + size_t idx, + struct pci_p2pdma_map_state *p2pdma_state) ... + if (dma_use_iova(state)) { ... + } else { ... + dma_addr = dma_map_page(dev, page, 0, map->dma_entry_size, + DMA_BIDIRECTIONAL); Thanks > > Best regards > -- > Marek Szyprowski, PhD > Samsung R&D Institute Poland >