From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E124C28B30 for ; Thu, 20 Mar 2025 23:52:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6A5D280002; Thu, 20 Mar 2025 19:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A19EC280001; Thu, 20 Mar 2025 19:52:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BA33280002; Thu, 20 Mar 2025 19:52:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 70900280001 for ; Thu, 20 Mar 2025 19:52:42 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C8D751A0BBC for ; Thu, 20 Mar 2025 23:52:42 +0000 (UTC) X-FDA: 83243581764.28.C043C42 Received: from mailout2.w1.samsung.com (mailout2.w1.samsung.com [210.118.77.12]) by imf12.hostedemail.com (Postfix) with ESMTP id E8F5640004 for ; Thu, 20 Mar 2025 23:52:39 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=BxoLCTZs; spf=pass (imf12.hostedemail.com: domain of m.szyprowski@samsung.com designates 210.118.77.12 as permitted sender) smtp.mailfrom=m.szyprowski@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742514760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V40Yt8b5UQP5CQRebMaM4XT/r0qYRmMZ1DD0U7kvnK4=; b=A0+i9hYw4l4GivCCU/PDloibIvtxN8xS3eADQna2ZmYgOGUTJVs9+mgyWtIPjIFh8Jfuec Wt7yl/elPIXWKaPJyZdJ98e+sgaSgtpZN3a9+27Q1arM0YHLw8u1M+fOn8AdV7N69ST7JN ZgKVcdAsoRlE024ZmuP81R7QZfwhBho= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=samsung.com header.s=mail20170921 header.b=BxoLCTZs; spf=pass (imf12.hostedemail.com: domain of m.szyprowski@samsung.com designates 210.118.77.12 as permitted sender) smtp.mailfrom=m.szyprowski@samsung.com; dmarc=pass (policy=none) header.from=samsung.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742514760; a=rsa-sha256; cv=none; b=sgr1IcBXZUeIorsvhrj2XoRucAz1rD9jqFO9o6RLHpII7X4HxyWhalOVhPNLV3Ejo6NjZL 6LM3imxkCvD9Qt1dFYrhj7KR3mmzqYhrUmuDcQQLKFB10eWXs/3TcDc2Yy+KqW8izv2sw4 PZmR7N0+RaR2Zf1bp3ISMl15n4RBEWM= Received: from eucas1p2.samsung.com (unknown [182.198.249.207]) by mailout2.w1.samsung.com (KnoxPortal) with ESMTP id 20250320235237euoutp02bdf4b51e84861cdb2b223a7e9d23ae3e~up-CijgvZ1443514435euoutp02e for ; Thu, 20 Mar 2025 23:52:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.w1.samsung.com 20250320235237euoutp02bdf4b51e84861cdb2b223a7e9d23ae3e~up-CijgvZ1443514435euoutp02e DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1742514757; bh=V40Yt8b5UQP5CQRebMaM4XT/r0qYRmMZ1DD0U7kvnK4=; h=Date:Subject:To:Cc:From:In-Reply-To:References:From; b=BxoLCTZsFrr5HpiEFgfCFVGODCKIfKbX3HyvX+POAUOf1BV746F+VZ9lBX4StFK1r h6AsoerRXR3ato2teMWP1yIOhu0jFo3UIRvXTU8ypM2b97DBeOXYM6d00OQub9e+Bl R0YCeDeXMsDhLivNl2ZXcYa+NUs9LNC8ecizvuMg= Received: from eusmges1new.samsung.com (unknown [203.254.199.242]) by eucas1p2.samsung.com (KnoxPortal) with ESMTP id 20250320235236eucas1p2d25a78b8b13dc4ab1b2ca298ada0dfd6~up-BT_ZSP1299312993eucas1p2O; Thu, 20 Mar 2025 23:52:36 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges1new.samsung.com (EUCPMTA) with SMTP id 33.34.20821.44AACD76; Thu, 20 Mar 2025 23:52:36 +0000 (GMT) Received: from eusmtrp1.samsung.com (unknown [182.198.249.138]) by eucas1p2.samsung.com (KnoxPortal) with ESMTPA id 20250320235235eucas1p20083fac3310cb507f47cd27594507feb~up-Ah0Xfl1299312993eucas1p2N; Thu, 20 Mar 2025 23:52:35 +0000 (GMT) Received: from eusmgms1.samsung.com (unknown [182.198.249.179]) by eusmtrp1.samsung.com (KnoxPortal) with ESMTP id 20250320235235eusmtrp147af0699b5659e6df38f53ef4ac28c3e~up-AhBruB1931419314eusmtrp12; Thu, 20 Mar 2025 23:52:35 +0000 (GMT) X-AuditID: cbfec7f2-b09c370000005155-97-67dcaa4406f9 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms1.samsung.com (EUCPMTA) with SMTP id 4F.88.19920.34AACD76; Thu, 20 Mar 2025 23:52:35 +0000 (GMT) Received: from [106.210.134.192] (unknown [106.210.134.192]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20250320235231eusmtip12043beb3ef63386f7348b314310b8aad~up_9CwvKL2854028540eusmtip1R; Thu, 20 Mar 2025 23:52:31 +0000 (GMT) Message-ID: <1034b694-2b25-4649-a004-19e601061b90@samsung.com> Date: Fri, 21 Mar 2025 00:52:30 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7 00/17] Provide a new two step DMA mapping API To: Jason Gunthorpe Cc: Leon Romanovsky , Robin Murphy , Christoph Hellwig , Jens Axboe , Joerg Roedel , Will Deacon , Sagi Grimberg , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Randy Dunlap Content-Language: en-US From: Marek Szyprowski In-Reply-To: <20250319175840.GG10600@ziepe.ca> Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA01SbUxTVxjOuff2tjQpuRQcB2UjK5MFNgrdNJ5kwMCw5U7/4H5sGcviGrkW t1K1tYgk2zrGl11RKLCWwko3y4eF8C1gFYQiK4QJDCNiBTsTZEztBsJ0fAxHe3Xj3/M+z/N+ PCeHhws7yO28I4oTjFIhlYtIPtH508podLJtWhY71BiIqpobSfT4qY5EDTNnSWTNSUezfQUA nW8YxNDqMkIVxiGAKg3zGDpd2cJFesckQDWeC1xUVX4cla3W4KjH9Rr6Id9KoOv2KhLdaXzK QdW197jomtlJojlHEYGseZvIM20gUP/CLAc1PfiTQF03cjgod3o3qi3nJ4bSs/1mjG40NwJ6 7sYcoC1tanrsTitB5171cOj2+ij63OXfMfr6NTXdZjtN0m2P9Fx6yLhG0O3Wr+j59gpAX7ql IelzZ0o5KUGp/Lg0Rn4kk1HGJHzKTz/v6CWPbYRkDRdWExowEKQFfjxI7YITXeOYFvB5Qqoe wG8vDZJeQUgtA6hxxbPCEoC3NXr8eUeBuxBnhToA6w1zBFssAqgtrfa1C6gEaC4x+DBB7YSa xRKc5QPgcMUs4cXbqDDodhm5XhxIJcO+TtYTRIlgr72E6x2KU51caJpfxLwCTgVD12y1D5OU BGo9Wt8CP0oMS8bmnnnCYJen6tmpPXzo7E7RAt4mToZVRRKWDoT3nR1cFofCkVKdLwCkCgC0 rLkxtijezP+bC7Cut+D06CrpHYRTkbDZHsPSSXD8r7tcdr4/nPIEsCf4Q32nAWdpASzMF7Lu CGhyNv23tn98Ai8GItOWVzFtCWnaEsb0/14LIGwgmFGrMmSMSqJgTopV0gyVWiETHzqa0QY2 P/bIhvNRN/j+/qLYATAecADIw0VBgqDC2zKhIE16KptRHj2oVMsZlQPs4BGiYMGPV/JkQkom PcF8zjDHGOVzFeP5bddgqbkxO+oK9yxULue/+apiwhhge6XBP+lJZsIv4nj9njMD+6s5kfbo vZaQfR91L67Y4tt7i12Wla+L392bvYTVt7uN6Ilj6hvzzNquy1+EvDEzaD8QR6jOmsqOl4X1 JYa/3KLTW9HFn4u4bcHY+8v+fusvuK/8un7h1MmWOOvBT4bX/7l44O/3djo/nH67qzYl1uGp e5Bz8+qSOWUsemXI4Hp8C6xt8D+2CMOSbdnNqff+GI35kp7MGi6TK16aVC7dVH23sC8p0ZaZ Jw+u0EVQD2vsD/PQ67EfhN9NM4QuROoGWnts68ShEUVTXYdjqnX3O11Z5ZPWiM8U+qjD2yTp 4bkvHhYRqnSpJApXqqT/ApX9M15HBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02SfUxTZxTG8957e1uqXS61pi/oYDZxc+oqBVpflrZjH3HXLDMkhmRzG6zC tTRAi70tkS1mjWMCzZyrFoFKWrTFjNK5rY1CABvo1EoMQyCSwQAH1spYikgWEoXJKM0S/vvl nOc5z8nJ4eHCfk4qT6c3MUa9pkxC8om7L8KTb7zrndBmDNrTUPNPPhItrX5LovbJsyTynCpB kd4agNrab2Ho+T8INTXeAehiwyyG6i7+zEXnQqMAtcaucVFz/XFkf96Koxvje9Cl0x4CjXQ1 k2jKt8pBriuPuGjAGSZRNHSGQJ5v1ig20UCgvoUIB139+wmBOu6f4qDqCTm6Us/P3U5H+pwY 7XP6AB29HwV0i99MD079QtDVN2McOvDDbtrd8xdGjwyYab+3jqT9i+e49J3GZYIOeL6iZwNN gO4es5C0+7vznDzREanSaDCbmFdKDKxJJflEhjKlshwkzczOkcqy9n/2ZqZcsk+tLGbKdJWM cZ/6c2lJWyhIVrxIOdFf6yIs4FeRFSTxIJUNax7U4lbA5wmpVgB7Fn/EE43tsP+ChZPgLXBl 1EomRE8A9J5d5MYbAkoNnbYGMs4EtRNantrwRD0Z9jdFiDhvpdLhg/HGdf0W6j3Yez2hEVES GOyyceNDcaqTC4e6wlgi4QwOH12eX4/GKTEcj7iwOJOUDFpj1vW0JEoKbYNRLKFRQOs1K0hw OuyINePfA6FjwyKODaMcGyyODZYWQHiBiDGz5dpyViZlNeWsWa+VFhnK/WDtp67ffhboBM65 p9IQwHggBCAPl4gEoto/tEJBsabqC8ZoKDSayxg2BORr17DhqVuLDGtPqTcVyhQZclm2IidD nqPIkogFB0YGjwkprcbElDJMBWP834fxklItWOl0L9ItF7g+euwDqsX+HcaV9kO/hXVfpqpe L25r+1Q79afD61DOvR8eyh1Gy3vempoVpvk387pVWSbvRNrC2LG9p2fGYkdP7OrUFAqTu6Em N7Ksyg8etCuSH6fGQuik+8LDg/zjoV0dtfXBqzvu3VhoKSlYiKmml8r61GI3/m/KtqMVhw+3 z/w+XjbQ6vmw+jw7D6O5Kfbp2qqu18YqOUpD3jv5O5eKOh7e4yuH82sCz3RzH0xs6t58+/K2 7gGKSb/lLvDZxZbiHuPbwqG6m/MnZ1ZW1XLMtp8NCzqrXHurJ0eDw1/XZR3p0XNe8i+9vCk3 cDfPKa50lr5a+XG9hGBLNLLduJHV/AehhEaR3AMAAA== X-CMS-MailID: 20250320235235eucas1p20083fac3310cb507f47cd27594507feb X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-RootMTR: 20250228195423eucas1p221736d964e9aeb1b055d3ee93a4d2648 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20250228195423eucas1p221736d964e9aeb1b055d3ee93a4d2648 References: <20250220124827.GR53094@unreal> <1166a5f5-23cc-4cce-ba40-5e10ad2606de@arm.com> <20250312193249.GI1322339@unreal> <20250319175840.GG10600@ziepe.ca> X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E8F5640004 X-Stat-Signature: w3idxwdto3rg658eacq4xshjbeqjqdqc X-HE-Tag: 1742514759-975129 X-HE-Meta: U2FsdGVkX1957rjx3H9WoLIMdpbCysoAWTHNTejd55+mNmNqtKDHpn9Kyp3tL0rF0Fu42MHsZ22d8OOLBzXsenRKkS2VuNkcIwJxc3sfosTFuitNI5DMq1jIg1uhrYweoXiszxKyMFNnNq/vcsN+9Zx6u0DLMKH6KvSk7BxSHA3tHC91IR5nxA7o70OS0z2twW4F76M1Q6rUHbijHG5HyYMPcp7N9oCDmYEhkFoQyuW6bxLoNcpFd1Qa7A8Gwpmz9VC+l5t6n2qCGeZa8AopZb+U9ydVJhaRk6PhlR822g0T0Ch+T+pgyOtcpGsWwX352/TeytLKHORoO9ok362giOc3zq7EYUFhDYZiB32Y6MHlKhd2DRgm5Yb9zHd7heEGBI07x5LFvx8OEMzp6jGCqUVYxE7/CGb8NIoFyExnT4+7o+mVtvw6Bv3A1Cbg/kzMgDQ4B5uM4SvRiYDxNtOJqnvz6HAVOXfshDlvPdhbzUrzD02tbtSO74NQc/ZEvzz4pEtlOKq9wszp6NrIz6hrJCGsi/y3zrk1w9ZLpz5urp4D8p1S8+mWTA275mKBgmfX2XW2CTA0dLYqqGmeyoKc8PUMyb3pDzki52+IE61osFMgcYLdqcKvesCfTFNHcnxtLU5g2dvVgucwjI6Gw1kiCDjd+ja9DnNphiyNiu6jP4Tpe1swIMInoNO7Aap/GWmX656dAZWRr51RVlPlAgkyr0h25+YtJcyPRmGexjIJwRGetm+eZvnmxVEam/VVUpeUNtxxQ7KYdtu3wH51NfIru1FcnwdbYbqA/JK2pVHN4vvS4fGUF/hELiQxz7hbkJQaj/PwFEdfH01CWoxvfttqgUFak9tc3YX0X8lD2/Kneeiy63lrVFdc1Dnc9FnLHgF4HtundOHgrfG9Pwt6xoR9qKjzTsBSU6xuky0Sqz53QwvBSrRAFbaAyNz1wDFK5x/FuwSfXL+2cxN7NV5fgMo joRVIsYA 2wR4AXTVnO011ZYis7fBWI0ZWoScgpPznrHK0wjv0tnTrl08MJ3bCGNbl7jGbLP5f6fK7leAg0k8RBnnZVH7X4qPQgH3i7lq5Bhd/oiyZhA4Dm5MnryRe2RT5KSqvvD8WwWej X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 19.03.2025 18:58, Jason Gunthorpe wrote: > On Fri, Mar 14, 2025 at 11:52:58AM +0100, Marek Szyprowski wrote: > >>> The only way to do so is to use dma_map_sg_attrs(), which relies on SG >>> (the one that we want to remove) to map P2P pages. >> That's something I don't get yet. How P2P pages can be used with >> dma_map_sg_attrs(), but not with dma_map_page_attrs()? Both operate >> internally on struct page pointer. > It is a bit subtle, I ran in to this when exploring enabling proper > P2P for dma_map_resource() too. > > The API signatures are: > > dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, > size_t offset, size_t size, enum dma_data_direction dir, > unsigned long attrs); > void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, > enum dma_data_direction dir, unsigned long attrs); > > The thing to notice immediately is that the unmap path does not get > passed a struct page. I see, thanks for pointing this out. It looks that I've encountered this issue too long time ago when I was implementing first iommu-to-dma-mapping glue for ARM arch. The lack of some kind of 'object' returned from dma_map* (and dma_alloc*) and passed back to dma_unmap* (and dma_free_*) required some non-trivial workarounds there and made the unmap/free path a bit more complicated. > So, lets think about the flow when the iommu is turned on. > > For normal struct page memory: > > - dma_map_page_attrs() allocates some IOVA and returns it in the > dma_addr_t and then maps the struct page to the iommu page table > > - dma_unmap_page_attrs() frees the IOVA from the given dma_addr_t > > If we think about P2P now: > > - dma_map_page_attrs() can inspect the struct page and determine it > is P2P. It computes a bus address which is not an IOVA, and does > not transit through the IOMMU. No IOVA allocation is performed. the > bus address is returned as the dma_addr_t > > - dma_unmap_page_attrs() ... is impossible. We just get this > dma_addr_t that doesn't have enough information to tell anymore if > the address is a P2P bus address or not, so we can't tell if we > should unmap an iova from the dma_addr_t :\ > > The sg path fixes this because it introduced a new flag in the > scatterlist, SG_DMA_BUS_ADDRESS, that allows the sg map path to record > the information for the unmap path so it can do the right thing. > > Leon's approach fixes this by putting an overarching transaction state > around the DMA operation so that map and unmap operations can look in > the state and determine if this is a P2P or non P2P map and then know > how to unmap. > > For some background here, Christoph gave me this idea back at LSF/MM > in Vancouver (two years ago now). At the time I was looking at > replacing scatterlist and giving new DMA API ops to operate on a > "scatterlist v2" structure. > > Christoph's vision was to make a performance DMA API path that could > be used to implement any scatterlist-like data structure very > efficiently without having to teach the DMA API about all sorts of > scatterlist-like things. Thanks for explaining one more motivation behind this patchset! Best regards -- Marek Szyprowski, PhD Samsung R&D Institute Poland