From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A631C43461 for ; Tue, 11 May 2021 16:05:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 89DA66191E for ; Tue, 11 May 2021 16:05:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 89DA66191E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CE35F6B0072; Tue, 11 May 2021 12:05:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C94236B0073; Tue, 11 May 2021 12:05:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0D908D0002; Tue, 11 May 2021 12:05:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0147.hostedemail.com [216.40.44.147]) by kanga.kvack.org (Postfix) with ESMTP id 8E7C26B0072 for ; Tue, 11 May 2021 12:05:31 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3E07299BD for ; Tue, 11 May 2021 16:05:31 +0000 (UTC) X-FDA: 78129425262.11.E894DD0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 78FB0A00019A for ; Tue, 11 May 2021 16:05:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620749130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=te4j0CaO/GGwAePTvy9qf9MhnS54BQnLL/PFFRuQ1v8=; b=Hqky/gW5/OFbT7DPxguflfM8JByEgMHMok5nvYo0xSUbuSyiLJ75b5Q+KvKCqPmEPd8BlW L44pD8y0byUYw1zq4enIT3NEpPLkjjImRUq8PhACbHCPFKXWKP5rhZdV4Lu4X2YcBnnY7P b30kc5nIP9L+F/jdMl5W9aqNrBzsdeU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-362-36MVrBJfO1-KLhYh32wKtA-1; Tue, 11 May 2021 12:05:25 -0400 X-MC-Unique: 36MVrBJfO1-KLhYh32wKtA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2629D805F03; Tue, 11 May 2021 16:05:22 +0000 (UTC) Received: from [10.3.115.19] (ovpn-115-19.phx2.redhat.com [10.3.115.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 895785D9F2; Tue, 11 May 2021 16:05:17 +0000 (UTC) Subject: Re: [PATCH 00/16] Add new DMA mapping operation for P2PDMA To: Logan Gunthorpe , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , John Hubbard , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy References: <20210408170123.8788-1-logang@deltatee.com> From: Don Dutile Message-ID: Date: Tue, 11 May 2021 12:05:16 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <20210408170123.8788-1-logang@deltatee.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Hqky/gW5"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf23.hostedemail.com: domain of ddutile@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=ddutile@redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 78FB0A00019A X-Stat-Signature: 5x566b1hz7tzoryk7or764byj4szzkcb Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=170.10.133.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620749120-499373 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/8/21 1:01 PM, Logan Gunthorpe wrote: > Hi, > > This patchset continues my work to to add P2PDMA support to the common > dma map operations. This allows for creating SGLs that have both P2PDMA > and regular pages which is a necessary step to allowing P2PDMA pages in > userspace. > > The earlier RFC[1] generated a lot of great feedback and I heard no sho= w > stopping objections. Thus, I've incorporated all the feedback and have > decided to post this as a proper patch series with hopes of eventually > getting it in mainline. > > I'm happy to do a few more passes if anyone has any further feedback > or better ideas. > > This series is based on v5.12-rc6 and a git branch can be found here: > > https://github.com/sbates130272/linux-p2pmem/ p2pdma_map_ops_v1 > > Thanks, > > Logan > > [1] https://lore.kernel.org/linux-block/20210311233142.7900-1-logang@de= ltatee.com/ > > > Changes since the RFC: > * Added comment and fixed up the pci_get_slot patch. (per Bjorn) > * Fixed glaring sg_phys() double offset bug. (per Robin) > * Created a new map operation (dma_map_sg_p2pdma()) with a new callin= g > convention instead of modifying the calling convention of > dma_map_sg(). (per Robin) > * Integrated the two similar pci_p2pdma_dma_map_type() and > pci_p2pdma_map_type() functions into one (per Ira) > * Reworked some of the logic in the map_sg() implementations into > helpers in the p2pdma code. (per Christoph) > * Dropped a bunch of unnecessary symbol exports (per Christoph) > * Expanded the code in dma_pci_p2pdma_supported() for clarity. (per > Ira and Christoph) > * Finished off using the new dma_map_sg_p2pdma() call in rdma_rw > and removed the old pci_p2pdma_[un]map_sg(). (per Jason) > > -- > > Logan Gunthorpe (16): > PCI/P2PDMA: Pass gfp_mask flags to upstream_bridge_distance_warn() > PCI/P2PDMA: Avoid pci_get_slot() which sleeps > PCI/P2PDMA: Attempt to set map_type if it has not been set > PCI/P2PDMA: Refactor pci_p2pdma_map_type() to take pagmap and device > dma-mapping: Introduce dma_map_sg_p2pdma() > lib/scatterlist: Add flag for indicating P2PDMA segments in an SGL > PCI/P2PDMA: Make pci_p2pdma_map_type() non-static > PCI/P2PDMA: Introduce helpers for dma_map_sg implementations > dma-direct: Support PCI P2PDMA pages in dma-direct map_sg > dma-mapping: Add flags to dma_map_ops to indicate PCI P2PDMA support > iommu/dma: Support PCI P2PDMA pages in dma-iommu map_sg > nvme-pci: Check DMA ops when indicating support for PCI P2PDMA > nvme-pci: Convert to using dma_map_sg_p2pdma for p2pdma pages > nvme-rdma: Ensure dma support when using p2pdma > RDMA/rw: use dma_map_sg_p2pdma() > PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() > > drivers/infiniband/core/rw.c | 50 +++------- > drivers/iommu/dma-iommu.c | 66 ++++++++++-- > drivers/nvme/host/core.c | 3 +- > drivers/nvme/host/nvme.h | 2 +- > drivers/nvme/host/pci.c | 39 ++++---- > drivers/nvme/target/rdma.c | 3 +- > drivers/pci/Kconfig | 2 +- > drivers/pci/p2pdma.c | 188 +++++++++++++++++++---------------= - > include/linux/dma-map-ops.h | 3 + > include/linux/dma-mapping.h | 20 ++++ > include/linux/pci-p2pdma.h | 53 ++++++---- > include/linux/scatterlist.h | 49 ++++++++- > include/rdma/ib_verbs.h | 32 ++++++ > kernel/dma/direct.c | 25 ++++- > kernel/dma/mapping.c | 70 +++++++++++-- > 15 files changed, 416 insertions(+), 189 deletions(-) > > > base-commit: e49d033bddf5b565044e2abe4241353959bc9120 > -- > 2.20.1 > Apologies in the delay to provide feedback; climbing out of several deep = trenches at the mother ship :-/ Replying to some directly, and indirectly (mostly through JohH's reply's)= . General comments: 1) nits in 1,2,3,5; =C2=A0=C2=A0 4: I agree w/JohnH & JasonG -- seems like it needs a device= -layer that gets to a bus-layer, but I'm wearing my 'broader then PCI' ha= t in this review; I see a (classic) ChristophH refactoring and cleanup in= this area, and wondering if we ought to clean it up now, since CH has do= ne so much to clean it up and make the dma-mapping system so much easier = to add/modify/review due to the broad arch (& bus) cleanup that has been = done.=C2=A0 If that delays it too much, then add a TODO to do so. 2) 6: yes! let's not worry or even both supporting 32-bit anything wrt p2= pdma. 3) 7:nit 4) 8: ok; 5) 9: ditto to JohnH's feedback on added / clearer comment & code flow (i= f-else). 6) 10: nits; q: should p2pdma mapping go through dma-ops so it is general= ized for future interconnects (CXL, GenZ)? 7) 11: It says it is supporting p2pdma in dma-iommu's map_sg, but it seem= s like it is just leveraging shared code and short-circuiting IOMMU use. 8) 12-14: didn't review; letting the block/nvme/direct-io folks cover thi= s space 9) 15: Looking to JasonG to sanitize 10) 16: cleanup; a-ok. - DonD