From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22861C433F5 for ; Mon, 31 Jan 2022 19:00:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89EEC6B00BB; Mon, 31 Jan 2022 14:00:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 84C4A6B00C0; Mon, 31 Jan 2022 14:00:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ECA76B00C4; Mon, 31 Jan 2022 14:00:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay036.a.hostedemail.com [64.99.140.36]) by kanga.kvack.org (Postfix) with ESMTP id 601556B00BB for ; Mon, 31 Jan 2022 14:00:55 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 268FAEBD for ; Mon, 31 Jan 2022 19:00:55 +0000 (UTC) X-FDA: 79091499270.03.3DF65D7 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf10.hostedemail.com (Postfix) with ESMTP id AC7D9C0002 for ; Mon, 31 Jan 2022 19:00:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:content-disposition; bh=JfiBWAhkX1yHT92Ca969hKJaoVoe9ZEugeTA+o2gzlk=; b=l8gfX2giRWGgLu5RJY5tnXex3G naOg0DFHyGRH+WLiZNH6sHyFr4bWsG1rAvdlZ1fmScq1SMfZDFIEnUYH8VczuVNJAw9nbR2ZHg/oq yJTHQQQl/ejTUOl5M+8lgqwY+8MnixFCjR2Spi6TJ1qAU3tOAr6Zvchw6CRPXgL437qt1wS6Z172Y dkvB2bUHt9IHazdxNJlQRzlYq1Zun08wMxYQTlew19LVw6U3nB3ecjgtyBQp29pl1CavUC8RSaymJ GqmzMSm6w+nK1bSbSduZGxfQ/6O6c76ZYO2FoIihh1jEypL8oHnl8n01OJnsZT5sASWHH3nFELW0W Q3IdSFGw==; Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1nEbuV-008jbJ-R5; Mon, 31 Jan 2022 12:00:20 -0700 To: Jonathan Derrick , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell References: <20220128002614.6136-1-logang@deltatee.com> From: Logan Gunthorpe Message-ID: <08564685-e027-30b2-2c6e-9faf1d95dc7a@deltatee.com> Date: Mon, 31 Jan 2022 12:00:14 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: rcampbell@nvidia.com, ckulkarnilinux@gmail.com, martin.oliveira@eideticom.com, robin.murphy@arm.com, ira.weiny@intel.com, helgaas@kernel.org, jianxin.xiong@intel.com, dave.hansen@linux.intel.com, jason@jlekstrand.net, dave.b.minturn@intel.com, andrzej.jakowski@intel.com, daniel.vetter@ffwll.ch, willy@infradead.org, ddutile@redhat.com, jhubbard@nvidia.com, christian.koenig@amd.com, jgg@ziepe.ca, dan.j.williams@intel.com, hch@lst.de, sbates@raithlin.com, iommu@lists.linux-foundation.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, jonathan.derrick@linux.dev X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v5 00/24] Userspace P2PDMA with O_DIRECT NVMe devices X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: AC7D9C0002 X-Stat-Signature: yjwnn3z94gieagb4z6pik3dcyzhf7fnk X-Rspam-User: nil Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=l8gfX2gi; spf=pass (imf10.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1643655650-656519 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-01-31 11:56 a.m., Jonathan Derrick wrote: >> This is relatively straightforward, however the one significant >> problem is that, presently, pci_p2pdma_map_sg() requires a homogeneous >> SGL with all P2PDMA pages or all regular pages. Enhancing GUP to >> support enforcing this rule would require a huge hack that I don't >> expect would be all that pallatable. So patches 3 to 16 add >> support for P2PDMA pages to dma_map_sg[table]() to the dma-direct >> and dma-iommu implementations. Thus systems without an IOMMU plus >> Intel and AMD IOMMUs are supported. (Other IOMMU implementations would >> then be unsupported, notably ARM and PowerPC but support would be added >> when they convert to dma-iommu). > Am I understanding that an IO may use a mix of p2pdma and system pages? > Would that cause inconsistent latencies? Yes, that certainly would be a possibility. People developing applications that do such mixing would have to weight that issue if latency is something they care about. But it's counter productive and causes other difficulties for the kernel to enforce only homogenous IO. Logan