From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ED4DD4335D for ; Thu, 7 Nov 2024 13:28:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCD506B008C; Thu, 7 Nov 2024 08:28:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D7CEA6B0092; Thu, 7 Nov 2024 08:28:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1D226B0096; Thu, 7 Nov 2024 08:28:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A2E3B6B008C for ; Thu, 7 Nov 2024 08:28:14 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1DEDA120715 for ; Thu, 7 Nov 2024 13:28:14 +0000 (UTC) X-FDA: 82759376658.22.E20202B Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by imf12.hostedemail.com (Postfix) with ESMTP id DF82E40008 for ; Thu, 7 Nov 2024 13:27:55 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Z3jSxjRK; dmarc=none; spf=pass (imf12.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.210.49 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730986032; a=rsa-sha256; cv=none; b=mK854N+MSLZHTlUuRwrc16TF8H5+Uz3Jar6y31SAYR0J+3PcXbLL3Em20cUIoFLq5mrddb FyUE6eSfJ3kZC+6iwtpPkkOOSGKelaJ6xWlonj7h1ui+emyp9qEUeRYTMlQXJItJ1UiqGI 6k2vrq4fiN2aE5UcM3nF8TsGe/+zp9s= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Z3jSxjRK; dmarc=none; spf=pass (imf12.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.210.49 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730986032; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wLc6Cp3t0rRD6tSmNJMVeZRKi1v5LF88UUzndPTrhYc=; b=pGWldYASoexR8mT+OenXXaBNA6Eie152QWm7eySKWHcmVE13VOqC7zYkkroTHwc0jNtvaK JH0ggoK1/haIwl9+lObzWstDIixS5OuW3YI2bG9x7vcT64X2uCJ/+ULRKSF4mXS41tPn1L LqR9l9pigIa5m/+yTA6YQ8EDZlirh7w= Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-7181b86a749so489980a34.3 for ; Thu, 07 Nov 2024 05:28:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1730986091; x=1731590891; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wLc6Cp3t0rRD6tSmNJMVeZRKi1v5LF88UUzndPTrhYc=; b=Z3jSxjRKif/6F+wQ8irjwKKpv1XcpXsWqgoitqVEEC3AZQDMYCdKHHXiVq1KYyrg0X Jgeu5buuJq8HJfuKuLRG9TmLygKJ86VjHEDkInADjaxNf3xJOU6Te9ToJdaZVu/p7hxG DLmtkNpG5sPixgaERukwdmTQIPoBeXnOFQhNbKhAiv2ZZNUhXxtY/+5SymH2L3x6o50r hm+GADpmWslIi0/TOkMUXvfFch5p8QawpbH4lDXXX2gbdDYgnlllCEJ1KFdqCk6LBw4n SpWE82dyVXnOL7Pc16kOIg1wyEiqtA7umkf+FfiJAj08APiwnIcA116kSaBHs+CxHRca 7xxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730986091; x=1731590891; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=wLc6Cp3t0rRD6tSmNJMVeZRKi1v5LF88UUzndPTrhYc=; b=hKflpzdEICbb+dzLNQ38nGIpUf2aXMBfrv8pSfFe4PONWZHwm/Cdb8SVTvm6DlayeI 1YMW4oRUyC5w3cBJLhncCr8Bn+yQlt5SmUqNSexH59vMkWU1BF9UG14kJ4f+Yc/8ERNn yjvpY1zNuJVL+KByzQS678YnoPGyJxYxKzbtd/DESSCjujnAWO5Pe8cU2P/GZQG+C/qL EahSVMRweW6Mz5jWbOP+jiYS0t17EHNL4qsOQdGHd1D19pktiEkbLATpAxFBJBP6wsHa PUx8y7Wf8gO6rCe3OdhUo3U0+8AE7Ry6dcf6eVxlY8qgp0D+WPEgGkYWBZ1ovd3dCBUj Ra6g== X-Forwarded-Encrypted: i=1; AJvYcCWyHYsza9NdKE13ZhluInXcF/4z803vkwcJjEMwWP0XToh0VGyJrGnLCaUo+nYnbMjYcwDXooXCxA==@kvack.org X-Gm-Message-State: AOJu0Yytca2a4Ykv/704gsfrcsC6LchLISvjqcN8Iclqm3qFL/Fg5F8C p+lgPrc1pV5fuk7IkJl3QkRcz/y4ZWevaPBOt7LpLTO/TKVfBW/wQ3a5dRit2iU= X-Google-Smtp-Source: AGHT+IF+tw+3lLnZf4WnNvvo/ldJVal46tJah/6dRwkTid9yDCpRNCjr9NuPNVZcVYNIHAZNA7U8ew== X-Received: by 2002:a05:6358:9696:b0:1c3:39ad:7c6c with SMTP id e5c5f4694b2df-1c5f99fec97mr1284772555d.17.1730986090941; Thu, 07 Nov 2024 05:28:10 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6d3961ecc4esm7389226d6.44.2024.11.07.05.28.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Nov 2024 05:28:09 -0800 (PST) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1t92YS-00000002FM3-2zqJ; Thu, 07 Nov 2024 09:28:08 -0400 Date: Thu, 7 Nov 2024 09:28:08 -0400 From: Jason Gunthorpe To: Christoph Hellwig Cc: Robin Murphy , Leon Romanovsky , Jens Axboe , Joerg Roedel , Will Deacon , Sagi Grimberg , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v1 00/17] Provide a new two step DMA mapping API Message-ID: <20241107132808.GK35848@ziepe.ca> References: <3567312e-5942-4037-93dc-587f25f0778c@arm.com> <20241104095831.GA28751@lst.de> <20241105195357.GI35848@ziepe.ca> <20241107083256.GA9071@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20241107083256.GA9071@lst.de> X-Rspam-User: X-Rspamd-Queue-Id: DF82E40008 X-Rspamd-Server: rspam11 X-Stat-Signature: tfjqegodg74m83r5eg49mu9ykeh4te49 X-HE-Tag: 1730986075-159871 X-HE-Meta: U2FsdGVkX1+2Nl+HVYk+JK2XD0gaVC5G1a2quUlETDUvqcyLqiQhPHs1/duSmX0C/BLx/1JFnkMyn/2vqKUY1wsxk4m2uy1gCHZUOsDxivU6SLdvmR/ajjTSYO3raofPmEw3W6WHCUkBmxx3Zp7XsVm0fuEcrJuuDZtTGlMpS/i+5YplPG6o229WlOWSyCMAy1LQ685AcCy/I00qb925w8tsEGbefgGMNh6+NFWNIVuT1m2GhlOFLB5yLJNelGSO1paOGVSYSmw6kIZmiN4PH/DHkSt5e16nAroTNt8S0Bin7mySP1b8+n7ZS6a0jHO+2DvR53CIvQMP4i10+TmIt0Fs7DjL6wOsog2vgLvW4C7vm7SR9eHP6VHoVMCo6+0f54+XXJS5GHnaWoShAwalCte0sNycNKnn6OH2b01v5770+186E37G7IK2H8s12REBAKrk3YQkrPJv+kUoyAttvwFMbETHhGNJP59lBG2Y7SzFlaKBJBzh932KK8Z9s/DvDEBj+XwMG3Ydh3Z1INYOg8pUKQJ1Wi+gXU9aVBc7gB8r9bF/zlC5Cr7r1jM/+og/ZgLNFvHcnsaXNONs1darRWCk3EEC6pPLCIksx+JhEim6qQaa9htQg30YJ0aYUhWhOQaOK2LRZBmTCipaJTs1sSuVfknRMG6FaHsJw488EwqUgALeSj4zQfAU+Uxhdo2mhfiU89NgPWwcmZYXgxnaNpgVQhEioy9jUzX8JkWJr1ckPJK6qixtXIab7emYg5y/59/CcEvGSbaWz9OJtpQB/NyicW9Lq/GknOr6lr9bTgTS4pimZ+w3z+cJG7K8c2EA4htlvs4XLq+KI5jWOcc3SK1MVv0uyOBH0WF1rYq7KxOo9QBN1yEjH10H3GLtzTa1MQuZB2qRrkvpFRLuorsKXVESyURxaJsXJ6fqL1nA79kAw0amwqR8lh1xOt5/8Dm+rCWG3mnZbTbbiEgF8ln 1lT+Y4nJ 5WXYlYyy0LedkaVjHUmB4K9Yfwde/DBjSSQXL81J39F+tYbiGqut2922aaC2onXryPwS4ZlB0EQveMmYw1LKJbrJ5j/Mt6hepPJW+sEMavIu9VEHiZMT3giF4k00L8PzfTRDy5apHwa5UqfzCsXwsvJ7bnbeiYmoN3uafpC0HKxUtn1xJMyBwMYXD5leI5jkTB2mgezq5gVOIEmxzBCankonH6RbwIbjjEH2um8kM75xcg/7qbwHPv1hoZvqIArnwrWu1m+5r0KvmOJ1ZG3mLB8Q3lvHKISW1tPjSVWvJASvymxfrbAMNBDowmBTdwptesJ3DhYi1ksqhOtE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Nov 07, 2024 at 09:32:56AM +0100, Christoph Hellwig wrote: > On Tue, Nov 05, 2024 at 03:53:57PM -0400, Jason Gunthorpe wrote: > > > Yeah, I don't really get the struct page argument. In fact if we look > > > at the nitty-gritty details of dma_map_page it doesn't really need a > > > page at all. > > > > Today, if you want to map a P2P address you must have a struct page, > > because page->pgmap is the only source of information on the P2P > > topology. > > > > So the logic is, to get P2P without struct page we need a way to have > > all the features of dma_map_sg() but without a mandatory scatterlist > > because we cannot remove struct page from scatterlist. > > Well, that is true but also not the point. The hard part is to > find the P2P routing information without the page. After that > any physical address based interface will work, including a trivial > dma_map_phys. Once we are freed from scatterlist we can explore a design that would pass the P2P routing information directly. For instance imagine something like: dma_map_p2p(dev, phys, p2p_provider); Then dma_map_page(dev, page) could be something like if (is_pci_p2pdma_page(page)) dev_map_p2p(dev, page_to_phys(page), page->pgmap->p2p_provider) >From there we could then go into DRM/VFIO/etc and give them p2p_providers without pgmaps. p2p_provider is some light refactoring of what is already in drivers/pci/p2pdma.c For the dmabuf use cases it is not actually hard to find the P2P routing information - the driver constructing the dmabuf has it. The challenge is carrying that information from the originating driver, through the dmabuf apis to the final place that does the dma mapping. So I'm thinking of a datastructure for things like dmabuf/rdma MR that is sort of like this: struct phys_list { enum type; // CPU, p2p, encrypted, whatever struct p2p_provider *p2p_provider; struct phys_list *next; struct phys_range frags[]; } Where each phys_list would be a single uniform dma operation and easily carries the extra meta data. No struct page, no serious issue transfering the P2P routing information. > > I saw the Intel XE team make a complicated integration with the DMA > > API that wasn't so good. They were looking at an earlier version of > > this and I think the feedback was positive. It should make a big > > difference, but we will need to see what they come up and possibly > > tweak things. > > Not even sure what XE is, but do you have a pointer to it? It would > really be great if people having DMA problems talked to the dma-mapping > and iommu maintaines / list.. GPU driver https://lore.kernel.org/dri-devel/20240117221223.18540-7-oak.zeng@intel.com/ Jason