From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBDFEC02190 for ; Tue, 28 Jan 2025 16:32:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E16E280238; Tue, 28 Jan 2025 11:32:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 443E6280236; Tue, 28 Jan 2025 11:32:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BDDF280238; Tue, 28 Jan 2025 11:32:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0A4E5280236 for ; Tue, 28 Jan 2025 11:32:36 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0BB521C6122 for ; Tue, 28 Jan 2025 16:32:35 +0000 (UTC) X-FDA: 83057403870.21.D6504EB Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by imf06.hostedemail.com (Postfix) with ESMTP id 649B4180019 for ; Tue, 28 Jan 2025 16:32:32 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nhaG1lsq; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf06.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 198.175.65.10) smtp.mailfrom=thomas.hellstrom@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738081952; a=rsa-sha256; cv=none; b=iQ2E99fYXK0MOPFjdj8yx/oXOm9cw6xw4Pr9MG/dqA9B7dtZDlVfIg4022do4d4tfqyq7z 17d/rCIfqFfqG/gLAtnD18+L37Zy8nicVFvNE/WQEHaDi8t8Sjl3ThKHTRGDiIZgDWHnib MRTBMId8NN5+lkYrgPkiXzdTiTAJTpQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=nhaG1lsq; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf06.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 198.175.65.10) smtp.mailfrom=thomas.hellstrom@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738081952; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QmV+7gtGvGbC0D7AD0FZxKqZ+uJvTV+fE2kcgk9KYwM=; b=LNbsn7VyfQV6kORwVrlUKvXGfOZe6ibiE7T8J0x96RR8kADClwxWAmrQiRj8gwWvFIS0Aw jL4YOnodg1IG2JQmhAKQqh8INVCHq5VeC0YHOmbZV1mL5omdS0pkRmmxIfpzuRX0AYxnp1 lkWnsqM/s8nlwIJxQEq80UKjTTz+1aQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738081953; x=1769617953; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=QmV+7gtGvGbC0D7AD0FZxKqZ+uJvTV+fE2kcgk9KYwM=; b=nhaG1lsql3R1PNF5lTzkoRZLoz45OwmdcfkRCrRB/gcYX1sO+INgJwSv NWa1BFsjIY7VaPeGpd2t5NHZYQTaDkxBS6rWRAHX271V7zTURoslBOVr0 yuRc1ttdiI3nZlJs+tjkw1YYxXpHPwDxJgGGq7NHGjdFADjijF2L5XQO/ lhvfxLpe+i/obiGjTDCv+6xmB8je4g4QaZKtVsuE7EiyXE7bO9TpGFsEF q1Tljd36q/26ARAPH3N1jBJ/eY8NK7riiGKLRQkDPHx4jOofKMjGF6AWj ywbBIm2CL9vpuZwVoYXyRYbwK6tErb6NDIffFw5Yvey+IAPNm1ms95jgl Q==; X-CSE-ConnectionGUID: xfY1fLqyQSiiPeKDJwyBTw== X-CSE-MsgGUID: 4Xg3ML2bTHSFH9gdYq2btQ== X-IronPort-AV: E=McAfee;i="6700,10204,11329"; a="56000665" X-IronPort-AV: E=Sophos;i="6.13,241,1732608000"; d="scan'208";a="56000665" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2025 08:32:31 -0800 X-CSE-ConnectionGUID: WBIIBAoqTTGl0aER7KaBzg== X-CSE-MsgGUID: ZX7aVbzyShGdDSqqUk7gUw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.13,241,1732608000"; d="scan'208";a="139672331" Received: from mwiniars-desk2.ger.corp.intel.com (HELO [10.245.246.161]) ([10.245.246.161]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2025 08:32:26 -0800 Message-ID: Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages From: Thomas =?ISO-8859-1?Q?Hellstr=F6m?= To: Jason Gunthorpe Cc: Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Date: Tue, 28 Jan 2025 17:32:23 +0100 In-Reply-To: <20250128151610.GC1524382@ziepe.ca> References: <20241201103659.420677-1-ymaman@nvidia.com> <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> <20250128151610.GC1524382@ziepe.ca> Organization: Intel Sweden AB, Registration Number: 556189-6027 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.3 (3.54.3-1.fc41) MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 649B4180019 X-Rspamd-Server: rspam10 X-Stat-Signature: fn1htkzjzq6bak7mnubywrgcxufacg38 X-HE-Tag: 1738081952-917843 X-HE-Meta: U2FsdGVkX1+VD0miJ5wcATEMEQSgJVG80fvVnquWfsbUXvErKpBQ2mo7k9aBs+madsZJlYDAtHSvS6brAQECJ75/6F9+zqb38OPJur2qT6pQ0PrT/m3eMxrZkJKFJVH85Wh0ShkTWqCY6WPdInQSp+K+9io65DVeE/mLvzyd6qPh9lImAOqley1a26hjudHAo/vP+YQWLGs2E+CHksTduK2mzSrSFvH/3kPkz5pEubDLRPW6qKty1yMjMxzSkAcwCZCKOgsN3brbC3aLRX4jF8cwMI/wYaJ+/pvmMQCuSxj1M3vIiF6c3ccbpaH4Gb4G76MvzliHHXSvG1/Q0TgZoaBdF9JA/qAnsE+DZ7wmIWjn482qOEAWRgHyc6LHMKrgSnJBK2TIYjdJ6IWe/dFz5m1LNu5t7oBl7kj3aUK196bf+ZAjQDxGAwHob3RZU3hkaWtkRmkGqCRaVq2zcsZRMPdXfOwadPZG4AqscUAmvVVQDGfNS2kNlhgFIcneP4nklInrUHOauMngb778qlV8H4+5TU/3iqzzBuOW0+hW5tItszNyIX5ldt6xSZZAtS8qSFa66Prz2S/W98WBOFXjUm84Qw0dKY+v+ZhqCN5hqrz9+cXltVmXtWWUsD1ufz6IHXb3qj/vX9DNreHcjpcbU8Bn64siGn6aqF38qAYxrIhTdYgiFwdzzC3utj0Xy6OcP0+uvT5Y9Dwm7KCHjeOXHUjNbg3eOhqU9/JmOzKsUEdqK727LQ0YGISI6YB9fZfq2oR+EG68o/WwxuXof7CcX2sEspEgghRcSVfJ2BGokTPEIFTSPud2YJopLN+pz5b9sAwZ3jNApq70ZRwYpoJXTFlq1UtY3r0b7uCzbdd4+kHtdw8kH/imAD7ysvzoMWIsurlr1WZd35qpTCY6qoHx4qElvbd4rzS5luVeGhFAPBWPLm7Ca+wcGAX2BTH+zOx+65mvK8bqLI4dDlveDpX rN4kh9uI 9ghplABXfnOlBIzRGSBhnUTX5PGrLUhx0O0Suv1gItq++PsIxhfn0TynsNHn4A6w2uS8tqFU2eW/XS/5Rwl26da5I7EfX2BBb0f7lTnDc8Zo4JRa8gOpSTV0HVUUUWZczQiTFirIuAOgXGDUnKkmNogahjRFdsM8LWhR7BxsnSuhTy420yFx98RzBDEPARiT4GLDrQyobNcKISuHapPbj5RRQ7RmP50s/KddptbPNQmWjBUtpF5TblAMWKzRUSoJsZqHsVml2miVt3bLH9QnJ8A+AGTCDuj11Y67b X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 2025-01-28 at 11:16 -0400, Jason Gunthorpe wrote: > On Tue, Jan 28, 2025 at 03:48:54PM +0100, Thomas Hellstr=C3=B6m wrote: > > On Tue, 2025-01-28 at 09:20 -0400, Jason Gunthorpe wrote: > > > On Tue, Jan 28, 2025 at 09:51:52AM +0100, Thomas Hellstr=C3=B6m wrote= : > > >=20 > > > > How would the pgmap device know whether P2P is actually > > > > possible > > > > without knowing the client device, (like calling > > > > pci_p2pdma_distance) > > > > and also if looking into access control, whether it is allowed? > > >=20 > > > The DMA API will do this, this happens after this patch is put on > > > top > > > of Leon's DMA API patches. The mapping operation will fail and it > > > will > > > likely be fatal to whatever is going on. > > > =C2=A0 > > > get_dma_pfn_for_device() returns a new PFN, but that is not a DMA > > > mapped address, it is just a PFN that has another struct page > > > under > > > it. > > >=20 > > > There is an implicit assumption here that P2P will work and we > > > don't > > > need a 3rd case to handle non-working P2P.. > >=20 > > OK. We will have the case where we want pfnmaps with driver-private > > fast interconnects to return "interconnect possible, don't migrate" > > whereas possibly other gpus and other devices would return > > "interconnect unsuitable, do migrate", so (as I understand it) > > something requiring a more flexible interface than this. >=20 > I'm not sure this doesn't handle that case? >=20 > Here we are talking about having DEVICE_PRIVATE struct page > mappings. On a GPU this should represent GPU local memory that is > non-coherent with the CPU, and not mapped into the CPU. >=20 > This series supports three case: >=20 > =C2=A01) pgmap->owner =3D=3D range->dev_private_owner > =C2=A0=C2=A0=C2=A0 This is "driver private fast interconnect" in this cas= e HMM > should > =C2=A0=C2=A0=C2=A0 immediately return the page. The calling driver unders= tands the > =C2=A0=C2=A0=C2=A0 private parts of the pgmap and computes the private in= terconnect > =C2=A0=C2=A0=C2=A0 address. >=20 > =C2=A0=C2=A0=C2=A0 This requires organizing your driver so that all priva= te > =C2=A0=C2=A0=C2=A0 interconnect has the same pgmap->owner. Yes, although that makes this map static, since pgmap->owner has to be set at pgmap creation time. and we were during initial discussions looking at something dynamic here. However I think we can probably do with a per-driver owner for now and get back if that's not sufficient. >=20 > =C2=A02) The page is DEVICE_PRIVATE and get_dma_pfn_for_device() exists. > =C2=A0=C2=A0=C2=A0 The exporting driver has the option to return a P2P st= ruct page > =C2=A0=C2=A0=C2=A0 that can be used for PCI P2P without any migration. In= a PCI GPU > =C2=A0=C2=A0=C2=A0 context this means the GPU has mapped its local memory= to a PCI > =C2=A0=C2=A0=C2=A0 address. The assumption is that P2P always works and s= o this > =C2=A0=C2=A0=C2=A0 address can be DMA'd from. So do I understand it correctly, that the driver then needs to set up one device_private struct page and one pcie_p2p struct page for each page of device memory participating in this way? >=20 > =C2=A03) Migrate back to CPU memory - then eveything works. >=20 > Is that not enough? Where do you want something different? >=20 > > > > but leaves any dma- mapping or pfn mangling to be done after > > > > the > > > > call to hmm_range_fault(), since hmm_range_fault() really only > > > > needs > > > > to know whether it has to migrate to system or not. > > >=20 > > > See above, this is already the case.. > >=20 > > Well what I meant was at hmm_range_fault() time only consider > > whether > > to migrate or not. Afterwards at dma-mapping time you'd expose the > > alternative pfns that could be used for dma-mapping. >=20 > That sounds like you are talking about multipath, we are not really > ready to tackle general multipath yet at the DMA API level, IMHO. >=20 > If you are just talking about your private multi-path, then that is > already handled.. No, the issue I'm having with this is really why would hmm_range_fault() need the new pfn when it could easily be obtained from the device-private pfn by the hmm_range_fault() caller? The only thing hmm_range_fault() needs to know is, again, whether to migrate or not. But I guess if the plan is to have hmm_range_fault() call pci_p2pdma_distance() on it, and we don't want the exporter to do that, it makes sense. >=20 > > We were actually looking at a solution where the pagemap implements > > something along > >=20 > > bool devmem_allowed(pagemap, client); //for hmm_range_fault > >=20 > > plus dma_map() and dma_unmap() methods. >=20 > This sounds like dmabuf philosophy, and I don't think we should go in > this direction. The hmm caller should always be responsible for dma > mapping and we need to improve the DMA API to make this work better, > not build side hacks like this. >=20 > You can read my feelings and reasoning on this topic within this huge > thread: >=20 > https://lore.kernel.org/dri-devel/20250108132358.GP5556@nvidia.com/ >=20 > > In this way you'd don't need to expose special p2p dma pages and > > the >=20 > Removing the "special p2p dma pages" has to be done by improving the > DMA API to understand how to map phsyical addresses without struct > page. We are working toward this, slowly. > pgmap->ops->dma_map/unmap() ideas just repeat the DMABUF mistake > of mis-using the DMA API for P2P cases. Today you cannot correctly > DMA > map P2P memory without the struct page. Yeah, I don't want to drag hmm into that discussion, although admittedly the idea of pgmap->ops->dma_map/unmap mimics the dma-buf behaviour. So anyway what we'll do is to try to use an interconnect-common owner for now and revisit the problem if that's not sufficient so we can come up with an acceptable solution. /Thomas