From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D538CC0218A for ; Tue, 28 Jan 2025 14:49:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFD90280230; Tue, 28 Jan 2025 09:49:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C852E28022B; Tue, 28 Jan 2025 09:49:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFEAD280230; Tue, 28 Jan 2025 09:49:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8CDBE28022B for ; Tue, 28 Jan 2025 09:49:17 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F276414164C for ; Tue, 28 Jan 2025 14:49:16 +0000 (UTC) X-FDA: 83057143512.09.858BFA2 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) by imf02.hostedemail.com (Postfix) with ESMTP id 7732980010 for ; Tue, 28 Jan 2025 14:49:14 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=luIhqPhv; spf=none (imf02.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 198.175.65.18) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738075754; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uNNV9AiWSNOX6ApUsgEFrkkAU4pDKZfbG0WkJNIOQ4k=; b=4pAqQY+z0XudjHCptQ2E6DPZv2x2ZMJwsYShuA9t+KsuU5RWhfwLGvTiiuV+vHNFiqgPki tyOcu0gFdLusW9AelkhSa0fdUINrnB8rCD0VsxD0RG3oMo182pPst27K/t3qRPgAeTEq9m 480NS4crSkjtnZoYnDEMk6Eu4ToeVZQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738075754; a=rsa-sha256; cv=none; b=ko0Hvr+vDcsArZW9WL1N6SwQxjurB0yPawM1/5W1GccqQgoGWaO/rpqxeyjxpvyJe/vyKF zB0CcD0GdM1XsImM+VILbRibDDjwYSttGGbsiT8+36Cf36Vb1IH+dCqgQJBEBzQ5WGYaku Z40Rs2Sfd8URBbHhqDlXor1W26y3HnE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=luIhqPhv; spf=none (imf02.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 198.175.65.18) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738075755; x=1769611755; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=uNNV9AiWSNOX6ApUsgEFrkkAU4pDKZfbG0WkJNIOQ4k=; b=luIhqPhvTzGSsojaaaHs15+E6+UKUiZ1mjLuI0M2ODbdVTMKdLz9dCpG fk6Kw4vLbkaSCo38EnjgGHSj6hYXEFsm4UyyRqy1CRoJaZ3fhnrqujfC7 SImEs5LfNKkJ+QNFGYbtinH77wToY0YKKUf6ZCY2LzeWzdeXW1vEQ3k02 83GIAWhWRYR+VBvVjz5SPQUhFMH+ruv6WKOYoqmRPi4C++UEixjl0BVh9 5gB1pMK2WAN4FOkqtbTG6M/Jsqz5Q3x2yC96PQWcIW+GQecJuccBTjG4j GgKifIno7mqLlKqvyEaEr1h7CGf/9WKKSm9JrilXvssrRQwlSdv3LXvSH Q==; X-CSE-ConnectionGUID: NoOueARARkSSnsovPFjNIQ== X-CSE-MsgGUID: piRUOR/gR/qFe3hM/e0YDw== X-IronPort-AV: E=McAfee;i="6700,10204,11314"; a="38662578" X-IronPort-AV: E=Sophos;i="6.12,310,1728975600"; d="scan'208";a="38662578" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2025 06:49:13 -0800 X-CSE-ConnectionGUID: hYHxh45jRjqWTCGeN8qB/A== X-CSE-MsgGUID: IlkXYCWZQdmDqhIdG2o5hA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.13,241,1732608000"; d="scan'208";a="109330374" Received: from mwiniars-desk2.ger.corp.intel.com (HELO [10.245.246.161]) ([10.245.246.161]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2025 06:49:08 -0800 Message-ID: Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages From: Thomas =?ISO-8859-1?Q?Hellstr=F6m?= To: Jason Gunthorpe Cc: Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Date: Tue, 28 Jan 2025 15:48:54 +0100 In-Reply-To: <20250128132034.GA1524382@ziepe.ca> References: <20241201103659.420677-1-ymaman@nvidia.com> <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> Organization: Intel Sweden AB, Registration Number: 556189-6027 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.3 (3.54.3-1.fc41) MIME-Version: 1.0 X-Stat-Signature: cinyz9jd6i5ejb1bqys5diu64n6nyk1k X-Rspamd-Queue-Id: 7732980010 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1738075754-419748 X-HE-Meta: U2FsdGVkX18QdAw3nTBbeNHAIlvHiXb6AJfcCR4sLhfRRBS0QlyZxm943cdN4lqA1Inm7jxDp/9GCtuTXxYfubq9QBRjFBdG+fg24tnjldS7FnOqxSBEvDMINyhONwIK2iRGi9eA6losnMlV3fcZdGWuSbK00nKktIk/7jTp4rZ75/oFujI9m4Em3/uj2aK8LheoqPNvzpJcpVifSbbqZ6d27a+zNFUayKbbsvsn7ASGGdhvPJEIsfYXuMwjpE57ccUPQ5bmSJm+RtuRqJIny6RFGucf9d+zbwSIsIbuhhqsRXO+KP2aDCOah524gBQmPWyi9jYOuy0Vsqm7dNvQUQVSr1q6SNDuc0kFZcNRmRP2MmqevKCTm2OIomd51xpsVqZIGJ/1CWVHsu0ESrqjitBvC7w9RyiMwL3XWzKiZmBPw4YsRrhyBUo64ba2UqInfQTba8vKvSNobGLBHSKup/mKpaRV1uLP39JUnHDK0Qon69yeR4RF9tZoZTl2TH7VaKIAO5KbOEkcBnCIgkr8gnkMygX+nVlVV/+pKojmIMf88C6bPvLvsTqZx7qcYk96JeZbb6DoxTrfVaXjg35bz4Omu78VK4Gf3F6X5YUqe1MDTYKzpWzp0yEb8EaGFF/TYmLgVBU2fkKXV2noNOLVM1wz7OTYZsK7RzJRsJC2sx41dPBHxx9rtum39QtZq35pt++wBM+vwLJrrY8qIPNex40Sp63qy5gcgyiT2p6Swhe7tW5IMwOnBq3wnjFUu4tw2B3B4fT8mzspoK4arBHtWhxbeyN8yaMoQ/KjQmmClBQW0zpPqNp4xWZFqNHD2sA5btYnclTwb4CDY3xmS2XYbsejoYSugYgEoTnl78YOoJO5yFz8jlmcMX3XHmcJXNnHs6EgB0t9A++O/W38KIMiZ7ABGxILqkZwJxHmDk54P5XO1RiPsbG0X64QQjZBI4RsBxkITNZQSQrhttOtNOO lJqiD2wK xJPWKfVQPgbtIlr9XQwQYISY+eeIcp4PuIgzNCBqb4wKJyDRLfTN2orJ6fN88FSA+cpeUAleIm2IRtAauYaHWDHutHOxhSowpaduT1V0Q4N7ilo5YRj4aYije/DEYY60XbuajdLL61XLGUDPIou2sghbGI7xFDFkFLxiOAiNp/9xtkf+q5e3DGj1x7w7tnRuLckTVPjdMT+pfG4SJqbsrMRYKoDcFSzH7sv8qLzumZkHEdBM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 2025-01-28 at 09:20 -0400, Jason Gunthorpe wrote: > On Tue, Jan 28, 2025 at 09:51:52AM +0100, Thomas Hellstr=C3=B6m wrote: >=20 > > How would the pgmap device know whether P2P is actually possible > > without knowing the client device, (like calling > > pci_p2pdma_distance) > > and also if looking into access control, whether it is allowed? >=20 > The DMA API will do this, this happens after this patch is put on top > of Leon's DMA API patches. The mapping operation will fail and it > will > likely be fatal to whatever is going on. > =C2=A0 > get_dma_pfn_for_device() returns a new PFN, but that is not a DMA > mapped address, it is just a PFN that has another struct page under > it. >=20 > There is an implicit assumption here that P2P will work and we don't > need a 3rd case to handle non-working P2P.. OK. We will have the case where we want pfnmaps with driver-private fast interconnects to return "interconnect possible, don't migrate" whereas possibly other gpus and other devices would return "interconnect unsuitable, do migrate", so (as I understand it) something requiring a more flexible interface than this. >=20 > > but leaves any dma- mapping or pfn mangling to be done after the > > call to hmm_range_fault(), since hmm_range_fault() really only > > needs > > to know whether it has to migrate to system or not. >=20 > See above, this is already the case.. Well what I meant was at hmm_range_fault() time only consider whether to migrate or not. Afterwards at dma-mapping time you'd expose the alternative pfns that could be used for dma-mapping. We were actually looking at a solution where the pagemap implements something along bool devmem_allowed(pagemap, client); //for hmm_range_fault plus dma_map() and dma_unmap() methods. In this way you'd don't need to expose special p2p dma pages and the interface could also handle driver-private interconnects, where dma_maps and dma_unmap() methods become trivial. >=20 > > One benefit of using this alternative > > approach is that struct hmm_range can be subclassed by the caller > > and > > for example cache device pairs for which p2p is allowed. >=20 > If you want to directly address P2P non-uniformity I'd rather do it > directly in the core code than using a per-driver callback. Every > driver needs exactly the same logic for such a case. Yeah, and that would look something like the above, although initially we intended to keep these methods in drm allocator around its pagemaps, but could of course look into doing this directly in dev_pagemap ops.=C2=A0 But still would probably need some guidance into what's considered acceptable, and I don't think the solution proposed in this patch meets our needs. Thanks, Thomas >=20 > Jason