From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01BB0D2A539 for ; Wed, 16 Oct 2024 17:44:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C6AC6B007B; Wed, 16 Oct 2024 13:44:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 429226B0082; Wed, 16 Oct 2024 13:44:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A1716B0083; Wed, 16 Oct 2024 13:44:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 093F76B007B for ; Wed, 16 Oct 2024 13:44:51 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F01A2807AE for ; Wed, 16 Oct 2024 17:44:41 +0000 (UTC) X-FDA: 82680190362.01.1433208 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf16.hostedemail.com (Postfix) with ESMTP id E5142180012 for ; Wed, 16 Oct 2024 17:44:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=VbtvK2Qz; spf=pass (imf16.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.48 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729100615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o35iiCDiZ5eaKfswFFD+TpBr1+3YZIM/Zzff16IB6bs=; b=8JWjp1nP/KmF/zpYjCiib54BillEX2XFat1glkE+NdLv52FdAMABYZvv0BmLrlGHqQoMh0 Q4UW4R+QK8viJj1ruXxB87xVWt4trT1fFrksQLo3bz8oMs8WAhDtmBzBnS9vsSy8BDephe PTuperLRUSi4/xx13VTVlVPVtXJTMuo= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=VbtvK2Qz; spf=pass (imf16.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.48 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729100615; a=rsa-sha256; cv=none; b=JIoOaC+KLCSJiVz3AgBVyd6y8AkqcYFXHWUK9SmN7KUKR6IRgKlUJvn3s7CTpj5+PJFiha th0z97VQoii9bUdJndpxFXhcgTSBfQPQkZh5uH5RRlZj7NJ7lkl3SkNXqDkIwWfjcct7Ts /cRpUwH/w08HK354qJKDnO6fxhV8JcQ= Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-6cbcc2bd7fcso634046d6.1 for ; Wed, 16 Oct 2024 10:44:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1729100688; x=1729705488; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=o35iiCDiZ5eaKfswFFD+TpBr1+3YZIM/Zzff16IB6bs=; b=VbtvK2QzBJaBPDZsf+Oqfn3zmYxQNPdx0cY5hOl+BKPZIo3BLIOBGi794k+Ljzsqlf 5Oh6prWAvwOrai0mwFW7aP7pLSFzyb26bheMgU55/2XZwBu58xM3fLCjWqQFH63zj0yJ boZhQA87+HuXBfCDcWoa8YfXm3aBLHtS9PNWLuj+cjjYh+vvEBGXvpiFTfeDFMFtvB6/ 0guiHbCm16QoicEAVcxEnvFc1zqW+b2Gyq125zEG/pxYVWNUirN6dnDAr4I6fa9AUc0a TpRXBzQmshJOImA3gleqxPw42Bqus9bp4IgR//bbxs3AGekCqt699rnBlAVd30hIRnEd fXCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729100688; x=1729705488; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=o35iiCDiZ5eaKfswFFD+TpBr1+3YZIM/Zzff16IB6bs=; b=UZK/yfNNXfYetQZQqfyZBqTE5Cr/QquzHQcBgSP805XOlMmHjYFbGdRH3E9uCb4UH0 h5GC99RuC1mmfWNGswTmd9+3o2HTYpjwgfd1VbSfRgNMd/aqhLbtHnSNxX0YugFZNaLs ToCzNmTfjX0JwPYuMiuosqgefyrkBvK1mDJx/i1ImIKyCZcGyiKB/kUSd5AXThkt//1Z ieSmQU8e//IUqyB0CPogZF5AO34SHk2EYQ6hwQEfMUZDlSEK+tdJZCOl8jpEljQ/6J1/ uhaAxnXWjUZXP7RrcGCaK5MQI/gBXvjnq1vxueqkjrcOTgBEj6WnhFJzY7IWMxJXWHmk TdqQ== X-Forwarded-Encrypted: i=1; AJvYcCUL62yJViI08MnpFuLRDo8rSncSuNiwLVm7PNlNFIsrJ1zB+sXK37BwJlEo2qP13qxU/+SQi7Nutg==@kvack.org X-Gm-Message-State: AOJu0YzP5O/vP5PTMFxsiXkDg7TTvQJgbdB6ZaP24HEqeH/0rsodPT8G RHb2MA6XnTQurz+CKd3Mw8Ju+rikvBvOgddIxJqNcXhqVuw7YFqHrQfXW98nzjs= X-Google-Smtp-Source: AGHT+IHcb2p/EJ+d36NHl0163jeyLVI0sDHNW9tlAjYiB2BbJs8req4vjOwcR9cJR+8TMnyYMuXpMQ== X-Received: by 2002:a05:6214:5b06:b0:6cb:d1ae:27a6 with SMTP id 6a1803df08f44-6cbf0095668mr259975216d6.24.1729100687504; Wed, 16 Oct 2024 10:44:47 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6cc22961bbasm19822836d6.108.2024.10.16.10.44.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Oct 2024 10:44:45 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1t184j-000h5k-5X; Wed, 16 Oct 2024 14:44:45 -0300 Date: Wed, 16 Oct 2024 14:44:45 -0300 From: Jason Gunthorpe To: Christoph Hellwig Cc: Yonatan Maman , nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, herbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, dri-devel@lists.freedesktop.org, apopple@nvidia.com, bskeggs@nvidia.com, Gal Shalom Subject: Re: [PATCH v1 1/4] mm/hmm: HMM API for P2P DMA to device zone pages Message-ID: <20241016174445.GF4020792@ziepe.ca> References: <20241015152348.3055360-1-ymaman@nvidia.com> <20241015152348.3055360-2-ymaman@nvidia.com> <20241016154428.GD4020792@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: E5142180012 X-Stat-Signature: fp3du1a95bch79pes6b5p7ap57sz777p X-HE-Tag: 1729100679-266049 X-HE-Meta: U2FsdGVkX1+QrTeabIoaNgTD6G6IP47oo7ZkY8nO6A4db4PcFXnGDSCLhC56p9pwoE8qQNO4qbrNTsqBbOmxqoveQ9N/7anRhKYJ/OIOFKWN9d2nsNPGLJSy54EtYIL8u7weqX6+9Tn4wFDfHMBlluxxqf/gYMJxAYhVFSqJ1730DH6fEp+6afhTxAQHyGjL0LNwfLKPCfWyPrkvrZZmZaYclCwRtKAXNbEZAKnbImyU0mtNDzxa/K3Z5JD0Tt0d9RTvhX2yZfdTmWZzfl10dMdhcMiVlaFMh+XifDG0I5lcioV/hiphk+y215i+20QkOInLoaXqBU4x6nCBRdxadvcNHPZK2QOPMfjokwdUZY1bi+tpuydm85CiUBHZlw0CgomgqAQY/EpZFX+Cgoqkfc1ULDWnad68yXee8LzyrFoVpOawS7lEjCMhfLZPaTu05FXRubZ57fZsRfaN/HGrtTQ2Q/qj1A1nkZfFGRA1SiehxajtrwTmUdY8puLv3xG5Wz9S1Qn5P85dJ/uFGt8fnGbBDiucX79gFkJBk1eL3Hz96rCapNRCDn3B2wOrmp01PedarIxnzP5xqRAz+GNcB/eQEfIRgqYM4X6lRGBRGITnbCYLAmXFjwln/rUxcaLd4HrA8l7K/kzlOp6z7+TQn1GUPfPQV34bbSVBXA5KbsmhGkzznNdNBIj65h4jpGUDAR9KSB2d+u/1bfLRwUXJCPMgMGDMjBkM23GL2cTrN4rR+m+yTkW9T6UvGYRXv3PEiK43krovbgLBrzAIUwqaUIjRNLncE58k10g7mmJ+wzQYN67Dp5LxElq7Ta4K6+tA8CusqCDmDGo8WxaCNyYgZZK/1bSyeM8aVpjKNu0QycRrZTNegLmoYmXAUZZ9ugZKEoTZprQkG7KIrbYTWLZq47mw9OhXAPcQaBl2F8vVyTSc+0TqX3RZtvga/Uhs9AypW3PpzzR10Hkx6iYw9oJ x7TEVrrn 8FGfW/eGMN+4TxjVaCInIJ1EST30bRrnzIY3Z0umWvq2zGLJJSReoOKUJlfldCgoPq+nvqV6h3DZQbvgZLQK3zVX2+95pYW5Nc8OWPDAtzn+L4fbBdRRCHcArnMP2plMGm7QYb3RGKwXSJmzkrmfY/iHuf9xOx3bJAoVMCvSLm6sue3WAAo0mosGdNEHUHNZb3ihw0YpSDxyJxPrPPpncOQRMig2BhrIGAUkAfLh0P9lr4r3n6xTfgW0iWEGp48Aak+5u1fsf5M3cYlwtp20/W+H8lq19XLAK30Pz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 16, 2024 at 09:41:03AM -0700, Christoph Hellwig wrote: > On Wed, Oct 16, 2024 at 12:44:28PM -0300, Jason Gunthorpe wrote: > > > We are talking about P2P memory here. How do you manage to get a page > > > that dma_map_page can be used on? All P2P memory needs to use the P2P > > > aware dma_map_sg as the pages for P2P memory are just fake zone device > > > pages. > > > > FWIW, I've been expecting this series to be rebased on top of Leon's > > new DMA API series so it doesn't have this issue.. > > That's not going to make a difference at this level. I'm not sure what you are asking then. Patch 2 does pci_p2pdma_add_resource() and so a valid struct page with a P2P ZONE_DEVICE type exists, and that gets returned back to the hmm/odp code. Today odp calls dma_map_page() which only works by chance in limited cases. With Leon's revision it will call hmm_dma_map_pfn() -> dma_iova_link() which does call pci_p2pdma_map_type() and should do the right thing. > > I'm guessing they got their testing done so far on a system without an > > iommu translation? > > IOMMU or not doens't matter much for P2P. The important difference is > through the host bridge or through a switch. dma_map_page will work > for P2P through the host brige (assuming the host bridge even support > it as it also lacks the error handling for when not), but it lacks the > handling for P2P through a switch. On most x86 systems the BAR/bus address of the P2P memory is the same as the CPU address, so without an IOMMU translation dma_map_page() will return the CPU/host physical address which is the same as the BAR/bus address and that will take the P2P switch path for testing. Jason