From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC31CD37499 for ; Thu, 17 Oct 2024 14:05:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 816CB6B0083; Thu, 17 Oct 2024 10:05:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79F1C6B0085; Thu, 17 Oct 2024 10:05:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 618F36B0088; Thu, 17 Oct 2024 10:05:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4001B6B0083 for ; Thu, 17 Oct 2024 10:05:12 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E720C1214BA for ; Thu, 17 Oct 2024 14:05:01 +0000 (UTC) X-FDA: 82683265938.28.82CDD4D Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf20.hostedemail.com (Postfix) with ESMTP id E3D581C0032 for ; Thu, 17 Oct 2024 14:04:57 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Uxnx0crN; dmarc=none; spf=pass (imf20.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.44 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729173750; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z7OJ+A0M+7sK83vQ9Fm7CrxVR6UTlfXfFle4pTLoIZk=; b=u3aGaWGNIqbpdmwzmgKVw2Dg4JZDf7dORS7uqBaR5srQDR58xQEjV5Kh8kOPNXlyW9+Psw NY33ptGUO0u0tx1REJ/hbpHAUlb0ST+w7VlDHsXHp/ydCRTEG2QCGlzqM+jtNLWZUdIR3w 39l/fgnObHRIpt0B4aLUzMIe9YH3QCY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729173750; a=rsa-sha256; cv=none; b=1ef1iKZJFTdPzTj+O11OQnpCWQezJFCF52wgTnehXQXjKOWLPGKb+vNkcdCv/kkdU3ZWRs QujiFLmUFYaIHmNUXTY1vbtvesWwD6B1Ne3x/4BD5I1E+jyf+rqnAfLWD8kAZYpyBpx9di VaFwLqNGa6eCPdeczf8nQc8AtlEMQlc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Uxnx0crN; dmarc=none; spf=pass (imf20.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.44 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-6cbf0e6414aso4552316d6.1 for ; Thu, 17 Oct 2024 07:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1729173909; x=1729778709; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=z7OJ+A0M+7sK83vQ9Fm7CrxVR6UTlfXfFle4pTLoIZk=; b=Uxnx0crNQAWQDQL8jbAFHnoxeJ029FlFKMuiEXnVItA6U0hA/6KVGJA+uSaNtp/4iW fRZt+7zLlvf/3Zrof3RJXe/x8BeIdjcj6o/mY1JWlAv4nuFKsatQh8ANnIsqGuVbzKcH M5qC1/geGX43XZByMRnqRagdI4RXrsvNM0wAGWpTPQkZdIdIAP6p88xcFEtBMGx0ACti Zyo9aSkD4G3I1s3kQ24z22Y9q+SmfRQH31G6tchTnRhp4heHRF7ZVc4Iumknu7l0QZE+ QiiC/j9UHO2Qphy2bjvuzMv98/ImLa9Vw9n/AgQDIibQUL+ygEMVQJ7tgRIcSh+0xoQP QlCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729173909; x=1729778709; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=z7OJ+A0M+7sK83vQ9Fm7CrxVR6UTlfXfFle4pTLoIZk=; b=XgUH/6lKBtHhLHwg2KnqzhMeSc6hfl1m/+ViGoYCsrQVmLB6FcFa6IYAbKIpyWakzE AYFMF0JLFITGdgWfcT7KlzzBtcnRu9M1kDTwee52I33bexzabbjkuslPHsCXXEy0MaP3 kYZ22SZa2gqIxsUYGsxbgTnOFiEILT0UzDnKacn3kwKmxZDt9VJ2wQMr3AoVwyIc6rRL h0TM2t3KckapAePG0niJWs2u80JBml21uDKe2Tmfq0s8die+DtY+VK3PvulxsKohiZTL WCs4uNiRGUShbJYmXyOidDvE0BsJuv0IWOoI8AGe+ZgLqKP8CmrH6ggR14PnJkrZjxPv EWkA== X-Forwarded-Encrypted: i=1; AJvYcCV0bGDFDJFSpw03opfMSIjGULoDucPT+W5iW+xmDoFN5WQRl5spOnrEEJ3j03X30fDHyxF4NxJDvw==@kvack.org X-Gm-Message-State: AOJu0YyH8LO6kuEAVX+9E0QnqgGuO/TRCKFJAEkubD4gC7D3HDOm7XwG m48HUXWutYz5JnSxkYsxf40s8+3smQM5B+OeX3rIjjoKEG1TFBTNZXtM1vfaT4Q= X-Google-Smtp-Source: AGHT+IHal8p/zFk/MmJsuAonYG12bCKHBoHHm6lVzKtXA3QkUgZLBGuCRV/wmp+GJALXP2rbtY7uRw== X-Received: by 2002:a05:6214:31a1:b0:6cb:c72d:6861 with SMTP id 6a1803df08f44-6cc2b8cbc62mr89798966d6.14.1729173908808; Thu, 17 Oct 2024 07:05:08 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6cc22921c60sm28522656d6.51.2024.10.17.07.05.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Oct 2024 07:05:08 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1t1R7j-0041fS-QX; Thu, 17 Oct 2024 11:05:07 -0300 Date: Thu, 17 Oct 2024 11:05:07 -0300 From: Jason Gunthorpe To: Christoph Hellwig Cc: Yonatan Maman , nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, herbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, dri-devel@lists.freedesktop.org, apopple@nvidia.com, bskeggs@nvidia.com, Gal Shalom Subject: Re: [PATCH v1 1/4] mm/hmm: HMM API for P2P DMA to device zone pages Message-ID: <20241017140507.GB948948@ziepe.ca> References: <20241015152348.3055360-2-ymaman@nvidia.com> <20241016154428.GD4020792@ziepe.ca> <20241016174445.GF4020792@ziepe.ca> <20241017130539.GA897978@ziepe.ca> <20241017134644.GA948948@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E3D581C0032 X-Stat-Signature: 8w8szzgqfib3cpsodrc3jzx7rgpmm1os X-Rspam-User: X-HE-Tag: 1729173897-142031 X-HE-Meta: U2FsdGVkX1+Jripy2FfUu2yXwD3dUNNT1qgSqPDocOe6VeCCkMtoUAqLPU6NrnhMnMoJSIviwuLMnJOv3Avfhj4G5KibrlOwn4ffiL26hNqsCYevhw/4tcKjTlg0aShuRapx9jwI6lr5mlVMg8k7BpikmkJILupM6gW8fOgx17jyTYIPySN4Rjln6Bf+dDIa+b6pCjBBm9UQGF81cmodpwPp55DUza36d5QzPX7ggKy769ddWmkyRN5lkR+S4YHwMaZ45dYDopZeRVIIZDc/iJ2ZMTMWA+MyUE/W13sjahwYq4a3Rp7FMqTIngJpkYRz+ikU3K7y5lKZOrXVk+sTfWilOlWk4xpwHpqgfU2EpXbjnqEkGp1S0uzAnvw15xOUrrK5eL9CVEaW81+Yl7CKTwAeCptOnA231C/TLCSRKuzCf/khR1q0t8iv3y1HehI9QFX7uvXqYcDP6VYWZqWPxGjVend7yqq4Lt9FsyAT1Pj7jkYsjOfiDTGXVPy2lkaIPpkO8B6+Tyn20qgXfwr4VgIkXEZVpQzNBJ2i5+c0vdSWVq4rVPx0mjyAnxtrJ4Mfac4XnHurmp3QyLcQTVW20rEBcRktDxHnABKccG8HMmIkkDDKdSX8rQmby+Gdyf+vy3cHjkK7sdTymA1hYXQTwFThDcyHegqSSB6bcNNwcp2vmvfullYKNpTuxZ9SopA1KQnXmx6wReSzKr0Yir6M7A2XN9IQTGmuREIGPojsHvl9CEgWsbOb2w7f+Op3jt/XMFmatlEj7uRk2moPIBRC26EiQpaI792jCYHC4jF8f1Lgm0BoffBlI6CHznqjAW0ZYGvn8bishyazBYZk+ma6TWbB+hJLLiN2ftKlRw+jdwSa8Qu9OuccMhLK0XiccV07a8XJse9oAFcNVfM+k5OG85zPu09hFTltnrSjjeWPQw0v5wIWd6QNXCaItXry9XccNj2RK4rnxOvkLatDsgh Y+nYc30x FjafZnbC9PwHCB/YitjunbD++qKAnx019HEB+DoafLvlDOWuEqZfy4vBCJGQy3T5ZfRl75hZW59Na+gSnYowk0jA4BWOxVaK1q7RFeCaGnp8mzK3637E5E8riDG76Kn8pycHC1qIXmq153APzRU1DvYWTgsXWYvyCNHIbp0+QzAy4n6lf9tWHC1YbINTBzHH0JOVMwUp2bdwyNjRyVPcbr6aPJCLumxVRys/U+AO7iunNcrTgrHlUJ+Y84ykWiwdVBXcvbYp4FPkTu716TmGOOkG6sRKxbhuTKTj/SHyx0Eki99Ac9c+YloEKzR+MCfAJaNCaD3YBh3E09bqSkLzKsTDKuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Oct 17, 2024 at 06:49:30AM -0700, Christoph Hellwig wrote: > On Thu, Oct 17, 2024 at 10:46:44AM -0300, Jason Gunthorpe wrote: > > On Thu, Oct 17, 2024 at 06:12:55AM -0700, Christoph Hellwig wrote: > > > On Thu, Oct 17, 2024 at 10:05:39AM -0300, Jason Gunthorpe wrote: > > > > Broadly I think whatever flow NVMe uses for P2P will apply to ODP as > > > > well. > > > > > > ODP is a lot simpler than NVMe for P2P actually :( > > > > What is your thinking there? I'm looking at the latest patches and I > > would expect dma_iova_init() to accept a phys so it can call > > pci_p2pdma_map_type() once for the whole transaction. It is a slow > > operation. > > You can't do it for the whole transaction. Here is my suggestion > for ODP: > > http://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/dma-split-wip OK, this looks very promising. I sketched something similar to the pci-p2pdma changes a while back too. BTW this: iommu: generalize the batched sync after map interface I am hoping to in a direction of adding a gather to the map, just like unmap. So eventually instead of open coding iotlb_sync_map() you'd flush the gather and it would do it. > For NVMe I need to figure out a way to split bios on a per P2P > type boundary as we don't have any space to record if something is a bus > mapped address. Yeah this came up before :\ Can't precompute the p2p type during bio creation, splitting based on pgmap would be good enough. Jason