From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C85DDC54E71 for ; Tue, 19 Mar 2024 17:54:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 130DF6B0083; Tue, 19 Mar 2024 13:54:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E0746B0085; Tue, 19 Mar 2024 13:54:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEBA16B0088; Tue, 19 Mar 2024 13:54:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DCEFA6B0083 for ; Tue, 19 Mar 2024 13:54:30 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 73A51161211 for ; Tue, 19 Mar 2024 17:54:30 +0000 (UTC) X-FDA: 81914538300.10.7901BBD Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) by imf23.hostedemail.com (Postfix) with ESMTP id 7CF65140012 for ; Tue, 19 Mar 2024 17:54:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Ls9j+IUN; spf=pass (imf23.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.167.179 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710870868; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D7H3WfkclcieHpBg89GTkZxaS3r7II1qq9xezXi0hVM=; b=lqcZOW7rnF+cUUKhstZPOzXN0FjHS4Odk6z/Dhh+pj6UD3edxFaf2ZctBFr0VpQ/Y6m8uy SFahhtdiu8bZKjeR8R0/BYvp5dTEqOWFrjXfeai1fWFMMVr4pkxyeE8iNuJzsDA1BaGxsN C47rWgmh4tFU+CbBERIEUhKsRWdDx5M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710870868; a=rsa-sha256; cv=none; b=AxeG0S8gZvQl5MuJuf7bKcdIKroiOc7q7i3dIToLbQuJ8lVQf9q0m3b1uF9QyD/VNfmMtG zu1LYyTdeDAa5J5M3ybyNsnsvhz5QXrSP8lhE/GXcQA6sbR1G4h7FXyAb2NFhN4M4vAcJB fCChYLR1DZ0tzlPvZgn4gs/+dQg4Ch0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Ls9j+IUN; spf=pass (imf23.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.167.179 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3c38f4e18eeso1113413b6e.1 for ; Tue, 19 Mar 2024 10:54:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1710870867; x=1711475667; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=D7H3WfkclcieHpBg89GTkZxaS3r7II1qq9xezXi0hVM=; b=Ls9j+IUNOJFYZrdPQVX90T3OFV/KjGVXWr3TbwHxen6bYUwJelN2A+TnBQsmKIhgBQ lA7jui5UrxwBI0cwbowoqp5WAuAjpsM+eEqXIlEV19nkpYX8PCFATDir4B6RkrWR8f/4 3yo4l9kL5jDDtTm/JeAA93QaQ2gc5pILR+p4fK8eFvjnKBrMz4drY75ytmkKCwJTSgsi lapSMCldFXZvGNNzdaFf3P6KhCsbuFjlq2Z3EBwHiZhTKQ5OsJYUCCKhnApVnqOMg8ll u6Ggb8mV2mWxwHTlsRMLaGqDZIns1vg2DKseb8ZFB1SO5E++Ifxe1w31VYEjrN4zOuiX U/SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710870867; x=1711475667; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=D7H3WfkclcieHpBg89GTkZxaS3r7II1qq9xezXi0hVM=; b=ws6d0czRH/wDKMaHGs6yz5RYnmkssVzowKKI0AJqsLn1ByaZXzZIvU8P8HE3lhKlsX NH038WQxBsXo+K+VyBqRRkGxooWPC3KpI0LqHO4m1gfx+ZJDyC78ILsV1Jb2Te7FgayY cGMB6NKrQ1pP/J5TG229877yFAKU1BX5RxcjPJru4bcyo3JRXuYPUgvwc5N24Bao6oNm CoI4whwpZjPZzmIFyx4M+N0fk8xWBqsvlVyh2Ig+PaiFxGXIfNEDdkSSzqMEaA31fpaL kECpyz1RHodp28nq0tDL1QU65uBSdzMfxvQSNewA+WFJyAlWToxxyjNIcjh+DG75xkmO dqBw== X-Forwarded-Encrypted: i=1; AJvYcCX4+nPqoEIgJB6zcC3ma7jbEhDGnivSMZ9qz0q7M3fdYTKbUKfcMTH57N6FZ8iaohnWdbyJMHwV0U4D3w/AUQZ0Btk= X-Gm-Message-State: AOJu0Yx9WkIlpnGfzqhccquY8KLTyHCNGY8pIJYKt6pa6YxfT1113+LR jrCdvqEX6B03WcW0utL7c7VOsVPXjO2HX0HUuAH1PPuV4wID2zyFZG+xgZ9M2qs= X-Google-Smtp-Source: AGHT+IFEKHu8FJxVRb4YxsELWM46dNSxNuN3njm0tBeD+R+fEyp1j8c0yaDa2gTZMCaYT5fxKBzgBQ== X-Received: by 2002:a05:6870:d20e:b0:222:99cb:2215 with SMTP id g14-20020a056870d20e00b0022299cb2215mr3580439oac.28.1710870867581; Tue, 19 Mar 2024 10:54:27 -0700 (PDT) Received: from ziepe.ca ([12.97.180.36]) by smtp.gmail.com with ESMTPSA id gh11-20020a056638698b00b00477716fcbb8sm2429986jab.40.2024.03.19.10.54.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 10:54:26 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rmbVk-001elO-4L; Tue, 19 Mar 2024 12:36:20 -0300 Date: Tue, 19 Mar 2024 12:36:20 -0300 From: Jason Gunthorpe To: Christoph Hellwig Cc: Leon Romanovsky , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240319153620.GB66976@ziepe.ca> References: <20240306154328.GM9225@ziepe.ca> <20240306162022.GB28427@lst.de> <20240306174456.GO9225@ziepe.ca> <20240306221400.GA8663@lst.de> <20240307000036.GP9225@ziepe.ca> <20240307150505.GA28978@lst.de> <20240307210116.GQ9225@ziepe.ca> <20240308164920.GA17991@lst.de> <20240308202342.GZ9225@ziepe.ca> <20240309161418.GA27113@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240309161418.GA27113@lst.de> X-Stat-Signature: u45amkcdso84ukbpkpzocsmc4n18sfso X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7CF65140012 X-Rspam-User: X-HE-Tag: 1710870868-182120 X-HE-Meta: U2FsdGVkX1/EhH3EnIMTnYy+m1VuArSEs/0fjSZVZnNye5b/UgHgng5hBSyjmZ9WHk4Xu5EGGem7UW8S/33pAm0R++P2+nSsS5zUGoBBgCUVEJU1PJi6FlhPRJDR65FTszOUA5C3zCHg2hT08oKqwIdgwelzwTwTD6Vie6cIuCUq6OpzhJr5lxFxZxG5pKfeNcRevxuPOUDXiFWP3zzVAhX3darJmH+Qysme+YMldUUpJvxHIrG5l1rXXBpwFyeaDyw9DpoO1dRXRLuMuB8q/jdvcTyO81Fzq984ktbOBiXgrdlCVs8k6qHWLhaAFTwUOnM35sN1Q0MSoEmsX6ZkDsFjFjmT2p8kxVZLx18qhk2vfYrcw1wG8HxsyiBdK5vvx6dIoJReq7GjKfMMgwb25hOUF4s3o1nAI1y3EIlTjSr3kxrZUpNH7/wuOBYXkiUTQLrURUoIuYDtJ8oFpZKoprAxoPJqsd6teDq1g7aLhFXAaR5tOhdDTxoQQPF3Rcd744b0clLy/NadcDN2m+GV/jFCE3xrxSU5aGEfcRS2phOOfO/5OGSqIUpBVBNein8VvdLpmYyKNDToRdS5c8EW0SLwxyokrh69H15lJTJxK/2/qN7oIU/gs1XqkyLID21b5Z4mn8Ot4SOj419PSci0/IUfsnxMf4DfAQoQaXEfuZNoE4ff8n7wahUAXLrDoy+pm4U5tZRTQb/iVT6EQQLOo6Se9zbOw9tKK6DNm44NzJP08QGLwK1mqf0uK6fn8iSsNLkI6imwaF6Vt6jdrNA88NmjHskbRKl2Bfb6Kn36KvHWpLwy2Pa/BV18vQRHrZJgvH419KmDXuGEVkKkTv4dZV3yD/XlkDtRo27xWPlX/zVHTAglvspOxdtMAdiVwSe4jnyCpOcT+RTWJ3xEtmINj14GaGogFrMS9tWM+SAVfgDD5M1tOJTtT/NGPp/LyjZ5PRxZjOzq+gDEGlyiMWK BLqarve7 zhP3ZVkN/Wqn5TWJcCPosN3ug70yCeC1cIhSoXkrUpJSWfDrc7lRHyzZkJbEdhoNGyUZ+rr1oQcqz/yIy6HhBtOgsELMob3CHqk6yhSkLs6X6aKjq2Yi89yypd9vL+7kwtud3fQ6DS2wX+nRO45rgNQBVSKwcVdudiF5f0euhyRmJ7poH47bq9MrVQ2MAzij81i8nahe/U4FOYajbmVjvi9pGDj+PTicsOAUUXSnZkuYVp3dSqrzB8TW13CR+a1YisADnFu9mj0pvxwdFZmaOI3NUBZXsS7CD5bYV40pYbUit+JrA3yHjhpcQC9gthWueETwfY+U+RJ9lknoL+d7gd/GQd8KEo8trAj+sXebGqlQVcUk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Mar 09, 2024 at 05:14:18PM +0100, Christoph Hellwig wrote: > On Fri, Mar 08, 2024 at 04:23:42PM -0400, Jason Gunthorpe wrote: > > > The DMA API callers really need to know what is P2P or not for > > > various reasons. And they should generally have that information > > > available, either from pin_user_pages that needs to special case > > > it or from the in-kernel I/O submitter that build it from P2P and > > > normal memory. > > > > I think that is a BIO thing. RDMA just calls with FOLL_PCI_P2PDMA and > > shoves the resulting page list into in a scattertable. It never checks > > if any returned page is P2P - it has no reason to care. dma_map_sg() > > does all the work. > > Right now it does, but that's not really a good interface. If we have > a pin_user_pages variant that only pins until the next relevant P2P > boundary and tells you about we can significantly simplify the overall > interface. Sorry for the delay, I was away.. I kind of understand your thinking on the DMA side, but I don't see how this is good for users of the API beyond BIO. How will this make RDMA better? We have one MR, the MR has pages, the HW doesn't care about the SW distinction of p2p, swiotlb, direct, encrypted, iommu, etc. It needs to create one HW page list for whatever user VA range was given. Or worse, whatever thing is inside a DMABUF from a DRM driver. DMABUF's can have a (dynamic!) mixture of P2P and regular AFAIK based on the GPU's migration behavior. Or triple worse, ODP can dynamically change on a page by page basis the type depending on what hmm_range_fault() returns. So I take it as a requirement that RDMA MUST make single MR's out of a hodgepodge of page types. RDMA MRs cannot be split. Multiple MR's are not a functional replacement for a single MR. Go back to the start of what are we trying to do here: 1) Make a DMA API that can support hmm_range_fault() users in a sensible and performant way 2) Make a DMA API that can support RDMA MR's backed by DMABUF's, and user VA's without restriction 3) Allow to remove scatterlist from BIO paths 4) Provide a DMABUF API that is not scatterlist that can feed into the new DMA API - again supporting DMABUF's hodgepodge of types. I'd like to do all of these things. I know 3 is your highest priority, but it is my lowest :) So, if the new API can only do uniformity I immediately loose #1 - hmm_range_fault() can't guarentee anything, so it looses the IOVA optimization that Leon's patches illustrate. For uniformity #2 probably needs multiple DMA API "transactions". This is doable, but it is less performant than one "transaction". #3 is perfectly happy because BIO already creates uniformity #4 is like #2, there is not guarenteed uniformity inside DMABUF so every DMABUF importer needs to take some complexity to deal with it. There are many DMABUF importers so this feels like a poor API abstraction if we force everyone there to take on complexity. So I'm just not seeing why this would be better. I think Leon's series shows the cost of non-uniformity support is actually pretty small. Still, we could do better, if the caller can optionally indicate it knows it has uniformity then that can be optimized futher. I'd like to find something that works well for all of the above, and I think abstracting non-uniformity at the API level is important for the above reasons. Can we tweak what Leon has done to keep the hmm_range_fault support and non-uniformity for RDMA but add a uniformity optimized flow for BIO? Jason