From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD6FEC4345F for ; Fri, 3 May 2024 16:42:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76EDF6B0089; Fri, 3 May 2024 12:42:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F7886B008A; Fri, 3 May 2024 12:42:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 571436B008C; Fri, 3 May 2024 12:42:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D46066B0089 for ; Fri, 3 May 2024 12:42:44 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 47191811FE for ; Fri, 3 May 2024 16:42:44 +0000 (UTC) X-FDA: 82077653448.05.445DD85 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf15.hostedemail.com (Postfix) with ESMTP id 41066A0016 for ; Fri, 3 May 2024 16:42:42 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=KUSasR8S; dmarc=none; spf=pass (imf15.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.210.50 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714754562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9MCkvEl6vfBsR3xGXbm5ipuSQzsyNXlO2daQS9IANas=; b=dtwgQ41yjFKrzN3FH3XWwoCUUBdQTLiNI+pbBEE3dCsN0y0m/WphmhNdjVieYP/3JUozMI zoKfSHcL3y6rHGTp3Ns6gDM0AVjXiSq4WCVL99lUP+8Whf2XAfmbMnly+GHOxE8r+8Q+z5 Fj4SpvGBHiZP2tVPQwX36iey6aCAzAc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=KUSasR8S; dmarc=none; spf=pass (imf15.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.210.50 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714754562; a=rsa-sha256; cv=none; b=U5BlOJm2tRCeqzYx6vt8UH7XbEa/EK+mRdc9SARkhRNGcpdqK0dtLKu78a/qEeEZKsOOo5 /A5KQhazEpv95Dw+1ZgRvOLTRU+Uu6mBKm2pABWwVmVsAnxSTLfWztqC1AqzXkrOJb3A0/ UnxeeMJ7MRlBHgAZO8eVuNN+l6Rs+Rs= Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-6ee32bb9c8eso2920392a34.1 for ; Fri, 03 May 2024 09:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1714754561; x=1715359361; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=9MCkvEl6vfBsR3xGXbm5ipuSQzsyNXlO2daQS9IANas=; b=KUSasR8SqCbBi7aaSZg0oeZJpxNdxOlvG+tLzRPNVLKhQTkgQwkuiVcNpwCl7h6OAF 3qwq5v2s65TXuJ159+F2XjZBHxaqzdGv7QXvSckIDXpNiXzHpciB8mmXED5UekH2et6H HXNG6vaImVR9U5EMZE099Soc8f8E7sq13aqdb2pXgplfMA3jA+c+v+z5p725I5LswqEL gvORSu/JMdcAGkiFE2iMo7J9rbPDmLUqF7hLIZpDRiN8ebwctnghs+wWZdpjZKpx/Mqq SmuCrrNr9goE+mjOhWXzH28KECZynavW2vtxJpNnRwi7/PUC2gUGNeg8ic6mjR8tNCWG AIkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714754561; x=1715359361; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=9MCkvEl6vfBsR3xGXbm5ipuSQzsyNXlO2daQS9IANas=; b=d3gpRoF6XIzVuWMoWByVpQemWUznGaT1VhGT96FcckULyj9mpkzvBkCuHGukn+H0z4 ByME47QCDyao6alNOvzOe6/XTiPEMAJoITb5a3jJ0XZ/fBq8tbqMqwoYbdeIL+5GORU9 1sbwP5ODLPqHluaiBDjYtjsAkEgbiehy0nmEJqO/NaGm0b2JjrhDIxzXJV942ldmx3dT KiXtwpfjLfunXVJHJnbUeowBvuu42DTaC5k1NHwq6eJWI4fhjxkgTRWwEwQ0xPBlczq/ erLgNbjOD9Fa8l/6zIbpB7ur4TiYPgnSXOomTx8aOk6yqSwCVFImvryFQ2cnockXvq3c dHaw== X-Forwarded-Encrypted: i=1; AJvYcCW0XgwTjMlZo4wvUUqzUjnHB12QH6DNsfzv78jGLV2cvyX88Kaul6wPYJe2bSRD8Q+kMYy8A9saRpp8Fwwgxe82/pE= X-Gm-Message-State: AOJu0Yz++gZEfHE0E/emipx8S++xdXK04x1E/zyXokduUIJN3Iiz5F3k rYKPfoaxaxyav6qKLh4pzx6elg6Nhvz76Badc5zJP8ucT79VapOhIHYCCB7xY24= X-Google-Smtp-Source: AGHT+IFVT2rAsXVdMpvr8pAPF1WuNA1CE6Xv9L67MOMLKugSh13tMbQmSyNxDuuJNWHb/QOcrYHVYQ== X-Received: by 2002:a05:6830:33fb:b0:6ee:1cc5:68c3 with SMTP id i27-20020a05683033fb00b006ee1cc568c3mr3499852otu.12.1714754561211; Fri, 03 May 2024 09:42:41 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-80-239.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.80.239]) by smtp.gmail.com with ESMTPSA id ca14-20020a056830610e00b006ee5b409f23sm701317otb.22.2024.05.03.09.42.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 May 2024 09:42:40 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1s2vzb-007ETb-4i; Fri, 03 May 2024 13:42:39 -0300 Date: Fri, 3 May 2024 13:42:39 -0300 From: Jason Gunthorpe To: "Zeng, Oak" Cc: "leon@kernel.org" , Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , "Brost, Matthew" , "Hellstrom, Thomas" , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , "Tian, Kevin" , Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-nvme@lists.infradead.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , "Williams, Dan J" , "jack@suse.com" , Leon Romanovsky , Zhu Yanjun Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps Message-ID: <20240503164239.GB901876@ziepe.ca> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: by3d1gq9k3mfczwm5cnhsm3j8ww8t9b1 X-Rspamd-Queue-Id: 41066A0016 X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1714754562-342412 X-HE-Meta: U2FsdGVkX18Eju6rx6tF6EcQI6jKebswVCTjBgbGBVEPcxQrE6xSob4d0QnYLC/ft0AWdzchhfPMV2GIjCWFNHkijlkfGr5TB5eipBsQ979iI1kOgP/Xg2P/CnvUxT/2zAT8sbSwnWn12upaoIZR9rihQaYrDsk3ULDZCVjMDt5AvKxGGCe45B4Bp3ZJzaB4NgUfPwNlzOckPvmKeZelbyAoOM8PHKWkcL/2jcb2JS+uE1fxZt294HJh2viBXsu5QKJ//QPCAKy0OiFW6l8iWQ6gczirM8AH2GBBi2xavlf5sQaWXu47ah4aWliGlzlrNNNGUbm4UZtGVZOb81efVB2W90yBXkppWJViP4pXY4K/aNGmDgurZHpXhkQDfPmhYpUyqF3drpU8k+Wmb4XsASRS+F7yk0fmdLaL6+e/2yhCWcjDkqYqWuIkZsBk+BWwA3glh1LEVh+4JPVunKJHP6DTkug+EdaTAe5oj/2hEWdWJdIL+iZjh+bo8wO/1xf/dSCJdNASOSvdPgqMM+VRuCiWkxYuGd9Q4zRigJw4VqEbuyb+OnI3/ylETo/58nMLLAkbiO2cvHk+EOsobszsAXqFYv82mKYDWW18BcA3IIOpT5i2ZueH3VWhXl2iYaR7FGtBFvWWagLEhv5vYKX+/FsFtY7rCZLUUc936nkPhQf9zFrjWMX0JTUDCBHzwPwdsYdGRyKa4y0zVnmYz0mZBIbaZB50rNRxGmRcA2EOoIT64eSlUFEquqz+AivG8HpWOckFO55S1cJPSNRGzHPmrza4FYsp8M+UUtbEuQD3ge7unR7nJFz0G1YcZCkamHxJorIGWfMPA65E2oDpeSvXvT0AhZVs8PMGlCBMV7S5p/ISvzNdsuRbQ/jjeNErpCRseSczJByK+Q44ISHC2oXymRVUUXuLv/DYj9lVwLjRkKJSRpS8ck8QIfyFUIKjAKKluVVW3dvscw6qJrr9jVZ df4Q/ZTm mlDHDrTBSWHUH6pHwRu5gwmn0g25G277M1Jz9P6+2p3oFtkAzGDGnGkHxnB3n0UPBh7Wi8p6pR/6Gg7yUHAOx9GkLYNDoVQntmQp0dLpVqQgib/PVBnPN0U0kSLpgOoQkaLRWWbtbjhkQVNuD2nYIrZsM5l9AziIoqD/DKbSCqzyPcEdPuE26er6t/kI5t2/tF6/C5QJCc/+5N/SWfPS0HCBkOS4DuGdL2/Cdfbou8Akww1YYRfX0Rql78n/7tlDXOhdK8oZ7eZW8nRLx5rtyWHI4UFpsmtXwL85/x/Y8G5EZWvXQ/Zcxkn79Ll9OeAERnj7cQZHh5iGS5KYHERgK3mf9yuTnsB1P7Cqr6g3hnpR7Gow= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, May 02, 2024 at 11:32:55PM +0000, Zeng, Oak wrote: > > Instead of teaching DMA to know these specific datatypes, let's separate > > existing DMA mapping routine to two steps and give an option to advanced > > callers (subsystems) perform all calculations internally in advance and > > map pages later when it is needed. > > I looked into how this scheme can be applied to DRM subsystem and GPU drivers. > > I figured RDMA can apply this scheme because RDMA can calculate the > iova size. Per my limited knowledge of rdma, user can register a > memory region (the reg_user_mr vfunc) and memory region's sized is > used to pre-allocate iova space. And in the RDMA use case, it seems > the user registered region can be very big, e.g., 512MiB or even GiB In RDMA the iova would be linked to the SVA granual we discussed previously. > In GPU driver, we have a few use cases where we need dma-mapping. Just name two: > > 1) userptr: it is user malloc'ed/mmap'ed memory and registers to gpu > (in Intel's driver it is through a vm_bind api, similar to mmap). A > userptr can be of any random size, depending on user malloc > size. Today we use dma-map-sg for this use case. The down side of > our approach is, during userptr invalidation, even if user only > munmap partially of an userptr, we invalidate the whole userptr from > gpu page table, because there is no way for us to partially > dma-unmap the whole sg list. I think we can try your new API in this > case. The main benefit of the new approach is the partial munmap > case. Yes, this is one of the main things it will improve. > We will have to pre-allocate iova for each userptr, and we have many > userptrs of random size... So we might be not as efficient as RDMA > case where I assume user register a few big memory regions. You are already doing this. dma_map_sg() does exactly the same IOVA allocation under the covers. > 2) system allocator: it is malloc'ed/mmap'ed memory be used for GPU > program directly, without any other extra driver API call. We call > this use case system allocator. > For system allocator, driver have no knowledge of which virtual > address range is valid in advance. So when GPU access a > malloc'ed/mmap'ed address, we have a page fault. We then look up a > CPU vma which contains the fault address. I guess we can use the CPU > vma size to allocate the iova space of the same size? No. You'd follow what we discussed in the other thread. If you do a full SVA then you'd split your MM space into granuals and when a fault hits a granual you'd allocate the IOVA for the whole granual. RDMA ODP is using a 512M granual currently. If you are doing sub ranges then you'd probably allocate the IOVA for the well defined sub range (assuming the typical use case isn't huge) > But there will be a true difficulty to apply your scheme to this use > case. It is related to the STICKY flag. As I understand it, the > sticky flag is designed for driver to mark "this page/pfn has been > populated, no need to re-populate again", roughly...Unlike userptr > and RDMA use cases where the backing store of a buffer is always in > system memory, in the system allocator use case, the backing store > can be changing b/t system memory and GPU's device private > memory. Even worse, we have to assume the data migration b/t system > and GPU is dynamic. When data is migrated to GPU, we don't need > dma-map. And when migration happens to a pfn with STICKY flag, we > still need to repopulate this pfn. So you can see, it is not easy to > apply this scheme to this use case. At least I can't see an obvious > way. You are already doing this today, you are keeping the sg list around until you unmap it. Instead of keeping the sg list you'd keep a much smaller datastructure per-granual. The sticky bit is simply a convient way for ODP to manage the smaller data structure, you don't have to use it. But you do need to keep track of what pages in the granual have been DMA mapped - sg list was doing this before. This could be a simple bitmap array matching the granual size. Looking (far) forward we may be able to have a "replace" API that allows installing a new page unconditionally regardless of what is already there. Jason