From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77132C3DA41 for ; Mon, 8 Jul 2024 16:52:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0BE26B00AD; Mon, 8 Jul 2024 12:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE3696B00AE; Mon, 8 Jul 2024 12:52:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAB4D6B00AF; Mon, 8 Jul 2024 12:52:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BD1A56B00AD for ; Mon, 8 Jul 2024 12:52:42 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 470D3A461D for ; Mon, 8 Jul 2024 16:52:42 +0000 (UTC) X-FDA: 82317179364.04.B47B581 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) by imf08.hostedemail.com (Postfix) with ESMTP id 5E56816000F for ; Mon, 8 Jul 2024 16:52:40 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=DIhVuijg; dmarc=none; spf=pass (imf08.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.178 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720457527; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=44iLSgmAA8Jj5OzeQp5q3uGTMuRJb4ek81IJRNbfP50=; b=o8mARkTIHU/AGTPcXQBg+AMAvUSUB+Kr95lNdLTFbT3AGlGssHSj0HP8OT7yJwfeOMpBDQ GfDvi8uY6to7I05IZO60jzm32XLeqkNP3btfVuZiYJ98DBtKKHun5AJ8pP4zau3kKcdzYT PdwvXV9gweIGkFhtinWJzRN4V0ytk1Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720457527; a=rsa-sha256; cv=none; b=xb2XZerA2C3P3EfxkbY94fLOxbb64i4sg2SPKeVtXyy7N00MArWgEQcLfUxg/ItEfTFA6e u4Q7CwwTAzgGZrHXDCyUgF+vOTOcVs5ZaV9EZ4L4ppjIpL1n0vMvgngyw/9xfBGp2vbf0M GpZJtxuAjhwzdIele1sewQFSlk/RjpI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=DIhVuijg; dmarc=none; spf=pass (imf08.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.178 as permitted sender) smtp.mailfrom=jgg@ziepe.ca Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-447f05bfdcfso7505251cf.2 for ; Mon, 08 Jul 2024 09:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1720457559; x=1721062359; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=44iLSgmAA8Jj5OzeQp5q3uGTMuRJb4ek81IJRNbfP50=; b=DIhVuijgMpzb9HwhSyGE6nZnD/aGvFASTpIdD4m2Jv4lY96ZXyNg+m4OipdliFwLnB yj2f56SzDBy0kPB4PD8DeOgeyEhETorMZJwio8agv2vzSts+hWBKOiBATnB2Y6RCj8UK DIG9tMabG14lMseJfPWDK2MxJj3WP0+UjsIUlfDSrdRYlZEm+dCXwP3sn43G2Go1S1g+ Llus8dGx97KaSe4iW1U1y4+h1lrv4joRf5fs1KETbX1WO8cGd6I7lZWA1JDBncP5App0 i/As2Xgps912+5fyuxjiMXwdF0Bt6gHHKTn1munbCzaukUg47Nh0Z4Sl6K2HYmhSg5C/ UeDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720457559; x=1721062359; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=44iLSgmAA8Jj5OzeQp5q3uGTMuRJb4ek81IJRNbfP50=; b=eHX+qfc/X//yPQYp06UIsQKZsACvGrNgaerf9H4/uWiEnqWqAKCGMpAn/JicqRl9O5 PJz0ehx38V+e3tgYO5ujXMVAdqR2U5y5T3mFsmFZ+vVcQeW2LdRcK5A8TnRmDHa7/1np f6uxYtR3TJklSBamKb5zmCcNi+dmt3D9D9uG6ADTd2X5Wp7Y8TXE6rq4+8RX2Ue/CYH6 AV/UIeBdq07vjEFy+iBeJE99VDGqkX8X4wYZkNr0bxJrZ0PotB7sTYu7C30MSre9DnWa UwZ/YqZ9o/EsQUgtJLFRzMv7vUL67Hs2P2nChXMPLqYby+cHSX3UGS0mUiRnCux73DDP 72rw== X-Forwarded-Encrypted: i=1; AJvYcCVvxGaONCGcDP4sTuM1SHd32ijqKlYgliLZQlNPy6MvHaiqkpfaL48/jbezZIMaaz74qxf+a6WpgWIcyFT4eTzSXMo= X-Gm-Message-State: AOJu0Yw8SiZHenvFLj9EQR+b0olzlheZPU9pRDaiv4V0sRbPx1NGGF+l Ju965VL4FIGD03finIIw94NmSgnByv9bJLafXEw691blsmfQsst8/l6Kz84NhNM= X-Google-Smtp-Source: AGHT+IHmTDKEE4X7UqSrEePQ2yW2AFHAIlc2Xn5bxdXHYF1pSNmcIuJQ4brop8xp0q/dd20hJkGD6Q== X-Received: by 2002:ac8:58c6:0:b0:447:e532:b370 with SMTP id d75a77b69052e-447fa8aefa5mr156831cf.10.1720457559408; Mon, 08 Jul 2024 09:52:39 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-80-239.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.80.239]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-447f9b40389sm1202611cf.36.2024.07.08.09.52.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Jul 2024 09:52:38 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1sQrbS-000Uky-FO; Mon, 08 Jul 2024 13:52:38 -0300 Date: Mon, 8 Jul 2024 13:52:38 -0300 From: Jason Gunthorpe To: Christoph Hellwig Cc: Leon Romanovsky , Jens Axboe , Robin Murphy , Joerg Roedel , Will Deacon , Keith Busch , "Zeng, Oak" , Chaitanya Kulkarni , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH v1 00/18] Provide a new two step DMA API mapping API Message-ID: <20240708165238.GE14050@ziepe.ca> References: <20240703054238.GA25366@lst.de> <20240703105253.GA95824@unreal> <20240703143530.GA30857@lst.de> <20240703155114.GB95824@unreal> <20240704074855.GA26913@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240704074855.GA26913@lst.de> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5E56816000F X-Stat-Signature: smkrzgjr6inxubesjbh1h8g7icnprbs3 X-Rspam-User: X-HE-Tag: 1720457560-500804 X-HE-Meta: U2FsdGVkX1+s7NCM3tbOelUw1CtRK41oKg9iO1ZjrO2t6f1rOtBLmGhAhcmz5FW3RUbmopoHgs6/a8/Yk0LRvMnS4LddSdoGVnXsamf+cqnIT81/E4JO0VhPdgdjiuHca3qUcWqufCDclBmhsLVrlOSo2u3KDqYrJdjufSdjqjfsehST9PjLr47gLwOrY7Uij5K9c16FwTVlWIHX9Vk5/7JZXMiltJ8THd/hLSIEwgsVCKfdz3fAR8aRsaIfsaXLOfAvGC6EKKzUelXHJJlfD+CDFOHwbiC+lxMZyM5zRLEyU1Kjm5bQnpaIpt2tVLsKhpQsy5rYTP8bs+/ITDu0A1+Rxu3KP9vhCLi7w7nITWFBm6kxzFEOS84jSWUY8q5XCf7316PT0pFjEkrom96WFxIVNyRw6emkOQCWy0nqKYWxeVeLyvxzXXAKxm1S19EmS8gt3RFOzXkebllXXKUcM9/zjnYr07u+1wlaxiz5Czk7nm8hPS1jsUWVRm9NVIZhFxH6S5REamNNljgd/NCMiX2K0dsA2sp1V0f2hH03RqT8nm6sRtSNXJaFNA6r4WaF0X7CTLw0bBEPKPaWtPoLWTO8vM4AaManrzeCkFknuuKYIviJzusGsRkoZNtWkwOAiFaQFxj/v8RMyyDgoxB4vwCq+3UsnT33PhNIJP5XOsjit+y7C5EMLOVFRAg6AnpVRTVMKAEEREJ7eK0iiDb8HRyxng/5qYn0wPxy1NWbKIGMeBa8Bg4o/udIplYHGZt/S10I8OkutvPwgOwyc6WJ1zST3/TI/DaGivZIQZcJZI/PgKXBrBXdT07IYIEeIaZD9gpHM9CP8N9M//QYdyKITSxPjrs/SZdjs8y12K7BH4CXySRbtvAEPkyfZe2XhIYIIAOEDfE9TpuWK18zMNjTqkLLaqquR0B8BjyeOPH1TLxxwUlWf6I6yVtB9zsSntdHYGIAdSnMLHIC50SEx/o xDwKk50f uNGN8pYXeCD/8solKoxUt6f+FI9UcB7pIR8RXdaCKLun2Ra3u/GfGPBNpUeH8EGlSpwf0x/3MVU7C2dhzh9csGOn/Ml0u46l98JRMnjiSjYcIpkdeESS/Fm+Y2XSPaoonjbhxUMeRVR0Q+z4GMm7Vj9hfU6+zXpgZoopSk/OnHWFJcitJGFhgTFS8DGgGq108C6Pk1FqbXhH53KEqtZTGzJ/tHD4Zh+ThPtof1R/fu9c9/tqWw7E7vrCvLwV3c0fN0yPcUVz6+3pz3+8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000777, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jul 04, 2024 at 09:48:56AM +0200, Christoph Hellwig wrote: > 1) The amount of code needed in nvme worries me a bit. Now NVMe a messy > driver due to the stupid PRPs vs just using SGLs, but needing a fair > amount of extra boilerplate code in drivers is a bit of a warning sign. > I plan to look into this to see if I can help on improving it, but for > that I need a working version first. It would be nice to have less. So much now depends on the caller to provide both the input and output data structure. Ideally we'd have some template code that consolidates these loops to common code with driver provided hooks - there are a few ways to get that efficiently in C. I think it will be clearer when we get to RDMA and there we have the same SGL/PRP kind of split up and we can see what is sharable. > Not quite as concerning, but doing an indirect call for each map > through dma_map_ops in addition to the iommu ops is not every > efficient. Yeah, there is no reason to support anything other than dma-iommu.c for the iommu path, so the dma_map_op indirection for this could just be removed. I'm also cooking something that should let us build a way to iommu map a bio_vec very efficiently, which should transform this into a single indirect call into the iommu driver per bio_vec, and a single radix walk/etc. > We've through for a while to allow direct calls to dma-iommu similar > how we do direct calls to dma-direct from the core mapping.c code. > This might be a good time to do that as a prep step for this work. I think there is room to benchmark and further improve these paths. Even the fast direct map path is not compiling down to a single load/store instruction per bio_vec entry as would be ideal. Jason