From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51C6FC369CB for ; Wed, 23 Apr 2025 18:09:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 123D66B0008; Wed, 23 Apr 2025 14:09:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D3026B000A; Wed, 23 Apr 2025 14:09:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB40A6B000C; Wed, 23 Apr 2025 14:09:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C7E3A6B0008 for ; Wed, 23 Apr 2025 14:09:43 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 101FD1A19CD for ; Wed, 23 Apr 2025 18:09:45 +0000 (UTC) X-FDA: 83366096730.16.32FDBE7 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf10.hostedemail.com (Postfix) with ESMTP id 40863C0010 for ; Wed, 23 Apr 2025 18:09:43 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=gIeTyX7a; dmarc=none; spf=pass (imf10.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.181 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745431783; a=rsa-sha256; cv=none; b=CnMrjG5yvghD4p+vZdmRPc0f/jmPYoJ4krtis1WBWLJcEzQYK48BBw7nGzwTKMlWb8X7Sr lwwPf+oY+SCMjlbYenBoyPLLL74nYUd/Jhx1NMKcYgWnG9R6CqNwxF7Wt4Dd6OsLuWZ8q6 phpSy5wQo+g86FeAooDdBhkcM/Tnzms= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=gIeTyX7a; dmarc=none; spf=pass (imf10.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.181 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745431783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kL2C8z2XNRWzrThEWzS/MGHmoZb+Ha7gcgLYUEwZduo=; b=X4dSs9qH82ka1+UW2TWsVl8u//zmLIzMu8HRqC36ce0aXZiJWrqsYfrgsMFlSWP3qCRTmU YrfwpiPprhs+neOdqt99vAtPqNY7gj5J6epa5xBETbk6UWqzWGdlCytTGKoezlLroWymUb uti5JESgjcDLX+b9CPlyageeTqP1HwE= Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-47664364628so2002481cf.1 for ; Wed, 23 Apr 2025 11:09:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1745431782; x=1746036582; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=kL2C8z2XNRWzrThEWzS/MGHmoZb+Ha7gcgLYUEwZduo=; b=gIeTyX7al1bxTfGDHXySpUNOTaB3eY6b6Sktq/G2W99Pcbx++5VT2srjzGsurQkYJp gF/8+7E7+zcTHJEcG/xSJhbz2w1GipME4Dd54nGIsXAoAoHv217rD2RPu3oDO9DTU4Rm zYqZbkCKATW8A01kWuhZrFIepNZlZtJbxyP3dvuT7DJ0XiY2V9ZjyvK9GaIQG1EQdnHB PG7fJzesuhMuSACGt332afLrUze1axMU00UQ1CUg+FFH7obmQpWygKlQnfyNjEdJf0TW 9X84FqOcp/ypKsWtWim4XmrugLzBfW8ccTy7ILIUYqeCZS6CHJttv299LOm+krCWJ8K6 FbgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745431782; x=1746036582; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=kL2C8z2XNRWzrThEWzS/MGHmoZb+Ha7gcgLYUEwZduo=; b=Ew3iwyZMPmKaq5hCFNFxNaF3WU2ffdhZcN17wsdmjwfJJlElaKU9YeYmKSZjCHJDmd sUvphoIBAG7L4rDRuwCanl11/JasM5S4zRstsA1RXgHaWDIkSBPzPSmEDflXGRxhCI4p ViL3oe/yLGgksNXRrdhCY4j99J0Ll8oUHjsV0hJKJm01KeKIYoFYM4S6Nvqvq7KCLhml d/fnxkk98VkcuiivOFx6wv0mRqbZVUoj1WDnEO8aFOrloW+CS1EGjLnIKGimi7sivfCa +skGvjwnFvp4+/Uuz30JlunTMGt9B+cbAaHF5FYIvl5uj0t+hACqH2IBwsYRA9wZYc16 Yp/w== X-Forwarded-Encrypted: i=1; AJvYcCXiWoJZrO9L5uEZ8JOWkq/eRUCSYmLM7Ja7f1aNp4bWXzUtnWshD2NJ+YFx8fREes2zwEzi5uV95g==@kvack.org X-Gm-Message-State: AOJu0YxAPMO4ZvJR1tGrpmcEgRBd4wh18S3Njs2WubvlsWXwZ2YB2QB2 paDAdbqv45LjIGP0IvnQKB3A5SdGTfASFEujMy/D9P1NYuMZCVwqWb5VL+W4qJs= X-Gm-Gg: ASbGncv8g071EZP6tIqM/4DXcYtW38bH9idZUiTPy92dsPY5yEFBLmgfCFtylEs+fov PTtS6N0oGW/ebXSLy43QmNzTJ8ENyhQFMfhWbzgmSousdExC7rsBIbWvujqWCJGir3Go82fqks0 6fjRBe8wxTtHaDT2y/YHRzpIUdtGUZPeDvVLN7Jl9X4VEXHJWGxp0upxA914pNgaj2bo8mki4u3 S1O8yHcfdVwuakoEdig2IKDy0Vehm6QQ3xKJ9JvoXo8mmP5vNr1Xxl3GpV7yXzQuhMg9xSBttOB 1btPXP4xYxGn061Cs/QJjkuWS2t/hGSpYKa2ar0E73+01JFKRNE9ImmHDXuZeVfEXTYCwd/rvMr bPrhKJDvD788E00Omjag= X-Google-Smtp-Source: AGHT+IH4xfxeCQIatfvhUgdAx8fM2pKJgYvLzginEQw7PV9Sr3aShEQ97+Q+stgMKvhEecV5S0qx4w== X-Received: by 2002:a05:622a:248d:b0:477:84f5:a0b with SMTP id d75a77b69052e-47e780b3d65mr2842901cf.2.1745431782337; Wed, 23 Apr 2025 11:09:42 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-167-219-86.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.167.219.86]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-47ae9c3b485sm71764181cf.27.2025.04.23.11.09.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Apr 2025 11:09:41 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1u7eXV-00000007LuW-1WeA; Wed, 23 Apr 2025 15:09:41 -0300 Date: Wed, 23 Apr 2025 15:09:41 -0300 From: Jason Gunthorpe To: Leon Romanovsky Cc: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch , Leon Romanovsky , Jake Edge , Jonathan Corbet , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: Re: [PATCH v9 17/24] vfio/mlx5: Enable the DMA link API Message-ID: <20250423180941.GS1213339@ziepe.ca> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 40863C0010 X-Stat-Signature: 8pdyrtmdbf6zb31excmhqk8hh9h4p9je X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1745431783-956824 X-HE-Meta: U2FsdGVkX1/+vdceIpTR2+CSvQh3Lg8WOtsD8XP4rP62C4O/W8zQYJI2j2Ury198pjkM8E1IQbSmoHyVDdBNzEnSI9J/s6JISZL1GkJCENIIGGGY1D7K5DNEiucCY3ngDnQmmCXKshmj1jyncSafsXgQbgjkrI7jW+cokGacrYpZXs0a8NcYTVSJ9Yko/nkgk1r7dLhXT2VgpDoGZRtXb1stqZZZ8flMiOJ5tFOMvD1TbOcce1JpG5lrhK5GhMBFrPFAvwj000AZmgeGXgOlMyMPzBwHnPMd714l8d+YPnyK2pxRks0vxgglYTX0OexhxkutXpy9jtDUmrVaSUCFfEAbWMuqvg6W7BYzrfV39fthFJ2EfP1iMRO9YqF1aRSFNDV4PCk5ubMywR81RV8w462DvW6BW6KlJE9rvbE/wpZrzhchMZKSZ7t0x6EJYKNQjBK51JPSjAI4UHFk2W4Or+NJGn/48TMGrCzdA4/J+wZ+BoxZXUh4JtgpilR/UubwpNvzduBWe0tzIIZ40ET8Boq2FtXd98g50E4N6ikn9hNd2LRaVxnrwn4kvFfqinMDDlxYmaQBgs4GjyCfHNZ2Ne47dRblW5osQ/0Z5oOviFPYyz25WLB1Bw1Gdn+T2aUyXh5k5DbnEcbrrxG4pB04CQJi0l4HmiBw6h7xeFaytgl94K1dZR6NLO8RYnpQBlT/nt1pOhTYchTRZpO8scGksFHOhNB/uTlAldfOeJIGxojUMwCRDgfh+ViVoe/oTSgJMwWAN5Jq+RQHUUWvoJBHYpBdE6rt1r++KSxb8WNzFz5fM4lGONanGUKHc2rosyZsU+zG2on54VSch5nljvQioscnKRDjPStQz0FJzCKImE50393pKNLKMMufBpL07tw3fiuhmnBXYQeLb20LXp+JBJEam5D++XJBZ5dMFaBmo08N8ou84BJSZf9Hqxh8RqkD4SZFiMoeW5hxEIccMf9 HAQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Apr 23, 2025 at 11:13:08AM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky > > Remove intermediate scatter-gather table completely and > enable new DMA link API. > > Tested-by: Jens Axboe > Signed-off-by: Leon Romanovsky > --- > drivers/vfio/pci/mlx5/cmd.c | 298 ++++++++++++++++------------------- > drivers/vfio/pci/mlx5/cmd.h | 21 ++- > drivers/vfio/pci/mlx5/main.c | 31 ---- > 3 files changed, 147 insertions(+), 203 deletions(-) Reviewed-by: Jason Gunthorpe > +static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, > + struct page **page_list, u32 *mkey_in, > + struct dma_iova_state *state, > + enum dma_data_direction dir) > +{ > + dma_addr_t addr; > + size_t mapped = 0; > + __be64 *mtt; > + int i, err; > > - return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); > + WARN_ON_ONCE(dir == DMA_NONE); > + > + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); > + > + if (dma_iova_try_alloc(mdev->device, state, 0, npages * PAGE_SIZE)) { > + addr = state->addr; > + for (i = 0; i < npages; i++) { > + err = dma_iova_link(mdev->device, state, > + page_to_phys(page_list[i]), mapped, > + PAGE_SIZE, dir, 0); > + if (err) > + goto error; > + *mtt++ = cpu_to_be64(addr); > + addr += PAGE_SIZE; > + mapped += PAGE_SIZE; > + } This is an area I'd like to see improvement on as a follow up. Given we know we are allocating contiguous IOVA we should be able to request a certain alignment so we can know that it can be put into the mkey as single mtt. That would eliminate the double translation cost in the HW. The RDMA mkey builder is able to do this from the scatterlist but the logic to do that was too complex to copy into vfio. This is close to being simple enough, just the alignment is the only problem. Jason