From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6DA0C2BD09 for ; Tue, 9 Jul 2024 19:03:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 660436B009C; Tue, 9 Jul 2024 15:03:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E9636B009D; Tue, 9 Jul 2024 15:03:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48B186B009E; Tue, 9 Jul 2024 15:03:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2863F6B009C for ; Tue, 9 Jul 2024 15:03:26 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CDFE441A07 for ; Tue, 9 Jul 2024 19:03:25 +0000 (UTC) X-FDA: 82321137570.16.7EC46C7 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf14.hostedemail.com (Postfix) with ESMTP id E7CBE10001E for ; Tue, 9 Jul 2024 19:03:23 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=FZ9veROL; spf=pass (imf14.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.46 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720551788; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0eX7Wo0KTUJfWUHtxoHH+B23ObEinslGP6IJ1acrkRc=; b=u3M40O4yZQRjUdX8BTMgIG/Vwa5VBhLNKspHbJPirU6/7JCnqIkeGYSQOgWWvm7sntECNK MHaDm6kLgSkm1AmhauCnyCLBP0ejUb0NAMq1PGqCrw3ZsNWt9q5w0V4q6LuXpIsa1x7HTq bcAVqke4ux5lP3CCjEpJf1zuXTZzYMk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=FZ9veROL; spf=pass (imf14.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.46 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720551788; a=rsa-sha256; cv=none; b=JcXEkzqshQj87SLOlANtD0DbQGMBptNTAQ6S7XQzaktwZzmNZ4vt7sEhsWwV+z8FTWwUon K3eiupdL/niKhzJX3BKh9YSaCyuFF2NgoF4R51augi9BE7jSdiAOSJ6n1MMew35BaqqqnP J8ct+fEPFvOIFEB8OHeQXPKqmi93XAw= Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6b5f2ac0fb9so24421996d6.2 for ; Tue, 09 Jul 2024 12:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1720551803; x=1721156603; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=0eX7Wo0KTUJfWUHtxoHH+B23ObEinslGP6IJ1acrkRc=; b=FZ9veROLEap6SX4M8d9ROCPVGIEKq750cEVM7/Kycf05xcCmq+myAK7kwc4O7kOLcK S2Q0nzZVa5vyGa/zTRXqmd059IxIU+xf+us+tP2JH/vYMtF7Lik8TouVmKpy+eRyeu5F KwPwoS5Mzo/PALvseTW2+0/B0qd/XVZdMyYLopUBXz87UDOejEuBO4syvMU4arSD2ooD Pna/pGN3e/GiJSeZUWFETEsv88qNglMVqIkVHl6UgYwZtC0bpznBg0+pEB37eqBfF2dN 0fnmBTv6Q4VC8pZ5V/yVFCcgzzwurH9kh5fRO5GsYerhszVtBhqDsqEBOuAvIsIRqdBR bcUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720551803; x=1721156603; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0eX7Wo0KTUJfWUHtxoHH+B23ObEinslGP6IJ1acrkRc=; b=XhvjVIa9rNShj1Z1hnqxVo/95qOfRFyidbrAqsaEf22mZN+E6RfRGMDcVZBwmM/74P tpEFtG9swLaIOL7JJc/U1xXtU9IxRXm3y1ifKYHNAVvesYhtcKSZ/zIVWqIZGagFWGTy PQ+F8hPP5LKT+uUFhmZoplISu5fAPCxKcg6H2crsyXIFDYlJn/rpi54YFc2rNxuZT9RE QJ3yvB9oyKRJfcN1G962wPRevMVUIGsscYUuLJJhvB1DiJ52NhgKing/is7U30Qv0/fr pUh7E00p2XXikux9l9Ejuo4z0yDA/idViUa1Q0tvraCPpcsIvyeNMCZun3aBPYwYAnDK U3xQ== X-Forwarded-Encrypted: i=1; AJvYcCVZWlCjAxD2ugFQbhWqSXH8sCgZuPIo6jX5ceGXAuWmthC1TKdf/uGru55xe7plEjz551INFlvFTk2B4gzSOsxxTNI= X-Gm-Message-State: AOJu0YwokLrGkmb3PEKsmYjt+cyEr2ngld+wG1wLXFej50lmPzeFm95h l5sSFpCQaLLbByb6tYGBpDx0dVSTtFW9TlXiFwUsekrzc132I2u8YsUPpGhnZ+w= X-Google-Smtp-Source: AGHT+IE9wk6d6lAYU4tvyeORxb5NkdX/FMtkoFI0Rc5O+LsyZzi6JOQvD31sQUESKmTir4qJcqoy3Q== X-Received: by 2002:a05:6214:250d:b0:6b4:4470:81a5 with SMTP id 6a1803df08f44-6b61bc7f74bmr43617486d6.2.1720551802811; Tue, 09 Jul 2024 12:03:22 -0700 (PDT) Received: from ziepe.ca ([128.77.69.90]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b61b9c4a1fsm11278546d6.12.2024.07.09.12.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jul 2024 12:03:22 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1sRG7U-002tXL-BB; Tue, 09 Jul 2024 16:03:20 -0300 Date: Tue, 9 Jul 2024 16:03:20 -0300 From: Jason Gunthorpe To: Christoph Hellwig Cc: Leon Romanovsky , Jens Axboe , Robin Murphy , Joerg Roedel , Will Deacon , Keith Busch , "Zeng, Oak" , Chaitanya Kulkarni , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH v1 00/18] Provide a new two step DMA API mapping API Message-ID: <20240709190320.GN14050@ziepe.ca> References: <20240705063910.GA12337@lst.de> <20240708235721.GF14050@ziepe.ca> <20240709062015.GB16180@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240709062015.GB16180@lst.de> X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: E7CBE10001E X-Stat-Signature: 7g94x7zet1k5u7f4fucixdpzbf31wwat X-HE-Tag: 1720551803-835820 X-HE-Meta: U2FsdGVkX19dhAZFw7GOy7ww8AsiqG9Ks6zdJV8xQE/4fbNhElF0656/h8d6Rkr9brs63evZdj+vP0sAZwunYJmcLogrikuLirLJDFCZUCiVN0fSpygJy+NXLNOICqNqb037KPIsFhBY4eZWD1aAFMsiC/El3kVKsX7Aa5meRZvFEgNkxT8KQOywAqmoBuzVmv2wZllySex/ZNjLMk9XDSgbDmXWqyffneJHFBb5oGD68kMCABT61k5Pkd3w0JnT6lUxyqgcUdhCykgfjyfBx+sA+o/W0HzC8K35JHHuEieGDrNH3xIhoFZ9gx3F3wsByIUrU78DhhC58Y89p15jSPxNlkPCsu/A+FK9msYPQezfm9XbcMv1oGbFEjnXVF1fvW6jOA8FhCZyAx7r+lD8GxJFtUVusmJuv7I8uYEI2nvsmS1aCL23qtdWiavytkhLMApxW4vebZ6zjem8/jUjDxB0J6EjLrRGmxRMajvx5yJDOO8dTECdWqrK+z2wb8TJ0KC/EEHbK9ohgKZJVC98iU0sbu48HP3DAYZdLNT/Cwru0vM+V/0kUpWXq8cH2HaqJgdf/8adwX+Hwe7gJ++nBpFTHfJ7lQvfFZ90PvVbBuyxBwbDea+SccB0F+CaCLGDVNkZUu2z2x0z+iVVv6pqGCrpRNpLyn0LHFEWE4YaYIns9uQeJDwTeBf/zvXFNc9g+T0xaBT9/07kZM0o+jWVzYzlx6yXwHnJSMuLVOQRCQ2i3o/2lg/12e/IUbYlWkvJraqNvIzVZlofkDBfhc2pxWMfsmGF2eHO5e3wnfd0RNbRIZoAPglLzxN5aGtUPJiormtTOrOBj0W+3Y4qv5OZ5L9a1zoG3a4vzgAk0WWwytbj6QSmJ+Y4sYy1hpCNvVARAmaKjn/etkvD8Q42uf+hIgBjmhmnmaPZF1JW/fDB1oBzEBalHHk4dbifYHjkfSQMkS1HiM+L1+NukXzPXif WOKZPp/F dw6aAu4nIbk+/cR6wNItA82GDtZqOp5p2C9EF4liGPx59bXzlK9zOD7pX2pvQSK4YUDj8Gm5VtsQU4mdPNo5HAmlCMgPnYnVe7ejuV9q0s5QWIuRr9KqCeTxOdcyVBXAjDn++uKXehTqLbzr41bLYv2FzRhgt3PdKxPSXbpFXjsfNv8i8HsRMgc7YvKf6PLe+SxaNW6KD6GfKI330xRC6I6f6HXXKd0/IAYBuPZTRfrhzkC4bObYbbwn39MuFpvB0dSjtRcjZcikGQJY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jul 09, 2024 at 08:20:15AM +0200, Christoph Hellwig wrote: > On Mon, Jul 08, 2024 at 08:57:21PM -0300, Jason Gunthorpe wrote: > > I understand the block stack already does this using P2P and !P2P, but > > that isn't quite enough here as we want to split principally based on > > IOMMU or !IOMMU. > > Except for the powerpc bypass IOMMU or not is a global decision, > and the bypass is per I/O. So I'm not sure what else you want there? For P2P we know if the DMA will go through the IOMMU or not based on the PCIe fabric path between the initiator (the one doing the DMA) and the target (the one providing the MMIO memory). Depending on PCIe topology and ACS flags this path may use the IOMMU or may skip the IOMMU. To put it in code, the 'enum pci_p2pdma_map_type' can only be determined once we know the initator and target struct device. PCI_P2PDMA_MAP_BUS_ADDR means we don't use the iommu. PCI_P2PDMA_MAP_THRU_HOST_BRIDGE means we do. With this API it is important that a single request always has the same PCI_P2PDMA_MAP_* outcome, and the simplest way to do that is to split requests if the MMIO memory changes target struct devices. Jason