From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F902C35FFA for ; Wed, 19 Mar 2025 17:58:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6746280002; Wed, 19 Mar 2025 13:58:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEF03280001; Wed, 19 Mar 2025 13:58:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943D5280002; Wed, 19 Mar 2025 13:58:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 70D00280001 for ; Wed, 19 Mar 2025 13:58:44 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8C5A5C0686 for ; Wed, 19 Mar 2025 17:58:44 +0000 (UTC) X-FDA: 83239060968.20.1C1793A Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) by imf01.hostedemail.com (Postfix) with ESMTP id 92EBD40007 for ; Wed, 19 Mar 2025 17:58:42 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=CreV74ry; dmarc=none; spf=pass (imf01.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.169 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742407122; a=rsa-sha256; cv=none; b=AwILj+CaRXHed7/GuBX0IU5SofCQIv4Zm4oYHS6PRr0d5cdid0rdyPXj+FVIbFnDkd1ECu iMR+7XK8QpvwCGUq6iJXXZ8Ehs5oPUhcQVIA3zWK90HNsW+QKaE8/IjPFY4tqwrJkyYDMa dO7WLnpyEorGdR2tgIc/O3yYhUfg3co= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=CreV74ry; dmarc=none; spf=pass (imf01.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.169 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742407122; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tk5ycb3N9meE73NCm46CD0AtZuFHAdivFMZfwt24RPU=; b=Flp8uzdEo4bngtBLov+YquC9+COsu5uJlKuZzC0t/W/bT71AbJMmGueTfCd0Q2fLaykpKL +lKqqIIKXdqtmsx7zJ6C7m4tGotZQgrZSpS2pwWN27HvMx2EqsyLYRFJLJjLHi92S/EOxv 0DWQPbMl45sfHYe7Uso5l6HzDFVit3E= Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c597760323so241751085a.3 for ; Wed, 19 Mar 2025 10:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1742407121; x=1743011921; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=tk5ycb3N9meE73NCm46CD0AtZuFHAdivFMZfwt24RPU=; b=CreV74ryhIFjasNBquv+I0olzFbTE7vaXYDxlX3o7+tRErT8QIMGEzNTx4/dPohOQS ZCINU2Yy3eq3YB0KdErP8EN2wBERiE9p8AIn+jyyY4Ei05wHU2RIu8RHYMU/TZvnXtZ8 9BkmJI2nqFNgcOa+/p8ReuJP1Z+WyNTq2rCQHEHJ19pn0kBUTkBK2Frmw8VRt7DQ7Ssg IvNhIY+/+RXTRW/ULNMQlK/pCL16IGslqFiWWumKlA6T25mMutuW0JqVxBzUCZV1c7BI 9p8oJNfguP1fAJ7AwQTeTyKOwLUvEZTEHQL9ErIT9Z3c+XHomkBr2v2IRahGlXZ0EV6h Bd/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742407121; x=1743011921; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tk5ycb3N9meE73NCm46CD0AtZuFHAdivFMZfwt24RPU=; b=WMQ8yd63gYMtXk9GpGb/d6VcQRzpiiUFi5oGzwQdP4u2SbfShjBOkEqnJqZx4nraHG 4hUv+l3G7KA6oeflmZVJ/pJQBl0M/5Q32iliVj1VTZnyr0PdJ2yn965Ng2ndpKzxTPUY eTFuqvy64yPHcnKheR9A5c8O8jZHYPOonZdBIm9YGgmqb0i+F/o3ld2QT6U546Zg75on yz+V7wtelr7hAzUEkHTFVtwcGhDG0V2XbgtYM2y4wLSEnxUK2IptDYz8+NCK0F5tothY FnRLUxeDdZS1JefuV/AbuMKKMslmUFTL6D13XndRnRJAxlruVBN6G916k1ryrxjb/Afp qPFg== X-Forwarded-Encrypted: i=1; AJvYcCVKM5rFEaGfV8Ish4U0tBf6OL9qDe6KbJIpWwvaUOlVZ53e/7b+vTXD4c5TpHIdtDXHgLor4kmjaQ==@kvack.org X-Gm-Message-State: AOJu0YwpMHOjUihCrHMxp5RiVRtspu0Z77jEb/HRwtQdvXVoJzfn2EBn LyYbqzwaeN4ZA3zvzLNAKNCoS/ByC8WlWo3cw4la/Y/ZBr7bXH82RUu7Jsk71Oa8u5Nc72OecdP 4 X-Gm-Gg: ASbGnctwTRqAUzJQPhPR9Jyfwp6uBEXAQ5oavYx+35kBdZ2T6A7GtDDDVEOooIdP2X7 sX1ekX6F+Mz+rhU1pTei3gxXGKNuI0O5YjYtKsQjQAUsC4B8MKSnMK/u9Xf2cXLFgZj+ZkTVm9M UakqkKZ/uV8y+DbazbF220S/Sj6xSy04rla3392DN6g3aUnhRdksipM815Pm+ysjE0jfmpc3atL Sa3i3Nj8hkh83nSeDGgA9auAqMFxTl3PMriAjAEXTHugZ39PauIFwwFGu+kpZw1t4SpYapHujY9 VXPq0g6LOrohNsBv/Lu4Grq4w6MUHUmLb5bJcwN7XBkQY1VjONQtIhY7oItwwvbILvThcFMA40e qzChxD/RKH347wEn3Dw== X-Google-Smtp-Source: AGHT+IH//pIx1EBbRFqDU6QPhXC6odo+Ppd2dshOt/mNgQMR0e3nRHsssO7Vo2UonDgQd/fB3+YCbw== X-Received: by 2002:a05:620a:3189:b0:7c5:5154:cb2 with SMTP id af79cd13be357-7c5a839688cmr568357985a.15.1742407121681; Wed, 19 Mar 2025 10:58:41 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-128-5.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.128.5]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c573d8a62asm885096585a.96.2025.03.19.10.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Mar 2025 10:58:40 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1tuxge-00000000Wns-1mXJ; Wed, 19 Mar 2025 14:58:40 -0300 Date: Wed, 19 Mar 2025 14:58:40 -0300 From: Jason Gunthorpe To: Marek Szyprowski Cc: Leon Romanovsky , Robin Murphy , Christoph Hellwig , Jens Axboe , Joerg Roedel , Will Deacon , Sagi Grimberg , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Randy Dunlap Subject: Re: [PATCH v7 00/17] Provide a new two step DMA mapping API Message-ID: <20250319175840.GG10600@ziepe.ca> References: <20250220124827.GR53094@unreal> <1166a5f5-23cc-4cce-ba40-5e10ad2606de@arm.com> <20250312193249.GI1322339@unreal> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: 92EBD40007 X-Rspamd-Server: rspam05 X-Stat-Signature: kgipamgyg9dmkbchhbkue1bsjgq7ejd3 X-HE-Tag: 1742407122-251333 X-HE-Meta: U2FsdGVkX1/iW3CrjwZKJfUp+tCqP8JtrAnjGjkqJdsPfgYnYJJtL6jDxkqBfpYNFyTHX5QOPUJ4mkM1uY+6k5MGTYDCKd8AlC/3qOYlm3Ab9YvbkNEeyO2sbKq4uptyGlsajkWO1Y0QfAtGt8SL5+NGHN7DTsad34B/85msTlaGcDN4vS7burWW2OX8FOyQY5jX6aBqodQnTk3USztoTaAQsMD0JRySbh+5OJMSlxxz0BN77pycRklxm7x+rP9YWDW4+ODKs3jT+8OMKZPPYArCLosdkJcOQhp557kOfKShOLF1pzGQ3Pgv6fz/Y9IGHM4ocz56vrpSV2MQuJg1aijhntKzrVxz60cX+c5nhUFLP+cEL6LUN9wrg32EszpXidt0jJDqPKChipKkVibfh6ZPp0j5mD+CXWxp8p03S29o05I8FtvEI/TpgyyB1I+NpTULs63aXt83p51wsySZXZHzNC2ReF6+ohVx/rV2p1vZDXMbS2wGjZwI8IORmY5thi+sd5JZ8kiIrHilFc+LFMOB+ubLtt9I0ir/AqLTz0/8oV96uipgBia8x7JAZQyIZZnOuDWCA81ZnfT3r7SQfK1CmU/MZnrbgISuWxVLXx7/iob6Qhog3tZuOQt2YvrJiCAli1UR7yCsfEe29fZCcmDAXJ2GMoXGND8YhnSFAu33eoIavZK7odfo9NLUSPNDFW7B3eREgJz2PabL7NFvt5f1Q4LFVv5oAwimTH9prJZ8fuAVz4j/M6L9LqfXDYhi78FBTilLdvOkB5vNn1V0mk47XUi2dOEFIU/0Yo03vZYO3AFwkzjq98EJSyPw7xDXB5LyBeh2S0R+quqEMmAN0ZfnwSKhivW53UFMzB4YQOBM5mhgX+dEfb9V2xn8a88+LHuNv+x4z7taH8h7i9mlvZlaI5ySvxNhSnzKBrli+rOjvzIFdGI0apKqTA54AYUAGGDYy4EqN7I9UXMu83t 7vEt1wf1 s2PuigshTaLyjHr4qiZQguIEt821MSzEIO/UcyQR+91NdjY0sgJRcTY3PAoqDzlMJey4RHV2c9PjdyIjd4kBFEOlehkPbGV2v9hBYWFaGc974N7XkpIB1ASak6GSHLaF4/lXVIHYur6hrAaDxYg8i+l1sCxxjamjZRd0iV+Fn/5AimIqJ9OY5HOL9ij3LIaTEXjd2DdBg4XS2W8jWgPqwCFxFHHY9u1caU57yanqI6QKjd1Z5qdhqyrQ4oJJzbgPgZdvdgH/ruzc0xus4Tyffb8J2Jfh2XsYciV/9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Mar 14, 2025 at 11:52:58AM +0100, Marek Szyprowski wrote: > > The only way to do so is to use dma_map_sg_attrs(), which relies on SG > > (the one that we want to remove) to map P2P pages. > > That's something I don't get yet. How P2P pages can be used with > dma_map_sg_attrs(), but not with dma_map_page_attrs()? Both operate > internally on struct page pointer. It is a bit subtle, I ran in to this when exploring enabling proper P2P for dma_map_resource() too. The API signatures are: dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, size_t offset, size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); The thing to notice immediately is that the unmap path does not get passed a struct page. So, lets think about the flow when the iommu is turned on. For normal struct page memory: - dma_map_page_attrs() allocates some IOVA and returns it in the dma_addr_t and then maps the struct page to the iommu page table - dma_unmap_page_attrs() frees the IOVA from the given dma_addr_t If we think about P2P now: - dma_map_page_attrs() can inspect the struct page and determine it is P2P. It computes a bus address which is not an IOVA, and does not transit through the IOMMU. No IOVA allocation is performed. the bus address is returned as the dma_addr_t - dma_unmap_page_attrs() ... is impossible. We just get this dma_addr_t that doesn't have enough information to tell anymore if the address is a P2P bus address or not, so we can't tell if we should unmap an iova from the dma_addr_t :\ The sg path fixes this because it introduced a new flag in the scatterlist, SG_DMA_BUS_ADDRESS, that allows the sg map path to record the information for the unmap path so it can do the right thing. Leon's approach fixes this by putting an overarching transaction state around the DMA operation so that map and unmap operations can look in the state and determine if this is a P2P or non P2P map and then know how to unmap. For some background here, Christoph gave me this idea back at LSF/MM in Vancouver (two years ago now). At the time I was looking at replacing scatterlist and giving new DMA API ops to operate on a "scatterlist v2" structure. Christoph's vision was to make a performance DMA API path that could be used to implement any scatterlist-like data structure very efficiently without having to teach the DMA API about all sorts of scatterlist-like things. Jason