From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E935C433EF for ; Wed, 29 Jun 2022 18:24:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B43E8E0021; Wed, 29 Jun 2022 14:24:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63CE58E0015; Wed, 29 Jun 2022 14:24:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B6B68E0021; Wed, 29 Jun 2022 14:24:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 357D18E0015 for ; Wed, 29 Jun 2022 14:24:52 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0539720C65 for ; Wed, 29 Jun 2022 18:24:51 +0000 (UTC) X-FDA: 79632099624.24.95E759A Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf10.hostedemail.com (Postfix) with ESMTP id 3901EC003F for ; Wed, 29 Jun 2022 18:24:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:From:References:Cc:To: MIME-Version:Date:Message-ID:content-disposition; bh=6f/hfdoWC+PgJWoBH5fqwFGRbVxwuN3ohOs8yiOPg/w=; b=KqlIdYHpF5kcUd1uj3HgVczqN5 +LiC/fe4thV0n4GWyYvjongR/yShAMienfuQ1vSKr2dlutBWKxI1nszo45XAnij64gY60aWl25yKn 9yS7YZF5VmQ5saw9wKgfKuTqMswh1/nBQISRAQ25FYKkr6Le3Ih3GarYYS3dES6F4h5ipa5KBXImW +5AYt7ZvYxGywfXkmA46cmlkEvd9ckJZYnEqkIBKI2tycRJV07BzAnX9mR41JUrGOel82aqdx3sqr 111sVLF6oie65Xr6Z42l/GsfBOMAbGWd8Y9f26Frw8Cb4XUZDZkK9lUUkK39WvPZjSo78/eCvKrz8 ZKirfsHg==; Received: from s0106ac1f6bb1ecac.cg.shawcable.net ([70.73.163.230] helo=[192.168.11.155]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1o6cMk-002URm-20; Wed, 29 Jun 2022 12:24:43 -0600 Message-ID: <9725fc38-cbe6-f660-8d5f-6ceb983e6976@deltatee.com> Date: Wed, 29 Jun 2022 12:24:36 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Content-Language: en-US To: Robin Murphy , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?UTF-8?Q?Christian_K=c3=b6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Chaitanya Kulkarni References: <20220615161233.17527-1-logang@deltatee.com> <20220615161233.17527-2-logang@deltatee.com> From: Logan Gunthorpe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 70.73.163.230 X-SA-Exim-Rcpt-To: robin.murphy@arm.com, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, dan.j.williams@intel.com, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, dave.b.minturn@intel.com, jason@jlekstrand.net, dave.hansen@linux.intel.com, jianxin.xiong@intel.com, helgaas@kernel.org, ira.weiny@intel.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, rcampbell@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v7 01/21] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656527091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6f/hfdoWC+PgJWoBH5fqwFGRbVxwuN3ohOs8yiOPg/w=; b=CdrjJJXwk5jfLzMJvAbviQM19AFtQ3RTbKiS9T2JpErmc4uNRxbC4EY45CGhnhz76Trh+E Bgc0ORpSC2kgMqzlw6J6zhMbCWNAImLbaMYjZWsj774wzrY4vdLFxAiXv4jsKampkVQGCG YIOtuyDZ2HO3yjdHgzUWeSuukl5knak= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=KqlIdYHp; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf10.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656527091; a=rsa-sha256; cv=none; b=vPYpmtdEWd8tamhg2CaDsBz2Qp3ODAZb3qIFHKExNDCs6yaGr+wpBFDUsiFSEuOqwybMIM mO6k7wxNQ4T8JwTAL+UnZV3txecDKNvHPkJIztRjItgRmw7o+brhWbBePJ9NZk0gjEDT8A JLK3SxmGoG/rJEUsi9EtWK07djxcADM= X-Rspamd-Queue-Id: 3901EC003F Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=KqlIdYHp; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf10.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: rrag7xr5nyjj4i6swgxca68cng7kxzy3 X-HE-Tag: 1656527090-743337 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-06-29 12:02, Robin Murphy wrote: > On 2022-06-29 16:39, Logan Gunthorpe wrote: >> On 2022-06-29 03:05, Robin Murphy wrote: >>> On 2022-06-15 17:12, Logan Gunthorpe wrote: >>> Does this serve any useful purpose? If a page is determined to be device >>> memory, it's not going to suddenly stop being device memory, and if the >>> underlying sg is recycled to point elsewhere then sg_assign_page() will >>> still (correctly) clear this flag anyway. Trying to reason about this >>> beyond superficial API symmetry - i.e. why exactly would a caller need >>> to call it, and what would the implications be of failing to do so - >>> seems to lead straight to confusion. >>> >>> In fact I'd be inclined to have sg_assign_page() be responsible for >>> setting the flag automatically as well, and thus not need >>> sg_dma_mark_bus_address() either, however I can see the argument for >>> doing it this way round to not entangle the APIs too much, so I don't >>> have any great objection to that. >> >> Yes, I think you misunderstand what this is for. The SG_DMA_BUS_ADDDRESS >> flag doesn't mark the segment for the page, but for the dma address. It >> cannot be set in sg_assign_page() seeing it's not a property of the page >> but a property of the dma_address in the sgl. >> >> It's not meant for use by regular SG users, it's only meant for use >> inside DMA mapping implementations. The purpose is to know whether a >> given dma_address in the SGL is a bus address or regular memory because >> the two different types must be unmapped differently. We can't rely on >> the page because, as you know, many dma_map_sg() the dma_address entry >> in the sgl does not map to the same memory as the page. Or to put it >> another way: is_pci_p2pdma_page(sg->page) does not imply that >> sg->dma_address points to a bus address. >> >> Does that make sense? > > Ah, you're quite right, in trying to take in the whole series at once > first thing in the morning I did fail to properly grasp that detail, so > indeed the sg_assign_page() thing couldn't possibly work, but as I said > that's fine anyway. I still think the lifecycle management is a bit off > though - equivalently, a bus address doesn't stop being a bus address, > so it would seem appropriate to update this flag appropriately whenever > sg_dma_address() is assigned to, and not when it isn't. Yes, that's pretty much the way the code is now. The only two places sg_dma_mark_bus_address() is called are in the two pci_p2pdma helpers that set the dma address to the bus address. The lines before both calls set the dma_address and dma_len. Logan