From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80C41C433EF for ; Tue, 5 Jul 2022 16:42:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CEE76B0073; Tue, 5 Jul 2022 12:42:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 180E96B0074; Tue, 5 Jul 2022 12:42:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 046CA8E0001; Tue, 5 Jul 2022 12:42:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E882C6B0073 for ; Tue, 5 Jul 2022 12:42:08 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id B7C5212059C for ; Tue, 5 Jul 2022 16:42:08 +0000 (UTC) X-FDA: 79653613536.22.3ADD6FA Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf01.hostedemail.com (Postfix) with ESMTP id 3A0CA40014 for ; Tue, 5 Jul 2022 16:42:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:From:References:Cc:To: MIME-Version:Date:Message-ID:content-disposition; bh=C8Lk5zmtX8e6o/NFY86TvQ45i0l4+E0mfisBAA1KmfA=; b=K6XgjV5zJgzurqFOh3H10ZC+lZ sUOdc71G0WJz5hTQAbh6bf2AcHhJclo/g4AOwGPkLNfCmRojltAtRExhvHJnoH9avPf8waMiFwdfR 6QfEUWvhSAqff6L/H0Hg4F7xEVNd+PGjGUzrlqyRDBPkMnt61QaopxIJ9uchDWmlJA7qsH+OIR+Fd pLimKgtUIBwdevI9TLxxwqXaUfrqq5GmOO+kWs84rj0e+uugVEPeqnUGI/rZ9/9lzs6c4QbXv5MML z1G4DBTFmxGcsqb2cTCmQ6DcD8WjWnmt6IOZySclbQ+N0osicOYOzPz+sx+SlezlGNGG4gwbElr4t eTgsjL8w==; Received: from guinness.priv.deltatee.com ([172.16.1.162]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1o8lcb-007Svi-Gn; Tue, 05 Jul 2022 10:41:58 -0600 Message-ID: Date: Tue, 5 Jul 2022 10:41:52 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Content-Language: en-CA To: Christoph Hellwig , Jason Gunthorpe Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, Stephen Bates , Dan Williams , =?UTF-8?Q?Christian_K=c3=b6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Bjorn Helgaas References: <20220615161233.17527-1-logang@deltatee.com> <20220615161233.17527-21-logang@deltatee.com> <20220629064854.GD17576@lst.de> <99242789-66a6-bbd2-b56a-e47891f4522e@deltatee.com> <20220629175906.GU23621@ziepe.ca> <20220705075108.GB17451@lst.de> <20220705135102.GE23621@ziepe.ca> <20220705161240.GB13721@lst.de> From: Logan Gunthorpe In-Reply-To: <20220705161240.GB13721@lst.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: hch@lst.de, jgg@ziepe.ca, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, dan.j.williams@intel.com, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, dave.b.minturn@intel.com, jason@jlekstrand.net, dave.hansen@linux.intel.com, jianxin.xiong@intel.com, helgaas@kernel.org, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, rcampbell@nvidia.com, bhelgaas@google.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v7 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657039328; a=rsa-sha256; cv=none; b=hvlETeqpt659HkvvKDRM2Th4QRF77m8OYuRiDX11My14abNAivbNtmsijxxmUko3N4d4KU fIh+CCeNMYjlQw39YvPwKdHGnANgEEufHRmlzumaKPRlyrc+AasLe267TRlWnhryhNfqnO St1ikfW2pQWFO0gm+SUtZ143kKJPrnU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=K6XgjV5z; spf=pass (imf01.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657039328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C8Lk5zmtX8e6o/NFY86TvQ45i0l4+E0mfisBAA1KmfA=; b=HUr2PPZ6Ic40HB24pPN73oWMyUzzCtMRGVQsKvIzE7dV47KGUoWIvYHumWZ2CfzeF4iwVt lxu19/MTr8FSoCxVpk8hYZ9zsfNRTvk/L2f/ucRvOzZkU83uLVOplQMgWQGtXx2kW189jb csXCC7+3yzW1gd6A9aSkFfnsNyD0wdg= X-Stat-Signature: snj3equfiytkch1rir17tijcc5t4f6oa X-Rspamd-Queue-Id: 3A0CA40014 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=K6XgjV5z; spf=pass (imf01.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1657039326-366082 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-07-05 10:12, Christoph Hellwig wrote: > On Tue, Jul 05, 2022 at 10:51:02AM -0300, Jason Gunthorpe wrote: >>> In fact I'm not even sure this should be a character device, it seems >>> to fit it way better with the PCI sysfs hierchacy, just like how we >>> map MMIO resources, which these are anyway. And once it is on sysfs >>> we do have a uniqueue inode and need none of the pseudofs stuff, and >>> don't need all the glue code in nvme either. >> >> Shouldn't there be an allocator here? It feels a bit weird that the >> entire CMB is given to a single process, it is a sharable resource, >> isn't it? > > Making the entire area given by the device to the p2p allocator available > to user space seems sensible to me. That is what the current series does, > and what a sysfs interface would do as well. Yes, I think Jason is assuming the sysfs file would behave like the existing mmio resource files where the process doing the mapping specifies the offset and length into the BAR. That is not what we want here, but I don't see why I don't see why we can't do the same thing in sysfs as we do with the char device with a bin_attribute->mmap() callback. mmapping the char device was convenient in user space, but it's not much more work to dig through sysfs and mmap an attribute from there. Using sysfs means we don't need all the messy callbacks from the nvme driver, which is a plus. But I'm not sure how we'd get or unmap the mapping of a sysfs file or avoid the anonymous inode. Seems with the existing PCI resources, it uses an bin_attribute->f_mapping() callback to pass back the iomem_get_mapping() mapping on file open. revoke_iomem() is then used to nuke the VMAs. I don't think we can use the same infrastructure here as that would add a dependency on CONFIG_IO_STRICT_DEVMEM; which would be odd. And I'm not sure whether there is a better way. Logan