From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9283AC25B74 for ; Thu, 9 May 2024 17:31:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AD846B008C; Thu, 9 May 2024 13:31:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25E0F6B0092; Thu, 9 May 2024 13:31:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14CA36B0093; Thu, 9 May 2024 13:31:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EDB4D6B008C for ; Thu, 9 May 2024 13:31:54 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 97661141524 for ; Thu, 9 May 2024 17:31:54 +0000 (UTC) X-FDA: 82099550148.28.6F9D90A Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf29.hostedemail.com (Postfix) with ESMTP id 9A0EA12001C for ; Thu, 9 May 2024 17:31:50 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=qyIg2OFG; dmarc=pass (policy=quarantine) header.from=deltatee.com; spf=pass (imf29.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715275910; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=KLv7MyynT4KBKlYpMAeljvNPeKJkIWm6jSoOqlGHsOk=; b=GKT/jF/gSDsEo9qzh/vU0Mb4qS08aoSVjbgZk0KvvdFC24Zxq/fusXy+JWtLQsVfULCZ7l uWj9OjC492mHuLVGlQQ8XTGDcDmT72GyD0vB1p/+uyIyesgLQl5sg0jeJlGvzoufJbsR2k CKRgwLOQ7KYYfjXUiT2UVrYzeR1X66Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715275910; a=rsa-sha256; cv=none; b=tYIMBHWEgKAZ+iJJ6t+ItzcgtKLhUhp3sK3Vk7C58MgnDxXsqqqcIYhvcW2n0/gGpOT7iy A3DJEcdLiAjSDpwCCniRZAzuE+vxtpniqQyq4jBfWoa6HR4dgXmW/omnOrIYgP/26LQrGW 8GD1tLHCURwM6xlTHkX9za3KJmkPeeY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=qyIg2OFG; dmarc=pass (policy=quarantine) header.from=deltatee.com; spf=pass (imf29.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:From:Cc:To:MIME-Version:Date:Message-ID :references:content-disposition:in-reply-to; bh=KLv7MyynT4KBKlYpMAeljvNPeKJkIWm6jSoOqlGHsOk=; b=qyIg2OFGPzM51pjipX4C+EC6qj 2l74tDOz8G9T0E66R4Zp5fo5B56QPOZL34MdAMsVZQpBXkFGucgJTkR3wxnL2EtDnzXBKssNeBszL ob3hr7t6+D2ePbiCBGuLeLyCUffY/8szTtEe3Ung3q2ebRxw1JOEW0qu/rO7gU2/gQNohgZjF1/RS QQy0vuKSJwY2IJJvwhYbCKfagJUCrLw3Tf834fSgR84t9FgBVrzoXIckTXR+wNSCq6/IMBr7gewG6 zE96Eu0IrqXy1Rhai0/ETTCYVwoYPVwpXr9S4H94lcV8iyuhHShtUmxfeziMz58jVA31MEcCG1aov kEzWxT3A==; Received: from d104-157-31-28.abhsia.telus.net ([104.157.31.28] helo=[192.168.1.250]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1s57cN-00ELue-Q8; Thu, 09 May 2024 11:31:44 -0600 Message-ID: Date: Thu, 9 May 2024 11:31:33 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-CA To: Jason Gunthorpe Cc: Martin Oliveira , Christoph Hellwig , Dan Williams , LKML , "linux-rdma@vger.kernel.org" , "linux-mm@kvack.org" From: Logan Gunthorpe Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 104.157.31.28 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, Martin.Oliveira@eideticom.com, hch@lst.de, dan.j.williams@intel.com, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org X-SA-Exim-Mail-From: logang@deltatee.com Subject: P2PDMA in Userspace RDMA X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) X-Rspamd-Queue-Id: 9A0EA12001C X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: tbfasigewewn5e874zkqy83n55jcf8jp X-HE-Tag: 1715275910-460515 X-HE-Meta: U2FsdGVkX19yFr1YoNHaPGeNkVf+0jDOV/55LibsAO0xeaAviCekhZEEpYrZdxb8QV5U+XwVQFx/h+9ZYyiz2EE3LWb4XrnOL4oiGf7TxDThlShGAvds1TdyLLByY/OQjJFilRmAaH122Dp+YBzTIsNrCp17TjbwFs41RvXMcHd9P8g4dL3FAdoCo6ba2Aylss3FXtG12CTOs1r+7eW1QgNEkhgxR7u8dRo0Em0HkoKFSDHhj/htAJ8epx/DHaXrvN/o2TjsfFOj5znY32kLfMjNhmxiMnQi7F0W8LURoBk4c/43fkXdxhPyejNKntGlWCPzwnurhqZtp0JCp7bm08ykMpOAJd8nPtUdwugihGZk31yNXgEyBjmNxWsqI1qiMTqMerptmKF88QsVD8SPdhTmNFJqMQG2/H1mGTS4O6/DG2sB/9cZYdsVpGHcNB/fYkRX4AHpT/DxxSX4mt5H/fbNVd7JzgtNiKLa+I70xYRLkJssS5H60dZQnaTrh38wvpuUv1pDxjIP4vfC2kZX5P+jfKNG1hskA1+7xsmpzdhkjfh0hgVGrdg1IlENG82qna3ztcnz20/kHWuu6Bi6oFe2IxZM6hDaysVH/gi6zRqRLf21YR2yaXauASUPLdyxhD5GC6+zTCAcuQ+TYNlTVUtO7QKNH2tMhQCB3+WeXBWifNo6ytRR0IvCi6DNdn7P9/pZOLOqPwhAQMYyJqrR0VbRzAo77qayRnYi2nHvhq12YVsfJ1MGflsd5ISLd5YEsCxLjBzOZxhuZg5eQJmQh0VlK76bkIbx01aJ0p0JBihQdVItQFlPxWb0HU5AYN+7MofS3yDucdEZTEsI4825KzXkpNe4Q44WAEF8Ve94MfgCRkR5s5B0+9L7s50objzgd/j6zToUs7UQkl7Uavnk1ISnxnrGqVef553/pVgADkCQcnOsGUecybLLn10i2J6vJ1aSzfNjjZejE7nphqW kUfnEkMi 15Dk6eREMyKmdn8RZryJ+jLwLZc0QG/0Xg4/p6GugbhLIOpSWyDHVcrarPyegXimXsjXQl1GbRvG8sMlv9wy8jJpe/L7Dq+wjLPjXV+QD8oeAzCszWrxD957J8POsqP87PYBFwOezVBS+LSKHVUkyHBXwrs5zTM2TZ1BCHs0uEqygZLJg5Z936GF4cldUtFZh/lY1OYkCM+DQIsWjiQlw3wKPatBdNO7uV+ut7c2XzQm1PQ2voL5C8/pJZ7Wmkx7ayJ/trdsrIvpMmwnH6FAVaKAq5r6RqXtG0hAOkOPq5iOu88fn6dK0Eq1iVtVFhTXUfPVuw6HYfG9tZXqgfW5wt6b33Z7MlYwNINB4LXyXs1Q6zOdPXgu4E55ROsVtMksMNm54JN4sMWjSCvNWhOzYVmTHnv3UNhsH3wicu4SxJoriz02eLx225A3g08kpCnoBGe9XJMwi+YqHO4kVQFyTvq4eGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Jason, We've become interested again in enabling P2PDMA transactions with userspace RDMA and the NVMe CMBs we are already exporting to userspace from our previous work. Enabling FOLL_PCI_P2PDMA in ib_umem_get is almost a trivial change, but there are two issues holding us back. The biggest issue is that we disallowed FOLL_LONGTERM with FOLL_PCI_P2PDMA out of concern that P2PDMA had the same problem as fs-dax. See [1] to review the discussion from 2 years ago. However, in trying to understand the problem again, I'm not sure that concern was valid. In P2PDMA, unmap_mapping_range() is strictly only called on driver unbind when everything is getting torn down[2]. The next thing that happens immediately after the unmap is the tear down of the pgmap which drops the elevated reference on the pages and waits for all page's reference counts to go back to zero. This will effectively wait until all longterm pins involving the memory have been released. This can cause a hang on unbind but, in your words, its "annoying not critical". Even if unmap_mapping_range() was happening outside of teardown, the P2PDMA code isn't like a regular filesystem in that it fully supports the elevated reference counts in the pgmap code and pages cannot be reused until the pin releases its reference counts and the count returns back to one. So I don't really see how there can be a UAF concern with this. Unless I'm really missing something, it seems P2PDMA does not have the same concern as fs-dax and I think it is safe to remove that restriction. Any objections? The other issue we hit when enabling this feature is the check for vma_needs_dirty_tracking() in writable_file_mapping_allowed() during the gup flow. This hits because the p2pdma code is using the common sysfs/kernfs infrastructure to create the VMA which installs a page_mkwrite operator()[4] to change the file update time on write. I don't think this feature really makes any sense for the P2PDMA sysfs file which is really operating as an allocator in userspace -- the time on the file does not really need to reflect the last write of some process that wrote to memory allocated using it. So I think removing the operator for P2PDMA makes sense, it's just a matter of creating the plumbing that allows P2PDMA to indicate this to kernfs. There may be a certain amount of bikeshedding on how this might be done, but it doesn't seem terribly complicated. Thoughts? Logan [1] https://lore.kernel.org/all/Yy4Ot5MoOhsgYLTQ@ziepe.ca/T/#u [2] https://elixir.bootlin.com/linux/v6.8.8/source/drivers/pci/p2pdma.c#L333 [3] https://elixir.bootlin.com/linux/v6.8.8/source/mm/gup.c#L1029 [4] https://elixir.bootlin.com/linux/v6.8.8/source/fs/kernfs/file.c#L435