From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 026D0C27C52 for ; Thu, 6 Jun 2024 08:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87BB26B009E; Thu, 6 Jun 2024 04:53:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82C296B00A1; Thu, 6 Jun 2024 04:53:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F2436B00AA; Thu, 6 Jun 2024 04:53:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5285B6B009E for ; Thu, 6 Jun 2024 04:53:59 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 010FC1215D8 for ; Thu, 6 Jun 2024 08:53:58 +0000 (UTC) X-FDA: 82199851398.12.2A0EEF9 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf27.hostedemail.com (Postfix) with ESMTP id E87A04000B for ; Thu, 6 Jun 2024 08:53:56 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HiszMVWX; spf=pass (imf27.hostedemail.com: domain of zyjzyj2000@gmail.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=zyjzyj2000@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717664037; a=rsa-sha256; cv=none; b=WBsipBWI5uCgrw1d5UxpvvChNl02InJyYFvx9UBrimiUgEp2BLHZSxpoloT1HUD0hWKd60 cgl89CMBlkPJIHpcU6eiMsuz6NN/BFvHuqu14n/VfXQPqhN2Wr7/KTR3UhMpervm5PVKbK 5FseBM/142km4IQK4ovbScpeos2FbhY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HiszMVWX; spf=pass (imf27.hostedemail.com: domain of zyjzyj2000@gmail.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=zyjzyj2000@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717664037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6VqKhsb7zNsKsO0Rwy1Ho9vMPyDMj69wLB9DOnaMWfE=; b=gJjMhediFph3KVzSyzvnmgSSlr6L9JFBrHDlIHXG4eAVD+bUlEIpOGh2VGHU2t1177fIZc 56xLzvjEBv5qeE9E5GxojhVKEP+U51fliaF69NO+xt31nt1VIMr+lZ9GXv1f5y1I4Y9xMT YzFKZIGvHuo/5Lf+LZxDg8lWW3J1ipQ= Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-4210aa00c94so7793955e9.1 for ; Thu, 06 Jun 2024 01:53:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717664035; x=1718268835; darn=kvack.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from:from:to :cc:subject:date:message-id:reply-to; bh=6VqKhsb7zNsKsO0Rwy1Ho9vMPyDMj69wLB9DOnaMWfE=; b=HiszMVWX8YPvfrjeKiX6BF6RpL9UgKjqNZceilvYt6fc2vE5OzFj0+PkowY3qFInJr Bxri3OFpWwasPF/Z2remOFEH7TSxd2ezyAUAXuRlqmAbY8unMnJZeW1f1Xc3hJsz4QIR uuqNxOo7BU7+e3GIz6KlFLKiBzf1pwafh6F82KZeBpvKD6cywMVYAd4fAmhm/anboazC hd4p7ErSmkmjHJxv35f8BTzFu9gLu3CYn+uE4u98Y1TkkRAi4KfRGELvYRX5DeiMZ8gY 3Qa+xHQT1UkJ8i541UIGqBtm8hjsNXLKGKngLskfptDrs5CSEtwtZOmv4iEm0sxr1T0i sRQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717664035; x=1718268835; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:subject:user-agent:mime-version:date:message-id:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6VqKhsb7zNsKsO0Rwy1Ho9vMPyDMj69wLB9DOnaMWfE=; b=Jj+dNMJHQQzIcGQqGSjHOc/BiZDyydlbAmbhllUCz6A5eMauLIIa1k1G2mD60xvEok Pp95dtY7CPuP0TdnRp7rCvM/T+MvMr+EyG/JcqvAsShSPrTu87IUUtTVQskZd+vYPplO k4CJ4TM618LAX6mcX88JaCDaFArEfougIRoML53NELL8LXNyOv/A8e+cTnLZXsLngTlq xUmh6Ak7M0bRYJIXuzoBUcEKsMeSy9YTsqPviPMAZ0bNL/vz6n9L0e6bIu2G60FWoLsF 8EU1KzNrLvC49P438PZ1kJw3YUgM3bxNApKuiT+yzMD5Za7wWFd2bXNLiIfJQs4/t75r bFfw== X-Forwarded-Encrypted: i=1; AJvYcCXheisPfi5vtUP157RlJqfaw9b2rH4zicqOAE5htpBDJ6ltf8qLOQCs05gnyIQ8RgYIYqOOmpkHpH81YxVYnohWPcA= X-Gm-Message-State: AOJu0YyNa6Rtehmx/dsjwWCaJSkqJUHsjYmvD6zNywMGK46WCW4kOHmf YIlFoMm0DM0svgIuDlHOmc3z6XSLgIFxjaZXYRtn2pFv0hm4wvwO X-Google-Smtp-Source: AGHT+IEYxPKC2k/qAgc09dTYL+ijHyJ1hOhPMH9D51cPSwf9aJhwmRtpen34WhDJZuIh5e9CdV1zsQ== X-Received: by 2002:a05:600c:45d2:b0:421:2168:8336 with SMTP id 5b1f17b1804b1-421562ef91cmr38175795e9.16.1717664035118; Thu, 06 Jun 2024 01:53:55 -0700 (PDT) Received: from [10.16.124.60] ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4215814e7bfsm48107375e9.40.2024.06.06.01.53.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 06 Jun 2024 01:53:53 -0700 (PDT) From: Zhu Yanjun X-Google-Original-From: Zhu Yanjun Message-ID: <285e6cdb-f765-4969-ab16-abc53ad66a95@linux.dev> Date: Thu, 6 Jun 2024 10:53:52 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/6] Enable P2PDMA in Userspace RDMA To: Martin Oliveira , linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Jason Gunthorpe , Leon Romanovsky , Bjorn Helgaas , Logan Gunthorpe , Greg Kroah-Hartman , Tejun Heo , "Rafael J. Wysocki" , Andrew Morton , Mike Marciniszyn , Michael Guralnik , Dan Williams , Ard Biesheuvel , Valentine Sinitsyn , Lukas Wunner References: <20240605192934.742369-1-martin.oliveira@eideticom.com> Content-Language: en-US In-Reply-To: <20240605192934.742369-1-martin.oliveira@eideticom.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E87A04000B X-Stat-Signature: 7w1re1p9wx8da9uj18cbj6gnef816ddt X-HE-Tag: 1717664036-488517 X-HE-Meta: U2FsdGVkX1+lAU7vawKSMDr3eCaTgUL65P6kGLXzvQONDvobxaKSoKC1BilZPpYazfZB+o5u3C5E0bKwEm4qpzJrj38qRiKJR5zxKG1z+jgifoT87fJ5qNmaroslU3+OwVFnnNbhq3RHb9GXZsbaEbHSB6mgp/NgcyiCtBmEgBq1MdmUgsXVFC/D7cxv+gWo6ZhCodoNn68/RRgFUXITRL4t1M1N4WsndeacBgZLp1MSs81VvMV/ZRqiogZDhVkorbma0CztoYXY2d5tReMYvD4m1/tCSSwFDi+ZRkTpm1pQOypFVtjrzwJTnvs3Ozq2uVT7eaEujwG+4MbEreFkVtpP7pXqM/ndtzCWV2bZu1nbTjDgb7UkPg4DO5jzY8WPfOtkazrAhr3i/ZFhHSb3dBq44P1ClDYejrN1CfrwTZH0htu/S2FRmvTvyrhlw5pRC8B9l8+E/EsUDFK2Wz3gYDqDAx1MEoq+xFJFPtFslQHGn/Z2N++txCnt/5TheF137IKj9WoGqF/CkhNi1oF57AfivLW9uLwhPS0y/1GGSjwV01PQro70h21eohfuRTzJILx20Azz82Jbb4kC+8pn7RuJ6DKbzOGsnM2+DmA6jWk9/pFMm4gmYwtUsRm5wkxpr+U7HDYa/TB7fn3CO91AoUotPWgmVKhpRc9kykB0mQjL+ttaC8VqZU/xjqog0bpmXXyMGG0ye1AVnjNk2neWXWho+tBf/Gn0T3ba3xNxsSUpwiF2Pg8i+IbruhUibYbeCfhZ/RUUWKK42QCweb+72p5nxAZrwBqlhHXIDPtd4YSg8S/Ic8TtdcBqwxXm47TNwP3X1EVVN/w4NuZXrA89QE83Q7cJvFvFWjlXbkykaFTti2MvzxNg/ANhdU61F9yHeabEWliFdaFqMu83TegA2Ljyscv1cA8TXitRTf/gLNja1J0wniv4ABiPz8SgIlUDWn3en3WeT5CgQGEVw/G 2euHxtRu YN+eeBZJmGbXDcqVfLCyr+DI1X73BD50TeR+talxwUDQfEpQ+c/Sys8SgqjQyVslY1jjBJPlZV3IIyvjF3wTX963T5OoIGiFFFf5rgDearGh/IdOR7Yajjo6/DNb0LkH57CZtnxJnWDdyCwV+BiqVTmsHYK63PQ+j8OJ6ncM53ZDRX8HtBY1N8duCjfslPzedbQXyprGzR9PgTDgBMx4vvSTj6+Xqb8Lgbpk8vdyvX1V92c7dkVzmkCass+JxY6+3CdDRzrkWX+qJNfZBvfNaRzsk36l+mJfFtoRQCGen8utEyDMqruZtFLPmhjJ8+08Fm2ueXhMTtsBtkw3Jje97XSdFAl5K6siTk5OdsFknrNkhQSKY5vq71R1IzCKwepIbRnURmgw/F3eE+subc5FI8QtfsaP+EM0aA8YWclVvAtfBm9Y6eGLRSNWFKB3FL+g3wE6oBMqalsl9mGlqMCgYPKLmlQxbY1+yD3Hwek0kRPBE+i4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 05.06.24 21:29, Martin Oliveira wrote: > This patch series enables P2PDMA memory to be used in userspace RDMA > transfers. With this series, P2PDMA memory mmaped into userspace (ie. > only NVMe CMBs, at the moment) can then be used with ibv_reg_mr() (or > similar) interfaces. This can be tested by passing a sysfs p2pmem > allocator to the --mmap flag of the perftest tools. Do you mean the following --mmap flag? " --mmap=file Use an mmap'd file as the buffer for testing P2P transfers. " I am interested in this. Can you provide the full steps to make tests with this patch series? Thanks a lot. Zhu Yanjun > > This requires addressing three issues: > > * Stop exporting the P2PDMA VMAs with page_mkwrite which is incompatible > with FOLL_LONGTERM > > * Fix folio_fast_pin_allowed() path to take into account ZONE_DEVICE pages. > > * Remove the restriction on FOLL_LONGTREM with FOLL_PCI_P2PDMA which was > initially put in place due to excessive caution with assuming P2PDMA > would have similar problems to fsdax with unmap_mapping_range(). Seeing > P2PDMA only uses unmap_mapping_range() on device unbind and immediately > waits for all page reference counts to go to zero after calling it, it > is actually believed to be safe from reuse and user access faults. See > [1] for more discussion. > > This was tested using a Mellanox ConnectX-6 SmartNIC (MT28908 Family), > using the mlx5_core driver, as well as an NVMe CMB. > > Thanks, > Martin > > [1]: https://lore.kernel.org/linux-mm/87cypuvh2i.fsf@nvdebian.thelocal/T/ > > Martin Oliveira (6): > kernfs: create vm_operations_struct without page_mkwrite() > sysfs: add mmap_allocates parameter to struct bin_attribute > PCI/P2PDMA: create VMA without page_mkwrite() operator > mm/gup: handle ZONE_DEVICE pages in folio_fast_pin_allowed() > mm/gup: allow FOLL_LONGTERM & FOLL_PCI_P2PDMA > RDMA/umem: add support for P2P RDMA > > drivers/infiniband/core/umem.c | 3 +++ > drivers/pci/p2pdma.c | 1 + > fs/kernfs/file.c | 15 ++++++++++++++- > fs/sysfs/file.c | 25 +++++++++++++++++++------ > include/linux/kernfs.h | 7 +++++++ > include/linux/sysfs.h | 1 + > mm/gup.c | 9 ++++----- > 7 files changed, 49 insertions(+), 12 deletions(-) > > > base-commit: c3f38fa61af77b49866b006939479069cd451173