From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF879C636D6 for ; Fri, 17 Feb 2023 05:52:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA1916B0072; Fri, 17 Feb 2023 00:52:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E51D06B0073; Fri, 17 Feb 2023 00:52:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1A066B0074; Fri, 17 Feb 2023 00:52:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C28A66B0072 for ; Fri, 17 Feb 2023 00:52:47 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 90AB641427 for ; Fri, 17 Feb 2023 05:52:47 +0000 (UTC) X-FDA: 80475714774.29.285159A Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by imf10.hostedemail.com (Postfix) with ESMTP id BD35DC0004 for ; Fri, 17 Feb 2023 05:52:45 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=i4sVg3Rt; spf=pass (imf10.hostedemail.com: domain of smfrench@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=smfrench@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676613165; a=rsa-sha256; cv=none; b=v7c/vrAwh6pBW2d+0hzgsE0HBlxivSCz+wKEfBBejZKKJXro3OP18D+q9oNHxWemf3sRkm 4N7bFxKSqCMgnXzQeJ0ENSH5qR8HTr1yermytUyQVMcW/Zglv5ua+OzTb54Z1Qi/meF6/j 558j9pyD/pIa7phkA0QGvREwISJTYPM= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=i4sVg3Rt; spf=pass (imf10.hostedemail.com: domain of smfrench@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=smfrench@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676613165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jhKEcQNHLahXXVVAZN/VNTQNbuPo76AKJbBDzHJgOUc=; b=1k8Ini6Cv+0FO+qoK1KTRfMxf2bORKf0u1byXCJCtiRbZ0ZZe3zNhz0dRrmuz7rBtIqXvd gXbbtn3pHM8WWf6W2/RBoyY8wPdWaviJQQOrk5wwGArrViqAgb9lU9eMlve99WlGmBdEoa IkLsxI8n48RcS5UDLoPxQY1pVQkvGUg= Received: by mail-lf1-f41.google.com with SMTP id i13so526144lfe.1 for ; Thu, 16 Feb 2023 21:52:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jhKEcQNHLahXXVVAZN/VNTQNbuPo76AKJbBDzHJgOUc=; b=i4sVg3Rt5qkkB82twz6yJ7yklKP63AMrO5uciaCNTatfIbYzWS8W0yu9GSv9ssFqJB hosNtRGyWss72q3YGpVwmbaVKxVGu6atglPUQ/iglRVjU/wKcYx9h1WYpbV1FM/lCHRi Zad6gTaXNW7kYPhkIASR/UVWUfK6C67bp+sJj9VwVe9Y28DxA4wp5mn51ejjbXNXzuze wWkJvUlmizecxndGdkkm81ywSm4nrISQpOdnP/rk1rzeqhfpQ+spGUrXp81aRucPBEuE sb8XK4ZASIte04pVDASai2uT0gRiFnn4/T4mWeLedzZIM+BCArUR8RqopmLVtaJA7wDY brpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jhKEcQNHLahXXVVAZN/VNTQNbuPo76AKJbBDzHJgOUc=; b=tj4R0W3wi/PX5Y0rTG5k6EioqN5JbHL+Vh5GinPZNGAl6QnwdlOnKJNHsdylEL2CTQ 6w6gEaS6zRC+ike1rdPtxUar3ZHyWeyxgQwjBE1Is0MEMlxD+ubsaXLEECnWBGnUhU2J NNRwZkaRT/uyT011Vr/I8msn+0aHkLZCyOMs2MUkpfC7kaH66S1QygUr/o5jETb13QXK +OuC1INHITgu1eUM2K0UuqT+EAPrToE5NV+YD4lyhdMExZqQmR0yj7iB7d9hDd7Tzz++ 6Ho5uQhILd6tzlGW7H0wVYIBK+y1Q6k8qciSXYORJvoohpnxvBOWdUjFiVCh9shROv2Y b83g== X-Gm-Message-State: AO0yUKVZVcZmNGIRXfe85ReyRVl8JyjBkAyW0tF1dnyDRx5qQT8cOQmP BcZwLFnY/Uk/J0Exl1cW5ZcekCK0wpy/9ef97tk= X-Google-Smtp-Source: AK7set/cB6L3GkTNc49elZ1VVrbsKguVoD4zFbvB6iRnustoBN4+jp+084rglthKUbV/WhcVCix1pklnzzgNYJB23eA= X-Received: by 2002:a05:6512:501:b0:4d5:ca32:7bc3 with SMTP id o1-20020a056512050100b004d5ca327bc3mr78862lfb.10.1676613163893; Thu, 16 Feb 2023 21:52:43 -0800 (PST) MIME-Version: 1.0 References: <20230216214745.3985496-1-dhowells@redhat.com> In-Reply-To: <20230216214745.3985496-1-dhowells@redhat.com> From: Steve French Date: Thu, 16 Feb 2023 23:52:32 -0600 Message-ID: Subject: Re: [PATCH 00/17] smb3: Use iov_iters down to the network transport and fix DIO page pinning To: David Howells Cc: Jens Axboe , Al Viro , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Christoph Hellwig , Matthew Wilcox , Jeff Layton , linux-cifs@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Queue-Id: BD35DC0004 X-Rspamd-Server: rspam01 X-Stat-Signature: hmdppnzre14h78qhh8eapapaiup8mh44 X-HE-Tag: 1676613165-834033 X-HE-Meta: U2FsdGVkX1/0wSzffb0IfxiLvJVvawQJOqQI17jDxL0sf5cmSCFPOpBfnL5r2+pxYlBssTYtseiyAmbHCjP6XLcv+qv609xfbiYOODY1jX1OS+dxuCL7jVGvlR3gAB2yT3Rj/ccmlFSu6UR8hkdlQw11f0H7JcU8PnFVVbLZ6dW4CuAb0AItlV6E3zomjrr15BEA+MqNC0qzPAwu8i3/dpb/VrEf7UmY9CxEWdVyNpWzRfp6JL3HxXNIIP4eAa0Vk+4k6/KYXCTwTjq+MpZ8/Nf4G0Oqg/sau5HSHLNzLK0NI5YXqpDN1TzOuEq0ShMxXmu3+OdrgF4CGFbFZTm75kMQ3CjFelbm6b6ED4ICu3cfzl5tlCFSs3EGN99Bgm1eTEFDz8qny9M+2LM+EIZl0VUFmWjxGuHwOpHNK7oaWUKj7dv5ZV4lB1ljBZI+d1VQILBmJHOXsNdRN0qjn5hDItm1WdBzKzp7FosGYyeyEa4iGgX6T+miTfzQlxFlzvhejK13DJmWg9X5nUqmjXamRsDMJ1Y21bHS9TOIaQ1HJ9HWa7jlO1wZYY72Iae+qDaV12HJMFNk9PyQW7EsD8Onovmo+3XDi7tZ7EJABkriD7Q7+730if/4TwccF89NkNJJMl1Og+8hhUDPLq30gYQ2T8bwx6u1qqkLYv/ER9FXKLM/oO62RzsHew7l4B+3ux/NmmpVlOCIv3QvpHaDCHjej+jChY/pVLb4KqchZW4KlE1wcb7QGeU8Xp9zCcmSe2wb0+WnP/JIEddiFQ0MloQep3uuJP5BVyJHfGmugTl0OB7eIOSUT98/85L1I+8s+W1AhEKPLO1jI4CDnWboc9x2BK8VCcKizMroNf98vO1QdBEAjZuJb25FLwmrvtfOidJHoSvkbar6/JTX1j8N2iCgf6cdupJj+dD0b8CFFM11plMP2DB4lXmTpFIPlqA2lI++looVXnYQe1dynabLive 5KEp2/bR sRUwvfPxtFaNNHQgAC0kCZ1N6fQJQ1K0XuT/nyHDZ7DwabJadpPgjg5akOpzAO2LjsosIhGcrO9+d0Ke739XD3Is0D8l9//cUFvJhHUSbmMn/OfShg6uvuoUGFkG2xFHy1K+AEtP5VxKJgfQ9ra1U4uGafMZTky5rXTLYFs0eQg0zBUxoU3DxX9+Hp5XTyrmuTY+aE5/yIB3wP6Z+6i62A6NXVjtkXKi6XITdmoCdHFwmZfFpcUF1dPh67z2uzncP7n8Fitczl6gdUCVOoKyLO70e65/wVpjYDIsWK0Z3pR0yOU2vvLITfMP0Kr+3oeG9TEOQI0pLV9r4nh1SZSBR+t8vddUaTahQeCDk70ZWt7rHISNmxqXgtS75LT58eMnrnFmXAUh7oz7ImyJJikkMS+tnMBOb+rNRHlJwYPvq2qs0BuP6wjolLGPuiOFO4QeaW8WuSqGsrKycrm3KmUTCYr2FgirlSo5G+Uh27FEU1jEIiyIABSswN0eEv1oKM7jmUf7OJqmrgnZNKwoNin3u3lZIK+rDC6jTqJ4rB4AbeZ3qnPXgYc+GpakJLV4YJn7zIe7CijW93Fj4ZzKZkmBohfHcuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: tentatively merged the first 13 of this series into cifs-2.6.git for-next (pending additional testing and any more review comments) On Thu, Feb 16, 2023 at 3:47 PM David Howells wrote: > > Hi Steve, > > Here's an updated version of my patchset to make the cifs/smb3 driver pass > iov_iters down to the lowest layers where they can be passed directly to > the network transport rather than passing lists of pages around. > > The series deals with the following issues: > > (-) By pinning pages, it fixes the race between concurrent DIO read and > fork, whereby the pages containing the DIO read buffer may end up > belonging to the child process and not the parent - with the result > that the parent might not see the retrieved data. > > (-) cifs shouldn't take refs on pages extracted from non-user-backed > iterators (eg. KVEC). With these changes, cifs will apply the > appropriate cleanup. Note that there is the possibility the network > transport might, but that's beyond the scope of this patchset. > > (-) Making it easier to transition to using folios in cifs rather than > pages by dealing with them through BVEC and XARRAY iterators. > > The first five patches add two facilities to the VM/VFS core, excerpts from > my iov-extract branch[1] that are required in order to do the cifs > iteratorisation: > > (*) Future replacements for file-splicing in the form of functions > filemap_splice_read() and direct_splice_read(). These allow file > splicing to be done without the use of an ITER_PIPE iterator, without > the need to take refs on the pages extracted from KVEC/BVEC/XARRAY > iterators. This is necessary to use iov_iter_extract_pages(). > > [!] Note that whilst these are added in core code, they are only used > by cifs at this point. > > (*) Add iov_iter_extract_pages(), a replacement for iov_iter_get_pages*() > that uses FOLL_PIN on user pages (IOVEC, UBUF) and doesn't pin kernel > pages (BVEC, KVEC, XARRAY). This allows cifs to do the page pinning > correctly. > > [!] Note that whilst this is added in core code, it is only used by > cifs at this point - though a corresponding change is made to the > flags argument of iov_iter_get_pages*() so that it doesn't take FOLL_* > flags, but rather takes iov_iter_extraction_t flags that are > translated internally to FOLL_* flags. > > Then there's a couple of patches to make cifs use the new splice functions. > > The series continues with a couple of patches that add stuff to netfslib > that I want to use there as well as in cifs: > > (*) Add a netfslib function to extract and pin pages from an ITER_IOBUF or > ITER_UBUF iterator into an ITER_BVEC iterator. > > (*) Add a netfslib function to extract pages from an iterator that's of > type ITER_UBUF/IOVEC/BVEC/KVEC/XARRAY and add them to a scatterlist. > The cleanup will need to be done as for iov_iter_extract_pages(). > > BVEC, KVEC and XARRAY iterators can be rendered into elements that > span multiple pages. > > Added to that are some cifs helpers that work with iterators: > > (*) Add a function to walk through an ITER_BVEC/KVEC/XARRAY iterator and > add elements to an RDMA SGE list. Only the DMA addresses are stored, > and an element may span multiple pages (say if an xarray contains a > multipage folio). > > (*) Add a function to walk through an ITER_BVEC/KVEC/XARRAY iterator and > pass the contents into a shash function. > > (*) Add functions to walk through an ITER_XARRAY iterator and perform > various sorts of cleanup on the folios held therein, to be used on I/O > completion. > > (*) Add a function to read from the transport TCP socket directly into an > iterator. > > Finally come the patches that actually do the work of iteratorising cifs: > > (*) The main patch. Replace page lists with iterators. It extracts the > pages from ITER_UBUF and ITER_IOVEC iterators to an ITER_BVEC > iterator, pinning or getting refs on them, before passing them down as > the I/O may be done from a worker thread. > > The iterator is extracted into a scatterlist in order to talk to the > crypto interface or to do RDMA. > > (*) In the cifs RDMA code, extract the iterator into an RDMA SGE[] list, > removing the scatterlist intermediate - at least for smbd_send(). > There appear to be other ways for cifs to talk to the RDMA layer that > don't go through that that I haven't managed to work out. > > (*) Remove a chunk of now-unused code. > > (*) Allow DIO to/from KVEC-type iterators. > > I've pushed the patches here also: > > https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-cifs > > David > > Link: https://lore.kernel.org/r/20230214171330.2722188-1-dhowells@redhat.com/ [1] > Link: https://lore.kernel.org/r/166697254399.61150.1256557652599252121.stgit@warthog.procyon.org.uk/ > Link: https://lore.kernel.org/r/20230131182855.4027499-1-dhowells@redhat.com/ # v1 > > David Howells (17): > mm: Pass info, not iter, into filemap_get_pages() > splice: Add a func to do a splice from a buffered file without > ITER_PIPE > splice: Add a func to do a splice from an O_DIRECT file without > ITER_PIPE > iov_iter: Define flags to qualify page extraction. > iov_iter: Add a function to extract a page list from an iterator > splice: Export filemap/direct_splice_read() > cifs: Implement splice_read to pass down ITER_BVEC not ITER_PIPE > netfs: Add a function to extract a UBUF or IOVEC into a BVEC iterator > netfs: Add a function to extract an iterator into a scatterlist > cifs: Add a function to build an RDMA SGE list from an iterator > cifs: Add a function to Hash the contents of an iterator > cifs: Add some helper functions > cifs: Add a function to read into an iter from a socket > cifs: Change the I/O paths to use an iterator rather than a page list > cifs: Build the RDMA SGE list directly from an iterator > cifs: Remove unused code > cifs: DIO to/from KVEC-type iterators should now work > > block/bio.c | 6 +- > block/blk-map.c | 8 +- > fs/cifs/Kconfig | 1 + > fs/cifs/cifsencrypt.c | 172 +++- > fs/cifs/cifsfs.c | 12 +- > fs/cifs/cifsfs.h | 6 + > fs/cifs/cifsglob.h | 66 +- > fs/cifs/cifsproto.h | 11 +- > fs/cifs/cifssmb.c | 15 +- > fs/cifs/connect.c | 14 + > fs/cifs/file.c | 1772 ++++++++++++++++--------------------- > fs/cifs/fscache.c | 22 +- > fs/cifs/fscache.h | 10 +- > fs/cifs/misc.c | 128 +-- > fs/cifs/smb2ops.c | 362 ++++---- > fs/cifs/smb2pdu.c | 53 +- > fs/cifs/smbdirect.c | 535 ++++++----- > fs/cifs/smbdirect.h | 7 +- > fs/cifs/transport.c | 54 +- > fs/netfs/Makefile | 1 + > fs/netfs/iterator.c | 371 ++++++++ > fs/splice.c | 93 ++ > include/linux/fs.h | 6 + > include/linux/netfs.h | 8 + > include/linux/pipe_fs_i.h | 20 + > include/linux/uio.h | 35 +- > lib/iov_iter.c | 284 +++++- > mm/filemap.c | 156 +++- > mm/internal.h | 6 + > mm/vmalloc.c | 1 + > 30 files changed, 2515 insertions(+), 1720 deletions(-) > create mode 100644 fs/netfs/iterator.c > -- Thanks, Steve