From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06631D68BCF for ; Fri, 15 Nov 2024 20:32:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66AA66B0088; Fri, 15 Nov 2024 15:32:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 61AF26B008A; Fri, 15 Nov 2024 15:32:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E12C6B0092; Fri, 15 Nov 2024 15:32:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 33B936B0088 for ; Fri, 15 Nov 2024 15:32:09 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E6DA6160184 for ; Fri, 15 Nov 2024 20:32:08 +0000 (UTC) X-FDA: 82789475874.02.642F137 Received: from bout3.ijzerbout.nl (bout3.ijzerbout.nl [136.144.140.114]) by imf08.hostedemail.com (Postfix) with ESMTP id 73F59160029 for ; Fri, 15 Nov 2024 20:31:36 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ijzerbout.nl header.s=key header.b=QeHMhT4Z; dmarc=none; spf=pass (imf08.hostedemail.com: domain of kees@ijzerbout.nl designates 136.144.140.114 as permitted sender) smtp.mailfrom=kees@ijzerbout.nl ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731702547; a=rsa-sha256; cv=none; b=ArNEJuANimKh9cnxPpQGEt5qcTzi4qOghQfCL6r64h2N+W0Y6BUWC2vciJ55uQnE/pon3u ElxY3puPvVQNGbysvFmpu/KtFGgdR6gihsSYaOU/d258Mfmii1o477RA3MtMEpp9dzHG2V 7j3STDNQE0xkQuIF5A6xzrRPupu1Qoc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ijzerbout.nl header.s=key header.b=QeHMhT4Z; dmarc=none; spf=pass (imf08.hostedemail.com: domain of kees@ijzerbout.nl designates 136.144.140.114 as permitted sender) smtp.mailfrom=kees@ijzerbout.nl ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731702547; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v5hJaYvL9y6KQTlCZ+gj9BUlS+HUfMaKtYKI2JbOETk=; b=v4JttE+Kc38ac1m1FZgUOpUxh5aJ1ctoCC3MzolkJ2+kswZvRl5czsHgn54F2EvHn6apkN deQrFetUPJmKvUDetEovJ+CIDfgGLoF1JqTkyXL+JRqd2XShMJKiMdIs+VlqsLW0UP3BMz W4W6gvBEbLG2BLj021JbyHDPOoNYUqQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijzerbout.nl; s=key; t=1731702725; bh=aW0kZXCjwMJkAvO9eGrMTuU564xR45DjHMlx+V4e670=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=QeHMhT4Zs6fgL1xTGbR5gC4hQqUmKKeyRJFxO6CGGf7WsIf7m4fxhFOcKeoK0KrMa HWuLBlIm2hXivW4vyOHHsZ2R8gfLbGyax+8HF4AMY+G+vp0ly7uX4qMLmxVMNdIfsw GN3/MV1yHiHGsVhiAly54eawmHiD1EC1TV5rvt3zKe3oSvjRgqbiMGzox+AgYKiICK p/pa8VsdyE7qHc4SiqawWHBxzK89GtQsK12xNh+Y3ZriJNsJ/LyH3dI9kHt7bau2uJ VlIJ5dQncKjSE9lG+ShI7oYB7LmpwL1zbScAmtVxtPMd1SQxtKSJ7zUavRnC1kWV+/ j0lhNwUHsLyQKY0k8s/okKEBXRAoP2rr0D09L0WvH+hO+0ERvYy/SWvKzYsPFz8VX8 s/mKq4LASMQ8xdNuxbwaMkuJ12rn8YpFb1iSSd8giG5FP/5jm5c918mouNSkD2/91Q OcLS182MwxzqpojOVoINTZ5JXbG/tl5qMcKIfPKlvMCAy0myE6EC3Q5FZKzvpXhZyX i8O8Q47umz52anLbdpycCdz7+f5pd4+Egishgq9vRBTJCK4CKl8BX1nIYlVJh1IYCT D6tQyiAKznHrv/y2Bjui8nj+WkPgHsSkEyGN7GNJEtc1Z09bFtiJg0VUioI3mV8Eq8 UFg72HmB4pDw9/qKIJBxNRiE= Received: from [IPV6:2a10:3781:99:1:1ac0:4dff:fea7:ec3a] (racer.ijzerbout.nl [IPv6:2a10:3781:99:1:1ac0:4dff:fea7:ec3a]) by bout3.ijzerbout.nl (Postfix) with ESMTPSA id D88A216029D; Fri, 15 Nov 2024 21:32:04 +0100 (CET) Message-ID: <17eb79fc-ccd9-4c85-bd23-e08380825c41@ijzerbout.nl> Date: Fri, 15 Nov 2024 21:32:02 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 23/33] afs: Use netfslib for directories To: David Howells , Christian Brauner , Steve French , Matthew Wilcox Cc: Jeff Layton , Gao Xiang , Dominique Martinet , Marc Dionne , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Eric Van Hensbergen , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org References: <20241108173236.1382366-1-dhowells@redhat.com> <20241108173236.1382366-24-dhowells@redhat.com> Content-Language: en-US From: Kees Bakker In-Reply-To: <20241108173236.1382366-24-dhowells@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: 6hb9hcjshykhpiajha5waser1t8t5q66 X-Rspamd-Queue-Id: 73F59160029 X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1731702696-449644 X-HE-Meta: U2FsdGVkX1/WNX5CZzRyL15zSLEC3Y2Om12LGuHm88rhh8YgHuzQR0jOmKWsFeFNg89JVUBBeYvHforxqPsHMqmw4hPrpLkfsrxNCoRcTGCNCVM0MlCKS7TKQOLe97rVie8Xw2LMp1RsZJH1jJgf7jEzXeMxCKxohRhH7JY1iZpnrFI8Uojlif8L/z+aikyOxQTWI3dIpYyLmKbvW51AZ83Zy6MhDCSxT+qK0UErS3CcoVhzubB8ZCSyOiNFnJO2VQogj+bqKCcrHAGENB+7yB4SCycfrbTL5DoNXNQ8PrdaOYYrxixTgi+FPnrFlEIXQfSCK149PKfe8R3ICltYDZA5tRbnWFJa/RXoBIuqyhOg0FIjV3HSNxz6skRc4DOK9osNvZThV0K5Fu4VkVB5ZnaDBTxhVEEZy41aX41AXIplFIQleSB7g0j8XgqAiS9DLTw3dzcEUQI8wtLe1SnBotqlPEtVuTw5E2TEmpfdAD+tsCueKzX3ngaHX2rq4voTpZaHkx69n3NkpCSBlZUHs6/8X3OLC2Su7nkKgJm0Dnp5WVvtw2d1bvUpVuaykmL06nb/gNhcpDXRmOiRNaMMHdNEuP4/Q81YvQNtfWxCwB+cK/7LD3FKEB+mBp7fsYlaVM/QrNbBAFwSCsJPGB5CBr6Pss6beTame+e5RjnYlph2NxB9GrXtmc65Nin8N4ZDtLIZdjwV/YPrxUdWK/QuPpsr6XDkmJrYU8mecrp2sP1etbPTCvj338baCPRe7PoctJOezjMerZJs1e7WS2KlXY9qMoKxbIhgG9zOn69xg+hAdfAYkS0SweS8f625i3bKSOkAlR0RZDIrguv2fDhG++SJl4J7PvkyP5H23zwBy0v5S8zezWj/htV0SadYJlCWM5TbJr42HMe9l+1K0A0GxNv2uQu55PFUmVyFGXZUBECETg5a7BGpLAKEI7Fa739ahq0/xPXM1kOOBtjwmUz ZZ5dus6G qmrM+y10FiQn7OZE1+V3Io5stMT8Lr78BTyBVXhUCTxQYZKALMAyfMYtopqPnZKDL20U+JO3T2kavcnSmkjX0qvPLZpU3+HySlVNVmK94mJ+tJo83I2ZyLBxrjBP2L4jMSjxwYwLMDcSNrvtdFm/M4Ot6xtCz1zntYrQx5jECiegCnePD1WcT6TsT4ktMjUcAPsC/8C56r3P/Ynv6Yn7N9LUG+WYuV370ZrAkAXVcAvEEXWNvdpCSuq4mm/XgORijoxnJldZmQr70fGzeZFzbl7lItZY0UQMB8pBfY5sPNrFMOGIfWOukmU/qKuqxFPuQAx7PBqEV5tYz4ToRfkxJkngq8nd3OI5Bx009DtnIans8vbguhxaBh5L3YoWwkU7hPKqlTgVbnJfECHaXwKuKOOu/eQq3zv6nGQxCW5IsswrXKQ2fir2nT61nhg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Op 08-11-2024 om 18:32 schreef David Howells: > In the AFS ecosystem, directories are just a special type of file that is > downloaded and parsed locally. Download is done by the same mechanism as > ordinary files and the data can be cached. There is one important semantic > restriction on directories over files: the client must download the entire > directory in one go because, for example, the server could fabricate the > contents of the blob on the fly with each download and give a different > image each time. > > So that we can cache the directory download, switch AFS directory support > over to using the netfslib single-object API, thereby allowing directory > content to be stored in the local cache. > > To make this work, the following changes are made: > > (1) A directory's contents are now stored in a folio_queue chain attached > to the afs_vnode (inode) struct rather than its associated pagecache, > though multipage folios are still used to hold the data. The folio > queue is discarded when the directory inode is evicted. > > This also helps with the phasing out of ITER_XARRAY. > > (2) Various directory operations are made to use and unuse the cache > cookie. > > (3) The content checking, content dumping and content iteration are now > performed with a standard iov_iter iterator over the contents of the > folio queue. > > (4) Iteration and modification must be done with the vnode's validate_lock > held. In conjunction with (1), this means that the iteration can be > done without the need to lock pages or take extra refs on them, unlike > when accessing ->i_pages. > > (5) Convert to using netfs_read_single() to read data. > > (6) Provide a ->writepages() to call netfs_writeback_single() to save the > data to the cache according to the VM's scheduling whilst holding the > validate_lock read-locked as (4). > > (7) Change local directory image editing functions: > > (a) Provide a function to get a specific block by number from the > folio_queue as we can no longer use the i_pages xarray to locate > folios by index. This uses a cursor to remember the current > position as we need to iterate through the directory contents. > The block is kmapped before being returned. > > (b) Make the function in (a) extend the directory by an extra folio if > we run out of space. > > (c) Raise the check of the block free space counter, for those blocks > that have one, higher in the function to eliminate a call to get a > block. > > (d) Remove the page unlocking and putting done during the editing > loops. This is no longer necessary as the folio_queue holds the > references and the pages are no longer in the pagecache. > > (e) Mark the inode dirty and pin the cache usage till writeback at the > end of a successful edit. > > (8) Don't set the large_folios flag on the inode as we do the allocation > ourselves rather than the VM doing it automatically. > > (9) Mark the inode as being a single object that isn't uploaded to the > server. > > (10) Enable caching on directories. > > (11) Only set the upload key for writeback for regular files. > > Notes: > > (*) We keep the ->release_folio(), ->invalidate_folio() and > ->migrate_folio() ops as we set the mapping pointer on the folio. > > Signed-off-by: David Howells > cc: Marc Dionne > cc: Jeff Layton > cc: linux-afs@lists.infradead.org > cc: netfs@lists.linux.dev > cc: linux-fsdevel@vger.kernel.org > --- > fs/afs/dir.c | 742 +++++++++++++++++++------------------ > fs/afs/dir_edit.c | 183 ++++----- > fs/afs/file.c | 8 + > fs/afs/inode.c | 21 +- > fs/afs/internal.h | 16 + > fs/afs/super.c | 2 + > fs/afs/write.c | 4 +- > include/trace/events/afs.h | 6 +- > 8 files changed, 512 insertions(+), 470 deletions(-) > > [...] > +/* > + * Iterate through the directory folios under RCU conditions. > + */ > +static int afs_dir_iterate_contents(struct inode *dir, struct dir_context *ctx) > +{ > + struct afs_vnode *dvnode = AFS_FS_I(dir); > + struct iov_iter iter; > + unsigned long long i_size = i_size_read(dir); > + int ret = 0; > > - do { > - dblock = kmap_local_folio(folio, offset); > - ret = afs_dir_iterate_block(dvnode, ctx, dblock, > - folio_pos(folio) + offset); > - kunmap_local(dblock); > - if (ret != 1) > - goto out; > + /* Round the file position up to the next entry boundary */ > + ctx->pos = round_up(ctx->pos, sizeof(union afs_xdr_dirent)); > > - } while (offset += sizeof(*dblock), offset < size); > + if (i_size <= 0 || ctx->pos >= i_size) > + return 0; > > - ret = 0; > - } > + iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0, i_size); > + iov_iter_advance(&iter, round_down(ctx->pos, AFS_DIR_BLOCK_SIZE)); > + > + iterate_folioq(&iter, iov_iter_count(&iter), dvnode, ctx, > + afs_dir_iterate_step); > + > + if (ret == -ESTALE) This is dead code because `ret` is set to 0 and never changed. > + afs_invalidate_dir(dvnode, afs_dir_invalid_iter_stale); > + return ret; > +} > [...]