From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5244C02181 for ; Mon, 20 Jan 2025 21:26:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C48B6B007B; Mon, 20 Jan 2025 16:26:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 573A46B0082; Mon, 20 Jan 2025 16:26:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43B136B0083; Mon, 20 Jan 2025 16:26:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2649A6B007B for ; Mon, 20 Jan 2025 16:26:34 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A1DB5C03A5 for ; Mon, 20 Jan 2025 21:26:33 +0000 (UTC) X-FDA: 83029114266.30.110ECB7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id B0E30A0002 for ; Mon, 20 Jan 2025 21:26:31 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=brHUMSvQ; spf=pass (imf25.hostedemail.com: domain of bcodding@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=bcodding@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737408391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B04i/gZ8lgb0zv2argciPqONFDUYHmh8OQrj2OyE9uo=; b=UN+1dVKQ7EMU0/195Fm+tao8Tg54NiKjpQyQ7XkzGhJCUf8fgGGKlyprW/RPeG6DKHcCWh HT67Te+dRQIKG7xjbm1LJEjt/hVhMytopFJ6UL0oohSnevjrX0WOL5LztC1EOnxqHdCw7D iDMySkqQG8uD7WinkxbwU2T9jBCneXU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=brHUMSvQ; spf=pass (imf25.hostedemail.com: domain of bcodding@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=bcodding@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737408391; a=rsa-sha256; cv=none; b=azGa8dc0bOn75PJsU5N/hDWY/gIoEpvGBqeJ8Ijhe2r6eHyLQCE6Ew63GREvSo4RLdRCc2 4yxDmH0XThAG00HBGFj5F/iZWu+y363iC/FUHzY4RXw65gExMlAWxwRPhPMJvBU6OZp/De m8qooChCsNjCI1EcAl6NR3CJ/H4VJcQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737408391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B04i/gZ8lgb0zv2argciPqONFDUYHmh8OQrj2OyE9uo=; b=brHUMSvQQRqBBMYaJlyD1b1a7iQvBHKgCM5PIFqF7TaTcHUA3B21XkKxOBIyCqJQ37pJv/ kGlkNEMkFDMHiBbb55yrfJiyN8FMxajA51vPenD5P+8mCoI8jqeJHca9KqDw3ch1lqPl2J uwMgH6XmhNlvzCBZass1NwIOIvBwR0Y= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-5-lXpB4fRHMqKvNM_Aia6CJg-1; Mon, 20 Jan 2025 16:26:27 -0500 X-MC-Unique: lXpB4fRHMqKvNM_Aia6CJg-1 X-Mimecast-MFC-AGG-ID: lXpB4fRHMqKvNM_Aia6CJg Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1EAC5195608A; Mon, 20 Jan 2025 21:26:25 +0000 (UTC) Received: from [100.85.132.103] (unknown [10.22.76.4]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9B3231955BE3; Mon, 20 Jan 2025 21:26:21 +0000 (UTC) From: Benjamin Coddington To: Amir Goldstein Cc: Shyam Prasad N , lsf-pc@lists.linux-foundation.org, linux-fsdevel , linux-mm@kvack.org, brauner@kernel.org, Matthew Wilcox , David Howells , Jeff Layton , Steve French , trondmy@kernel.org, Shyam Prasad N Subject: Re: [LSF/MM/BPF TOPIC] Predictive readahead of dentries Date: Mon, 20 Jan 2025 16:26:19 -0500 Message-ID: <4A6F89B6-5E6C-4DFD-AC3A-CD80F6E4B1EB@redhat.com> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B0E30A0002 X-Stat-Signature: 8fnafmrxzzdbhwrpnsr39j9dofk9qe8i X-Rspam-User: X-HE-Tag: 1737408391-150678 X-HE-Meta: U2FsdGVkX1+mvYSOod0luMBXVz1LkIW1xMnWbbzkz/feAM0EZ3q6jQeWxno6YX5t9yHJh8ZHJOM41qGyYwkxXY9TaGYj1FUnWlIaxbuSzFbrtQqoi+RKJq7/3pkch1GlEENNLoWpoV9phUBK7laG12IDxPfKf8LUI4B72nWPxswT+RRKtTULt33JUwaA4wsdoUEd+c2siu9mVX7lnz2alkl8MbzyIbNXE77CR6YpyB2utTOz5+0dIvuTaSOr1fUsdBLf9rRmBTnfQGlH6g4dF0D2mRpTqtuoDIvCFL2yaOqlkTa08Z61aI5JNF+zBlSYslkL64dKXKU/kKJdGQxLegecmD5qF55IhWFSpCQQdwMDpK4w+6b5GKes7xKHoafhhiL1py+yCd2TuOwANR1N/0weaVs10rbD7U4D3xuAg5iyPzFrn39Adr3NnaCGFABeT+6P1T24Wlpt78LZFHL8h8PdqtweQ7cmRH3OX4zIX0feSK8QOs1T4CY5y5GYcqYbyfv1wk0/wEgoazs+S0UP94TbSE/P6qvhXVKvW/pQTnm/jQRTLS8iC4G6iHjxOmjWpqvHlNd0SltnFQfegLi5UB8+QidtQpO81HmXTbi6UArBa7pZf19565RNsZsAlierez2Q2jLVhRtM6rYcoscsACDy+Fv4i+nSfkycQQz2r0Z1WgZ+6PQj0joY5+L5J0F3xya5OvmPJ+KbsjoOaMl7K9Uk1pO0jfT5USwPynZnQpqSl3Y72oM4zs/f8VQtY2Kmypye3T43vnypSynM6kxbaVFy3Kjg+pU7N5C+1O631KAl0YQrbtvSfBcrjgK/X+/fIvYvzaETh64cwqnYz5U5mhfNgTNmAzPGDZws/U4joq2EFAXwJFMlNKu/qoyIyy4v5ekcz6CSjpnq/+njMB/V3Hf8ZhR5OAMHkEXNLy/lkrxZnJ3GJWhQCZvD9bgHfDGfOnFV2CwiKHQuVQ8mzg6 JkVXv1G6 kjJos7bMALFXIx3MNdrrpZUEoFPpQxVz2GEjMDV9+thxF40RF3iKH7TWt3av3JdQkG1tFGw2dDC2voKZnXPy6XMaL48EYs3vYvw68z8KbfhLg/Vi5SOJIimDc1Ss3x+33NYxKfGtWtYlOX29mKKKmX7COOjHYJtI9PTeBH4J9IxC6MqroAIg78mL2rOcn1GRS3AeoBzxUH5ozSYrCuIlYXA/silOEUmwybftKbUxvadfSkQhUm4bmFM62awjBydNstqc/HPHAgnyy592mp5Zpfj68nY8R8QoaQPVyh38XGEJZDK0QqLGbNH/jGMB2jhcYQKC7OhhTczNzxfEjwLofI0xG9yAtf3v2Jb+xDSAWGXxQUyY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 14 Jan 2025, at 8:24, Amir Goldstein wrote: > On Tue, Jan 14, 2025 at 4:38 AM Shyam Prasad N wrote: >> >> The Linux kernel does buffered reads and writes using the page cache >> layer, where the filesystem reads and writes are offloaded to the >> VM/MM layer. The VM layer does a predictive readahead of data by >> optionally asking the filesystem to read more data asynchronously than >> what was requested. >> >> The VFS layer maintains a dentry cache which gets populated during >> access of dentries (either during readdir/getdents or during lookup). >> This dentries within a directory actually forms the address space for >> the directory, which is read sequentially during getdents. For network >> filesystems, the dentries are also looked up during revalidate. >> >> During sequential getdents, it makes sense to perform a readahead >> similar to file reads. Even for revalidations and dentry lookups, >> there can be some heuristics that can be maintained to know if the >> lookups within the directory are sequential in nature. With this, the >> dentry cache can be pre-populated for a directory, even before the >> dentries are accessed, thereby boosting the performance. This could >> give even more benefits for network filesystems by avoiding costly >> round trips to the server. >> > > I believe you are referring to READDIRPLUS, which is quite common > for network protocols and also supported by FUSE. > > Unlike network protocols, FUSE decides by server configuration and > heuristics whether to "fuse_use_readdirplus" - specifically in readdirplus_auto > mode, FUSE starts with readdirplus, but if nothing calls lookup on the > directory inode by the time the next getdents call, it stops with readdirplus. > > I personally ran into the problem that I would like to control from the > application, which knows if it is doing "ls" or "ls -l" whether a specific > getdents() will use FUSE readdirplus or not, because in some situations > where "ls -l" is not needed that can avoid a lot of unneeded IO. Indeed, we often have folks wanting dramatically different behavior from getdents() in NFS, and every time we've tried to improve our heuristics someone else shouts "regression"! We can tune the NFS heuristic per-mount, but it often makes the wrong choice.. As you say letting the application make the call would be ideal. POSIX_FADV_ ? Ben