From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D09B2C02180 for ; Wed, 15 Jan 2025 11:27:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69502280003; Wed, 15 Jan 2025 06:27:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64231280002; Wed, 15 Jan 2025 06:27:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 509D4280003; Wed, 15 Jan 2025 06:27:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 320D8280002 for ; Wed, 15 Jan 2025 06:27:20 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D1E734568D for ; Wed, 15 Jan 2025 11:27:19 +0000 (UTC) X-FDA: 83009460198.01.D7DD7D1 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf08.hostedemail.com (Postfix) with ESMTP id E2585160016 for ; Wed, 15 Jan 2025 11:27:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bJW44lHQ; spf=pass (imf08.hostedemail.com: domain of nspmangalore@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=nspmangalore@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736940438; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tC/USVkD7u5A+CkiGS1ymR6/SBGLkkgxeJaxuDF7ATE=; b=NG5PkgWrSmFxAXpu4WyGFGWLi/CUTms9aYMcUHUFfclOZCh4thrFUKRulin03oKuGqj7oX Z1ye85G+JrFDAyZ+lbqHXCwtvUvfBJ/nzMDlzo6X4eW9MTrqYasAcUiS/dsrBJ51STj7FH mw8iEp+8lRybxrHaxU1aa7Q/qB4Bj2s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736940438; a=rsa-sha256; cv=none; b=gVEWlDo3dsuxxDE4PnPAiqvlK3B3krKo5980HyhXxUMipZmvszeXD6/Wqh0ErgEoIUz2pA LmCSYI22QiweTYd2AJmrMKmKmM14H93AfezcJAG+RUtm34zA7FeDDNIs/B79tV0FHnPJaU /m43ThihKLbFxvh6IklRIASEAGCZWRk= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bJW44lHQ; spf=pass (imf08.hostedemail.com: domain of nspmangalore@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=nspmangalore@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-aae81f4fdc4so1249688266b.0 for ; Wed, 15 Jan 2025 03:27:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736940436; x=1737545236; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=tC/USVkD7u5A+CkiGS1ymR6/SBGLkkgxeJaxuDF7ATE=; b=bJW44lHQujgk8NiYJHPKZVomUudPNR6/2zbKJxjc5BQTqDBT5LAPYhgBfIMSvU9/0I wduSzayk6+ZDwJ46i8iBpFwBROYU+q09wSHXsp2urpuUPzqWfSpbVNzOBlfxVXFqGp/m qGtsQ7I7e+Zq1VGRSxniIdr9xjwxxFmFZrqpei+hTZkDNafARzCnBJqn1wwoVIHiXRpz XvoBif9FtONYFNiYxkMLEj/AhGPKjnpPHQcYGc1qSe6szV1pqwPGZkBdYSTciKm//fB+ 3UKQPSIwcKjQU2wt3u68O4alsnclgt/ctKKXm78SIL21/Fejfe6wxEqa+edpIMgHTDqA IP6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736940436; x=1737545236; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tC/USVkD7u5A+CkiGS1ymR6/SBGLkkgxeJaxuDF7ATE=; b=XFC6atym3Dj3hDmZsUmChSglk6t1abgPHSMnxGyvFKXX69qiZgxJ72E6XsGU2DPoyj alNWTz18Fv/7YIrZtfFsvgRn8Ozz0fLFK0Vxm85tvM9i32tjDHMTgrdoA91Yb4UgtoLX 0rz0IsKCJjE2WiDSVHMl8Frjgv7oJUi0zLr6GvYa92rYMzw6zUPtddgyEd4yTupSJ6+k AlZ12ZCa3lX2RtPZkQFYXI7HXIcJx/T8PiVmbKKiPnGlkQFub6i0ZA07CUwSrdkgQycm jm5sPosSYQGaQKVlgfvck6t78OwPWUVUHR/Sj7+wKW30AzixktKJ9NRpZRvtJaN+MJmw kT+g== X-Forwarded-Encrypted: i=1; AJvYcCWrz7mrlSN/eG44jCuqdAjG+Lp9Sa4phmowMQX9ziF0AVW2eYk0o334/QIlFoek5778yD0ouzoSpA==@kvack.org X-Gm-Message-State: AOJu0Yy+ifqfAPWh6ZlVfjFLnAqDJSqu6mvxAk+LLop+LVF3359od4Wx nZlk+fj6Eh88GSa5yyDKJR+4YuE38DsQm1F1Gz2LyJ9sYBxM2ClpnpoLjo+YVKycu7tT98DVhDG aWEMEYQTF3BbshWV/i2ZN7o8a+5g= X-Gm-Gg: ASbGncv58z8tMuNIAjk2tl+SZtHlFDc3lS8SMjkx6O5UpSINdkdcjx51u+jbafcNWM8 QbMfG39trQxYJvfW4WL0uH/SsCiBxEt1E3MY40/D9ImSA7yPRhYHU+UoBOWYTcabHo1NJ X-Google-Smtp-Source: AGHT+IH45zSej/+IjVUCeTIS1QCOIqGy6FhxMLz5CePH9bJAqfiiXOrNw0HIxitnpdaxI/L4g20icyJLfjqZwsNYArg= X-Received: by 2002:a17:907:8690:b0:aac:742:28e2 with SMTP id a640c23a62f3a-ab2ab6a8df6mr2651790266b.6.1736940436237; Wed, 15 Jan 2025 03:27:16 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Shyam Prasad N Date: Wed, 15 Jan 2025 16:57:05 +0530 X-Gm-Features: AbW1kvZ2dB3cdO4krmhiL-fP1yIMfDiH076-m9Adzt1SEjJKzvFXgfysprgQ93w Message-ID: Subject: Re: [LSF/MM/BPF TOPIC] Predictive readahead of dentries To: Amir Goldstein Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel , linux-mm@kvack.org, brauner@kernel.org, Matthew Wilcox , David Howells , Jeff Layton , Steve French , trondmy@kernel.org, Shyam Prasad N Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: E2585160016 X-Stat-Signature: odkq5yeabdr7io4yqcw3a5fw9ap4q66r X-Rspam-User: X-HE-Tag: 1736940437-48012 X-HE-Meta: U2FsdGVkX18hdkIyXYDXbG7pm0QKVx2tMtvW0yvGFrgQTitr8n4d+ukBOzjuS+94d0sK1FGAHRWqA0EYORHfZOpXKOZprlT/beIh1N/WFGEl6kybjjYezki4maSK1lVqmloNAnP5QHrlfA5DHCr7FOZDuE8ZK/L+8XSVAOza9UqRc8CCBlxGrbD5Rij/88PdWUJwsCaT6ct0JCLCNQe1TCyy7CLSBedLNqYJpkQbDZ+Yq2i2FdQLbiiGRrDKxWJ6gyjaBYkzcmb80nLOCMjGjrRo+otUh/DxZ2Cim+bQVvbrfWO6ZD1O+Vzin6yyW29wLXdvXPLO+7OzNIcXZiP6pgN73/9m2g9MNGTLOuQtDvGD7Z1jjMIM5hPCGwDNUR6kqr1jMFHK1wOQBcx4ng0dI2hMwx5dPVAGq7jP2oNrzG7BBWxR7E/cfgg0ZHoneAXKDkIKO8bpsvMJTc7WXVSE5jfaGxxsiycdIFGMcgZh1nDW8qdaciWlwVn4FZT7VSo8CuhYTgDqflRz3IPFQyLk4Sf7Q5Wj9AHQXsf+UVvXFPTxGFw2zGywjrGF3EtKV5M5hRaXFlqs6Jnk5X6zZ3EkEQI8ZXTnlZMACFako5EkPXd3dJHRxUielK2C93nWqGH0FisUw7flhTuq78xdrheX5xd12Hp+Nf3/Mccr4KoQ9Jga56jrLNr4NtXPL6ywXGgE474hvn5cibwsxI/sNOCn5mxj5wUOTnQurPwIV/NbUBE1/8JoRKiEoegIqKkTCHiroe2UTsZJSmrexT+Zp9wTENSkOPsOdEuTbSXIo3b5EM98j7Fmb8+O0wIQeKpBJcCRE1CCasNIu18PVM9EMnyL4mSVVC0uCP/hVDzXySZp2ngtVRu9XfbUv8uyAKlOnDq6BJQ1yYwmx9GY2WtBlwffx7dRTKEHZzNQ2pVA6AFDVQuDndHYulEFOT82tDIvTk8IFzAsIu3TmYgTcDg9uDk 2Waigawa 1cgZN5OaICu6ZfATamVqggdZt8oc4pPRLecSvuO9JaWkjfbNuVG5LlI4BR5+lbqFRJkqbNgowWV7kHvEHJZCSjX33b0hwvTZ/Y2ijHnz7cnVmkusQIVZPENOP1y+Iyg1xmCzhbryB3bQnYZUjY3MyPUI0DHEvH9EYT5tdKV9V99ENyLmhDlkMUJEYv3Ye3H2citFuzgRA+fF3HfHrAyWMWhtbCjTJk8/rC658kFE7SxJ+P3ksTkH78s5WCkfEbEKQvuaKaUS3KdZMTd7hANzN3ciN71WG1z05gjJTX4jcfTjdDwiADnTuRoAYXgIHnIMWlkuvdyWjFCwDcCFjPRWU0R+WmxnEG9eF2pUQHN2+rXgEN9HPA2gdQWzRkaKlNNbDAXfU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000634, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jan 14, 2025 at 6:55=E2=80=AFPM Amir Goldstein = wrote: > > On Tue, Jan 14, 2025 at 4:38=E2=80=AFAM Shyam Prasad N wrote: > > > > The Linux kernel does buffered reads and writes using the page cache > > layer, where the filesystem reads and writes are offloaded to the > > VM/MM layer. The VM layer does a predictive readahead of data by > > optionally asking the filesystem to read more data asynchronously than > > what was requested. > > > > The VFS layer maintains a dentry cache which gets populated during > > access of dentries (either during readdir/getdents or during lookup). > > This dentries within a directory actually forms the address space for > > the directory, which is read sequentially during getdents. For network > > filesystems, the dentries are also looked up during revalidate. > > > > During sequential getdents, it makes sense to perform a readahead > > similar to file reads. Even for revalidations and dentry lookups, > > there can be some heuristics that can be maintained to know if the > > lookups within the directory are sequential in nature. With this, the > > dentry cache can be pre-populated for a directory, even before the > > dentries are accessed, thereby boosting the performance. This could > > give even more benefits for network filesystems by avoiding costly > > round trips to the server. > > > > I believe you are referring to READDIRPLUS, which is quite common > for network protocols and also supported by FUSE. This discussion is not completely about readdirplus, but definitely is a part of it. I'm suggesting doing the next set of readdir() calls in advance, so that the data needed to serve those are already in the cache. I'm also suggesting artificially doing a readdir to avoid sequential revalidation of each dentry; or a readdirplus to avoid stat of each inode corresponding to these dentries > > Unlike network protocols, FUSE decides by server configuration and > heuristics whether to "fuse_use_readdirplus" - specifically in readdirplu= s_auto > mode, FUSE starts with readdirplus, but if nothing calls lookup on the > directory inode by the time the next getdents call, it stops with readdir= plus. > > I personally ran into the problem that I would like to control from the > application, which knows if it is doing "ls" or "ls -l" whether a specifi= c > getdents() will use FUSE readdirplus or not, because in some situations > where "ls -l" is not needed that can avoid a lot of unneeded IO. > > I do not know if implementing readdirplus (i.e. populate inode and dentry= ) > makes sense for disk filesystems, but if we do it in VFS level, there has= to > be at an API to control or at least opt-out of readdirplus, like with rea= dahead. That would be a great knob to have for network filesystems. We have to rely on heuristics today to predict which of these patterns the workload is using. > > Thanks, > Amir. --=20 Regards, Shyam