From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30BDFD5A6C2 for ; Tue, 26 Nov 2024 01:48:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98D456B0082; Mon, 25 Nov 2024 20:48:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 93D3E6B0085; Mon, 25 Nov 2024 20:48:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82C996B0088; Mon, 25 Nov 2024 20:48:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 65F906B0082 for ; Mon, 25 Nov 2024 20:48:53 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BF02880F1A for ; Tue, 26 Nov 2024 01:48:52 +0000 (UTC) X-FDA: 82826562420.18.2B2CC14 Received: from granite.fifsource.com (granite.fifsource.com [173.255.216.206]) by imf26.hostedemail.com (Postfix) with ESMTP id BDC58140004 for ; Tue, 26 Nov 2024 01:48:47 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of phil@fifi.org designates 173.255.216.206 as permitted sender) smtp.mailfrom=phil@fifi.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732585729; a=rsa-sha256; cv=none; b=SZE367jjz77s/aCWhmWm0bu/sjOXZGf/PKG4fm76sNwJfB++wlLsQuqp2mKvAaevL4MQ4C +XFwHWUUHSadHwi4PF2lYG3hbw+jNG0GweJQdsDd7MhzDQ28X0s19+BI2mEUjeq0x55ptM TD7AHQwqP49sZblzQzCWSXyhUqGKxVs= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of phil@fifi.org designates 173.255.216.206 as permitted sender) smtp.mailfrom=phil@fifi.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732585729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x1Q1jqiMb0qZANI1diCAbnUoCUvWMoZSXXu1B14MoLE=; b=zGlND10Bn8Q/mF1/Wj8nRLsotYwSh4F/i/0uz3ZjoqU5eWvvPaUvLXKivhCfFKgz3QCspt 52bDtSm2nqPOeKXJnqKBFYI/GBcW9z6+T5JSXtD3CtQBUu3SFJNuK7xCY/53CoAa//+d21 fO1RIAho0z0fw11WZOYOwAjD7aGKTuI= Received: from ceramic.fifi.org (107-142-44-66.lightspeed.sntcca.sbcglobal.net [107.142.44.66]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by granite.fifsource.com (Postfix) with ESMTPSA id 345DC4076; Mon, 25 Nov 2024 17:48:49 -0800 (PST) Message-ID: <3b1d4265b384424688711a9259f98dec44c77848.camel@fifi.org> Subject: Re: Regression in NFS probably due to very large amounts of readahead From: Philippe Troin To: Anders Blomdell , Jan Kara , "Matthew Wilcox (Oracle)" , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Jan Kara Date: Mon, 25 Nov 2024 17:48:48 -0800 In-Reply-To: <49648605-d800-4859-be49-624bbe60519d@gmail.com> References: <49648605-d800-4859-be49-624bbe60519d@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.4 (3.52.4-2.fc40) MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BDC58140004 X-Rspamd-Server: rspam01 X-Stat-Signature: 97e16sygdmoyoxzwnza49pq9irjhagi4 X-HE-Tag: 1732585727-677896 X-HE-Meta: U2FsdGVkX1/CvSMeL/BDJH4g7KR2zeSy/qVD3vxhqhpgQoe1j8ILkF3yLXSx7QgziiA7U8eP9FKSrXFuZCnGy1WZ4mdmXN8bGoj5aGrDVmczCaSaVZJrYDc+QjIOXtWAio7GhNezTS8czjNa3XkSVwTo3x9V0IIkLILzlhwshIlzEtzRQO3sryNKbW3FMpTF+mzteS/5et1ZoQgINHsW4PIuXs9Y6T736OSQ9MU6vW3BqVmIU422M3B5Mr6Co/IfqbSiha/AjZPGzPXl5C8eQ3ApDBjYNenS5k3v47u/NEyKmxVtmoXV8c4VMHJyh/lyGuT2z3f2+B9xiAMoHIZUhbD2aiLXx51AIlLPuVMGVbfAsjBvxLyLK5NweZ3qDhO8W0KxEZBIBHdeJNnLI19EeSHZaBSOrGk8/rfoaK+miDtiILFtn2PbhZesHRWZoTAzugqkjfF+nDdubfVziQQgRWMSlCJx1mLy+mR8SSkIwrQvQvpTqmQ9XFvjqRwKpJjrKyQNPiaroD0gYUQKpImCiGMknK4s7knY4/I6E0i+ySqcRE+fjOhVICowc6DJ78IXH0QRX9xx8BurNAqntkymGKS5cq/x+4THMK0DcRgvSrqh5P7Ylh9Uxd2Hjk5zOY4cT1AwmoePOPmLNVu5ntZg4Y9uiWs+Y9sqqlif10gyfq8q3a+w1bQKNtpKZtxBemL4YOfWhNXa4lEhGX/4lXwkdDUWYCV+SilqnTZOjI7UI0Qr9XmKeCzvL2xrtHIcMbgsJb7uO+hz4EanlLJGX7jePZ10UsnhU4fxxTs7wIDNGVJacHCtBBntFO6DXV0xzVZlZ+fTJhRICsf607Seu+sb3aPuGkA5w95awmNwhOaNDSov3MuXZ8gc/hRo5MKQW+rlgpVDj+BDh1gHu9rXLLPr1/yLQrD4bNrKcOgNIC5WiDsbTAnwuY2h5OgatDPe90xM7Ypxe03nrKYquUGMUz5 /2OU21RI Tufy12wyy3rb3+VhpLIeizcMr18xgi+zpqe0ZAVAlzKKWJJjjBaTT6bgb25C8tTdkLcXfGUOnANc6+slCW12Z8CMn70hnHbX7gkqQSCjU5dUQy++SALkjY0yuUTR5R0wvO93oi0/coazElZsCJNvXFfqeDvFcpUskmGfqlBY03o/B4Ek= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, 2024-11-23 at 23:32 +0100, Anders Blomdell wrote: > When we (re)started one of our servers with 6.11.3-200.fc40.x86_64, > we got terrible performance (lots of nfs: server x.x.x.x not > responding). > What triggered this problem was virtual machines with NFS-mounted > qcow2 disks > that often triggered large readaheads that generates long streaks of > disk I/O > of 150-600 MB/s (4 ordinary HDD's) that filled up the buffer/cache > area of the > machine. >=20 > A git bisect gave the following suspect: >=20 > git bisect start 8< snip >8 > # first bad commit: [7c877586da3178974a8a94577b6045a48377ff25] > readahead: properly shorten readahead when falling back to > do_page_cache_ra() Thank you for taking the time to bisect, this issue has been bugging me, but it's been non-deterministic, and hence hard to bisect. I'm seeing the same problem on 6.11.10 (and earlier 6.11.x kernels) in slightly different setups: (1) On machines mounting NFSv3 shared drives. The symptom here is a "nfs server XXX not responding, still trying" that never recovers (while the server remains pingable and other NFSv3 volumes from the hanging server can be mounted). (2) On VMs running over qemu-kvm, I see very long stalls (can be up to several minutes) on random I/O. These stalls eventually recover. I've built a 6.11.10 kernel with 7c877586da3178974a8a94577b6045a48377ff25 reverted and I'm back to normal (no more NFS hangs, no more VM stalls). Phil.