From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B5F3D5A6EC for ; Tue, 26 Nov 2024 08:01:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0BAF6B0083; Tue, 26 Nov 2024 03:01:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BCD36B0085; Tue, 26 Nov 2024 03:01:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8838D6B0088; Tue, 26 Nov 2024 03:01:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 64F046B0083 for ; Tue, 26 Nov 2024 03:01:43 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1253A1C7D24 for ; Tue, 26 Nov 2024 08:01:43 +0000 (UTC) X-FDA: 82827501918.10.9861390 Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by imf06.hostedemail.com (Postfix) with ESMTP id 12C00180018 for ; Tue, 26 Nov 2024 08:01:37 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZWTzgIH+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.208.182 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732608099; a=rsa-sha256; cv=none; b=AuCO1WV0yC+LEC4MMZsQ193aUa9pRMxSeZ4Ma+mgyBQf55AyruTByZlH4TepkGMSbgxU8v p0u9YMRU2rGnz94LwI0BqXyjaAWuehmGDfxDf/M3PN/VxWnhFrXLz1vfcqp2capuJ7UF5T 9OUfxRwDBeUwQ1MuXhe4JpOKTiGVYTw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZWTzgIH+; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.208.182 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732608099; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TLhcUgcaehzCh657DFQOY7eTZBplqKp16QE7H3wG/RI=; b=NY3TIlumm99Fx7yCizYrKVgb3fujRX/V0xlDAEoQMKcbGKQ7fk6DM6IKrsGOJ5OcyIIS3H QoA/8GSIQGp9pzNEdNACydEigs7IEI9E4133Q94u3JVlgmkIRnarWSL7jr31ClgtDeOPh0 vRWKTZpqEtYJVu82t4R2Jw8t+o+Qalo= Received: by mail-lj1-f182.google.com with SMTP id 38308e7fff4ca-2ffc76368c6so25678861fa.0 for ; Tue, 26 Nov 2024 00:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732608099; x=1733212899; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id:from :to:cc:subject:date:message-id:reply-to; bh=TLhcUgcaehzCh657DFQOY7eTZBplqKp16QE7H3wG/RI=; b=ZWTzgIH+RFiLd8C9oJhsJEYvXGfxZwJ6AUHmaGULs+WT4RaRfG2sYMisO+ii6rvHyR uA1xJw4WPPcFwUR4S6ajquJiTG5IiMwDc959Q7P0gMm8IglrHfpjjshkS+5uta2p4s95 elG/4wFSBbAtKBmpTzD4OrpV16ETNlCIFor9+rlwU/X5APvwwKzCSF/wsgxtit4gdRgR tC2h1+34T2AYe3y6lvCvhJuPqs9ZbVulXKAoc+Y0FcSOaR2AxJ59RYxknydsx8E5n46v 2mjj53VZZmVDc5DR6H/ruHchmjKTQtSZ4zjTHLaMFNjTnBW5gc17NVB30btRt6j6qq1I Fd/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732608099; x=1733212899; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TLhcUgcaehzCh657DFQOY7eTZBplqKp16QE7H3wG/RI=; b=T/fAHDmenYEJChDO+H7ejML/E7TBo14GLGlXW/Q6h8ost7dk3F0pfL7wHN1ij5eS9D zjS3MNSJN0whvZekp9paf8a30VuYeP92HPVxaHeUrJH1xeTJvM3KK/yNtrgk1Me0YLhD LnadvjVG4ysikwBIHMtHPQ3hnjtbKGFWnlk4a6D+HYpFOtr4oluanTy6uCNgWhD0stFY KCSGASVQSychbqpI70A+9owaX3RTTAiRkMfB85XTcpHSk+YdyI3F1/S3wWTNuMQYXbUr OVT8YWhkJGVFkKyMRO3aq/alCxvnnPE2fvnYb9+nIQ2izNqQBLHM0h7egDJsjTI6KzkQ W/bQ== X-Forwarded-Encrypted: i=1; AJvYcCXR7pkgD025oozbstkHkNuxDZxbBJtUFrflPHGf78j8TVMIJYVvKpWqprBmTLQRD4kLkuTYR4ECwA==@kvack.org X-Gm-Message-State: AOJu0YwIwp/e2We6pwX8YTmHu1dnq3yimNraH+kq1QGLD4AkD8DrufMW K5EbZqkxJI8OBjNxyTJuDoBwIBuaUtlZk5Lw3gOF37XEIkTl1EWL X-Gm-Gg: ASbGncvWwF6HZ6YkL27oy+viZAMPJRMY8mepMHzj3eNw8NYp7NCh39Zzu1mdN82Ef77 /MrUb9bsRn2vzkZfAFzYzVSiFqNRv4y1bi4iopoAYwxwX12WUgVa3oRzP4tNhDHY/zBfkb0aR+R GX/aJMhjCD2NTKxt8av57TWVCVnrKQfb0UKZ/ozx1268Fl3aOm0lcenzXoR9aOrvN3Utf29ugPS pKrXraRG2IAxoKrqlP8JaMq7EBsQ5TEUd4YmaDJ+1WiiUsH7zjKAFcqzjS1758nLWrmSDR7cLHE HJepzj7NpVdHRdbrMV5RJ6SBsio= X-Google-Smtp-Source: AGHT+IE4dWqX6FYS8GpTyQPwG7QbFUjZkKoqX8mHebeNCaYOS/7Fu3gJfNC6CUKQ2XORHy1/kXb/oA== X-Received: by 2002:a05:651c:888:b0:2fb:3df8:6a8c with SMTP id 38308e7fff4ca-2ffa71259cdmr113863541fa.23.1732608098684; Tue, 26 Nov 2024 00:01:38 -0800 (PST) Received: from ?IPV6:2a00:801:2f3:e48d:c8bb:deb3:99d1:7504? ([2a00:801:2f3:e48d:c8bb:deb3:99d1:7504]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-2ffa538f23csm17380591fa.111.2024.11.26.00.01.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Nov 2024 00:01:38 -0800 (PST) Message-ID: <4bb8bfe1-5de6-4b5d-af90-ab24848c772b@gmail.com> Date: Tue, 26 Nov 2024 09:01:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Regression in NFS probably due to very large amounts of readahead To: Philippe Troin , Jan Kara , "Matthew Wilcox (Oracle)" , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <49648605-d800-4859-be49-624bbe60519d@gmail.com> <3b1d4265b384424688711a9259f98dec44c77848.camel@fifi.org> Content-Language: en-US From: Anders Blomdell In-Reply-To: <3b1d4265b384424688711a9259f98dec44c77848.camel@fifi.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 12C00180018 X-Rspamd-Server: rspam11 X-Stat-Signature: qwxr3for7iyxzjy1krksfe3r1oqjgzd1 X-HE-Tag: 1732608097-46884 X-HE-Meta: U2FsdGVkX1+7PTFt6LyBCP8gQPk4eNIehbcdBTntxQn3KBQ5/h7+itsiXwd9coSkDQe+JjQCD2iSLXSrvvILLTG+AQVyf6DDBOM7UiIYCKu8CKYdvYIqLUydOZFrBX1hMUmrDHEV5rzbvlzvofXRC2Qqbv17y+OMZ5+IL8HBs4Oo0bNlSERWnDquTnwlhzOotsLqrOJhpea7QcWVjPykHlaMGlnx3gGBvpEmCkwE1iJ+Xt/M7Ulnu2QEFu3ywzxniNXaoF50hzwRvbdy820Zla/XMC2SikE3Ih08btif3ynOnG+X+Rry2PoLGHVTeVqUKEUDNydGYpLkigl+rjLFAmftqwPo4EV3SMNwl8H9v2Q+H7OW9L+FipzuE678H062/YjQyhazhC6JK1SGGEz2kxm1MbGnVATvCqrRRwm1DZWby6smS4XsGx+01diGVz5jblHY904mp1OxKZrIDwMLngQCMWefqvf9kv1gp2TT4Bm0Wp/+WEkTCSzaA3AORLqRbnpHKVF6r8HgPsMuGwtkewBkc30JYCD3KczehyDhmIg4kKQ66C4kc8YLSxy95kVT8m3hD8F8++ZN12V3pydGH+aSa3h8bRrOrWMuRii3FLAyKLQlOY1mlHTo/K1hqTCN0Dx6y1P7od74tQ6mXAaJsBSwFjy+fUPMfwuGLKb+N3n0mGDDGaPdqbKm34j8B58M6b4lsZjFFTmmF9a0MTvtlB+OdIIsl1XGMOn+fQpusC2A49mtyU4wu9XY2gicNJfTDriAVhQoMxJThd9kMsNAjXUK6wpsYbceayX8FotawnOeZr6K8VovIChZw+E7vJbTRiyN91vl5RkzhnLsrIeX2rcifZfu8fSdnY20Kt5RN8rpU0nHGEVqqmHulYBAExNqwic+7gSM0k7nneoVqnPXdQeKmuETiR1Z0JH0eDB7D0oITxs2xtBHWz/iPRN0SAM4JV6owjB/Sf44HN4fqGj HpDBz0AI e0TRJFO/8yyhyGk3wVEjBshkBGew80Gk09aNmrtFz72RxJBII7N6u/kDo7FUhpeLNkML6lmAW8+l5IJGaz7+uZwPteNDXS/qdhGxEKUwSXHa36oNlX1yW/34c+yOt0wlgBShNTDxC03b4fddkRtfMyvCedVHSxJs/rsx9K5SA12Ki8lUsVdgwANUDhyLiPmHAg/u/H3oj63YikAL+TqVw6CAaqfi1M77MJxO6HF9juehk83oCY2mWepqKcOjtVOUC6yfn6jd20UQvVKvJjhHAVIkVIFTOl3X7nIMOMhL56DWgABduPywhISrC5B0QXCobzA8zAq3Q1afUtzhqxMnNMmgGNVMmnrVH+Yjog2G+FfZ7fH0lATChZtnXYAxRGIimcSj10uEcQcv7HLqbxDv4JSxE8EbGeL1rtKfInlHpZI0V8nZL2YJDNZzyF53odLoGjp6GAtC5EUttqXpEy7GtkvMfzwbw4yYZ8cKQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-11-26 02:48, Philippe Troin wrote: > On Sat, 2024-11-23 at 23:32 +0100, Anders Blomdell wrote: >> When we (re)started one of our servers with 6.11.3-200.fc40.x86_64, >> we got terrible performance (lots of nfs: server x.x.x.x not >> responding). >> What triggered this problem was virtual machines with NFS-mounted >> qcow2 disks >> that often triggered large readaheads that generates long streaks of >> disk I/O >> of 150-600 MB/s (4 ordinary HDD's) that filled up the buffer/cache >> area of the >> machine. >> >> A git bisect gave the following suspect: >> >> git bisect start > > 8< snip >8 > >> # first bad commit: [7c877586da3178974a8a94577b6045a48377ff25] >> readahead: properly shorten readahead when falling back to >> do_page_cache_ra() > > Thank you for taking the time to bisect, this issue has been bugging > me, but it's been non-deterministic, and hence hard to bisect. > > I'm seeing the same problem on 6.11.10 (and earlier 6.11.x kernels) in > slightly different setups: > > (1) On machines mounting NFSv3 shared drives. The symptom here is a > "nfs server XXX not responding, still trying" that never recovers > (while the server remains pingable and other NFSv3 volumes from the > hanging server can be mounted). > > (2) On VMs running over qemu-kvm, I see very long stalls (can be up to > several minutes) on random I/O. These stalls eventually recover. > > I've built a 6.11.10 kernel with > 7c877586da3178974a8a94577b6045a48377ff25 reverted and I'm back to > normal (no more NFS hangs, no more VM stalls). > > Phil. Some printk debugging, seems to indicate that the problem is that the entity 'ra->size - (index - start)' goes negative, which then gets cast to a very large unsigned 'nr_to_read' when calling 'do_page_cache_ra'. Where the true bug is still eludes me, though. /Anders