From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3F6CD3B98B for ; Tue, 26 Nov 2024 13:24:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2F156B0083; Tue, 26 Nov 2024 08:24:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDF4A6B0085; Tue, 26 Nov 2024 08:24:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA7256B0088; Tue, 26 Nov 2024 08:24:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8EB236B0083 for ; Tue, 26 Nov 2024 08:24:11 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E561D1611C1 for ; Tue, 26 Nov 2024 13:24:10 +0000 (UTC) X-FDA: 82828314576.15.9F8874E Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by imf21.hostedemail.com (Postfix) with ESMTP id 6D76A1C0011 for ; Tue, 26 Nov 2024 13:24:00 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=knfKKn67; spf=pass (imf21.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732627443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q1iBvzimN1iv1fee77YElBhVaU2CCj11UuGl0oezYDc=; b=G3448exdlTO86LDmE+wAVtyiJShAvmZsxH3P7sq5m6LlXw2CJ1u2rr4J8RfVmoRwOf+Yoz Whj3lp/wSYFvpnRaW/a9gRYH/vv4fHY5xUA2fgcrEpF61vZNm9+yt+2q6sbVwpWENiLxEN 4LLADQ1d1u4T5CzHXLeEe7qmHcnnmKQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=knfKKn67; spf=pass (imf21.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732627443; a=rsa-sha256; cv=none; b=XNPADvoWuQk0JStfvxIwW74uKjGA1cdrgzwUgXfxkw5V737gF7fR58VbGDKUp7AW9CrMCq mCh02QkN+4C5GHEAmqwIZ/y2dqzvmsKfWXI9OiwyVj8o91qrjhph0rDN9kaRmtfMuf88Pb KiQh6UppJ/HeI5CC6KfR+r1Mi93zy5g= Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-53deefc2ceeso128074e87.0 for ; Tue, 26 Nov 2024 05:24:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732627447; x=1733232247; darn=kvack.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:from:to :cc:subject:date:message-id:reply-to; bh=q1iBvzimN1iv1fee77YElBhVaU2CCj11UuGl0oezYDc=; b=knfKKn67jRp/PrlhrMxVyDY23mKCI3FwZ93cX/OpnbShGxx3SC/dqRazR8BHsHiVTt VRPagSNKlHamo9T24veQxjl/Ro4rrd8S4Jy5kSTUPIXfA6Pu7irzwjWoA2t30IF6bIFZ jXHgGYVH7dl7P7vH4hxTEJ3eT9vtffPbXEYl4vdLzpWKIGRB+1A6bJqniYXUY3316zf0 qZrfFcshe+uIa3U8H51g+j4KbvRvLxQixhBOeSzVJF29GEa7wm+HSlmU0zdHIeNebab3 SgdysU1ICIo7X19NjLPunmz+5+caIcrupGbaF3+D/+9NEU19TlDeV9/e0P0DaZQKYvXo kbEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732627447; x=1733232247; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q1iBvzimN1iv1fee77YElBhVaU2CCj11UuGl0oezYDc=; b=IuxtdDRlz4ZpcUsb+Y+Zb93qbRuDEEdOK+8fTSuH4XpsA7Cz2JWtey+wLnGNl5N2i+ n0n1scXgNXXC5Bl0UfVufhHtXYFNA4MCjtx8nX4QT1p/x61b/B4qo57rJIONJurGCegS jY1VWOZG84CPxCHbBZ8Jm+5M/e5jACTYjvTL4Jt12c/a9UwFQ/R1A5DEZLqVBZqmfh1T mJUhoMccqXECw/1WMq5vvAjTwROs2dJrjJQbkmTdHlHbmjScHHEdn0KUUk3UEaRnLsMY Ucn/FWAnwd1cJo8jXMGaRPJPqQ4KQFXWazBpr0liERd8NnU2/JHPZxmvcKnQW+Ixez28 EDPg== X-Forwarded-Encrypted: i=1; AJvYcCUap91T1+Lq58RMyeFD98NpNmymSf5AAOQbr9vNG0+UTt3EIlBfYtIvSBp2nHYAz6tYrEm98tBbJQ==@kvack.org X-Gm-Message-State: AOJu0YyTTNTNYtMw5RYja+HxLYwUUPwhXVdYYUb5lBwMRgkUSnUqto6A xEsIg7YoVCPxhILwBSs2IhLwE3GrcJNqey8bKTFA1UHtthKD3uAh X-Gm-Gg: ASbGncuLB06Utr70YRj1GklzmZrNO9/3QKBKb8r5hLppGN1bTDLRTkBt2hMHvUSRXUw GFBjbefJXM6x+snpc/ofFnQmDmXdeiygAhF+hnYGEQrQOqnneRjzDTEAEA9ubWLU90rgt/fcKq+ w1aAtM7FYv0aXfYf0L9gdyfjB/xlYbn5gJnVif4S8s/rn0jX54EZdYREqLmdbLMez+RDogn3W1J hPkvg3GrdazY9gaAeaTiiw4SDU3eus8bGPJdjXAosf6NQ8NiFLyiX5fTfKhGPTqIS17Rbb2coxh /tHpLndK X-Google-Smtp-Source: AGHT+IEob9+z/w1hvogkiR/Kp//LkJYdHtQ9ArarJa3hPnE5n/peMRVpRUWIrYf7qSruTyw0ncGebA== X-Received: by 2002:a05:6512:2203:b0:53d:e592:5415 with SMTP id 2adb3069b0e04-53de5926f10mr5468823e87.34.1732627446804; Tue, 26 Nov 2024 05:24:06 -0800 (PST) Received: from [130.235.83.196] (nieman.control.lth.se. [130.235.83.196]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53ddb348709sm1543432e87.100.2024.11.26.05.24.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 Nov 2024 05:24:06 -0800 (PST) Message-ID: Date: Tue, 26 Nov 2024 14:24:05 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Regression in NFS probably due to very large amounts of readahead From: Anders Blomdell To: Jan Kara Cc: Philippe Troin , "Matthew Wilcox (Oracle)" , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <49648605-d800-4859-be49-624bbe60519d@gmail.com> <3b1d4265b384424688711a9259f98dec44c77848.camel@fifi.org> <4bb8bfe1-5de6-4b5d-af90-ab24848c772b@gmail.com> <20241126103719.bvd2umwarh26pmb3@quack3> <6777d050-99a2-4f3c-b398-4b4271c427d5@gmail.com> Content-Language: en-US In-Reply-To: <6777d050-99a2-4f3c-b398-4b4271c427d5@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6D76A1C0011 X-Stat-Signature: oy5kiam6j1btymyg8juqr3atjenqynxs X-Rspam-User: X-HE-Tag: 1732627440-576099 X-HE-Meta: U2FsdGVkX18DYgIxTB+Hn0xRZT6/vwJsYmXjZ3wtNA6H7ZnWy/hNJ7nAl8wPmMd/zldPQiwuMIHm9bFDuXYuRRhD7x3FAvjbwC57iqbErWm8Q+dCCq0KKbvsWbVF1vm/vDPcW7L07PRQvE2MCuf9vmz9L3SfA1xgdau/7DloM3K7rujT8mLOw4woE7lsHfneV+0viy0mNqr1h9z4K0Pw6Ud9Ke63Wqt707neBZVWG6TWCTpI7vBVaRcuU1sdqx72VOsL3FTrRPq5xeEWQ5xr97sJuu368Ry+P2cKxmZd/eDM6OAOHYW2QbTUcsI0AFAWm0J8l6xMxjD27UhuRJdx4rqnLh0oVGadS4xa6oJYXmxfQAJZJ1oLnJtz8NSUxqCpUdPIEuws7cVvegwfxUOgedUCtLXOh2xz3fpp1/8PUKbFOVx2CpU+82koDYmNwzhEaUizSjiV6xM0ksndJsImXxZjTjBTRHjXrA9jLXfm/jYpYIzZBARtTZPS92W+kXn3t0EBNDHK/khEfhuOnusNtXeGkfvjvg+ePW4bctiTgtU58QKEjYUduO3BsgCoYzgQPORIUTb1RejhtSiI5j0UR89BBOOus2Prp/HEXrw1lVQlQbTGp7sTmjS1ie4BF6rVgKIhbKBzjazhp2npgyuoWOXqHfZwqRifgHFoa/tdygKgYiwuL1kHOLiUVYHYtUKHmcUVXJjcDL0mPkjiYI7GGi78owy4eEZwov3H/raMSfW51E6JJuGx2i9VymfQ0ZqGV3ATCGJhXvBd3+iQZCLtFx7DrYuF9vQUKZm5cH3jigxH/xs9FEHjMKZUjebCt+s3ylsHxqS0JQKeUU6A+FritXmDD8DouC35DI89d1JgPFVvVyQUH3h5HI+z2pgP6oZwg235EyHrRCDHbgQ4Vr5U5vAvWrFWZSJWNhDmk9q0hVpbbCMnyXq5r5RIOAyKSKzf/LUGXW+aIMSQ1O5oE4d B+7sNPEY 8qWMgx7/VNzTfRmkC0LjL2xc4repRtDDSYoxB32ITWz+ue8ITf042dVXySV9Vyr4NhLrZRFC5aUnSYS4ioGmVR5hoVDObAXq5e+IFBVg0g+/wz+Fx3EV6mAK6E5e8L6gEHH5/mCUP9frxo/ymfWp0PzgCRysSiizlxAq8G3Mr4c3C2sDhd+vQYpKua9r4/Cv5VBLZEy19Fw9CD8DN5blf58IruDrjtQ/8cSzMWo/SaR5LaE9brJa2QHQwQ2nUPMzrS2lOA/7aEfJG4TycbFfAnSXQXDtE3s0erNl/7V4xez5/mJuWDb5m1xwAhdm9H8DZO5rHPDSibDdrjIQ0XvEJLfY9zmWa9LQfrMHHe0bNrvKn67tkylknEP2EB4LN15yIoT7JBto/t6Re8ExiGLQxC6ZXYdlnUUuHxB4hyhsgLYxvsVsTQfVDUcZP5f0QtWh2ADwn2dvHPkMY3aQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-11-26 13:49, Anders Blomdell wrote: > > > On 2024-11-26 11:37, Jan Kara wrote: >> On Tue 26-11-24 09:01:35, Anders Blomdell wrote: >>> On 2024-11-26 02:48, Philippe Troin wrote: >>>> On Sat, 2024-11-23 at 23:32 +0100, Anders Blomdell wrote: >>>>> When we (re)started one of our servers with 6.11.3-200.fc40.x86_64, >>>>> we got terrible performance (lots of nfs: server x.x.x.x not >>>>> responding). >>>>> What triggered this problem was virtual machines with NFS-mounted >>>>> qcow2 disks >>>>> that often triggered large readaheads that generates long streaks of >>>>> disk I/O >>>>> of 150-600 MB/s (4 ordinary HDD's) that filled up the buffer/cache >>>>> area of the >>>>> machine. >>>>> >>>>> A git bisect gave the following suspect: >>>>> >>>>> git bisect start >>>> >>>> 8< snip >8 >>>> >>>>> # first bad commit: [7c877586da3178974a8a94577b6045a48377ff25] >>>>> readahead: properly shorten readahead when falling back to >>>>> do_page_cache_ra() >>>> >>>> Thank you for taking the time to bisect, this issue has been bugging >>>> me, but it's been non-deterministic, and hence hard to bisect. >>>> >>>> I'm seeing the same problem on 6.11.10 (and earlier 6.11.x kernels) in >>>> slightly different setups: >>>> >>>> (1) On machines mounting NFSv3 shared drives. The symptom here is a >>>> "nfs server XXX not responding, still trying" that never recovers >>>> (while the server remains pingable and other NFSv3 volumes from the >>>> hanging server can be mounted). >>>> >>>> (2) On VMs running over qemu-kvm, I see very long stalls (can be up to >>>> several minutes) on random I/O. These stalls eventually recover. >>>> >>>> I've built a 6.11.10 kernel with >>>> 7c877586da3178974a8a94577b6045a48377ff25 reverted and I'm back to >>>> normal (no more NFS hangs, no more VM stalls). >>>> >>> Some printk debugging, seems to indicate that the problem >>> is that the entity 'ra->size - (index - start)' goes >>> negative, which then gets cast to a very large unsigned >>> 'nr_to_read' when calling 'do_page_cache_ra'. Where the true >>> bug is still eludes me, though. >> >> Thanks for the report, bisection and debugging! I think I see what's going >> on. read_pages() can go and reduce ra->size when ->readahead() callback >> failed to read all folios prepared for reading and apparently that's what >> happens with NFS and what can lead to negative argument to >> do_page_cache_ra(). Now at this point I'm of the opinion that updating >> ra->size / ra->async_size does more harm than good (because those values >> show *desired* readahead to happen, not exact number of pages read), >> furthermore it is problematic because ra can be shared by multiple >> processes and so updates are inherently racy. If we indeed need to store >> number of read pages, we could do it through ractl which is call-site local >> and used for communication between readahead generic functions and callers. >> But I have to do some more history digging and code reading to understand >> what is using this logic in read_pages(). >> >>                                 Honza > Good, look forward to a quick revert, and don't forget to CC GKH, so I get kernels recent  that work ASAP. BTW, here is the output of the problematic reads from my printk modified kernel, all the good ones omitted: nov 13:49:11 fay-02 kernel: mm/readahead.c:490 000000002cdf0a09: nr_to_read=-3 size=8 index=173952 mark=173947 start=173941 async=5 err=-17 nov 13:49:12 fay-02 kernel: mm/readahead.c:490 000000002cdf0a09: nr_to_read=-7 size=20 index=4158252 mark=4158225 start=4158225 async=20 err=-17 nov 13:49:16 fay-02 kernel: mm/readahead.c:490 0000000036189388: nr_to_read=-8 size=4 index=17978832 mark=17978820 start=17978820 async=4 err=-17 nov 13:49:19 fay-02 kernel: mm/readahead.c:490 00000000ce741f0d: nr_to_read=-5 size=8 index=3074784 mark=3074771 start=3074771 async=8 err=-17 nov 13:49:21 fay-02 kernel: mm/readahead.c:490 00000000ce741f0d: nr_to_read=-4 size=6 index=3087040 mark=3087030 start=3087030 async=6 err=-17 nov 13:49:23 fay-02 kernel: mm/readahead.c:490 0000000036189388: nr_to_read=-2 size=16 index=16118408 mark=16118405 start=16118390 async=10 err=-17 nov 13:49:24 fay-02 kernel: mm/readahead.c:490 0000000036189388: nr_to_read=-10 size=16 index=20781128 mark=20781118 start=20781102 async=16 err=-17 nov 13:49:24 fay-02 kernel: mm/readahead.c:490 0000000036189388: nr_to_read=-13 size=16 index=20679424 mark=20679411 start=20679395 async=10 err=-17 nov 13:49:25 fay-02 kernel: mm/readahead.c:490 0000000036189388: nr_to_read=-9 size=4 index=20792116 mark=20792103 start=20792103 async=4 err=-17 nov 13:50:22 fay-02 kernel: mm/readahead.c:490 000000009b8f0763: nr_to_read=-7 size=4 index=4172 mark=4167 start=4161 async=1 err=-17 nov 13:50:24 fay-02 kernel: mm/readahead.c:490 00000000295f3a99: nr_to_read=-7 size=4 index=4108 mark=4097 start=4097 async=1 err=-17 nov 13:50:24 fay-02 kernel: mm/readahead.c:490 00000000295f3a99: nr_to_read=-7 size=4 index=4428 mark=4417 start=4417 async=4 err=-17 nov 13:56:48 fay-02 kernel: mm/readahead.c:490 000000009b8f0763: nr_to_read=-10 size=18 index=85071484 mark=85071456 start=85071456 async=18 err=-17 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -485,7 +485,21 @@ void page_cache_ra_order(struct readahead_control *ractl, if (!err) return; fallback: - do_page_cache_ra(ractl, ra->size - (index - start), ra->async_size); + long nr_to_read = ra->size - (index - start); + if (index > mark) { + printk("%s:%d %p: " + "nr_to_read=%ld " + "size=%d index=%ld mark=%ld start=%ld async=%d err=%d", + __FILE__, __LINE__, + ractl->mapping->host, + nr_to_read, + ra->size, index, mark, start, ra->async_size, err); + } + if (nr_to_read < 0) { + printk("SKIP"); + return; + } + do_page_cache_ra(ractl, nr_to_read, ra->async_size); } static unsigned long ractl_max_pages(struct readahead_control *ractl, Regards /Anders