From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8692E6688E for ; Sat, 23 Nov 2024 22:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0890A6B0082; Sat, 23 Nov 2024 17:32:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 038036B0083; Sat, 23 Nov 2024 17:32:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1A3F6B0085; Sat, 23 Nov 2024 17:32:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C40076B0082 for ; Sat, 23 Nov 2024 17:32:50 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4B83C121727 for ; Sat, 23 Nov 2024 22:32:50 +0000 (UTC) X-FDA: 82818810858.15.8E4C5F3 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by imf17.hostedemail.com (Postfix) with ESMTP id 69DCF40012 for ; Sat, 23 Nov 2024 22:32:47 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Onn5iKGH; spf=pass (imf17.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732401167; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Nk4VKU+4UkZZl7xiP/BV0EFMMFVeUg9V2zY8tgSiJcE=; b=028LY1ppm4t6T+83I2A0sKHIeCGNZQRDxDCobig5vcaqk/Eui4oZbgrltkF6B5rB21ZX1B yrE3/bIedMglZnnFnUx6jideth85aZTJXCVIRrcospRV0JOgCF/ZDiRkDw2X+rMd7x2R1H 3YoUSTJv8hrKtQzeXk3GopqnsQjKpIs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732401167; a=rsa-sha256; cv=none; b=PadTy5Cu+hvO8oQQZVNdrgwAa1LkyfAU2cZag4lnu5wuHOdfYH4UmMl9MEpSvx/ExKSqqq QDf42vd9BpavvUonCYVkqxFA0bLo7gI6BJL/WSZsBuj+O8uGPK00a3nzn/boFqD0jOAPrt zfb+Z+zonMoIoATXYvEkW1luXEy8vN4= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Onn5iKGH; spf=pass (imf17.hostedemail.com: domain of anders.blomdell@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=anders.blomdell@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-53da2140769so3853317e87.3 for ; Sat, 23 Nov 2024 14:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732401166; x=1733005966; darn=kvack.org; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=Nk4VKU+4UkZZl7xiP/BV0EFMMFVeUg9V2zY8tgSiJcE=; b=Onn5iKGHTbXLdx35tqcA6qCx21yMG+DS/IuwRh/YeiMhqpOQMubya9izwHwesSbqMT hufXwue8Pbp47C6W+h7OzoHw8LcJZr+M74w43rPcrhZo2jRgN6x7a200wWotwdiVkjYd vJw4vjqCh1bR3DR8pRkEP4oozSK3JpOkELLb4x94oLqCwWFp6Foo1TZKz76YyLl04jeI Lq3ZTLkcUQk60s220+Y45B/e5nEoYEMiVTA5K8ydz0ECVqzGm1baUOSrOxc0bqzyQKgc V1GMSdVuUff/k56EkP4jcosa2Xxz2zs3Ei2w0cjuo0FaihyBDdbVx9OfbE3k9u6d/anP pmVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732401166; x=1733005966; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Nk4VKU+4UkZZl7xiP/BV0EFMMFVeUg9V2zY8tgSiJcE=; b=hDt1i/zrLyW6VSevEH6kkfAdXf71N/2pBTUcsqXOqZuUUhbn75M6CgbUA+Iywu5YWp xywUt6Og7hlsXTYnSYt789BjD00YJ3WGspbAZWmQh5wwtsrWECQpy0bAtIkfQCdWzXv0 tSEBG8y0hdELHyoxgNxX1YKPxMPeLUQesXfM0qmiq3cUsiUNNmn6PIdUo/SnliQHS0jD eBi/ypymSSX0p0ReOi6ut76Duqj8ImD/ufPBcZ919jt51P7Via7qvOkLKQimxDQjbxFO s5a/ibj9QBPzidFa1tQ+IuFzyTOcs/VIExXe10Do8XuMo8HDVy0U8RVVSLxQjQG2tMGV WXCA== X-Forwarded-Encrypted: i=1; AJvYcCUs9NDCA00Aeb2UfVkJJYRL/ZWafSoLKwX8c3Wdc12VQIl8o43t6DOsbKt0JnC6LqelNS/bh6R9fg==@kvack.org X-Gm-Message-State: AOJu0YwHX5dsfK1LgetG/v0CFsuHotseK+FEZGIli2GGECW9k4wwr2iV 21awlp7CiUL5Jl+O29Crg1pie8Lrrw5SsVUR1g3+UG0gHW98CwXc X-Gm-Gg: ASbGnct+DTFf7hIJg1hVQnNyq+jxBCQmf9Ka4feQm5PkcpfLX1/71FuPX33c8Cie5mA Y9wOSB0A5LcifZk7dSirAogzQbeMIbmW76ERVsGOEgJfk+Xvy9bzzXdpmHpvwbDB3R5ckF149NV GYwAMkOQLuUkUIhr7V8b5zpJS01qH0tDLzcoJ/GniREzuO40BVrG2LcCDXhnVMWVDS3hwxbni9Z 0JFF6+9EBmhP1K43agq4HG6TBORv4bSgJ6qfj1M7QHxKYJfCG+yV46OrJiZK5DBhTaPDC8l38bC htH4e/Ci87ITSlgKN0rSrQ== X-Google-Smtp-Source: AGHT+IGTZX1caXgox/FS6gjjWk+Y9HI77Uft+ZNVRiPY45h1eXn0pFbaPstf2x2zy2pf6Tod7cKVwQ== X-Received: by 2002:a05:6512:282b:b0:53d:c2f6:8399 with SMTP id 2adb3069b0e04-53dd39b5533mr3975249e87.53.1732401166140; Sat, 23 Nov 2024 14:32:46 -0800 (PST) Received: from [192.168.68.111] (c83-253-44-175.bredband.tele2.se. [83.253.44.175]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53dd244571dsm1090310e87.46.2024.11.23.14.32.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 23 Nov 2024 14:32:45 -0800 (PST) Message-ID: <49648605-d800-4859-be49-624bbe60519d@gmail.com> Date: Sat, 23 Nov 2024 23:32:41 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Jan Kara , "Matthew Wilcox (Oracle)" , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org From: Anders Blomdell Subject: Regression in NFS probably due to very large amounts of readahead Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: w53kyrby47t3z55zbb4cyego39sghk66 X-Rspam-User: X-Rspamd-Queue-Id: 69DCF40012 X-Rspamd-Server: rspam02 X-HE-Tag: 1732401167-313248 X-HE-Meta: U2FsdGVkX182nZ/q8SYvvYVMfEjU4IFamqj5ThL4/7yRUdfKdDPARh74PA7LiX/82iw3mjjiuLzCINcku0u60HdOil1LVB9KxYfeJjqwF3wAvBkBT2PnbBbmllo+zMhu/sEpE5FVDbicS6KIMzpSn2ujyDOMK3+yUD1/FDTGO00tqlxz509Lm0YXI/UgQofMYar1ZNE11UeJ/egw4EGD1C6gtO6yPJE2hpN5Sqjdh+KXlcNHkouHiPl0CaDwoZqY4BDd3yFLDHP2GVpfnK6ghePC9Way5aH8wX/PwfDvBV27DXt/QPI0LAvsY6yYN+defCRd6ipkSkPdSGriEx27XNvL9VsQiJhcrFvicpb3vvxcIM3IVva25u7taA8gwosy7DWValR6qpaO9ERgIb4tROFzCUoREIeLmDBiINi3iIvnNxNWYl5kuCVtzYRulCMyNLyQZ0adfu9i0YS1WJKOjVJp5mvsHFvQnkiOebkhU6/4kkc9rdPnnHBw0r7dCT04PuS17fNhALCWKNArZjD/f32Dc5FRPKrFAMRiuVo0lWp6S1deHPPE25Wal7dt/GAiEwuBI35wRj8p3qf9EjUycAQKUFqPHOM+yGvI0AbFrn0KtJ1tz2M/IVxgVvU+OZrPLtVx7oqEgaO7IKvVdVLqTTU2I2VuepMxPODgtXjUEdSHoI7QADM0HYLHrTpvmsSn/wQ0SF71YELZ+fAInym/GGemUoRw0QXsKlZI9CsNw0iPSgctO1ov5AHfoio5fVPuIIGGntQeYL2CQO2KHEoEUA0ByRq56GBvafQqBlCv9du1QQj/iSPBS+lgsiMZOfNie6OYdHOAMbEUqa/c6pICCDbnzBCDL4RFWMEgBtUKG9awJFjutSXVvyEs1Y7D9KmQ0Mbbl8nroFnWNj41x7EyOf0sIOcJRW8Qj6ITM8zKJ9A/YLSRjy1fBx4v9G6UiqcNtg5RFZKRsOP0H5pE/r6 +X0FDb8T rwT1QBUNroTF2Rof7/Lt3K/aZLOwO0B/qhwuExLdKEabgnLsSiZXH5C0T5ycDr2vIH4O69bSTSnSCvvvaNkzZWlnsopcA7xOjKPR88xuOjBm50uCuv2UhbzsxvqUhSMnp4FWDzd9PiQY1ndLIATtz9xTVKb4u3oR21Qg5g6tbNiRY0yLYFFiEScjJZzr6KglziLyuBRjAKW2GRoDxdb50TABtA2PML9w+eMhuZcxlQTWtFAePpFZBeSacZG5tuJNZcPZ97O2zHQW9XAxhuuhOpU/BXOXkIJS7FfNkBnZ4RsWcIEJAuzlD+j+fcp2Soc6u+T9AqrmqC74jgmH7NDp3+0oN2k9hO2AcSlSNPfZKcNDO7h0NzBQOc5Nh4+ZJY2dX++ottuOCAzm65f7rcnilYUEHKVsfwOD55Uyoy2qUyqTlFQV/CB4Nv9o5VwlJ0GT1Eh8e74I1j3QiHn8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When we (re)started one of our servers with 6.11.3-200.fc40.x86_64, we got terrible performance (lots of nfs: server x.x.x.x not responding). What triggered this problem was virtual machines with NFS-mounted qcow2 disks that often triggered large readaheads that generates long streaks of disk I/O of 150-600 MB/s (4 ordinary HDD's) that filled up the buffer/cache area of the machine. A git bisect gave the following suspect: git bisect start # status: waiting for both good and bad commits # bad: [8e24a758d14c0b1cd42ab0aea980a1030eea811f] Linux 6.11.3 git bisect bad 8e24a758d14c0b1cd42ab0aea980a1030eea811f # status: waiting for good commit(s), bad commit known # good: [8a886bee7aa574611df83a028ab435aeee071e00] Linux 6.10.11 git bisect good 8a886bee7aa574611df83a028ab435aeee071e00 # good: [0c3836482481200ead7b416ca80c68a29cfdaabd] Linux 6.10 git bisect good 0c3836482481200ead7b416ca80c68a29cfdaabd # good: [f669aac34c5f76b58e6cad1fef0643e5ae16d413] Merge tag 'trace-v6.11-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace git bisect good f669aac34c5f76b58e6cad1fef0643e5ae16d413 # bad: [78eb4ea25cd5fdbdae7eb9fdf87b99195ff67508] sysctl: treewide: constify the ctl_table argument of proc_handlers git bisect bad 78eb4ea25cd5fdbdae7eb9fdf87b99195ff67508 # good: [acc5965b9ff8a1889f5b51466562896d59c6e1b9] Merge tag 'char-misc-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc git bisect good acc5965b9ff8a1889f5b51466562896d59c6e1b9 # good: [8e313211f7d46d42b6aa7601b972fe89dcc4a076] Merge tag 'pinctrl-v6.11-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect good 8e313211f7d46d42b6aa7601b972fe89dcc4a076 # bad: [fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c] Merge tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm git bisect bad fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c # good: [f416817197e102b9bc6118101c3be652dac01a44] kmsan: support SLAB_POISON git bisect good f416817197e102b9bc6118101c3be652dac01a44 # bad: [f6a6de245fdb1dfb4307b0a80ce7fa35ba2c35a6] Docs/mm/damon/index: add links to admin-guide doc git bisect bad f6a6de245fdb1dfb4307b0a80ce7fa35ba2c35a6 # bad: [a0b856b617c585b86a077aae5176c946e1462b7d] mm/ksm: optimize the chain()/chain_prune() interfaces git bisect bad a0b856b617c585b86a077aae5176c946e1462b7d # good: [b1a80f4be7691a1ea007e24ebb3c8ca2e4a20f00] kmsan: do not pass NULL pointers as 0 git bisect good b1a80f4be7691a1ea007e24ebb3c8ca2e4a20f00 # bad: [58540f5cde404f512c80fb7b868b12005f0e2747] readahead: simplify gotos in page_cache_sync_ra() git bisect bad 58540f5cde404f512c80fb7b868b12005f0e2747 # bad: [7c877586da3178974a8a94577b6045a48377ff25] readahead: properly shorten readahead when falling back to do_page_cache_ra() git bisect bad 7c877586da3178974a8a94577b6045a48377ff25 # good: [ee86814b0562f18255b55c5e6a01a022895994cf] mm/migrate: move NUMA hinting fault folio isolation + checks under PTL git bisect good ee86814b0562f18255b55c5e6a01a022895994cf # good: [901a269ff3d59c9ee0e6be35c6044dc4bf2c0fdf] filemap: fix page_cache_next_miss() when no hole found git bisect good 901a269ff3d59c9ee0e6be35c6044dc4bf2c0fdf # first bad commit: [7c877586da3178974a8a94577b6045a48377ff25] readahead: properly shorten readahead when falling back to do_page_cache_ra() I would much appreciate some guidance on how to proceed to track down what goes wrong. Best regards Anders Blomdell