From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEC82C0218C for ; Fri, 24 Jan 2025 18:21:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9462728008B; Fri, 24 Jan 2025 13:21:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 87FBB280079; Fri, 24 Jan 2025 13:21:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6116628008B; Fri, 24 Jan 2025 13:21:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 40831280079 for ; Fri, 24 Jan 2025 13:21:53 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BD6171A15F7 for ; Fri, 24 Jan 2025 18:21:52 +0000 (UTC) X-FDA: 83043164064.21.6A5A3F1 Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) by imf13.hostedemail.com (Postfix) with ESMTP id 54C9F20009 for ; Fri, 24 Jan 2025 18:21:50 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=auristor.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of marc.c.dionne@gmail.com designates 209.85.218.47 as permitted sender) smtp.mailfrom=marc.c.dionne@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737742910; a=rsa-sha256; cv=none; b=5KMquzX+o4eikXkonto3P8qays1WL3VPdJenpouI5fuBgBH4TZVeoDLaz3HoCfR9yfSwRZ cqw8VUQyX7YwZxR1WxYVk74FpmWtPseu074X6PMfWc+d5a3MistO++0OHP8r2yZ69/bfec NgKbJBfsciPr0v1HBOjPo8fq8oH71I0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=auristor.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of marc.c.dionne@gmail.com designates 209.85.218.47 as permitted sender) smtp.mailfrom=marc.c.dionne@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737742910; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LXZkbsZymQDEeD5R6N901lei4FIsTKs9xGaTy0mc5JE=; b=QKlQwRdnLrrJc5g2+eEPyOFAObokDf1/I/5FnHXFDBZPggIf3rAO5PMLVCw0XJaRTpRcSc 08BAxDABbDngjnNMjYFDuU1p5wGFllw6KTFFu+OLU76WiUYyEr8Zl0mYqWZu0jrxw7NEPe bj3TdTrjMsjI9i0NmPKW1a9j/o/sSqY= Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-aaee0b309adso349666666b.3 for ; Fri, 24 Jan 2025 10:21:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737742909; x=1738347709; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LXZkbsZymQDEeD5R6N901lei4FIsTKs9xGaTy0mc5JE=; b=plMqEyT/IiZ2yB1FxT50Ej3Lcyxpg2UW/L1k5SJNz4SX73w6Ze/lp2NsJzX+0cvBKM VpSxOea5iFJwShQQDv2ptZ6InCO9cWtzIutmTdBl9380frTAGm6glNmr3LWjmNeIJX/k 71oLZ8f0BYsxxx1ZSQzkOjIxPBmDb/T5m4YOouXDYmJJttciIjgLmZt8t8pPBqgir6qs s//tD5jEbevnpU600wWDhFz32pHWj8jXToMQSKQFCM+WakvM1r5xkWVeXDzPTRgtiOrs pY2gq7Jz0zwGKc/NecqG4nrKl9QZ8feOssXIqNB2LUpi/RDK8FuL+TfSM5shpcMg69a6 QiJA== X-Forwarded-Encrypted: i=1; AJvYcCXqFMiFuH/PlCH4xxK16dlJImAJvX9XM+3b25vx6vIvokKTbSjcZ1NqQ65uOaZbvA4XPOI6gqxMlw==@kvack.org X-Gm-Message-State: AOJu0YxbOViMVTfpNXoGvLTbGE+Lyb/yv6Bb/B588L5+pKnbImm/FYQj 3M1BRB3x+nzSa/eDRavzMK3ujLJPgGJIZlxhh+d5T4gZYooLCHGaYwOJQSe8 X-Gm-Gg: ASbGncvMPmzRd2BAZEDklvl/7k7V12GSWsnHKr0lT2dh/HXE7EPRljgUnDUhhdXW4oN nkuCg120PQe1L1TLjsE+t+p0AK2Y6n8qbMGqasQJkjTxfqm0KLkj0leIFeKoY7/jS4UzjmBU/Ca 61/vdywpP68yErAKqhhJik1DlMqvcQbMHa+X1CueGY0b07KtGBnHY+l3D1lo32CSxVcgY49zicJ wjvC8Uz6TlyfIYSMB3U8MW60LBGgzuEXKjV8L7qVgsJQ4I8YjSXZgAlp3Uip+tK2G8jFLo+h3dp vsSaTdqvA1cFHMhxf8J+yFYZ8digJRZXsPjWzE858+eEkVAB X-Google-Smtp-Source: AGHT+IGaYDZmEaZ+8zVj7aVstrBcDhkJaIxiLZkhtMemw4ChaUGVN4d0/FNYSOxUpTJ7q0Gs5ArEhA== X-Received: by 2002:a17:906:2cc4:b0:ab3:a3b4:f91c with SMTP id a640c23a62f3a-ab3a3b4f9b3mr2243749466b.34.1737742908601; Fri, 24 Jan 2025 10:21:48 -0800 (PST) Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com. [209.85.218.41]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ab6760b70a7sm163138366b.116.2025.01.24.10.21.47 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Jan 2025 10:21:48 -0800 (PST) Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-aa684b6d9c7so427799966b.2 for ; Fri, 24 Jan 2025 10:21:47 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXEzHlI78xzMtAhvIc4vKudv0ZCJ5OkWz0zciTrOXzQsAYMkBdzmnVyP/oPf8pSxqm/jbdFc59CoA==@kvack.org X-Received: by 2002:a17:907:7d9f:b0:aae:df74:acd1 with SMTP id a640c23a62f3a-ab38b0b9b7dmr3175435766b.11.1737742907278; Fri, 24 Jan 2025 10:21:47 -0800 (PST) MIME-Version: 1.0 References: <20241216204124.3752367-1-dhowells@redhat.com> <20241216204124.3752367-28-dhowells@redhat.com> In-Reply-To: From: Marc Dionne Date: Fri, 24 Jan 2025 14:21:35 -0400 X-Gmail-Original-Message-ID: X-Gm-Features: AWEUYZlb36GF3Tcj0JvJQORKz4x4XpHrlQbMsxXnEx1A7tVtE0HWDV4nwAVkVJo Message-ID: Subject: Re: [PATCH v5 27/32] netfs: Change the read result collector to only use one work item To: Ihor Solodrai Cc: David Howells , Christian Brauner , Steve French , Matthew Wilcox , Jeff Layton , Gao Xiang , Dominique Martinet , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Eric Van Hensbergen , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 54C9F20009 X-Rspamd-Server: rspam10 X-Stat-Signature: bt59dwwk41m88nf756bpu9effwaynm3j X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1737742910-369234 X-HE-Meta: U2FsdGVkX1+7mUZyqGgcW27TZPkHPWZRE6keVYgEzlMcJR0k5zk48z1TVchZ5KhSLY4QnHV5V3cLgpD77FWBzCAA1HMR9x+mizNQhbLOzsKcplWxfdcoRWlmGS1VQAqkhCe/iyaWFOcqLwCEvYFsuNA8k+nPsw3DaAECwdjhRi38ATIKplSWrAFZxk0YsfKAYvCo08Tt6keOJVYl1VA9OwO4DY+zSU1DXocegXhAn0zy4Px+N1QnK79lwmwi9IkwgldaDZtGJCqWMMHzYg1GQqoQ6Q8Esekr0G+oOfj/4Xj/6It6YG7U5v6UmWT4opLgpcr1sZbmobBNG50Hk1Yk2Hw8Z/n+xq3ejqEbaIVWQI/lvsEJsfFNiBgr70oLAWN5xT4Tt7plBYMx07JEBah+0lWaxJJKnTlOJmuitiAubqQ+am2OGECNdJa488ACGnWnCOz0T2/bK+WQpPpjj3ICb3Q+22a42cvSHm/vTDOeih51jKiB4g8KYUzfZQrml6CNH7aBanYt54Hl0VJHo2HpSNlLFKJ3tAU8i0gJ7WuwLic4TAxAmHX8mkPH2b3TITFlqOwV4FY4Nqde0IowXXo8NHT/dIJwCVpkd2DvtuLkCmklHrZx5Calm924QjoOK+deqeFL+NBe//XlXsMhdwAtt/EdKCcdUSnCWIKbo2Do43KldN8UA9DIuq45pbaiJPphR85NoMSQzWfQhkoSbrRHUZ91EOAImxyyGTqpK6VkIDNlEdXpxsimBHuGb//elghoDCcPcG+5EMKZ2Adho9xvdZma9gMMJxdGgWBc4x42MC/tlfraB32DoFfYzhEfWZOxY5Ddn1geX1bA7A6YbwOKHVSsZOvyjrUWGZm3+bBcpEWgive/9ey5stxXUksNMWtL1Py+TZWQTYhpSFuHy/KNCIL1+vhzrem4/Fdk6ssyfBdO+nG/rPly3EhN0XmX21t2Is7tJ6BtWX3sCkYY64W eXBmitRI JVMRYWbwsMTYkIE9E84RwxiXmraZ9U+0c7vnobc1IoLreIXSWO5J+ilb5oyl3UinJQ19MTk+KWes9ubn9RDc1TUxGVc4l08BlxYFdIgGHOZGd0GYGuV0WAtQ3YVuffaiw62hKgs8GTOlDISfyYVKqk6MrUizZcSLZPdQGFOcht+YtuS65Zg+lyZh0OodqSe16oBrCvzC7LxAB7Rh175qeoAcerLHp5YREbLdEyAs4+WZ8XQAa0MhsJYwOTbIHrIJsFjpiq3ftge1TgE7fOxJWPJKAH0VRSaTjKmobK0vwAobhAM2zsqi7ElZes/Dmn3Z4nvN5xKu1GAMpbp1BfxhTRxEWavcwgIT7DAOOYNiCaFwUK8kbr1HlbQ/vXqACo5w1tIav91gvpW5uKka90TtyBqK0yfZrX0SgBl6u/nmxo6nHv/4CtJGAKpTOJ6F3ajIj3HyycjZ2KaTQPZQxIBCut4Xt5l98kJ1w8EB9T2ETKce/QgrDRiQHadILQY4OrghKc2RwQ/MbG8+L3go56uccwgnQ5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.004699, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 24, 2025 at 2:00=E2=80=AFPM Ihor Solodrai = wrote: > > On Monday, December 16th, 2024 at 12:41 PM, David Howells wrote: > > > Change the way netfslib collects read results to do all the collection = for > > a particular read request using a single work item that walks along the > > subrequest queue as subrequests make progress or complete, unlocking fo= lios > > progressively rather than doing the unlock in parallel as parallel requ= ests > > come in. > > > > The code is remodelled to be more like the write-side code, though only > > using a single stream. This makes it more directly comparable and thus > > easier to duplicate fixes between the two sides. > > > > This has a number of advantages: > > > > (1) It's simpler. There doesn't need to be a complex donation mechanism > > to handle mismatches between the size and alignment of subrequests and > > folios. The collector unlocks folios as the subrequests covering each > > complete. > > > > (2) It should cause less scheduler overhead as there's a single work it= em > > in play unlocking pages in parallel when a read gets split up into a > > lot of subrequests instead of one per subrequest. > > > > Whilst the parallellism is nice in theory, in practice, the vast > > majority of loads are sequential reads of the whole file, so > > committing a bunch of threads to unlocking folios out of order doesn't > > help in those cases. > > > > (3) It should make it easier to implement content decryption. A folio > > cannot be decrypted until all the requests that contribute to it have > > completed - and, again, most loads are sequential and so, most of the > > time, we want to begin decryption sequentially (though it's great if > > the decryption can happen in parallel). > > > > There is a disadvantage in that we're losing the ability to decrypt and > > unlock things on an as-things-arrive basis which may affect some > > applications. > > > > Signed-off-by: David Howells dhowells@redhat.com > > > > cc: Jeff Layton jlayton@kernel.org > > > > cc: netfs@lists.linux.dev > > cc: linux-fsdevel@vger.kernel.org > > --- > > fs/9p/vfs_addr.c | 3 +- > > fs/afs/dir.c | 8 +- > > fs/ceph/addr.c | 9 +- > > fs/netfs/buffered_read.c | 160 ++++---- > > fs/netfs/direct_read.c | 60 +-- > > fs/netfs/internal.h | 21 +- > > fs/netfs/main.c | 2 +- > > fs/netfs/objects.c | 34 +- > > fs/netfs/read_collect.c | 716 ++++++++++++++++++++--------------- > > fs/netfs/read_pgpriv2.c | 203 ++++------ > > fs/netfs/read_retry.c | 207 +++++----- > > fs/netfs/read_single.c | 37 +- > > fs/netfs/write_collect.c | 4 +- > > fs/netfs/write_issue.c | 2 +- > > fs/netfs/write_retry.c | 14 +- > > fs/smb/client/cifssmb.c | 2 + > > fs/smb/client/smb2pdu.c | 5 +- > > include/linux/netfs.h | 16 +- > > include/trace/events/netfs.h | 79 +--- > > 19 files changed, 819 insertions(+), 763 deletions(-) > > Hello David. > > After recent merge from upstream BPF CI started consistently failing > with a task hanging in v9fs_evict_inode. I bisected the failure to > commit e2d46f2ec332, pointing to this patch. > > Reverting the patch seems to have helped: > https://github.com/kernel-patches/vmtest/actions/runs/12952856569 > > Could you please investigate? > > Examples of failed jobs: > * https://github.com/kernel-patches/bpf/actions/runs/12941732247 > * https://github.com/kernel-patches/bpf/actions/runs/12933849075 > > A log snippet: > > 2025-01-24T02:15:03.9009694Z [ 246.932163] INFO: task ip:1055 blocke= d for more than 122 seconds. > 2025-01-24T02:15:03.9013633Z [ 246.932709] Tainted: G = OE 6.13.0-g2bcb9cf535b8-dirty #149 > 2025-01-24T02:15:03.9018791Z [ 246.933249] "echo 0 > /proc/sys/kerne= l/hung_task_timeout_secs" disables this message. > 2025-01-24T02:15:03.9025896Z [ 246.933802] task:ip stat= e:D stack:0 pid:1055 tgid:1055 ppid:1054 flags:0x00004002 > 2025-01-24T02:15:03.9028228Z [ 246.934564] Call Trace: > 2025-01-24T02:15:03.9029758Z [ 246.934764] > 2025-01-24T02:15:03.9032572Z [ 246.934937] __schedule+0xa91/0xe80 > 2025-01-24T02:15:03.9035126Z [ 246.935224] schedule+0x41/0xb0 > 2025-01-24T02:15:03.9037992Z [ 246.935459] v9fs_evict_inode+0xfe/0x= 170 > 2025-01-24T02:15:03.9041469Z [ 246.935748] ? __pfx_var_wake_functio= n+0x10/0x10 > 2025-01-24T02:15:03.9043837Z [ 246.936101] evict+0x1ef/0x360 > 2025-01-24T02:15:03.9046624Z [ 246.936340] __dentry_kill+0xb0/0x220 > 2025-01-24T02:15:03.9048855Z [ 246.936610] ? dput+0x3a/0x1d0 > 2025-01-24T02:15:03.9051128Z [ 246.936838] dput+0x114/0x1d0 > 2025-01-24T02:15:03.9053548Z [ 246.937069] __fput+0x136/0x2b0 > 2025-01-24T02:15:03.9056154Z [ 246.937305] task_work_run+0x89/0xc0 > 2025-01-24T02:15:03.9058593Z [ 246.937571] do_exit+0x2c6/0x9c0 > 2025-01-24T02:15:03.9061349Z [ 246.937816] do_group_exit+0xa4/0xb0 > 2025-01-24T02:15:03.9064401Z [ 246.938090] __x64_sys_exit_group+0x1= 7/0x20 > 2025-01-24T02:15:03.9067235Z [ 246.938390] x64_sys_call+0x21a0/0x21= a0 > 2025-01-24T02:15:03.9069924Z [ 246.938672] do_syscall_64+0x79/0x120 > 2025-01-24T02:15:03.9072746Z [ 246.938941] ? clear_bhb_loop+0x25/0x= 80 > 2025-01-24T02:15:03.9075581Z [ 246.939230] ? clear_bhb_loop+0x25/0x= 80 > 2025-01-24T02:15:03.9079275Z [ 246.939510] entry_SYSCALL_64_after_h= wframe+0x76/0x7e > 2025-01-24T02:15:03.9081976Z [ 246.939875] RIP: 0033:0x7fb86f66f21d > 2025-01-24T02:15:03.9087533Z [ 246.940153] RSP: 002b:00007ffdb3cf93f= 8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7 > 2025-01-24T02:15:03.9092590Z [ 246.940689] RAX: ffffffffffffffda RBX= : 00007fb86f785fa8 RCX: 00007fb86f66f21d > 2025-01-24T02:15:03.9097722Z [ 246.941201] RDX: 00000000000000e7 RSI= : ffffffffffffff80 RDI: 0000000000000000 > 2025-01-24T02:15:03.9102762Z [ 246.941705] RBP: 00007ffdb3cf9450 R08= : 00007ffdb3cf93a0 R09: 0000000000000000 > 2025-01-24T02:15:03.9107940Z [ 246.942215] R10: 00007ffdb3cf92ff R11= : 0000000000000202 R12: 0000000000000001 > 2025-01-24T02:15:03.9113002Z [ 246.942723] R13: 0000000000000000 R14= : 0000000000000000 R15: 00007fb86f785fc0 > 2025-01-24T02:15:03.9114614Z [ 246.943244] That looks very similar to something I saw in afs testing, with a similar stack but in afs_evict_inode where it hung waiting in netfs_wait_for_outstanding_io. David pointed to this bit where there's a double get in netfs_retry_read_subrequests, since netfs_reissue_read already takes care of getting a ref on the subrequest: diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c index 2290af0d51ac..53d62e31a4cc 100644 --- a/fs/netfs/read_retry.c +++ b/fs/netfs/read_retry.c @@ -152,7 +152,6 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags); } - netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); netfs_reissue_read(rreq, subreq); if (subreq =3D=3D to) break; That seems to help for my afs test case, I suspect it might help in your case as well. Marc