From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BD4DC3DA7F for ; Mon, 12 Aug 2024 13:02:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD04E6B007B; Mon, 12 Aug 2024 09:02:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C58D56B008C; Mon, 12 Aug 2024 09:02:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B20DA6B0092; Mon, 12 Aug 2024 09:02:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 93EE36B007B for ; Mon, 12 Aug 2024 09:02:09 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 10709161CED for ; Mon, 12 Aug 2024 13:02:09 +0000 (UTC) X-FDA: 82443606378.30.1C7312F Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) by imf21.hostedemail.com (Postfix) with ESMTP id 28D281C003A for ; Mon, 12 Aug 2024 13:01:57 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of c.ebner@proxmox.com designates 94.136.29.106 as permitted sender) smtp.mailfrom=c.ebner@proxmox.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723467641; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=57zqsOKK3Kli7364S/sNBd+NFiy1TJuLpY8zXMwjRYM=; b=azvYPk4qu40zrPqn2pe6JReJpJUp5Ix83Z06KWANoLeKNiOggo7wof0UQSMH+8rhbkHqDT ofuI2DGyjjI2ksd2X9fmIgAJA+gsPwgzVC+QWv5z9jgW4k+Mp9yBCfycRKg5qC8sh/KTid gEw4aMqyX0xaX6GvyDn9+1eFfooRH8c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723467641; a=rsa-sha256; cv=none; b=V+fZynEtSXxztanv7W8aN+4txDsV/rhecIxApg/2G2OwU0YXaZVR04l86ZZT++nRnAHK9E y8k1EmH7QKg+8Cu34YU2/YxQO6PG5QzA2itK+TVAaCaLIJG1Pv2v+63dkUZk2NbTdY4O6U dcVwR7NeonukFuT0jTCEMEVnOVB822I= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of c.ebner@proxmox.com designates 94.136.29.106 as permitted sender) smtp.mailfrom=c.ebner@proxmox.com Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 35255464CE; Mon, 12 Aug 2024 15:01:55 +0200 (CEST) Message-ID: Date: Mon, 12 Aug 2024 15:01:52 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 14/40] netfs: Add iov_iters to (sub)requests to describe various buffers From: Christian Ebner To: David Howells , Jeff Layton , Steve French Cc: Matthew Wilcox , Marc Dionne , Paulo Alcantara , Shyam Prasad N , Tom Talpey , Dominique Martinet , Eric Van Hensbergen , Ilya Dryomov , Christian Brauner , linux-cachefs@redhat.com, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org References: <20231221132400.1601991-1-dhowells@redhat.com> <20231221132400.1601991-15-dhowells@redhat.com> Content-Language: en-US, de-DE In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 28D281C003A X-Stat-Signature: tuwq4p35gpcmpi8a9ekadkkbrofsu9qj X-Rspam-User: X-HE-Tag: 1723467717-377773 X-HE-Meta: U2FsdGVkX1+IFqvUR058Rf95Dp3/PO936I23+MIVj4/+DdceRNzidtDCxufxA8tX53NC7TY9MaaFtsxl06hcyZkVcJTzu83RiX/JxTJAlN95eem+HD2dpqXZIEA/RHymykfrRTiPuM+35WMf8U68yyAdH10lBrThdJ9VLZh3vyU7xsBneU202J6EkN1B1GfllRqErQWzuLLTzBdmxyT4O2ueRw0N4GSvlXWSsp7Gg6SxkKfYMk3GJTXrrzJgxSaVwKfdczawN2R8NRFuf3z/l92klNeBDFoYCvE/IHAmDXD/0HZRk6SoUnw6i0js38HXRnOoYtHfFRB6aIkcDEf7ai/RczVV1QVQAq04X+5wkVnc0NXGgRlESlsSjrk9yCBW6lICljpmBFEgZJvGZZ1N2wd5rerFsn2PV0sBPb3Yyotv+aVDl9V+LiAeGdFbc+utCxXi8qU6D7JJ6sVxfI2LKy7Mp79lJZJrK39Tzb11O6DAMdLYdNz/FaSbj/9bgnkBX/MqIbbgYz5ouYvpAid7I6wn8pypXB5KUpdqUYqkoisKEOdw9ywVRHW07nFEm2LKtTEWhFoJfOqb6enDZs0ALm40J4H/hXIVCSKUfNYYx8Zy5zplLGvWET1+mFPJK8nrfAy42cmjiLeCy0ppOy1E5lf/2kzBHkJ1jRqAyvGm5axL3BUbkopZzQPOIVYFaQ1q0LxHvtG1M1W4G0URumbJz49F9NupkzcZf5n+VwGUkZQ9qwt3EzR1+zGpY8AEgOgSHeeQAppk/H8xxojLvTAcYUL9iQlI9U9sHYW57+l3Lr+PlwMbtKY+vf6whq4LZiGOkvojPORQqbeUInI6/x6o9KI5iTspE8bpJuAsQ4m4Lxo8Z1sWZdUqALS4/CDgcxEnYG+aVhKCvq9EThY/X/TC2iUFQ9EoVmYB2gNRaUOx8UVXLS2R8ZHwzLQnRHgL49uH6juRIls8eWO4mXw0STa EJ2fAA46 yIS+6Loc//mN2G3+Q/dMAeQf1kQVDfz5QDOVTbvl1S7XVo5D1JoYpc78Dg/xuCsBkHq3pw5iy/8bXs/n5R45O8DBm31lzThoTKfw3zxljUiIhE67jG5mI9CJaHJNLBbYHCF42vx2SYzy7+mp41DMsxNKG2Iw/NGab+qjZ2LmtJmnkjzjQdS29MmoaN6twOE8EMakaBLKjbK2u7y9C9aVRJeROgdBzlhvMQujwbL67SZ8HDJzuxVVKJpCITuLZ4mO64bfnxdPIum4caK0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 8/8/24 10:07, Christian Ebner wrote: > Hi, > > recently some of our (Proxmox VE) users report issues with file > corruptions when accessing contents located on CephFS via the in-kernel > ceph client [0,1]. According to these reports, our kernels based on > (Ubuntu) v6.8 do show these corruptions, using the FUSE client or the > in-kernel ceph client with kernels based on v6.5 does not show these. > Unfortunately the corruption is hard to reproduce. > > After a further report of file corruption [2] with a completely > unrelated code path, we managed to reproduce the corruption for one file > by sheer chance on one of our ceph test clusters. We were able to narrow > it down to be possibly an issue with reading of the contents via the > in-kernel ceph client. Note that we can exclude the file contents itself > being corrupt, as any not affected kernel version or the FUSE client > gives the correct contents. > > The issue is present also in the current mainline kernel 6.11-rc2. > > Bisection with the reproducer points to this commit: > > "92b6cc5d: netfs: Add iov_iters to (sub)requests to describe various > buffers" > > Description of the issue: > > * file copied from local filesystem to cephfs via: > `cp /tmp/proxmox-backup-server_3.2-1.iso > /mnt/pve/cephfs/proxmox-backup-server_3.2-1.iso` > * sha256sum on local filesystem: > `1d19698e8f7e769cf0a0dcc7ba0018ef5416c5ec495d5e61313f9c84a4237607 > /tmp/proxmox-backup-server_3.2-1.iso` > * sha256sum on cephfs with kernel up to above commit: > `1d19698e8f7e769cf0a0dcc7ba0018ef5416c5ec495d5e61313f9c84a4237607 > /mnt/pve/cephfs/proxmox-backup-server_3.2-1.iso` > * sha256sum on cephfs with kernel after above commit: > `89ad3620bf7b1e0913b534516cfbe48580efbaec944b79951e2c14e5e551f736 > /mnt/pve/cephfs/proxmox-backup-server_3.2-1.iso` > * removing and/or recopying the file does not change the issue, the > corrupt checksum remains the same. > * Only this one particular file has been observed to show the issue, for > others the checksums match. > * Accessing the same file from different clients results in the same > output: The one with above patch applied do show the incorrect checksum, > ones without the patch show the correct checksum. > * The issue persists even across reboot of the ceph cluster and/or clients. > * The file is indeed corrupt after reading, as verified by a `cmp -b`. > > Does anyone have an idea what could be the cause of this issue, or how > to further debug this? Happy to provide more information or a dynamic > debug output if needed. > > Best regards, > > Chris > > [0] https://forum.proxmox.com/threads/78340/post-676129 > [1] https://forum.proxmox.com/threads/149249/ > [2] https://forum.proxmox.com/threads/151291/ Hi, please allow me to send a followup regarding this: Thanks to a suggestion by my colleague Friedrich Weber we have some further interesting findings. The issue is related to the readahead size passed to `mount.ceph`, when mounting the filesystem [0]. Passing an `rasize` in the range of [0..1k] leads to the correct checksum, independent of the bisected patch being applied or not. Ranges from (1k..1M] lead to corrupt, but different checksums for different `rasize` values, and finally `rasize` values above 1M lead to a corrupt, but constant checksum value. Again, without the bisected patch, the issue is not present. Please let me know if I can provide further information or debug outputs in order to narrow down the issue. Best regards, Chris [0] https://docs.ceph.com/en/reef/man/8/mount.ceph/#advanced