From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A48C52D6F for ; Tue, 27 Aug 2024 03:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D0756B0093; Mon, 26 Aug 2024 23:32:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 680B96B0095; Mon, 26 Aug 2024 23:32:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56E736B0096; Mon, 26 Aug 2024 23:32:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 38ABE6B0093 for ; Mon, 26 Aug 2024 23:32:51 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C6C381411E3 for ; Tue, 27 Aug 2024 03:32:50 +0000 (UTC) X-FDA: 82496603700.15.789A2F4 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) by imf12.hostedemail.com (Postfix) with ESMTP id DFBD240003 for ; Tue, 27 Aug 2024 03:32:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aghUYc3d; spf=pass (imf12.hostedemail.com: domain of smfrench@gmail.com designates 209.85.167.48 as permitted sender) smtp.mailfrom=smfrench@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724729454; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2xbzKRYt6Qy3ZNV+HFmoHf+KHGRhq2St3sDinQito6s=; b=jVlVI8X3OhsN1af0kKTcl4s8OWBzFwvFx/TP9wvrA+EmzXm0OAFKWB+ovgOe9kpOzGL/01 YEEZCVIPFtTp1qMn093/wvD/Y+30Ol/xdp5ZzDrkOYOsMecEtCuMJnzS0tX63+HJbCTmok QnD1ScZSNCtx4QVz9b19lCHu5OwJxns= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aghUYc3d; spf=pass (imf12.hostedemail.com: domain of smfrench@gmail.com designates 209.85.167.48 as permitted sender) smtp.mailfrom=smfrench@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724729454; a=rsa-sha256; cv=none; b=d15U3/hc+t0lCYUH9m/bumV9pC8IVAx2i/CC4TyJEFepJGJyL0+s+AStqkmdL10697HOkS 3d8CGhA8A91LbteOg9LmnNNHH8Dls3+OHoecR7bcTQ1Xp4/ctTnTW5y2OrhlPQz2my481e uN+r73F1VRVMz6jHsMT6kGpJXQGLtjk= Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-5334c4cc17fso6935751e87.2 for ; Mon, 26 Aug 2024 20:32:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724729567; x=1725334367; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2xbzKRYt6Qy3ZNV+HFmoHf+KHGRhq2St3sDinQito6s=; b=aghUYc3drHF4+avtt8TVgdTtXZlgYDH0HB3UjqKP9sH4ghZOS+sawwi8zytBTJ5CJC YLsZ5m2vGH/ptuQlhiGq08vijDnEXXZlY3woABhOxaASWrWgt6rsQKmgM10QMb5ZW2Aq RZgzTq1O7of2mqYb1fcgTKjA7MZrKi5H4RfE8udiCrayUN43yP0RHVYBhcvLLF4sYVax Llcb4SGArUJrq1nsuPKC0y3GkdyVQan6fjAiJFdLkEYSFXJiPWZbdVvXCj+gr0A3Lsyp QPvFpjpR8Za/HhR2+XgGjThaeQ+xfd41BcJA2++qCMlftkTPrpDcDw11zSGMNj3cm1DK VkiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724729567; x=1725334367; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2xbzKRYt6Qy3ZNV+HFmoHf+KHGRhq2St3sDinQito6s=; b=rvqflFvo2D3pYrQWhbImAZorBk1onH4bMu86QrOY4+9rq7JTevSqqmI6P5CPaKOjOs pRHDlrWPREPs/KA2WYUdTJPn9UP5YHi6P7bizPvInsukDeATyxaezwfiqX+jqmyynRVq OQfsoXKFKBkrnXzOyUhSi51VocoN2arRhvGhp4F0QmCl2n6O6SxR0iKqto0QGQKaWxoL Rxc1cPA6mFrEBk9SEgHimS6t7uq5twujswDZq0SdJdj8GvWhXIA79hJQG06X54mEG7D+ M9BMyJamNtszdftDzGGPWTE2pGrDtv1mPWhwHHR79WMBTweHN9JU30GagNOx+Zyz5iAJ oHVA== X-Forwarded-Encrypted: i=1; AJvYcCXkxMlEV4Bvuf4VVf4Q4G0XR/eav+SeuJbyoK2Aj6WNcC4pXyBGt4SogbWSx0EtzRyW9ndvRYDWww==@kvack.org X-Gm-Message-State: AOJu0Yx97kWFr2EX+nC3u5KhWV2Yb/h6aPK9aW2FOuU06xMMM4KGD0y8 ixJTZe632gZYq1YjylKYmzm8/g5rSk5N4irMB74RO9h6PBv4bJUjrZGa07Fk48KS6SCokxdpA1Q 1ogW6XacpaLdIEvEWBDVXSroDnHsLvwUR X-Google-Smtp-Source: AGHT+IGlOWGqqXh1+AAccZrVim+pgf7gaTU+IPPNG2QORVDEu9THbJF91fyJWwiE2/xBAQh6UFqMh3R4jhgIzDZyGwA= X-Received: by 2002:a05:6512:230a:b0:52b:9c8a:734f with SMTP id 2adb3069b0e04-5344e500664mr813134e87.50.1724729566569; Mon, 26 Aug 2024 20:32:46 -0700 (PDT) MIME-Version: 1.0 References: <37fncjpgsq45becdf2pdju0idf3hj3dtmb@sonic.net> In-Reply-To: <37fncjpgsq45becdf2pdju0idf3hj3dtmb@sonic.net> From: Steve French Date: Mon, 26 Aug 2024 22:32:35 -0500 Message-ID: Subject: Re: [REGRESSION] cifs: Subreq overread in dmesg, invalid argument & no data available in apps To: Forest Cc: David Howells , Steve French , Shyam Prasad N , Rohith Surabattula , Jeff Layton , linux-cifs@vger.kernel.org, netfs@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, regressions@lists.linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DFBD240003 X-Stat-Signature: d39zx7snx9s1g5zgkast99fwykqirmpg X-Rspam-User: X-HE-Tag: 1724729568-280394 X-HE-Meta: U2FsdGVkX18Wu769yR0YvbtdkB4ePp9zHZRrdY2ixiNao6EiGyeR6GX3eUkU2sA+PbsPnc++7nLX8hkwu8xmJsviTcSLA2jq5uZqXL2W4//jdOMKiVzNveRf9bz1s1t5tiEB3sbcgEJTg9v3Hmp7/Z7NWT025/Qo43yYVeF9EvB2SlIPTfav0rSCO7YqihdJTFTm+cKsvw/NQOrcGOb7mUkMnkznL0BrQ6moFNzW2dTOaJVRf6tgYDZPRtIILVzpdBJiskn+cnEIWEgjxjjQ/bhBaqx2v5yPo389ntSUsQc5Y+9csg3kfa6WUi1nodBE/ogcF3UeSY2rLyT22sYTsTe5yfAya2eG6lnt9CAfeIJHAg2fD9iwFBSjksJusplH6SdkPf0l8xb2eeWKK3ntF99Nyu2nIsx3whaDxfkaxgTDrMdmgR2qSn565Pvl3hZxw4JbgGSYADDdPQzGg8BvM0bJ5DC9//4loSi6TSYbEtg7qdqGW+YuqHO4kYosbwai3gmmHK7kEZs/8rOUI1oeWO+54RnFAbH+bXZ1PMbtGIfm+ioBhg36dkuLm4Er89m5OHz8fYTaqVNm9EcaELyqEBwHpOJY/pdFOE0HSykbiATp7OD5utJ5FObWRslHZtHbY05UjC14lJLCmPPTmkdpigFROlWXUODgOYlAkEllqg7jOiA2ZpUjYSeAiHj5wDwHC1snpDU+kmvFvS7VJeNQrJ9amHGcRptqoN6z2r2oMumEmGKz5mu8fWXRX8vYrsfctAUs0glPq732fAvXHjL0LXsO0T0nyQmQY68YBfTSI0DcuoDXRiENu7krgD4nFDlaXKr/6M1OWgKK1/hHjganOhQ7Q2zsn3AG1Qp06hcxWWSS6r22EnIoU2ix7jL4Xq/FzKn/OALds5O4PJuGs68kIsYmh7arIcfSzRFgcfk6WHVUZe5zAnhyE/D3qx09cfQIukWQmfZUnWHtdC/2gj6 6l+1oRx+ oSrK8vDAS442yDfwk/Avf5HhOvKjAbm0QOUGUsvv5MAAHXzqb26THRqNE5wnERI6RLjhLnSh/unr+9Q1MZ34yiW9vuWIKGxn0IYEaVO4eiFsHbXCo1KyIRj3rQyrxuFB5V50p5702k5vBfYwqDyhQ6wMgZxYQvkkPYYHAxGg6wA8uG2V6dEYfzZ0KyxXb9Ig/i5hUwelaiXnRrPLM1WnYPjIMkXoBBOQav4Dg3r5D7VDzwo43jnJOijMAE8y7Hlb+7K6zLwT/i2Z8jHmjLsXB9agTTFNqAwnnUBY0fckWLV2uKn6q/8zRTcMPlghAHbrUVAR7iltxykVIvMp6V7XjETvBeh4AkzDZ6odLHSVX2KuvWKhNmR9yAXVfvtvQmF+t0mrujcg06lscV6ynPvQF+kA2GaBeSZNYHf2ssXstbZBAgpddLT06jMHWAx69a1vFsTh7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Yes - thanks for reporting this. I can reproduce this (although slightly different error on "cat foo" with 6.11-rc5 vs. later patches in for-next - I get ""No data available" now with some of David's additional patches instead of "Invalid argument" which you get on 6.11-rc5). This problem looks related to something that David has discussed earlier - on the wire (with 6.11-rc5) we see SMB3 READ with the server returning "STATUS_INVALID_PARAMETER" - there are actually two reads in a row that are similar (1MB at offset 0), the first succeeds, the second request has a "credit charge" of 1 (instead of 16 which is what I would have expected) which is likely related to the cause of the problem (and the second read fails). It looks like some of this is fixed with David's patch: https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/commi= t/?h=3Dnetfs-fixes&id=3D78d0d91398ad7bc37e73cdda65602ac8b6d675bb 0014-cifs-Fix-lack-of-credit-renegotiation-on-read-retry.patch It is possible that the second part of this is fixed with: https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/commi= t/?h=3Dnetfs-fixes&id=3De81de4d9db1c25dc3e9feb21548fc5f3e2b3ad8e Is there an easy way for you to retry with those two patches? On Sun, Aug 25, 2024 at 6:26=E2=80=AFPM Forest wrote: > > #regzbot introduced: e3786b29c54c > > Dear maintainers, > > Recent kernel release candidates have a cifs regression that produces > unexpected errors in userspace and a WARNING (with stack trace) in dmesg. > > I can consistently reproduce it with these commands on a mounted Samba > share: > > > $ echo hello > foo > $ ls -l foo > -rw-r----- 1 user user 6 Aug 25 15:41 foo > $ cat foo > cat: foo: Invalid argument > $ xxd foo > 00000000: 6865 6c6c 6f0a 0000 0000 0000 0000 0000 hello........... > 00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > (...more null bytes...) > 00001fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 00001ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > xxd: No data available > > > The xxd command above also triggers these kernel log messages... > > Subreq overread: R3[1] 312 > 8192 - 7956 > WARNING: CPU: 30 PID: 421 at fs/netfs/io.c:495 > > ...followed by the usual module list and stack trace. > > > The bug is present in kernel v6.11-rc5, but not the 6.10 series. > > Git bisect says: > > e3786b29c54cdae3490b07180a54e2461f42144c is the first bad commit > commit e3786b29c54cdae3490b07180a54e2461f42144c > Author: Dominique Martinet > Date: Thu Aug 8 14:29:38 2024 +0100 > 9p: Fix DIO read through netfs > > > Here's the full dmesg output when I run xxd on kernel v6.11-rc5: > > [ 48.137018] ------------[ cut here ]------------ > [ 48.137021] Subreq overread: R3[1] 312 > 8192 - 7956 > [ 48.137029] WARNING: CPU: 30 PID: 421 at fs/netfs/io.c:495 netfs_subre= q_terminated+0x276/0x2d0 [netfs] > [ 48.137046] Modules linked in: rfcomm algif_hash algif_skcipher af_alg= cmac nls_utf8 cifs cifs_arc4 nls_ucs2_utils cifs_md4 dns_resolver netfs nf= t_masq nft_chain_nat nf_nat > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables nfnet= link nvme_fabrics essiv authenc crypto_null snd_seq_dummy snd_hrtimer snd_s= eq snd_seq_device qrtr zstd > zram bnep binfmt_misc nls_ascii nls_cp437 vfat fat mt7921e snd_hda_codec_= realtek mt7921_common snd_hda_codec_generic mt792x_lib snd_hda_scodec_compo= nent mt76_connac_lib > snd_hda_codec_hdmi mt76 btusb snd_hda_intel amd_atl btrtl intel_rapl_msr = snd_intel_dspcfg intel_rapl_common snd_intel_sdw_acpi btintel amd64_edac ed= ac_mce_amd mac80211 btbcm > snd_hda_codec asus_nb_wmi eeepc_wmi btmtk asus_wmi kvm_amd snd_hda_core s= parse_keymap bluetooth libarc4 snd_hwdep platform_profile kvm cfg80211 snd_= pcm battery wmi_bmof rapl > snd_timer sp5100_tco ccp pcspkr watchdog snd k10temp rfkill soundcore joy= dev sg evdev nct6775 nct6775_core hwmon_vid msr parport_pc ppdev lp parport= loop efi_pstore > [ 48.137103] configfs ip_tables x_tables autofs4 ext4 mbcache jbd2 btr= fs dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy a= sync_pq async_xor async_tx > xor raid6_pq libcrc32c crc32c_generic raid1 raid0 md_mod hid_generic usbh= id hid amdgpu amdxcp drm_exec gpu_sched drm_buddy i2c_algo_bit drm_suballoc= _helper drm_display_helper > sd_mod cec crct10dif_pclmul rc_core crc32_pclmul xhci_pci drm_ttm_helper = crc32c_intel ttm ahci xhci_hcd drm_kms_helper libahci r8169 ghash_clmulni_i= ntel libata sha512_ssse3 > realtek nvme mdio_devres sha256_ssse3 drm usbcore scsi_mod sha1_ssse3 i2c= _piix4 libphy video nvme_core i2c_smbus usb_common scsi_common crc16 wmi gp= io_amdpt gpio_generic button > aesni_intel gf128mul crypto_simd cryptd > [ 48.137148] CPU: 30 UID: 0 PID: 421 Comm: kworker/30:1 Not tainted 6.1= 1.0-rc5 #3 > [ 48.137150] Hardware name: ASUS System XXXXXXXXXX > [ 48.137151] Workqueue: cifsiod smb2_readv_worker [cifs] > [ 48.137176] RIP: 0010:netfs_subreq_terminated+0x276/0x2d0 [netfs] > [ 48.137182] Code: 66 ff ff ff 0f 1f 44 00 00 e9 5c ff ff ff 48 89 f1 0= f b7 93 86 00 00 00 8b b5 ac 01 00 00 48 c7 c7 78 81 7a c2 e8 ba 68 2f da <= 0f> 0b 48 8b 43 70 31 d2 4c > 8d ab 98 00 00 00 66 89 93 84 00 00 00 > [ 48.137183] RSP: 0018:ffffad8942637e58 EFLAGS: 00010282 > [ 48.137185] RAX: 0000000000000000 RBX: ffff9d09639a7200 RCX: 000000000= 0000027 > [ 48.137186] RDX: ffff9d107e721788 RSI: 0000000000000001 RDI: ffff9d107= e721780 > [ 48.137187] RBP: ffff9d094bf38a00 R08: 0000000000000000 R09: 000000000= 0000003 > [ 48.137187] R10: ffffad8942637ce8 R11: ffff9d109de3cfe8 R12: 000000000= 0000001 > [ 48.137188] R13: ffff9d095fcf6000 R14: ffff9d09639a7208 R15: 000000000= 0000000 > [ 48.137189] FS: 0000000000000000(0000) GS:ffff9d107e700000(0000) knlG= S:0000000000000000 > [ 48.137190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 48.137191] CR2: 000055d1a3402760 CR3: 00000006b0222000 CR4: 000000000= 0750ef0 > [ 48.137192] PKRU: 55555554 > [ 48.137193] Call Trace: > [ 48.137194] > [ 48.137197] ? __warn+0x80/0x120 > [ 48.137201] ? netfs_subreq_terminated+0x276/0x2d0 [netfs] > [ 48.137207] ? report_bug+0x164/0x190 > [ 48.137210] ? prb_read_valid+0x1b/0x30 > [ 48.137213] ? handle_bug+0x41/0x70 > [ 48.137215] ? exc_invalid_op+0x17/0x70 > [ 48.137216] ? asm_exc_invalid_op+0x1a/0x20 > [ 48.137220] ? netfs_subreq_terminated+0x276/0x2d0 [netfs] > [ 48.137225] ? netfs_subreq_terminated+0x276/0x2d0 [netfs] > [ 48.137230] process_one_work+0x179/0x390 > [ 48.137233] worker_thread+0x249/0x350 > [ 48.137235] ? __pfx_worker_thread+0x10/0x10 > [ 48.137237] kthread+0xcf/0x100 > [ 48.137240] ? __pfx_kthread+0x10/0x10 > [ 48.137242] ret_from_fork+0x31/0x50 > [ 48.137244] ? __pfx_kthread+0x10/0x10 > [ 48.137246] ret_from_fork_asm+0x1a/0x30 > [ 48.137250] > [ 48.137251] ---[ end trace 0000000000000000 ]--- > --=20 Thanks, Steve