From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8948CC6FA90 for ; Thu, 22 Sep 2022 06:53:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA44D6B0071; Thu, 22 Sep 2022 02:53:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D53DB6B0072; Thu, 22 Sep 2022 02:53:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1B5C940007; Thu, 22 Sep 2022 02:53:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B02F36B0071 for ; Thu, 22 Sep 2022 02:53:12 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7C5B414090D for ; Thu, 22 Sep 2022 06:53:12 +0000 (UTC) X-FDA: 79938804624.27.4A53543 Received: from vulcan.natalenko.name (vulcan.natalenko.name [104.207.131.136]) by imf27.hostedemail.com (Postfix) with ESMTP id C76BA40003 for ; Thu, 22 Sep 2022 06:53:11 +0000 (UTC) Received: from mail.natalenko.name (vulcan.natalenko.name [IPv6:2001:19f0:6c00:8846:5400:ff:fe0c:dfa0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by vulcan.natalenko.name (Postfix) with ESMTPSA id 7DD05105AE02; Thu, 22 Sep 2022 08:53:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=natalenko.name; s=dkim-20170712; t=1663829589; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=iHPCHBmgMOO01eOltUkBnrrO+2vU5QB2hFCzwA+i6iY=; b=QZ7toBLbm1GBRjqzrDX3YJYGbVDZBrjX+Si+TwtsVedCx0mvonmPnqDmsXEec/1RRTTYV3 +fXFHjly6sTcVQw/o2Qh+G2IJETnzHdmMGHoJVDN3MRBXst1NPLtmde7w3rpabKiKCTSVV nZR127cFl0drOYDCeVb6C5RTJ6mUxJs= MIME-Version: 1.0 Date: Thu, 22 Sep 2022 08:53:09 +0200 From: Oleksandr Natalenko To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Seth Jennings , Dan Streetman , Vitaly Wool , Andrew Morton , Miaohe Lin Subject: Panic/lockup in z3fold_zpool_free Message-ID: <6f834faddf927b6fa5a8edd8adfde63c@natalenko.name> X-Sender: oleksandr@natalenko.name Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663829592; a=rsa-sha256; cv=none; b=ndx8LrPHnzjAleg5WjkpkxNWeexUx8sirJXd2+GredATBOiaobT0K3mtw3gkxQBtO1SppW UsWGRWaRNOBJW+T6s70pyAiH5x5ixHUTx6g97D7eRMdLbzKO7PFSLJvIUxmqOYCmWXc9OF Yb8wlN+v2zOCygc2UtPkMQe+gt2YWJc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=natalenko.name header.s=dkim-20170712 header.b=QZ7toBLb; spf=pass (imf27.hostedemail.com: domain of oleksandr@natalenko.name designates 104.207.131.136 as permitted sender) smtp.mailfrom=oleksandr@natalenko.name; dmarc=pass (policy=reject) header.from=natalenko.name ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663829592; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=iHPCHBmgMOO01eOltUkBnrrO+2vU5QB2hFCzwA+i6iY=; b=IuvXWJ3lJbQW8p7Q/DCv7tDqa4iRFvcVPkOq/Szlyft5eDLd20RjVfgwc4hSnG9acD/aeT Vad6S8Z49HnEKId6Uls/sAlb7XDXRLXJ4GYH5IMR0nNLfN/TPib5lXVULahLsyaBeQMPvH wEFnLTOXEhSzKAUOEmbtooofInhjrqU= X-Rspamd-Queue-Id: C76BA40003 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=natalenko.name header.s=dkim-20170712 header.b=QZ7toBLb; spf=pass (imf27.hostedemail.com: domain of oleksandr@natalenko.name designates 104.207.131.136 as permitted sender) smtp.mailfrom=oleksandr@natalenko.name; dmarc=pass (policy=reject) header.from=natalenko.name X-Rspamd-Server: rspam11 X-Stat-Signature: i8th9iic7uprefbo87jdfmixiob6h6xa X-HE-Tag: 1663829591-941709 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello. Since 5.19 series, zswap went unstable for me under memory pressure, and occasionally I get the following: ``` watchdog: BUG: soft lockup - CPU#0 stuck for 10195s! [mariadbd:478] Modules linked in: netconsole joydev mousedev intel_agp psmouse pcspkr intel_gtt cfg80211 cirrus i2c_piix4 tun rfkill mac_hid nft_ct tcp_bbr2 nft_chain_nat nf_tables nfnetlink nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 fuse qemu_fw_cfg ip_tables x_tables xfs libcrc32c crc32c_generic dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net aesni_intel serio_raw net_failover ata_generic virtio_balloon failover pata_acpi crypto_simd virtio_blk atkbd libps2 vivaldi_fmap virtio_pci cryptd virtio_pci_legacy_dev ata_piix virtio_pci_modern_dev i8042 floppy serio usbhid Unloaded tainted modules: intel_cstate():1 intel_uncore():1 pcc_cpufreq():1 acpi_cpufreq():1 CPU: 0 PID: 478 Comm: mariadbd Tainted: G L 5.19.0-pf5 #1 12baccda8e49539e158b9dd97cbda6c7317d73af Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014 RIP: 0010:z3fold_zpool_free+0x4c/0x5e0 Code: 7c 24 08 48 89 04 24 0f 85 e0 00 00 00 48 89 f5 41 bd 00 00 00 80 48 83 e5 c0 48 83 c5 28 eb 0a 48 89 df e8 b6 8d 9f 00 f3 90 <48> 89 ef e8 bc 8b 9f 00 4d 8b 34 24 49 81 e6 00 f0 ff ff 49 8d 5e RSP: 0000:ffffbeadc0e87b68 EFLAGS: 00000202 RAX: 0000000000000030 RBX: ffff99ac73d2c010 RCX: ffff99ac4e4ba380 RDX: 0000665340000000 RSI: ffffe3b540000000 RDI: ffff99ac73d2c010 RBP: ffff99ac55ef3a68 R08: ffff99ac422f0bf0 R09: 000000000000c60b R10: ffffffffffffffc0 R11: 0000000000000000 R12: ffff99ac55ef3a50 R13: 0000000080000000 R14: ffff99ac73d2c000 R15: ffff99acf3d2c000 FS: 00007f587fcd66c0(0000) GS:ffff99ac7ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f587ce8bec8 CR3: 0000000005b48006 CR4: 00000000000206f0 Call Trace: zswap_free_entry+0xb5/0x110 zswap_frontswap_invalidate_page+0x72/0xa0 __frontswap_invalidate_page+0x3a/0x60 swap_range_free+0xb5/0xd0 swapcache_free_entries+0x16e/0x2e0 free_swap_slot+0xb4/0xc0 put_swap_page+0x259/0x420 delete_from_swap_cache+0x63/0xb0 try_to_free_swap+0x1b5/0x2a0 do_swap_page+0x24c/0xb80 __handle_mm_fault+0xa59/0xf70 handle_mm_fault+0x100/0x2f0 do_user_addr_fault+0x1c7/0x6a0 exc_page_fault+0x74/0x170 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x556e96280428 Code: a0 03 00 00 67 e8 28 64 ff ff 48 8b 83 b0 00 00 00 48 8b 0d da 18 72 00 48 8b 10 66 48 0f 6e c1 48 85 d2 74 27 0f 1f 44 00 00 <48> c7 82 98 00 00 00 00 00 00 00 48 8b 10 48 83 c0 08 f2 0f 11 82 RSP: 002b:00007f587fcd3980 EFLAGS: 00010206 RAX: 00007f587d028468 RBX: 00007f587cb1a818 RCX: 3ff0000000000000 RDX: 00007f587ce8be30 RSI: 0000000000000000 RDI: 00007f587cedd030 RBP: 00007f587fcd39c0 R08: 0000000000000016 R09: 0000000000000000 R10: 0000000000000008 R11: 0000556e970961a0 R12: 00007f587d1f17b8 R13: 00007f5883595598 R14: 00007f587d1f17a8 R15: 00007f587cb1a928 ``` This happens on the latest v5.19.10 kernel as well. Sometimes it's not a soft lockup but GPF, although the stack trace is the same. So, to me it looks like a memory corruption, UAF, double free or something like that. Have you got any idea regarding what's going on? Thanks. -- Oleksandr Natalenko (post-factum)