From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 778E2C36002 for ; Mon, 24 Mar 2025 18:24:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A7C9280003; Mon, 24 Mar 2025 14:24:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0592D280001; Mon, 24 Mar 2025 14:24:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8951280003; Mon, 24 Mar 2025 14:24:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C758E280001 for ; Mon, 24 Mar 2025 14:24:52 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2688357E24 for ; Mon, 24 Mar 2025 18:24:53 +0000 (UTC) X-FDA: 83257270866.25.7128E9E Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) by imf06.hostedemail.com (Postfix) with ESMTP id 28C00180021 for ; Mon, 24 Mar 2025 18:24:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=XZNvTSzy; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf06.hostedemail.com: domain of yosry.ahmed@linux.dev designates 95.215.58.181 as permitted sender) smtp.mailfrom=yosry.ahmed@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742840691; a=rsa-sha256; cv=none; b=tsQVZmMBylR3xPNAPDOzXLV64FRri6y8Lmd4a20+mJ6AY7AKqcmIiGF37RDZSCu4wMqq8a xYCEW4gU9L/90Sd0/T3YAtO7uAwCDmb57MJpmBvR3yesuVj/FN/8wgPUI4QQiUzXE4AYB1 dcnIstGxhxtOFkr5cBHwKoJUT/kWiIA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=XZNvTSzy; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf06.hostedemail.com: domain of yosry.ahmed@linux.dev designates 95.215.58.181 as permitted sender) smtp.mailfrom=yosry.ahmed@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742840691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l7VKYzfTXlG5xci0GzpcktTSnkwrSI94S7s9++n5OOI=; b=xsIWzWk67+ProIdcuFJWpBOFTtaGzQs9QvVKLsOpWSEHFDXOCVvIw4DNlDTOvTk9+0IWz/ iLfw4dlyyuCcuo+XeneHojObhrYZqp3VcJEUJ1VGTBaySnLPe4T1s++TNd7KL6ddGAyWev peEmgMU3wV4/bc3KVxoQSXFIycNHAkw= Date: Mon, 24 Mar 2025 18:24:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1742840689; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l7VKYzfTXlG5xci0GzpcktTSnkwrSI94S7s9++n5OOI=; b=XZNvTSzyOsrL8edQJlNO8H7jocIFUnvod3AQ70/aMOD7KGokoyP+7yElsToUoJyLk4Ze8+ Xyi5jYHcBkIiS6tgbHJYDkMrn4umY/IPj0bbJz9pUuVCVynsasGz2utp6I8HcNuAjyFoXB huLfDk/2FaHGDMDwAUznVhIdJNr7QQk= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Yosry Ahmed To: Nhat Pham Cc: Chris Murphy , Linux List , Andrew Morton Subject: Re: 6.14.0-rc6 lockdep warning kswapd Message-ID: References: <3f156789-759e-4195-8881-c5c74ced85d4@app.fastmail.com> <1977c271-daba-464e-a6f7-13e6778b0882@app.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 28C00180021 X-Stat-Signature: gehfm3ebff4tbmg9u1grsph4myzn16ay X-HE-Tag: 1742840690-871876 X-HE-Meta: U2FsdGVkX1+QCauveFgkSmrqIIO/EipTsk7ga/9/cnR2+xU6x4CQizRYKdBfLtMeTDpGisuNTU31CLIYDRuiUHdbmIar2MgGXkjepP+32zn8dNL9X05Fds942zpaw3LF2E7kXsh7+aYGvKUZbAdaab4tF+I5CQRGQJqEOd3rwlRUn++WnK+Fo0bIjoAqtIN7yhO2TmsKUAFTe1AlyGDA8z0rGVUqrBF4sKxaqJb57I+T0fUsijfovC8NxfsoZdSouPGIZ1sRmKvQTPnDm9jwHD3wUkrmf5PbM45coej3448sax57EO91q8CFrVEf9Q0KTClDFvSPUrrUG3hrAuZiMm4OWefKVAqUBlXFq7yooSFSgxOEnm7Jm52zbDt/o7URmbJv3QEE6SZnlX4W6EGyWusJNlwZ4ONH/NqCTjox8Y6B1YkyHzM8k6GAaMZFY5mXW/YERjeHtN72rVhWbLof2YCEBxyQxYr+rszZblGKMrOsisxEyzeEnZsRQ5pcEDxrpL9AAcoRq+FUW+O4fEmJiiFrlXSHaHY6qctpo0FRFEAcsgGuhThYguSGYAwPPyoEbrS3jyABd0AIxi5V3bOdTIWUXcfTr5hjbbvLDm47oBTj9GDJXuRWPrFnWYGjrKX8dTxYciXLbOPdiVP+vGYjvFkK0X2HpX7WTB578n18JJ7G4AVUFkdxUo7EfAjMS1ZXopYrPGPZdfT/xFtQ/9ffRn2NLAbWYMbC4IS6QSnAwka0ZIvP+5JJEDwBSzIWjIVu23cXNgfHd/z+4fI95vLRNPDbhRh8juyOIZMAQPMMSNkOszo3M4t0wBRAZzRFD/LY+1e3UW1ebFV8+AHy9X+iPGXiw4TXbflxgWXltebodhndHmkHJRcae2FQPakigkulJ6PS4qB1hHCHt2OWhlnS8voVItNVkQg/7C7oL2ANjoKaojsvR0+PFU9hBT/t/lePz+Jd3yYrMQRI2qByvR0 BaHrPgN3 kkoxI2mgq7QEJfNQaUu5iQIf7VaRtcILnFYW+IiA61vEiWLdP3UzI7EnYq7b6fzKS9OuRAHGqx58eo3v65euFoVwzTYsGSVDoQgnIwYb4iMFC4hWwI9aFiTwrc7IjySGDkKzh3L+mm9HXwYfjIj4EsPXU1K19XhU771OYzqUs9xYCJWehC/4ERHlDKmed9lOpcGl4BmzRZM7/a0Ar82xRnaadNO+kou5FllotsDd9rZHPiiWBmvXmG9pjcpLHFPIsnWsSSENzjQuD5dVRbXXvzSCtzjtLqsScBiMl+cJP7Wjd3gslZ34qB6czXOdUPzySIebCov99dp2rL/L9UNOFovEjC/+bFjeNX/R4EeB09QEYJZFuRFp/gQx3Ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001554, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Mar 24, 2025 at 01:06:32PM -0400, Nhat Pham wrote: > On Fri, Mar 21, 2025 at 7:25 PM Chris Murphy wrote: > > > > > > > > On Thu, Mar 20, 2025, at 10:28 PM, Nhat Pham wrote: > > > > > I don't think this is in rc6 yet. Can you apply the patch and test again? > > > > At the moment I don't have a way to test the patch. > > > > Also I failed to mention this is swapfile on btrfs on dm-crypt. > > > > -- > > Chris Murphy > > I checked-out mm-unstable and reverted Yosry's fix. With my usual > "firehose" stress test script, I was able to trigger: > > [ 99.324986] > [ 99.325146] ====================================================== > [ 99.325569] WARNING: possible circular locking dependency detected > [ 99.325983] 6.14.0-rc6-ge211b31825f4 #5 Not tainted > [ 99.326383] ------------------------------------------------------ > [ 99.326803] kswapd0/49 is trying to acquire lock: > [ 99.327122] ffffd8893fc1e0e0 (&per_cpu_ptr(pool->acomp_ctx, > cpu)->mutex){+.+.}-{4:4}, at: zswap_store+0x3ec/0xf40 > [ 99.327812] > [ 99.327812] but task is already holding lock: > [ 99.328202] ffffffffa654fa80 (fs_reclaim){+.+.}-{0:0}, at: > balance_pgdat+0x3cd/0x760 > [ 99.328726] > [ 99.328726] which lock already depends on the new lock. > [ 99.328726] > [ 99.329266] > [ 99.329266] the existing dependency chain (in reverse order) is: > [ 99.329851] > [ 99.329851] -> #2 (fs_reclaim){+.+.}-{0:0}: > [ 99.330415] fs_reclaim_acquire+0x9d/0xd0 > [ 99.330908] __kmalloc_cache_node_noprof+0x57/0x3e0 > [ 99.331497] __get_vm_area_node+0x85/0x140 > [ 99.331998] __vmalloc_node_range_noprof+0x13e/0x800 > [ 99.332573] __vmalloc_node_noprof+0x4e/0x70 > [ 99.333062] crypto_scomp_init_tfm+0xc2/0xe0 > [ 99.333584] crypto_create_tfm_node+0x47/0xe0 > [ 99.334107] crypto_init_scomp_ops_async+0x38/0xa0 > [ 99.334649] crypto_create_tfm_node+0x47/0xe0 > [ 99.334963] crypto_alloc_tfm_node+0x5e/0xe0 > [ 99.335279] zswap_cpu_comp_prepare+0x78/0x180 > [ 99.335623] cpuhp_invoke_callback+0xbd/0x6b0 > [ 99.335957] cpuhp_issue_call+0xa4/0x250 > [ 99.336279] __cpuhp_state_add_instance_cpuslocked+0x9d/0x160 > [ 99.336906] __cpuhp_state_add_instance+0x75/0x190 > [ 99.337448] zswap_pool_create+0x17e/0x2b0 > [ 99.337783] zswap_setup+0x27d/0x5a0 > [ 99.338059] do_one_initcall+0x5d/0x390 > [ 99.338355] kernel_init_freeable+0x22f/0x410 > [ 99.338689] kernel_init+0x1a/0x1d0 > [ 99.338966] ret_from_fork+0x34/0x50 > [ 99.339249] ret_from_fork_asm+0x1a/0x30 > [ 99.339558] > [ 99.339558] -> #1 (scomp_lock){+.+.}-{4:4}: > [ 99.340155] __mutex_lock+0x95/0xda0 > [ 99.340431] crypto_exit_scomp_ops_async+0x23/0x50 > [ 99.340795] crypto_destroy_tfm+0x61/0xc0 > [ 99.341104] zswap_cpu_comp_dead+0x6c/0x90 > [ 99.341415] cpuhp_invoke_callback+0xbd/0x6b0 > [ 99.341744] cpuhp_issue_call+0xa4/0x250 > [ 99.342043] __cpuhp_state_remove_instance+0x100/0x250 > [ 99.342423] __zswap_pool_release+0x46/0xa0 > [ 99.342747] process_one_work+0x210/0x5e0 > [ 99.343054] worker_thread+0x183/0x320 > [ 99.343346] kthread+0xef/0x230 > [ 99.343605] ret_from_fork+0x34/0x50 > [ 99.343881] ret_from_fork_asm+0x1a/0x30 > [ 99.344179] > [ 99.344179] -> #0 (&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex){+.+.}-{4:4}: > [ 99.344732] __lock_acquire+0x15e3/0x23e0 > [ 99.345040] lock_acquire+0xc9/0x2d0 > [ 99.345315] __mutex_lock+0x95/0xda0 > [ 99.345597] zswap_store+0x3ec/0xf40 > [ 99.345875] swap_writepage+0x114/0x530 > [ 99.346183] pageout+0xf9/0x2d0 > [ 99.346435] shrink_folio_list+0x7f8/0xdf0 > [ 99.346750] shrink_lruvec+0x7d2/0xda0 > [ 99.347040] shrink_node+0x30a/0x840 > [ 99.347315] balance_pgdat+0x349/0x760 > [ 99.347610] kswapd+0x1db/0x3c0 > [ 99.347858] kthread+0xef/0x230 > [ 99.348107] ret_from_fork+0x34/0x50 > [ 99.348381] ret_from_fork_asm+0x1a/0x30 > [ 99.348687] > [ 99.348687] other info that might help us debug this: > [ 99.348687] > [ 99.349228] Chain exists of: > [ 99.349228] &per_cpu_ptr(pool->acomp_ctx, cpu)->mutex --> > scomp_lock --> fs_reclaim > [ 99.349228] > [ 99.350214] Possible unsafe locking scenario: > [ 99.350214] > [ 99.350612] CPU0 CPU1 > [ 99.350916] ---- ---- > [ 99.351219] lock(fs_reclaim); > [ 99.351437] lock(scomp_lock); > [ 99.351820] lock(fs_reclaim); > [ 99.352196] lock(&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex); > [ 99.352588] > [ 99.352588] *** DEADLOCK *** > [ 99.352588] > [ 99.353021] 1 lock held by kswapd0/49: > [ 99.353468] #0: ffffffffa654fa80 (fs_reclaim){+.+.}-{0:0}, at: > balance_pgdat+0x3cd/0x760 > [ 99.354017] > [ 99.354017] stack backtrace: > [ 99.354316] CPU: 0 UID: 0 PID: 49 Comm: kswapd0 Not tainted > 6.14.0-rc6-ge211b31825f4 #5 > [ 99.354318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 > [ 99.354320] Call Trace: > [ 99.354321] > [ 99.354323] dump_stack_lvl+0x82/0xd0 > [ 99.354326] print_circular_bug+0x26a/0x330 > [ 99.354329] check_noncircular+0x14e/0x170 > [ 99.354331] ? lock_acquire+0xc9/0x2d0 > [ 99.354334] __lock_acquire+0x15e3/0x23e0 > [ 99.354337] lock_acquire+0xc9/0x2d0 > [ 99.354339] ? zswap_store+0x3ec/0xf40 > [ 99.354342] ? find_held_lock+0x2b/0x80 > [ 99.354346] __mutex_lock+0x95/0xda0 > [ 99.354348] ? zswap_store+0x3ec/0xf40 > [ 99.354350] ? __create_object+0x5e/0x90 > [ 99.354353] ? zswap_store+0x3ec/0xf40 > [ 99.354355] ? kmem_cache_alloc_node_noprof+0x2de/0x3e0 > [ 99.354358] ? zswap_store+0x3ec/0xf40 > [ 99.354360] zswap_store+0x3ec/0xf40 > [ 99.354364] ? _raw_spin_unlock+0x23/0x30 > [ 99.354368] ? folio_free_swap+0x98/0x190 > [ 99.354371] swap_writepage+0x114/0x530 > [ 99.354373] pageout+0xf9/0x2d0 > [ 99.354381] shrink_folio_list+0x7f8/0xdf0 > [ 99.354385] ? __mod_memcg_lruvec_state+0x20f/0x280 > [ 99.354389] ? isolate_lru_folios+0x465/0x610 > [ 99.354392] ? find_held_lock+0x2b/0x80 > [ 99.354395] ? mark_held_locks+0x49/0x80 > [ 99.354398] shrink_lruvec+0x7d2/0xda0 > [ 99.354402] ? find_held_lock+0x2b/0x80 > [ 99.354407] ? shrink_node+0x30a/0x840 > [ 99.354409] shrink_node+0x30a/0x840 > [ 99.354413] balance_pgdat+0x349/0x760 > [ 99.354418] kswapd+0x1db/0x3c0 > [ 99.354421] ? __pfx_autoremove_wake_function+0x10/0x10 > [ 99.354424] ? __pfx_kswapd+0x10/0x10 > [ 99.354426] kthread+0xef/0x230 > [ 99.354429] ? __pfx_kthread+0x10/0x10 > [ 99.354432] ret_from_fork+0x34/0x50 > [ 99.354435] ? __pfx_kthread+0x10/0x10 > [ 99.354437] ret_from_fork_asm+0x1a/0x30 > [ 99.354441] > > With Yosry's fix, it goes away. > > So I guess: > > Tested-by: Nhat Pham Thanks for confirming this! Although I think responding with your Tested-by on the original patch is probably easier for Andrew to track down.