From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8464AC36002 for ; Mon, 24 Mar 2025 17:08:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54871280004; Mon, 24 Mar 2025 13:08:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CC85280001; Mon, 24 Mar 2025 13:08:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 347EB280004; Mon, 24 Mar 2025 13:08:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 13982280001 for ; Mon, 24 Mar 2025 13:08:55 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 709381C9278 for ; Mon, 24 Mar 2025 17:08:55 +0000 (UTC) X-FDA: 83257079430.25.AB5E7AC Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf17.hostedemail.com (Postfix) with ESMTP id 21DEB40011 for ; Mon, 24 Mar 2025 17:06:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=m6CCEQee; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742836006; a=rsa-sha256; cv=none; b=2DbVe6jFxPHzmbiEC5Igk8AMN909meLOnuGETI+qoz/QLhmC6NiZlJeiwhpCCCx0VKy39P 325EC4ggiawt2ghnVUUGCBXQQH7kP1jMhhzHrazeN5iO3FAOzf5RGJqOgq6unI1VBd3xCd 2lEFYAKc4Y2G7QZyXobj4hHy1IAvqjk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=m6CCEQee; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742836006; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jGViCVbUvvtpJhhhZ4DYIUD2pF50STbhBBYMDi/CL3Y=; b=Ln5ljDB8cAliAY4414i5dJ90aT75jDDNUzRZsvkzU5/5iEFrcRJPyWgfulhbFHtLbnDR51 PwqSzA5TmFiRkPwzCRzF9Xd5MKTbGLjJ2Wx4N2QG+QydKAV4UcVc2DYZ5UdkdO28GCxtC8 ubV8GNZDnEHlkaYq2BxqVaRBiIOu7og= Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-476ae781d21so41718921cf.3 for ; Mon, 24 Mar 2025 10:06:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1742836004; x=1743440804; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=jGViCVbUvvtpJhhhZ4DYIUD2pF50STbhBBYMDi/CL3Y=; b=m6CCEQeeP43s94bqtfzopKuDza6OjEjfydiaVm1kgbw+3zckUok5UJh5HMSYEc/Bg1 yIlB/hZpnq4lckweyfHfOd/jQW57Z3v1jnNTQ/qiSHbRlOC96djJ6RRTGDJdG+F4yKk9 RKO7RQM/d6VOrKMKgaWkvtn+cwn+cBu7Sd0eXZ1gbJgjT0bXOxgZ7Baaf1f7NKXHQYc7 VHZRbZfVJUXJLTxSThTI++Tf059V2an3Z9huU+po2UNQph0kvqzaNnEy0RcxCpBQAcwT Q0iMu5Iqp2fVADLNNlzYfNxd1YtxRZ8VVgxsVu30Jhdi0WvzAomd++J79kIoRX+T/QHL wVUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742836004; x=1743440804; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jGViCVbUvvtpJhhhZ4DYIUD2pF50STbhBBYMDi/CL3Y=; b=ukJObkvBATph3Iq7LvWDOYzb13ikWLYnpGeY09vHKbZcVl1oB/X7bvN6M9JQWTRLNH XE3mOyWv1ENSdoHY/8B8jRmS1FfXASNtBtW4Zbp2ZHDArLcFNduwKNdH2ZE5Mc4FCgq5 sDIpJJBJseZuWVuJyXT8HCHreYTx76G6q7EzGKggjw+0vD5rXEmmoP0sfa34LgoL9vuZ 7MxfHWnGAsv6SaDK1wmNLej50y8jYCQTpHN4jnF5zLd6/W6+P5qxVTYcyx1MItewZ7RL lTE6sK3szx5bf8gFNC1jvjZEoCBbljHjP19kbHFlU0dFDrVt3tgZvKRZp2ZTxTPjwZgf jefQ== X-Gm-Message-State: AOJu0YzSkuCKFfHTbA9iv7rBED4drpuIQ0uay2Cx1qFZNUU0ewfKECBC 2aqD2huSd8J6CN8J65qVjG02B5+9B6havwKzn3Hmyqvm6lTweZj1FIRl2fRM50fIa/CYoPVm4II TZHMK9VUfzgh9Bezypf5hNNegSOA= X-Gm-Gg: ASbGnctEG1CSOSRDIVEtY/EjgjeYsPlXnFnSfKTAN0fMadsmV5maxqvJAC92KSgGbwN grbadqmdMj6PSY0Npcul4G5/y9nqhVaop2KkFtJjm9PPDkH8Ig+8ciZtLSmh/CPjTtSqfrB08mE gU5kqhW9yc2P16v7Vlmdc97awBAQ== X-Google-Smtp-Source: AGHT+IFhNAuJAgtq8tpyNXfars+8YnjVYyKGWdFmMBxG5l4bPrS0d0wSL38xBU8KR2RlakY3JjCzu/p46+7A8TVoj2Y= X-Received: by 2002:ad4:5c42:0:b0:6e8:f91a:c5a with SMTP id 6a1803df08f44-6eb3f2d5384mr245865736d6.22.1742836004008; Mon, 24 Mar 2025 10:06:44 -0700 (PDT) MIME-Version: 1.0 References: <3f156789-759e-4195-8881-c5c74ced85d4@app.fastmail.com> <1977c271-daba-464e-a6f7-13e6778b0882@app.fastmail.com> In-Reply-To: <1977c271-daba-464e-a6f7-13e6778b0882@app.fastmail.com> From: Nhat Pham Date: Mon, 24 Mar 2025 13:06:32 -0400 X-Gm-Features: AQ5f1JqBhMcsLXMBWcRUY4_14JqpqKaPowQMv9S99ji662Y0bP02-qIAod56FLc Message-ID: Subject: Re: 6.14.0-rc6 lockdep warning kswapd To: Chris Murphy Cc: Linux List , Andrew Morton , Yosry Ahmed , Yosry Ahmed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 21DEB40011 X-Stat-Signature: q6afez3ex4pr81ihfaph3payc6h4hqfq X-HE-Tag: 1742836005-792201 X-HE-Meta: U2FsdGVkX1/59z7xSKpDjM8sR6stddYB208KMKXjaZXpfs3g2IjVe5t0Wdr5IZhk9fDTas7gd4EioZwTH3+nzlIdfV3GJQku2SMdi9K3wrwEWezwBGtXJmTQyQdxNnvNtPdzNT0NB3w7xQph881YEht/b13CIej5uTT2AHudRQIEC9Wlp/wg5YN8qH4e4NIaftyptY1uCZmNbLMfMCgzxT8GR9iJunxM5bv2CnBj50IEyAe8cHgKiZ8ucnQ3qG8gsXwLkAcq8OzR0cu+JLkfzlMUruiabws8Ljh+F4OJ8fxCptxhjcXgmelgXq28RMrHFnBqgSA9JuDwgra0ssAzftYLsoJW24au5SqKepd6FCOCJAOpuAqqkKXfiDAhBblq2dB2SMfbEYIJUSfqblK+zP9sHwRd9yMc78bZUDIT+jNJOCbMpCzeXlBvInOI+sr8J6bPgahID7pltjkkn9q/fXH+l0p7NJ/dXgHjFGc6kWySygObWxwY1WN4sAK/hMhyd0YszeFzhjNNx8FtgSlQ1KQGoa9NRQ2W5GfJZS3tiiU902jKwfdLST3WAiLvtFga1YXpsC5b9zba59l0ZYuFb2iIw8C+aMcC3dZ6j3UYueO1IV2idZwxMr5omNcXZwQvJPVb8lvqA9/hdcBuPRSAZE8a/0Ra/mSt2S+ZrJyZJWutRXozOTBi4KdyemEBJNDUNrJ8Ihz3uGTA7fm460inrTEu1lGfbYQWG1z5Q1ZiqmUPsATumcd+K+5P+LMMw2K6+RTED2Bfe9bRaAYAp66V07xIlpJfS7fGDzQDlJx4MB35PsAx2XpGpCQ5JpEA6l5AdbuUHmV1bUZNVvJanaJD+Hpkq59Akj9fOp/IKPIWsfGfRN0uykLy3HrNxfnylWE6NKD3OQGPEo1DmsQpCGG83LGK+czoNnDQSLMXwLLXSf3nXRkfkMUAoxwnTtExiS5ZR4wyj4NxOEsqvqrNHjv Kf4fShQo j+AF84J0nkyThbVYM08Qw8IW2q3eqUqVq1R+fLZeCRmFKb0KIDqEsU2WRGNtrIf5d9i54Ow5IFAU2rBxThDhjOvvXlnakvQgf0+p75zuHgicTPGVwu5/CD/+5Ce5FRFWouymEIXmzO9riz1doQ1HeZgrkvymR/wCb7CeDfnRHSwuZiud5dHBXYPx2uM60aQ5KM7TP6L1EulXYAIAY9S4ihC4S7NDA13cPQQUtjzwR57wK/GgxBdksYe9os5u7pHyJR9cnRZjzx9ZUq9bIKAKlAQCBVfqwUITjE+Vp5RPQQyauWeFySM5mdUDelydebNw5TGR7/YssobEjmEHf0aoWxqPV1fHsoI14pWU64Y27zcD7IuJyh5lORcdRpr/nWB42iiOr0FTqzszwvRHmKImGPPvnDGL+ftWiCHTrJTy7C/vo8rGV+X3fTxpAMVgLIGcUIxcebrHLtmIe0Dc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.295302, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Mar 21, 2025 at 7:25=E2=80=AFPM Chris Murphy wrote: > > > > On Thu, Mar 20, 2025, at 10:28 PM, Nhat Pham wrote: > > > I don't think this is in rc6 yet. Can you apply the patch and test agai= n? > > At the moment I don't have a way to test the patch. > > Also I failed to mention this is swapfile on btrfs on dm-crypt. > > -- > Chris Murphy I checked-out mm-unstable and reverted Yosry's fix. With my usual "firehose" stress test script, I was able to trigger: [ 99.324986] [ 99.325146] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D [ 99.325569] WARNING: possible circular locking dependency detected [ 99.325983] 6.14.0-rc6-ge211b31825f4 #5 Not tainted [ 99.326383] ------------------------------------------------------ [ 99.326803] kswapd0/49 is trying to acquire lock: [ 99.327122] ffffd8893fc1e0e0 (&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex){+.+.}-{4:4}, at: zswap_store+0x3ec/0xf40 [ 99.327812] [ 99.327812] but task is already holding lock: [ 99.328202] ffffffffa654fa80 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x3cd/0x760 [ 99.328726] [ 99.328726] which lock already depends on the new lock. [ 99.328726] [ 99.329266] [ 99.329266] the existing dependency chain (in reverse order) is: [ 99.329851] [ 99.329851] -> #2 (fs_reclaim){+.+.}-{0:0}: [ 99.330415] fs_reclaim_acquire+0x9d/0xd0 [ 99.330908] __kmalloc_cache_node_noprof+0x57/0x3e0 [ 99.331497] __get_vm_area_node+0x85/0x140 [ 99.331998] __vmalloc_node_range_noprof+0x13e/0x800 [ 99.332573] __vmalloc_node_noprof+0x4e/0x70 [ 99.333062] crypto_scomp_init_tfm+0xc2/0xe0 [ 99.333584] crypto_create_tfm_node+0x47/0xe0 [ 99.334107] crypto_init_scomp_ops_async+0x38/0xa0 [ 99.334649] crypto_create_tfm_node+0x47/0xe0 [ 99.334963] crypto_alloc_tfm_node+0x5e/0xe0 [ 99.335279] zswap_cpu_comp_prepare+0x78/0x180 [ 99.335623] cpuhp_invoke_callback+0xbd/0x6b0 [ 99.335957] cpuhp_issue_call+0xa4/0x250 [ 99.336279] __cpuhp_state_add_instance_cpuslocked+0x9d/0x160 [ 99.336906] __cpuhp_state_add_instance+0x75/0x190 [ 99.337448] zswap_pool_create+0x17e/0x2b0 [ 99.337783] zswap_setup+0x27d/0x5a0 [ 99.338059] do_one_initcall+0x5d/0x390 [ 99.338355] kernel_init_freeable+0x22f/0x410 [ 99.338689] kernel_init+0x1a/0x1d0 [ 99.338966] ret_from_fork+0x34/0x50 [ 99.339249] ret_from_fork_asm+0x1a/0x30 [ 99.339558] [ 99.339558] -> #1 (scomp_lock){+.+.}-{4:4}: [ 99.340155] __mutex_lock+0x95/0xda0 [ 99.340431] crypto_exit_scomp_ops_async+0x23/0x50 [ 99.340795] crypto_destroy_tfm+0x61/0xc0 [ 99.341104] zswap_cpu_comp_dead+0x6c/0x90 [ 99.341415] cpuhp_invoke_callback+0xbd/0x6b0 [ 99.341744] cpuhp_issue_call+0xa4/0x250 [ 99.342043] __cpuhp_state_remove_instance+0x100/0x250 [ 99.342423] __zswap_pool_release+0x46/0xa0 [ 99.342747] process_one_work+0x210/0x5e0 [ 99.343054] worker_thread+0x183/0x320 [ 99.343346] kthread+0xef/0x230 [ 99.343605] ret_from_fork+0x34/0x50 [ 99.343881] ret_from_fork_asm+0x1a/0x30 [ 99.344179] [ 99.344179] -> #0 (&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex){+.+.}-{4:4= }: [ 99.344732] __lock_acquire+0x15e3/0x23e0 [ 99.345040] lock_acquire+0xc9/0x2d0 [ 99.345315] __mutex_lock+0x95/0xda0 [ 99.345597] zswap_store+0x3ec/0xf40 [ 99.345875] swap_writepage+0x114/0x530 [ 99.346183] pageout+0xf9/0x2d0 [ 99.346435] shrink_folio_list+0x7f8/0xdf0 [ 99.346750] shrink_lruvec+0x7d2/0xda0 [ 99.347040] shrink_node+0x30a/0x840 [ 99.347315] balance_pgdat+0x349/0x760 [ 99.347610] kswapd+0x1db/0x3c0 [ 99.347858] kthread+0xef/0x230 [ 99.348107] ret_from_fork+0x34/0x50 [ 99.348381] ret_from_fork_asm+0x1a/0x30 [ 99.348687] [ 99.348687] other info that might help us debug this: [ 99.348687] [ 99.349228] Chain exists of: [ 99.349228] &per_cpu_ptr(pool->acomp_ctx, cpu)->mutex --> scomp_lock --> fs_reclaim [ 99.349228] [ 99.350214] Possible unsafe locking scenario: [ 99.350214] [ 99.350612] CPU0 CPU1 [ 99.350916] ---- ---- [ 99.351219] lock(fs_reclaim); [ 99.351437] lock(scomp_lock); [ 99.351820] lock(fs_reclaim); [ 99.352196] lock(&per_cpu_ptr(pool->acomp_ctx, cpu)->mutex); [ 99.352588] [ 99.352588] *** DEADLOCK *** [ 99.352588] [ 99.353021] 1 lock held by kswapd0/49: [ 99.353468] #0: ffffffffa654fa80 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x3cd/0x760 [ 99.354017] [ 99.354017] stack backtrace: [ 99.354316] CPU: 0 UID: 0 PID: 49 Comm: kswapd0 Not tainted 6.14.0-rc6-ge211b31825f4 #5 [ 99.354318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 99.354320] Call Trace: [ 99.354321] [ 99.354323] dump_stack_lvl+0x82/0xd0 [ 99.354326] print_circular_bug+0x26a/0x330 [ 99.354329] check_noncircular+0x14e/0x170 [ 99.354331] ? lock_acquire+0xc9/0x2d0 [ 99.354334] __lock_acquire+0x15e3/0x23e0 [ 99.354337] lock_acquire+0xc9/0x2d0 [ 99.354339] ? zswap_store+0x3ec/0xf40 [ 99.354342] ? find_held_lock+0x2b/0x80 [ 99.354346] __mutex_lock+0x95/0xda0 [ 99.354348] ? zswap_store+0x3ec/0xf40 [ 99.354350] ? __create_object+0x5e/0x90 [ 99.354353] ? zswap_store+0x3ec/0xf40 [ 99.354355] ? kmem_cache_alloc_node_noprof+0x2de/0x3e0 [ 99.354358] ? zswap_store+0x3ec/0xf40 [ 99.354360] zswap_store+0x3ec/0xf40 [ 99.354364] ? _raw_spin_unlock+0x23/0x30 [ 99.354368] ? folio_free_swap+0x98/0x190 [ 99.354371] swap_writepage+0x114/0x530 [ 99.354373] pageout+0xf9/0x2d0 [ 99.354381] shrink_folio_list+0x7f8/0xdf0 [ 99.354385] ? __mod_memcg_lruvec_state+0x20f/0x280 [ 99.354389] ? isolate_lru_folios+0x465/0x610 [ 99.354392] ? find_held_lock+0x2b/0x80 [ 99.354395] ? mark_held_locks+0x49/0x80 [ 99.354398] shrink_lruvec+0x7d2/0xda0 [ 99.354402] ? find_held_lock+0x2b/0x80 [ 99.354407] ? shrink_node+0x30a/0x840 [ 99.354409] shrink_node+0x30a/0x840 [ 99.354413] balance_pgdat+0x349/0x760 [ 99.354418] kswapd+0x1db/0x3c0 [ 99.354421] ? __pfx_autoremove_wake_function+0x10/0x10 [ 99.354424] ? __pfx_kswapd+0x10/0x10 [ 99.354426] kthread+0xef/0x230 [ 99.354429] ? __pfx_kthread+0x10/0x10 [ 99.354432] ret_from_fork+0x34/0x50 [ 99.354435] ? __pfx_kthread+0x10/0x10 [ 99.354437] ret_from_fork_asm+0x1a/0x30 [ 99.354441] With Yosry's fix, it goes away. So I guess: Tested-by: Nhat Pham