From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C8DFC54E58 for ; Thu, 21 Mar 2024 03:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF6526B0083; Wed, 20 Mar 2024 23:55:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7EA66B0087; Wed, 20 Mar 2024 23:55:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1EC96B0088; Wed, 20 Mar 2024 23:55:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9CCAA6B0083 for ; Wed, 20 Mar 2024 23:55:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 674BFC1513 for ; Thu, 21 Mar 2024 03:55:00 +0000 (UTC) X-FDA: 81919680360.03.04A022F Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf21.hostedemail.com (Postfix) with ESMTP id 87F901C000A for ; Thu, 21 Mar 2024 03:54:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=KliRpXQR; spf=pass (imf21.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710993298; a=rsa-sha256; cv=none; b=PaT4WVkiFbIlsK6v4u30D0tMQQzHVrSTa9iH9wc7EuHpuuWf0nYTQtoBycATAITIBThmpR iMe4DOxLT0XHpeHHN5BJbSUrogGBy00NyJLREqJ6gG6sKV05gxHgVsF4NyldVZLpEauYus ex1DHZkdgQtt28fcJtFBPkZyYCg+r4w= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=KliRpXQR; spf=pass (imf21.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710993298; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=4BvvXd5CfTi3QPbpMLqaIgWABiJrQL6kpFAbiTyYM9E=; b=IzE0cDSHnAFAgbhtjIFfz3JRH6wP+1uwOvhl7nMPkm5hBtloQamwLL8ToilgIgShexo/Bz UcDfJlR86fyU0uN8KWhXQtCqcSFWfvHjxSqCPy0Rfb54EqKpo+i7BjoeDUeoUhNNAE0vpq PQedWgWDQh2ei8OlOOgH3BH5AIW7kPo= Date: Wed, 20 Mar 2024 23:54:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1710993296; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=4BvvXd5CfTi3QPbpMLqaIgWABiJrQL6kpFAbiTyYM9E=; b=KliRpXQRlUCP07Aus8kd4KvGBYua10DLwgKh4qwB+lnTsOsLhH22nnBjVRza0XPZB32Dt7 OroshGdTwwrhvoeYH6um4pAPoj1P+W+hR0LlbGQUKPfWdFE+GACYhfvnaBJPvO/W4gz3hL SCHVv2OjDjamFopep7wK+QTPhHapYQg= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org Subject: zswap doing io in GFP_NOIO reclaim context Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 87F901C000A X-Stat-Signature: qi357xty8gph6mumanxfo6su39p4qba4 X-Rspam-User: X-HE-Tag: 1710993298-478662 X-HE-Meta: U2FsdGVkX1/cUDJ1IwH15GZL6rQbIK/glvphiSKJ5i+gs1SEY8DbKVIkDwboYUQ68XEQvH3BCG8Wqbu695ocv9vu3A5yLAJgJXVlMOE/irkH5gx/ha+gi6Xlcskym3qnLHwl3fDT2Pc5lrEknfGSu3L8cwFlpXTIQXor09Z0T+uG0vS4r8R/+EWlbgXQwHyZoDHP2kr6b0bq1TINu0ythi91WolKRf1Y8Oz8lRAbpOO+TnQOmvSvChjgpUFNnCs6cIf4ihsk2TjnBi8ImdrML5YyuSGicLmf3xSE2iqRl8aE2PtaMgIKvwAj4OG7lnqVDb0zrYPLiLEmhgKpBuN2joricJ+JdB1l2rB0MhuMRkmAhNev9Dr4HsbWXa6aPU2G/lLn+c+C8fHqTTGBdy0Uo+5ZJrsJ9g6pzKPxa25pdDlk0pIo3357dxhttwjV/m0PM/fxD3+CiGJU9+2S8f4DObLwjGFlsrE88hMRcRAcHaMZcBIYZ5ObhiGuZn3dAJGErEAb4TwyoDask/XC8qKfUdMOqMGj868M4zM8ocoOxf56gBinoq37YGCmKIomQeyl9NfxiRZ7Nx0ZTNtKQz7oDEbjouLEIQAOPVBYDHzrOKUvnGiIAtHc1CAOehRcUmMdYtiIKnKyyv910SHZ8Tuzvq/kaKYkGozHkpos2GK2htl5fDLD0PQ9pUYhJgB8iiFgUv3Y5wEyEBrp8K8lbLkvKyjnceYtKhCoE6ufUUdC29PZCQXejcd0pMXoRsLIHw2YN4GnMSXHCdJG0De5di3K7tgh3s/fhGKgXJw7dIM2H1PlgKgv8f34qxSFIRP3I20BdCq8UwF6tovzhELTsTXnG8YrMGHBC17NiOTep74luK98QZdvenoTSTg6sc8yu8FG9Z5CXUP3UUHSmElpX2klE5tZyhE8hyMaZCpdJEOwiaGiOFPcuQP/bdWq+cn8hLVoWG8mKp0ySeYEcKCJDip 56K+CyOM WqXozu8K1JbHlYBcxPcBoCInN1OTYxIaSE/OBqjuNM2NwImEJkmkwNLwbJ3MZf/1tFHHP+Jc/8m+TjJMe98oMnmsbwssE0JxHTbW5t8fFiX7DeNEqQQfFdzkjsB7mC9j4yryR X-Bogosity: Ham, tests=bogofilter, spamicity=0.006334, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: just got this bug report, things wildly backed up in bcachefs and do some digging and it looks like zswap is to blame [10264.128242] sysrq: Show Blocked State [10264.128268] task:kworker/20:0H state:D stack:0 pid:143 tgid:143 ppid:2 flags:0x00004000 [10264.128271] Workqueue: bcachefs_io btree_write_submit [bcachefs] [10264.128295] Call Trace: [10264.128295] [10264.128297] __schedule+0x3e6/0x1520 [10264.128301] ? ttwu_do_activate+0x64/0x200 [10264.128303] schedule+0x32/0xd0 [10264.128304] schedule_timeout+0x98/0x160 [10264.128306] ? __pfx_process_timeout+0x10/0x10 [10264.128308] io_schedule_timeout+0x50/0x80 [10264.128309] wait_for_completion_io_timeout+0x7f/0x180 [10264.128310] submit_bio_wait+0x78/0xb0 [10264.128313] swap_writepage_bdev_sync+0xf6/0x150 [10264.128315] ? __pfx_submit_bio_wait_endio+0x10/0x10 [10264.128317] zswap_writeback_entry+0xf2/0x180 [10264.128319] shrink_memcg_cb+0xe7/0x2f0 [10264.128320] ? xa_load+0x8c/0xe0 [10264.128321] ? __pfx_shrink_memcg_cb+0x10/0x10 [10264.128322] __list_lru_walk_one+0xb9/0x1d0 [10264.128324] ? __pfx_shrink_memcg_cb+0x10/0x10 [10264.128325] list_lru_walk_one+0x5d/0x90 [10264.128326] zswap_shrinker_scan+0xc4/0x130 [10264.128327] do_shrink_slab+0x13f/0x360 [10264.128328] shrink_slab+0x28e/0x3c0 [10264.128329] shrink_one+0x123/0x1b0 [10264.128331] shrink_node+0x97e/0xbc0 [10264.128332] do_try_to_free_pages+0xe7/0x5b0 [10264.128333] try_to_free_pages+0xe1/0x200 [10264.128334] __alloc_pages_slowpath.constprop.0+0x343/0xde0 [10264.128337] __alloc_pages+0x32d/0x350 [10264.128338] allocate_slab+0x400/0x460 [10264.128339] ___slab_alloc+0x40d/0xa40 [10264.128341] ? mempool_alloc+0x86/0x1b0 [10264.128343] ? finish_task_switch.isra.0+0x94/0x2f0 [10264.128345] ? __schedule+0x3ee/0x1520 [10264.128345] kmem_cache_alloc+0x2e7/0x330 [10264.128347] ? mempool_alloc+0x86/0x1b0 [10264.128348] mempool_alloc+0x86/0x1b0 [10264.128349] bio_alloc_bioset+0x200/0x4f0 [10264.128351] ? __queue_work.part.0+0x1a5/0x390 [10264.128352] bio_alloc_clone+0x23/0x60 [10264.128354] alloc_io+0x26/0xf0 [dm_mod 7e9e6b44df4927f93fb3e4b5c782767396f58382] [10264.128361] dm_submit_bio+0xb8/0x580 [dm_mod 7e9e6b44df4927f93fb3e4b5c782767396f58382] [10264.128366] __submit_bio+0xb0/0x170 [10264.128367] submit_bio_noacct_nocheck+0x159/0x370 [10264.128368] bch2_submit_wbio_replicas+0x21c/0x3a0 [bcachefs 85f1b9a7a824f272eff794653a06dde1a94439f2] [10264.128391] btree_write_submit+0x1cf/0x220 [bcachefs 85f1b9a7a824f272eff794653a06dde1a94439f2] [10264.128406] process_one_work+0x178/0x350 [10264.128408] worker_thread+0x30f/0x450 [10264.128409] ? __pfx_worker_thread+0x10/0x10 [10264.128409] kthread+0xe5/0x120 dm is using GFP_NOIO for that allocation, so zswap is clearly busted. We're already under generic_make_request(), so that submit_bio_wait() that zswap kicked off is never going to return. We need to think about how to add some assertions so that we know reclaim context is being honoured...