From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47BF3D10C11 for ; Mon, 28 Oct 2024 13:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D26086B007B; Mon, 28 Oct 2024 09:22:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD59E6B0083; Mon, 28 Oct 2024 09:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA7226B0085; Mon, 28 Oct 2024 09:22:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 994B46B007B for ; Mon, 28 Oct 2024 09:22:30 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 48261402D5 for ; Mon, 28 Oct 2024 13:22:30 +0000 (UTC) X-FDA: 82723074378.06.890903C Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf05.hostedemail.com (Postfix) with ESMTP id 0059210000D for ; Mon, 28 Oct 2024 13:21:43 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=K9Bw2Cr7; spf=pass (imf05.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730121591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0iVa6DyRidyOL+QxARBUWZ4PRk4jvzaoNq/H8Rqef6E=; b=0JYY62OYKuwX0U0ByUBYNq33YCNj2KshIoxCnzbmalIEz5SzCPZv4d8IG+HA035tUITWR0 1ug2gW8GNcgqMJBrYFI+ipg/BYJMivUua8F/XHLjpDh5QpmlT8w1+j3d3OuBR6Sbx+c/Eb 7JGx8eihI5jq6lEb8WMgcCFfQC8aABE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730121591; a=rsa-sha256; cv=none; b=EoJ6Ob6HufSCdnA8okHVz1seBXpx+1c5JR7Lo/tYoMp9dnR9PsS/uD5sFMlK0vx0tLZmTa YVaxN1eJYu7WRX0BwtQ3AGBFsnwcXhXymDdDnLWDWw8eoHsTFYB+Mxw34FX75hZIhKqI8s WKzjJJPGajlIttKXmPEmk3lo2CFYnP0= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=K9Bw2Cr7; spf=pass (imf05.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-a9acafdb745so773834866b.0 for ; Mon, 28 Oct 2024 06:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730121747; x=1730726547; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=0iVa6DyRidyOL+QxARBUWZ4PRk4jvzaoNq/H8Rqef6E=; b=K9Bw2Cr7lzOrjLwjR9efxZgc9WpucfsN2fWTXNS0DkOxnAZU9nviiEb8A2lnIuWfJ6 s7auZjrCXmus2Kl9vU4dV2Y9jad23K00XsDk4iPIc90yKd9JzTVa8k4PpjrkmcQ+Ge7N zZpuT+xKB3ihXJGlzYoJ0S5/4k+FOfuv9mXhcmfy0O7mv9LOAS10nA1nizIIT91MoZWt ZsEcdg//WI42pgs+eqxhPdPpug7kT47cfeHflAjPs0CGCJk9c7ySMdtJVKi247LHmCgh 3f2einLbIYGArLR4FlUi59//n3nRGHjzjzCCBtBpxuqiaoRF/OLRMIgahBAQYlJKnWuF Ymvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730121747; x=1730726547; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0iVa6DyRidyOL+QxARBUWZ4PRk4jvzaoNq/H8Rqef6E=; b=C0PfPBGG2PyW0mQ1877WaHIvFmCVT3LDAH42IlIszz+FeYwO2g6nR20/Fe2+ysOzFF HMvPeOEe8mOFwGURDAteiBmcrQ1CQQXvPPkktjVT1UfEIxHvZUcBLzTOVOI6SCz3qqx2 k2C93YuvyTKfLfFtwKerPcToZAPa1VZO2hmluXcrIJbL3trU0z9BYFAcNpRcEosQ68ZD VnRlEj7iD9BTs4I7n01nL3Xq/dKptUjz7Le6eqY0TlDUdMQULMgcoaYJ10d43zHKWm6v gxZbgyjYPQzUeo9lK7ToQEoZuJePoiqtR6vrd36WawQUwJde6mCTin5PKJEUBdKVizuK ItOA== X-Gm-Message-State: AOJu0YzhddJw39zKEJx6QgOiA4c+LcKk5hxsYzyTc7iKdELa6QIb7eCN ZzTe739RTX6MQ6jxfOEgq14toiHPPfriJXy6ZV0O3mzm7DWNBafX X-Google-Smtp-Source: AGHT+IFw8M7ulax5T7IES0b1QZHEmtEa3Xyc1Mb9w9bXuTUciTcVogecjHHxveA+Km5+5BHeZH/YNg== X-Received: by 2002:a17:907:3f89:b0:a99:ff33:9ba5 with SMTP id a640c23a62f3a-a9ad1a5eecamr1476974866b.24.1730121746339; Mon, 28 Oct 2024 06:22:26 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:eb:d0d0:c7fd:c82c? ([2620:10d:c092:500::5:1494]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9b1f298dc4sm376806366b.124.2024.10.28.06.22.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 28 Oct 2024 06:22:25 -0700 (PDT) Message-ID: <6da1b9a9-dc77-48f2-8ab2-7b672e6c11ad@gmail.com> Date: Mon, 28 Oct 2024 13:22:25 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 5/6] mm/list_lru: split the lock to per-cgroup scope To: Kairui Song Cc: linux-mm@kvack.org, Andrew Morton , Matthew Wilcox , Johannes Weiner , Roman Gushchin , Waiman Long , Shakeel Butt , Michal Hocko , Chengming Zhou , Qi Zheng , Muchun Song , Nhat Pham , Yosry Ahmed References: <20240925171020.32142-1-ryncsn@gmail.com> <20240925171020.32142-6-ryncsn@gmail.com> <7334b158-8567-4c66-8176-57b879ff8882@gmail.com> Content-Language: en-US From: Usama Arif In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 0059210000D X-Stat-Signature: 51ze7ihef6hf1no35um9i475rwhq786y X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1730121703-535966 X-HE-Meta: U2FsdGVkX1+aHi3ycm5mzeejrJa0pSPd+BqZc2urb5QOJL+mucSyf5kIhLHhqOevFsrSUI8mGjGKfF0tEWp/b2r4ysDCrb1jSklcAV/Y9xSlxfytf6NNo4R9A09BZ1B51r7gHPXsFoxHtGLDK4/2einayGiZZu8lEjeYpbeOCOjg+OeqicJvOw6V3+0wHiGY+b1i0fxcxCQt6Op7F3Tsr+yBtehOH/wj6LIv221v4syPC7pHsHiDVqK1AurCbTCvuDjidBeEZMe2387xF0/VUYlEKsUiVY4LuSJaty71WbWh/hBxmTyZLkvvuMyNXV0V7avUuh1KfEXQvbrOKgrDALPOwFoXL9pIhWwVdNlXg1FbCcgFIEn9rcSLfALXNJGtUANyz+SgxiUZYD07NeMbSQmbwt1Zxw+J0/8Dp+Foa5OE2ninicBqj6pO5f3Tzq4zH2mxxMMtDxDYme7n52ZPkcelHNCEQJr1wHQ3pz8CCzYQp48a7y52fSUsYykLSUQMGJrsPCznSGa8nWsU9tmD/1dOLHQBwxaWOrkGjo/TMyOWJCT8GYD/sAyzibzmL/D/hYXSKl/KpgYPQGlM8XK5G3d4PwjEqiFXZIFlGjR30zQes9n0+2p/ZerFmV+BXMlpGc3W+wuYak5uKAKHnoGe45IIscTjGWfsgYRfsoGA+y45r9Una932MVCWmZUq6kbsCI3yumyPswMWn0ktvHic84k1u0WKLbBXgNQamLCvTDkUt9Fb1sd9IfuMD5kZoSnIvrUCZsAqltdhYmz42SSLcYfdAS0kBWfW4pOvwqX13GtHkV35jTQb25Txh9RatgqaEIGRB3vO2+9s4X8bvhsm8zanmha+gWNpmS35n0gNsq8MrK6Oq5yPdV9UFI0uO74m2PoS9hZXmJ/IR7hrDHpUeJY8CAd4idllIJKqYxGn4cA1Wo/jFlWyQ+YH32IBOCsYdU+xDwkiB3ogzIdJaOS 95hJ11ox tq90dWQ91mg59cwiL2EiGYhSHo2kVFH+jZ+4+EEJmlORmi7sTTyXW2xeg/r19/lRW6qCjqNKPUM6+igfy2q0VzGS3dk20nOjG7bj8/Re30iZWHkP/YofyDSO8AbDXXz4ulO1bxsQRhKaNCQYKFgEROplcOQduVhaluCnbb3TCEplrZwuEAJHaYqU+35drrS0r40god1A8WpbOnh+pYe0znvkUtosFcQUDFr0eRmEm/0tEOKNQ0R2DLYhIBOi+oS35XmJ8aFENJ6Pdw+Cex4dw25thv+nmydKXTItdFNgGlMqVQx8X2xsCBuif55SV+FZ04f+DAPafyg3juQ9jW3Yr34+r2QJ4z2G4hWviCCGQnx+akeTh66XgUCBk2K4hS6xQV7EDW58mZuUTPP9nRrItHF6jNhRnV2oPy+fpZBwK6btBIrjlnQ3PB/NvWSC6w/BK8r4+6BgVLecZRrxdoWyaiQnccytF6+HDVn9JxxUGyxdwkKw/v3SZQgo1gGGcGz6tzZdjTE3n+HG3O7g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 27/10/2024 17:26, Kairui Song wrote: > Hi Usama, > >> >> Hi Kairui, >> >> I was testing zswap writeback in mm-unstable, and I think this patch might be breaking things. >> >> I have added the panic below >> >> 130.051024] ------------[ cut here ]------------ >> [ 130.051489] kernel BUG at mm/list_lru.c:321! >> [ 130.051732] Oops: invalid opcode: 0000 [#1] SMP >> [ 130.052133] CPU: 1 UID: 0 PID: 4976 Comm: cc1 Not tainted 6.12.0-rc1-00084-g278bd01cdaf1 #276 >> [ 130.052595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.el9 04/01/2014 >> [ 130.053276] RIP: 0010:__list_lru_walk_one+0x1ae/0x1b0 >> [ 130.053983] Code: 7c 24 78 00 74 03 fb eb 00 48 89 d8 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f 5d c3 41 c6 07 00 eb e8 41 c6 07 00 fb eb e1 0f 0b <0f> 0b 0f 1f 44 00 00 6a 01 e8 44 fe ff ff 48 83 c4 08 c3 66 2e 0f >> [ 130.055557] RSP: 0000:ffffc90004a2b9a0 EFLAGS: 00010246 >> [ 130.056084] RAX: ffff88805dedf6e8 RBX: 0000000000000071 RCX: 0000000000000005 >> [ 130.057407] RDX: 0000000000000000 RSI: 0000000000000022 RDI: ffff888008a26400 >> [ 130.057794] RBP: ffff88805dedf6d0 R08: 0000000000000402 R09: 0000000000000001 >> [ 130.058579] R10: ffffc90004a2b7e8 R11: 0000000000000000 R12: ffffffff81342930 >> [ 130.058962] R13: ffff888017532ca0 R14: ffffc90004a2bae8 R15: ffff8880175322c8 >> [ 130.059773] FS: 00007ff3f1e21f00(0000) GS:ffff88807dd00000(0000) knlGS:0000000000000000 >> [ 130.060242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 130.060563] CR2: 00007f428e2e2ed8 CR3: 0000000067db6001 CR4: 0000000000770ef0 >> [ 130.060952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 130.061658] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 130.062425] PKRU: 55555554 >> [ 130.062578] Call Trace: >> [ 130.062720] >> [ 130.062941] ? __die_body+0x66/0xb0 >> [ 130.063145] ? die+0x88/0xb0 >> [ 130.063309] ? do_trap+0x9d/0x170 >> [ 130.063499] ? __list_lru_walk_one+0x1ae/0x1b0 >> [ 130.063745] ? __list_lru_walk_one+0x1ae/0x1b0 >> [ 130.063995] ? handle_invalid_op+0x65/0x80 >> [ 130.064223] ? __list_lru_walk_one+0x1ae/0x1b0 >> [ 130.064467] ? exc_invalid_op+0x2f/0x40 >> [ 130.064681] ? asm_exc_invalid_op+0x16/0x20 >> [ 130.064912] ? zswap_shrinker_count+0x1c0/0x1c0 >> [ 130.065172] ? __list_lru_walk_one+0x1ae/0x1b0 >> [ 130.065417] list_lru_walk_one+0xc/0x20 >> [ 130.065630] zswap_shrinker_scan+0x4b/0x80 >> [ 130.065856] do_shrink_slab+0x15f/0x2f0 >> [ 130.066075] shrink_slab+0x2bf/0x3d0 >> [ 130.066276] shrink_node+0x4f0/0x8a0 >> [ 130.066477] do_try_to_free_pages+0x131/0x4d0 >> [ 130.066717] try_to_free_mem_cgroup_pages+0x143/0x220 >> [ 130.067000] try_charge_memcg+0x22a/0x610 >> [ 130.067224] __mem_cgroup_charge+0x74/0x100 >> [ 130.068060] do_pte_missing+0xaa8/0x1020 >> [ 130.068280] handle_mm_fault+0x75d/0x1120 >> [ 130.068502] do_user_addr_fault+0x1c2/0x6f0 >> [ 130.068802] exc_page_fault+0x4f/0xb0 >> [ 130.069014] asm_exc_page_fault+0x22/0x30 >> [ 130.069240] RIP: 0033:0x7ff3f19ede49 >> [ 130.069441] Code: c9 62 e1 7f 29 7f 00 c3 66 0f 1f 84 00 00 00 00 00 40 0f b6 c6 48 89 d1 48 89 fa f3 aa 48 89 d0 c3 48 3b 15 c9 a3 06 00 77 e7 <62> e1 fe 28 7f 07 62 e1 fe 28 7f 47 01 48 81 fa 80 00 00 00 76 89 >> [ 130.070477] RSP: 002b:00007ffc5c818078 EFLAGS: 00010283 >> [ 130.070830] RAX: 00007ff3efac9000 RBX: 00007ff3f02d1940 RCX: 0000000000000001 >> [ 130.071522] RDX: 00000000000005a8 RSI: 0000000000000000 RDI: 00007ff3efac9000 >> [ 130.072146] RBP: 00007ffc5c8180c0 R08: 0000000003007320 R09: 0000000000000007 >> [ 130.072594] R10: 0000000003007320 R11: 0000000000000012 R12: 00007ff3f1f0e000 >> [ 130.072981] R13: 000000007ffa1e74 R14: 00000000000005a8 R15: 00000000000000b5 >> [ 130.073369] >> [ 130.073496] Modules linked in: >> [ 130.073701] ---[ end trace 0000000000000000 ]--- >> [ 130.073960] RIP: 0010:__list_lru_walk_one+0x1ae/0x1b0 >> [ 130.074319] Code: 7c 24 78 00 74 03 fb eb 00 48 89 d8 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f 5d c3 41 c6 07 00 eb e8 41 c6 07 00 fb eb e1 0f 0b <0f> 0b 0f 1f 44 00 00 6a 01 e8 44 fe ff ff 48 83 c4 08 c3 66 2e 0f >> [ 130.075564] RSP: 0000:ffffc90004a2b9a0 EFLAGS: 00010246 >> [ 130.075897] RAX: ffff88805dedf6e8 RBX: 0000000000000071 RCX: 0000000000000005 >> [ 130.076342] RDX: 0000000000000000 RSI: 0000000000000022 RDI: ffff888008a26400 >> [ 130.076739] RBP: ffff88805dedf6d0 R08: 0000000000000402 R09: 0000000000000001 >> [ 130.077192] R10: ffffc90004a2b7e8 R11: 0000000000000000 R12: ffffffff81342930 >> [ 130.077739] R13: ffff888017532ca0 R14: ffffc90004a2bae8 R15: ffff8880175322c8 >> [ 130.078149] FS: 00007ff3f1e21f00(0000) GS:ffff88807dd00000(0000) knlGS:0000000000000000 >> [ 130.078764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 130.079095] CR2: 00007f428e2e2ed8 CR3: 0000000067db6001 CR4: 0000000000770ef0 >> [ 130.079521] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 130.080009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 130.080402] PKRU: 55555554 >> [ 130.080713] Kernel panic - not syncing: Fatal exception >> [ 130.081198] Kernel Offset: disabled >> [ 130.081396] ---[ end Kernel panic - not syncing: Fatal exception ]--- >> >> Thanks, >> Usama >> > > Thanks for the report. I converted list_lru_walk callback to keep the > list unlocked when LRU_RETRY and LRU_REMOVED_RETRY is returned, but > didn't notice shrink_memcg_cg in zswap.c could return LRU_STOP after > it unlocked the list. > > The fix should be simple, is it easy to reproduce? Can you help verify? > > diff --git a/mm/list_lru.c b/mm/list_lru.c > index 79c2d21504a2..1a3caf4c4e14 100644 > --- a/mm/list_lru.c > +++ b/mm/list_lru.c > @@ -298,9 +298,9 @@ __list_lru_walk_one(struct list_lru *lru, int nid, > struct mem_cgroup *memcg, > ret = isolate(item, l, cb_arg); > switch (ret) { > /* > - * LRU_RETRY and LRU_REMOVED_RETRY will drop the lru lock, > - * the list traversal will be invalid and have to restart from > - * scratch. > + * LRU_RETRY, LRU_REMOVED_RETRY and LRU_STOP will drop the lru > + * lock, the list traversal will be invalid and have to restart > + * from scratch. > */ > case LRU_RETRY: > goto restart; > @@ -318,14 +318,13 @@ __list_lru_walk_one(struct list_lru *lru, int > nid, struct mem_cgroup *memcg, > case LRU_SKIP: > break; > case LRU_STOP: > - assert_spin_locked(&l->lock); > goto out; > default: > BUG(); > } > } > -out: > unlock_list_lru(l, irq_off); > +out: > return isolated; > } Hi Kairui, With this fix there are no more crashes. Thanks for the quick fix. Just FYI, to test it, just enable zswap and zswap shrinker (echo Y > /sys/module/zswap/parameters/shrinker_enabled) and build the kernel in a memory constrained environment (memory.max 1G). Thanks, Usama