From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1326FE7717F for ; Mon, 16 Dec 2024 18:39:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83D876B00B3; Mon, 16 Dec 2024 13:39:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81CBF6B00B4; Mon, 16 Dec 2024 13:39:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DA806B00B7; Mon, 16 Dec 2024 13:39:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4EB6E6B00B3 for ; Mon, 16 Dec 2024 13:39:55 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D550D1A0222 for ; Mon, 16 Dec 2024 18:39:54 +0000 (UTC) X-FDA: 82901685132.27.4B79135 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf26.hostedemail.com (Postfix) with ESMTP id 98E11140006 for ; Mon, 16 Dec 2024 18:39:30 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=o5qUqpFL; spf=pass (imf26.hostedemail.com: domain of sashal@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734374360; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AoeaMd8FrJRB9+IkG6miKbZoClIvmwR13AbsCmCSeGI=; b=ByszjtvKnL1WFlJ4EYbLD0ZyWJIeWYja1YB5CdECgkAr1vrcgNACXmZM6VbMi6fPo1kmFD HjUgN7gwMs9PYFD9X9HM9ERynsQm2aJGpcSxl3wa9sFRF3sHiBhG8UeYVPkTKbraoS13pI P37Hz9QJZFSO/vk+pStXkHFYM19DNCs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734374360; a=rsa-sha256; cv=none; b=KxjMXN7G+/k9VUKab6IJRm8jLhFteCtIHR70Y9QVVWzOWXg47DqbQB6yb2aCCGpcoS8jMl pQ31SnW6FNzdG5XbNanrHvJOFgHaqdqSXHLmtadSj1moA1l4hqXidSgoLJ1XkNdrGGjmVF lREtt/hPCfbkUOgJ4PvlQioU7oxDxQ4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=o5qUqpFL; spf=pass (imf26.hostedemail.com: domain of sashal@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 87056A419BF; Mon, 16 Dec 2024 18:38:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE6C6C4CED0; Mon, 16 Dec 2024 18:39:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1734374392; bh=v8zC+hRT7LWTpMFOk/xRpKN/iJbd5ILLztijeSa/UiA=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=o5qUqpFLiBM1qUWGHPYlrUP71YlgMOtSzT5syC/Dg426bmqllwd0WQ1FUA7gLTo+y I8oy86wXEoHaJLX0lVuDVr1QxUwaiaNpYMLU7T5YBTDgdJmH3dh/ev1m2fzQHZKuh6 i93BnHn5vUS1Z0+d85NPt0pnwW3+vxSlBZpnuTdgxe2fslS/ol7CnGe5/i0GELuV8A ct/3L6S1Szvwxf81TWy5xc4z+z/PmKCIBT/KEJpiaXoMOeGgPHqy3ankY1aR07fLGy 8+7fkSnWlSYYwceykF9NXkxsDTPlWjNWAgKSgVz8VTy1ngp0WwMJ7UlwnrhyeqbwI3 D6DpPJqw85Rag== Date: Mon, 16 Dec 2024 13:39:50 -0500 From: Sasha Levin To: Yu Zhao Cc: Kairui Song , syzkaller-bugs@googlegroups.com, syzbot , akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [syzbot] [mm?] WARNING in lock_list_lru_of_memcg Message-ID: References: <675d01e9.050a0220.37aaf.00be.GAE@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 98E11140006 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: exoqtj5h8udrfh9dwyo73ewzcm3194z4 X-HE-Tag: 1734374370-736688 X-HE-Meta: U2FsdGVkX1/Sy1FIbE4KljrWZE20mJ1fdQon9E+n9+usvG1Q9PNUwkccAZ4FqNGYHXcywRJ7xM2DYAt/d1njIxKcGC14W4d+PTKd+c72Suy/kmrhtUgTsdVwu4eX9Vb7jipDFKUO3jpHhTMs0ewY7HKv6M99qQ7oJA+qqlZ5hNnnAonFfkQodhhh0YLCr/CHJ/TAPB7KhIq+mrsmrwQ+26MhgOqbV4JDeaReH1n7VMpflWbCAJwrIrD8msmqLghUcQuI9V513GPCnfKC23WepfSsAQBEE6hpS6zbNrR8+tex6lxpMlsUxKiNPnxJ2W78I/jnQ/G5fYYWhH791yYWWSWcvlPbmMDUWCiixVDsaOBguKI7pz85V+NzYdrFcfLiXCR6XxCK9pSsQrIcNBQiBjMaYPf0cg7h1pfig8vzo1oXMCynlsYpY2sU8kTqdIyqqL4wnsb1WSKeZR3z3Rt94zsIwEirMbQqCJOs5IfqQ0g9MHgpsmUie3OxvH8Pq5KmzHlYjMlgeybnAM8+FkpVV15WZBbKVvmcECHZTS+KQ0vWyYhok79MnlQUry6CurEmoq9qdyxvQVKNqsTvah/YXhCsCKNNxG2uqFS3Ahp0Yv9opo+8k8jOLWhb2lFQUJDYnj/v+k49pKVuIyzk5jAhp308S9yH/5qECV5YMgmRda/G+Nvq0Ub85HnPTH3a5Chfx5rSx9DfeXysUSWF7ErJQoEA/cvtsyeGKNMi5aj/V7IU43iDky53d2LNH8P9Ex+Z2WOTZMA+Qwdd97SRq64gEEHiFWdYpfpTQucWZaw2Mskf5+9waN4jqxcrexymnpnkJT2YW2gE/CuzTVxrcMiqWYF+CPIRK9hSoXeeEMLpxxkbzK92l44eEhCvGejQOObZ61jBF6d2lfDc+RMbopdjCKmYKQcej4evdrFyEYznJfdISHdfBLltmtYYEnVOKIhq21BHTR21P0vfl2a8mrP Yu6qCzBC ndrMm+UPZSqHndP+llDV/ws3CvtQM0rST7pJbiK0fpfZVg/BunfurNJnkMgyJ9Ln704BnEDbInT7JovqKM/Adwgyh40Lg9CQ1P1anmlz7AAmvJTF19QrYL4lzA/I70NDFIlICOZFdUbCr9a9QDmaneUH2hL1Tru96tjTtLZZCnSECjtkc4Z7XhT7Du+kbIk+BDHhwWxSapM9Dk2AU1PrxuLR6Fdz3ubk8rTJMU1eM/s258lkL2Y3nBUjFU1aZ2yjqsSZ77IKqJpQBm3TWt38P+M3ZMssadCAbWy20Rioa+XZfaTNGohx5/dVLbrhL76cDWFed3MA/ldj7d8aNHR2vDKHPkIq6JWMK4SDLQz2YT/GEb03l79xbubFbYi0pnJ7P2zmT8HH7Q/Kt+n/TaJoU5ZK218UA57JG92hPHwxGukjczZsk0CFeukQRQjLZmjWBcjYSsqMpRXQMAnvSTKVHXGLN7L2C3dLwlICSW/ZR1Zdm2123WKmDuMk9Uk/aPtHLyaUKoX9+6rwaCmF96fYooE3gkK9jUR4UntukNAuF0Fl51KI/SMJ+6lMyTA/+ysYZDKVt6YIdXZ2gdl4mglVzB9omLUhTOJJJkwoXaqwQsvMXSafbSR5MutdGoM3U4iruPXUNOoFLSeKIBQGv/ZGjqMAepBuj8EDvhJa/LH3ir6g1hTNjOuSOG7jP2tCtdjvP7YdCYOhpwbEmIF5GWLrZQcBBNeTZjssmqioeedn0yT7K1fzBwdMBia2GwZwy5H7nmK+TPHiFkqp1+jFD8Bh3kVG1Kc4v0v5gXc1WovepdBKKxmP2VSkEZ5fDctIQOLK1LaGQpXbumeK2GmaR2SRhXdzpz4VGjhcXxOJP3T/wkCdpnQIzIwdW2WhObzuKEU8+tWI1TMCGHO/jLp3W+fKIKgyqoaWsocwbPtpLbXsawbSvILoE3ewJnCRhLLu7Fa1nPuaAwYyk0yZKbud3Fjymu4yXiT7U 9b8VtYap FpJOpsBUhovQuGoxGoVPVnHaWyN1U8P6FkweLYCCTLn81XpeFWlLl3HmQD0kdiIg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Dec 15, 2024 at 07:45:38PM -0700, Yu Zhao wrote: >Hi Kairui, > >On Sun, Dec 15, 2024 at 10:45 AM Kairui Song wrote: >> >> On Sun, Dec 15, 2024 at 3:43 AM Kairui Song wrote: >> > >> > On Sat, Dec 14, 2024 at 2:06 PM Yu Zhao wrote: >> > > >> > > On Fri, Dec 13, 2024 at 8:56 PM syzbot >> > > wrote: >> > > > >> > > > Hello, >> > > > >> > > > syzbot found the following issue on: >> > > > >> > > > HEAD commit: 7cb1b4663150 Merge tag 'locking_urgent_for_v6.13_rc3' of g.. >> > > > git tree: upstream >> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=16e96b30580000 >> > > > kernel config: https://syzkaller.appspot.com/x/.config?x=fee25f93665c89ac >> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=38a0cbd267eff2d286ff >> > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 >> > > > >> > > > Unfortunately, I don't have any reproducer for this issue yet. >> > > > >> > > > Downloadable assets: >> > > > disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7feb34a89c2a/non_bootable_disk-7cb1b466.raw.xz >> > > > vmlinux: https://storage.googleapis.com/syzbot-assets/13e083329dab/vmlinux-7cb1b466.xz >> > > > kernel image: https://storage.googleapis.com/syzbot-assets/fe3847d08513/bzImage-7cb1b466.xz >> > > > >> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: >> > > > Reported-by: syzbot+38a0cbd267eff2d286ff@syzkaller.appspotmail.com >> > > > >> > > > ------------[ cut here ]------------ >> > > > WARNING: CPU: 0 PID: 80 at mm/list_lru.c:97 lock_list_lru_of_memcg+0x395/0x4e0 mm/list_lru.c:97 >> > > > Modules linked in: >> > > > CPU: 0 UID: 0 PID: 80 Comm: kswapd0 Not tainted 6.13.0-rc2-syzkaller-00018-g7cb1b4663150 #0 >> > > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >> > > > RIP: 0010:lock_list_lru_of_memcg+0x395/0x4e0 mm/list_lru.c:97 >> > > > Code: e9 22 fe ff ff e8 9b cc b6 ff 4c 8b 7c 24 10 45 84 f6 0f 84 40 ff ff ff e9 37 01 00 00 e8 83 cc b6 ff eb 05 e8 7c cc b6 ff 90 <0f> 0b 90 eb 97 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c 7a fd ff ff 48 >> > > > RSP: 0018:ffffc9000105e798 EFLAGS: 00010093 >> > > > RAX: ffffffff81e891c4 RBX: 0000000000000000 RCX: ffff88801f53a440 >> > > > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 >> > > > RBP: ffff888042e70054 R08: ffffffff81e89156 R09: 1ffffffff2032cae >> > > > R10: dffffc0000000000 R11: fffffbfff2032caf R12: ffffffff81e88e5e >> > > > R13: ffffffff9a3feb20 R14: 0000000000000000 R15: ffff888042e70000 >> > > > FS: 0000000000000000(0000) GS:ffff88801fc00000(0000) knlGS:0000000000000000 >> > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> > > > CR2: 0000000020161000 CR3: 0000000032d12000 CR4: 0000000000352ef0 >> > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> > > > Call Trace: >> > > > >> > > > list_lru_add+0x59/0x270 mm/list_lru.c:164 >> > > > list_lru_add_obj+0x17b/0x250 mm/list_lru.c:187 >> > > > workingset_update_node+0x1af/0x230 mm/workingset.c:634 >> > > > xas_update lib/xarray.c:355 [inline] >> > > > update_node lib/xarray.c:758 [inline] >> > > > xas_store+0xb8f/0x1890 lib/xarray.c:845 >> > > > page_cache_delete mm/filemap.c:149 [inline] >> > > > __filemap_remove_folio+0x4e9/0x670 mm/filemap.c:232 >> > > > __remove_mapping+0x86f/0xad0 mm/vmscan.c:791 >> > > > shrink_folio_list+0x30a6/0x5ca0 mm/vmscan.c:1467 >> > > > evict_folios+0x3c86/0x5800 mm/vmscan.c:4593 >> > > > try_to_shrink_lruvec+0x9a6/0xc70 mm/vmscan.c:4789 >> > > > shrink_one+0x3b9/0x850 mm/vmscan.c:4834 >> > > > shrink_many mm/vmscan.c:4897 [inline] >> > > > lru_gen_shrink_node mm/vmscan.c:4975 [inline] >> > > > shrink_node+0x37c5/0x3e50 mm/vmscan.c:5956 >> > > > kswapd_shrink_node mm/vmscan.c:6785 [inline] >> > > > balance_pgdat mm/vmscan.c:6977 [inline] >> > > > kswapd+0x1ca9/0x36f0 mm/vmscan.c:7246 >> > > > kthread+0x2f0/0x390 kernel/kthread.c:389 >> > > > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 >> > > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 >> > > > >> > > >> > > This one seems to be related to "mm/list_lru: split the lock to >> > > per-cgroup scope". >> > > >> > > Kairui, can you please take a look? Thanks. >> > >> > Thanks for pinging, yes that's a new sanity check added by me. >> > >> > Which is supposed to mean, a list_lru is being reparented while the >> > memcg it belongs to isn't dying. >> > >> > More concretely, list_lru is marked dead by memcg_offline_kmem -> >> > memcg_reparent_list_lrus, if the function is called for one memcg, but >> > now the memcg is not dying, this WARN triggers. I'm not sure how this >> > is caused. One possibility is if alloc_shrinker_info() in >> > mem_cgroup_css_online failed, then memcg_offline_kmem is called early? >> > Doesn't seem to fit this case though.. Or maybe just sync issues with >> > the memcg dying flag so the user saw the list_lru dying before seeing >> > memcg dying? The object might be leaked to the parent cgroup, seems >> > not too terrible though. >> > >> > I'm not sure how to reproduce this. I will keep looking. >> >> Managed to boot the image and using the kernel config provided by bot, >> so far local tests didn't trigger any issue. Is there any way I can >> reproduce what the bot actually did? > >If syzbot doesn't have a repro, it might not be productive for you to >try to find one. Personally, I would analyze stacktraces and double >check the code, and move on if I can't find something obviously wrong. > >> Or provide some patch for the bot >> to test? > >syzbot only can try patches after it finds a repro. So in this case, >no, it can't try your patches. > >Hope the above clarifies things for you. Chiming in here as LKFT seems to be able to hit a nearby warning on boot. The link below contains the full log as well as additional information on the run. https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.13-rc2-232-g4800575d8c0b/testrun/26323524/suite/log-parser-test/test/exception-warning-cpu-pid-at-mmlist_lruc-list_lru_del/details/ -- Thanks, Sasha