From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 743F6E7717F for ; Tue, 17 Dec 2024 18:19:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0090F6B0088; Tue, 17 Dec 2024 13:19:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFB766B0089; Tue, 17 Dec 2024 13:19:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9C156B008A; Tue, 17 Dec 2024 13:19:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BA7306B0088 for ; Tue, 17 Dec 2024 13:19:53 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 35E8CAF12E for ; Tue, 17 Dec 2024 18:19:53 +0000 (UTC) X-FDA: 82905263742.24.55D2BB8 Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by imf05.hostedemail.com (Postfix) with ESMTP id 513FE10000F for ; Tue, 17 Dec 2024 18:18:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NkG8PbG3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734459562; a=rsa-sha256; cv=none; b=62MoLbqUuwnbuup0w8OUv5kEaMZOEkD1WhUqWJBgDb7W1UzWkKCQGDRKAp8e1aiiqrJ0al HNBcfLIbqsLZmPzF68ucHXODgEbH5ElssZ+FfIaNXKEDJmHgCIiae3LkIiBgbla4cVKb+G TgjdNRShE2l8z2fVQyc+cNyQcF4nk2c= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NkG8PbG3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734459562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+P5DKogCq0v/GbW8OSiapHzXqgUsSnRh6uUJF+XX4Dc=; b=Kztw4RkA6wyq5qOhr55pIhJhbA/2mgloYCpROk3CqgqjQ7cRIs1c3Td3fdZmJ+oSgOqG1I yshCJEbuwZsxyJ31uBHD5cUIstykUF1ShrT6myPCPU89Nf5j2eVYdKzbXPXIYyyK5Q63H6 mP6NuwWzsSChIvuXe1kQ55nPpMC2xdU= Received: by mail-lj1-f181.google.com with SMTP id 38308e7fff4ca-303489e8775so34366081fa.3 for ; Tue, 17 Dec 2024 10:19:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734459589; x=1735064389; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=+P5DKogCq0v/GbW8OSiapHzXqgUsSnRh6uUJF+XX4Dc=; b=NkG8PbG3rtIzPVA/8EUUrWQ3DCQ67BQs57xt97urrBXprNDMtfwrYxzl7ZkuzDYHtJ iPkgu5sdu7F93EtZ5607d3LdwzJcCRVyUvvgsJmP1QeSgak6XwCq9KedZor36HocHkxR M1IhhWPmGsVUfJ1jzuWmyeFZDm3S9TyaL6WTzSO6QxOZpVlhbnvcFLoaT1V84LddKMJ+ 4UbUyyPO3/l8L8PGSI+zAbtOwcX8bHJLKeUlieJne1oXPIv6WOkSqpvRdwRv2Vl0RjMP xujvQ2UOv0wTUcDJhnjPXqQ2DDEs2FbdqOzONJ6c/suj5ibR1SI0JZ4dQvliADVqRA2p ERUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734459589; x=1735064389; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+P5DKogCq0v/GbW8OSiapHzXqgUsSnRh6uUJF+XX4Dc=; b=dbiiCKJU02xJ7pXxzOEN5qlBpJviKbf/jd7LAVGpjogyUvCrIspKPYOshhUhoAKo48 c3h/89IaV4wSLnuj8Jiv5y2CAeXRUEcctdNVI9Hi6MO3ddSd0kdmX5EV+9Rkb4g60wUZ ryxVNdhjLdFyvoC9NkdQV66JPwIO5q/5lIsnG2/zCWfeZVv1LQCb5qzsKFMlOSm5oYAM w37e0w+fp0y0hRD7VYsoYVPFVqpdiPUk+NAWeIHdXk2knOwpwfMedxUzIF73cFDSewip /GjtGuTT1yJXMy7FZKGNS81pz+bpqiZEAGekMS3ckGddst/+GotGXpcQEkk94JJGe/UW zXRQ== X-Forwarded-Encrypted: i=1; AJvYcCVvH3KuO3SqA1G4xhcrnWBdYFcl/LdZP1/zLONx8EpKu0ZNULxml7pvz6iDRQcuVgljDoVHokQgZw==@kvack.org X-Gm-Message-State: AOJu0YxEY8mOhSHXycPgK8Ei3vN/IU9tds7koMw4UXUBZeTVjzE20N0j GwKt9F8uMTXXff9GWJPlJ25j9zpoM42V1AdcKXTB1feyO+x/Z7qKjy20Lj/Dc0S2ZhKzF1fJ4Qs wmXKXXy8sXfWLx7B1sgHUn76vg8U= X-Gm-Gg: ASbGncv22ko4wsqhkyWYaB5hm4iEwak9MxKISc2chG3EyYa5ROiRzpkyeRiGM2hGRbS w74UrwfZrigguiUuKu+bcpZn5ZBUoxsXD2r/5sQ== X-Google-Smtp-Source: AGHT+IGf4SqADEjGfG+Ae8Wp6YpZfVo404b3p/VZyDrOJ4qRxgh5WLnRYubpxjPk1DK6HanjOIoImSB9jFtsULQit94= X-Received: by 2002:a2e:be1a:0:b0:302:48fd:6922 with SMTP id 38308e7fff4ca-3044db51c69mr249791fa.37.1734459588991; Tue, 17 Dec 2024 10:19:48 -0800 (PST) MIME-Version: 1.0 References: <675d01e9.050a0220.37aaf.00be.GAE@google.com> In-Reply-To: From: Kairui Song Date: Wed, 18 Dec 2024 02:19:32 +0800 Message-ID: Subject: Re: [syzbot] [mm?] WARNING in lock_list_lru_of_memcg To: Sasha Levin Cc: Yu Zhao , syzkaller-bugs@googlegroups.com, syzbot , akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 513FE10000F X-Stat-Signature: yqr113uhbackk9jghco3b8aojahindh4 X-Rspam-User: X-HE-Tag: 1734459532-501300 X-HE-Meta: U2FsdGVkX1+3RmSCxBtG/iBEkzqDhieF/TKKVKjTvUv59XiOUtTqTsS9Q7ccXgjwp4XWcYAUdjgX5yro6lhAIdQI3ihMdUwyasT/bLpKzXdx5lG31zi195HyGSLK20jSVZ2eEHfzluPDSUGnIT/gyFM/e+OZJpVyfXXdwON/YcqYtJJTT8reg6f7JTiIdRWRA3Hgjy22kwNWzSRto3ISWR+bkrRaNWYAUGb80noLP+MCZ6lBP8rINieQI7JZ25F/u/Cq68VuTJdpfchWnXqeLIYEnLIBduBYqONnJLUh9tYLyvVKnGLqv05aL2Dj1+YqeSNdHMRkPZpUCchaGw8LAHBECxuJJb5RGJ9aNsNEog78ID8pd8LI5mq6bN+a++oGmrwigBlAENFdtTNZystM0/4CrT6YLNGtwhp6eH0RoI0ivIJrEsbXdd211Kj9ZeA9turNHt68rdArh5mulfGvNrVhZZyIM6qPGI65AhDymnLLFDGBq2xy4JMEPzJCxESKE4bAnvQD4YHXsFKuj3/LU/AFFW6Jnxs+PbnfCCfcky34rYkIVNw8sy82LOC+56ZqTQCo1xFj20GAKv+MK1B5A7QdgBhMWnYWTCEWHQBnpPO5GwTD82Zzh1S9bb4lC5JBVXcJpjoteOPP3SWN6h2SO6Ix/FgsJf4s5Lvs+1v977Fl1cyXLJETXPAoWqBF02B8p03Ml/+7epq/trlI8eJSZk0hp1pXKaXAVseFld8+hoP3cq7rio+ZwDjbJk4ICNTy+ZANCKFNyzxJGO+XXE246tj/tBDetsqMna8bFD+q/5+INkxiOj5gTwIghG+IiLhLfjbWM9p4PH2EREMX4HSuQy8EScTxjVEJVw3WBTUBYsWOTzaIhLQ217Xv9mklfrA0prk5Pul/0V2mtipN5gqkZaYYJVM+kzgT5CnMRiyVsI8ifKk6odRn6IRWiUEd8N5Y/WXe2UccLGohvzmtKC6 uWWjXTSR cxM9QhpqKRRMT1C30VN8YAAW9XhkhjbLHMS78JB96Ao7xNRla7Sf4VSAfzXYdBU6liDHt/51NJm77zew3eHmZ37RWvF7x8275x+3dfiWWddXME+VqgXmL/IBTCKs1V9FltotsQs3w2J6qoTPWUEpfSzU9SPxJFR436FjgA0tzVfQgCa+gfV20g6nS1+oofoIjD3XhygaXJnhcY8C+dIquihwfQiOlCwKlgJVG/6Vu/Kvjd0CKiw7BIo7M1hZiJGVLmEKO13bc6DJqfAi1dzurSORW2CO7x1NRYpUk/p8c8ISEooLYEqrYGrMaq+i/NUYEC9a88xw7eH1vi0lZCxpA4aFWAwByqAdv6sg1FnlJcBvFxgM+kdYEQTY3blILC3qkpQ3atbHBPGkaPU/IdH9StvZn7H7RScPCImY/dTUrBz8shIO+xvn+68a6ltoYxpzWkt7EnOBLC6iI7jjX44vWLskaEZwbwgHtGZtBy8l3/ORzfypLrozyD+0j1VxQq39b5kPhn9G7+juTV2jMFoesNTFd++xSm+iYXYnO3ht0VvwDNBPzZqWbMbMrhEw0osXCVsWk8f0Mt8AfM50m09M9hpjK/QNIQEBwvL3d8QDDTuYepSlTAsBoMwTqJwhOicb6rVckFcrbFq70BfmgkeZYiE8RSQby2hZzOb8NkybFPI6UodsbWVJDBUnwNskjOXGV9159xFHyeB2AVSyRW7QRqsqbtGC9Gn+1Y4aQsrdyfb9y3gw1/Cdjq9c+E2sHjsLQIFepBOg8C5j2tB/4cvjdlAe2WQI+iDSzzaA02gEMOHVOb9QoGEtF/Xei1kztzUgPvz50SOAOCJZLYY9EFFNoVIMzEatJRLU0kFyWvVtuaDxUaEB3gYD8r6HfI5U9tHSPGtB1xU1tJ4YR4BkvU5aJOUlU9klx2dAthSQedcqiXPExeLb/lsrhs9z7jM5AbwSaBk1HzuGWGrFOXtIiwTT29CGetWQO 4Pbq8JJK MKzo1wsdthCCsmixlGQicI7I7nRoHeExC1MYtqjkDytiyNR0xwGKf6bDCu7BIfra4HrssC4Yd1Od8/keQzpyxxlmU/Be25g7rtIuoIlU0HXJWynochFoFWsmp3ZLwLL8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Thanks! Looking Sasha Levin =E4=BA=8E 2024=E5=B9=B412=E6=9C=8817=E6=97= =A5=E5=91=A8=E4=BA=8C 02:39=E5=86=99=E9=81=93=EF=BC=9A > > On Sun, Dec 15, 2024 at 07:45:38PM -0700, Yu Zhao wrote: > >Hi Kairui, > > > >On Sun, Dec 15, 2024 at 10:45=E2=80=AFAM Kairui Song = wrote: > >> > >> On Sun, Dec 15, 2024 at 3:43=E2=80=AFAM Kairui Song = wrote: > >> > > >> > On Sat, Dec 14, 2024 at 2:06=E2=80=AFPM Yu Zhao = wrote: > >> > > > >> > > On Fri, Dec 13, 2024 at 8:56=E2=80=AFPM syzbot > >> > > wrote: > >> > > > > >> > > > Hello, > >> > > > > >> > > > syzbot found the following issue on: > >> > > > > >> > > > HEAD commit: 7cb1b4663150 Merge tag 'locking_urgent_for_v6.13= _rc3' of g.. > >> > > > git tree: upstream > >> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=3D16e9= 6b30580000 > >> > > > kernel config: https://syzkaller.appspot.com/x/.config?x=3Dfee2= 5f93665c89ac > >> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=3D38a0cb= d267eff2d286ff > >> > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutil= s for Debian) 2.40 > >> > > > > >> > > > Unfortunately, I don't have any reproducer for this issue yet. > >> > > > > >> > > > Downloadable assets: > >> > > > disk image (non-bootable): https://storage.googleapis.com/syzbot= -assets/7feb34a89c2a/non_bootable_disk-7cb1b466.raw.xz > >> > > > vmlinux: https://storage.googleapis.com/syzbot-assets/13e083329d= ab/vmlinux-7cb1b466.xz > >> > > > kernel image: https://storage.googleapis.com/syzbot-assets/fe384= 7d08513/bzImage-7cb1b466.xz > >> > > > > >> > > > IMPORTANT: if you fix the issue, please add the following tag to= the commit: > >> > > > Reported-by: syzbot+38a0cbd267eff2d286ff@syzkaller.appspotmail.c= om > >> > > > > >> > > > ------------[ cut here ]------------ > >> > > > WARNING: CPU: 0 PID: 80 at mm/list_lru.c:97 lock_list_lru_of_mem= cg+0x395/0x4e0 mm/list_lru.c:97 > >> > > > Modules linked in: > >> > > > CPU: 0 UID: 0 PID: 80 Comm: kswapd0 Not tainted 6.13.0-rc2-syzka= ller-00018-g7cb1b4663150 #0 > >> > > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-= debian-1.16.3-2~bpo12+1 04/01/2014 > >> > > > RIP: 0010:lock_list_lru_of_memcg+0x395/0x4e0 mm/list_lru.c:97 > >> > > > Code: e9 22 fe ff ff e8 9b cc b6 ff 4c 8b 7c 24 10 45 84 f6 0f 8= 4 40 ff ff ff e9 37 01 00 00 e8 83 cc b6 ff eb 05 e8 7c cc b6 ff 90 <0f> 0b= 90 eb 97 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c 7a fd ff ff 48 > >> > > > RSP: 0018:ffffc9000105e798 EFLAGS: 00010093 > >> > > > RAX: ffffffff81e891c4 RBX: 0000000000000000 RCX: ffff88801f53a44= 0 > >> > > > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000= 0 > >> > > > RBP: ffff888042e70054 R08: ffffffff81e89156 R09: 1ffffffff2032ca= e > >> > > > R10: dffffc0000000000 R11: fffffbfff2032caf R12: ffffffff81e88e5= e > >> > > > R13: ffffffff9a3feb20 R14: 0000000000000000 R15: ffff888042e7000= 0 > >> > > > FS: 0000000000000000(0000) GS:ffff88801fc00000(0000) knlGS:0000= 000000000000 > >> > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> > > > CR2: 0000000020161000 CR3: 0000000032d12000 CR4: 0000000000352ef= 0 > >> > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 000000000000000= 0 > >> > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 000000000000040= 0 > >> > > > Call Trace: > >> > > > > >> > > > list_lru_add+0x59/0x270 mm/list_lru.c:164 > >> > > > list_lru_add_obj+0x17b/0x250 mm/list_lru.c:187 > >> > > > workingset_update_node+0x1af/0x230 mm/workingset.c:634 > >> > > > xas_update lib/xarray.c:355 [inline] > >> > > > update_node lib/xarray.c:758 [inline] > >> > > > xas_store+0xb8f/0x1890 lib/xarray.c:845 > >> > > > page_cache_delete mm/filemap.c:149 [inline] > >> > > > __filemap_remove_folio+0x4e9/0x670 mm/filemap.c:232 > >> > > > __remove_mapping+0x86f/0xad0 mm/vmscan.c:791 > >> > > > shrink_folio_list+0x30a6/0x5ca0 mm/vmscan.c:1467 > >> > > > evict_folios+0x3c86/0x5800 mm/vmscan.c:4593 > >> > > > try_to_shrink_lruvec+0x9a6/0xc70 mm/vmscan.c:4789 > >> > > > shrink_one+0x3b9/0x850 mm/vmscan.c:4834 > >> > > > shrink_many mm/vmscan.c:4897 [inline] > >> > > > lru_gen_shrink_node mm/vmscan.c:4975 [inline] > >> > > > shrink_node+0x37c5/0x3e50 mm/vmscan.c:5956 > >> > > > kswapd_shrink_node mm/vmscan.c:6785 [inline] > >> > > > balance_pgdat mm/vmscan.c:6977 [inline] > >> > > > kswapd+0x1ca9/0x36f0 mm/vmscan.c:7246 > >> > > > kthread+0x2f0/0x390 kernel/kthread.c:389 > >> > > > ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 > >> > > > ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 > >> > > > > >> > > > >> > > This one seems to be related to "mm/list_lru: split the lock to > >> > > per-cgroup scope". > >> > > > >> > > Kairui, can you please take a look? Thanks. > >> > > >> > Thanks for pinging, yes that's a new sanity check added by me. > >> > > >> > Which is supposed to mean, a list_lru is being reparented while the > >> > memcg it belongs to isn't dying. > >> > > >> > More concretely, list_lru is marked dead by memcg_offline_kmem -> > >> > memcg_reparent_list_lrus, if the function is called for one memcg, b= ut > >> > now the memcg is not dying, this WARN triggers. I'm not sure how thi= s > >> > is caused. One possibility is if alloc_shrinker_info() in > >> > mem_cgroup_css_online failed, then memcg_offline_kmem is called earl= y? > >> > Doesn't seem to fit this case though.. Or maybe just sync issues wit= h > >> > the memcg dying flag so the user saw the list_lru dying before seein= g > >> > memcg dying? The object might be leaked to the parent cgroup, seems > >> > not too terrible though. > >> > > >> > I'm not sure how to reproduce this. I will keep looking. > >> > >> Managed to boot the image and using the kernel config provided by bot, > >> so far local tests didn't trigger any issue. Is there any way I can > >> reproduce what the bot actually did? > > > >If syzbot doesn't have a repro, it might not be productive for you to > >try to find one. Personally, I would analyze stacktraces and double > >check the code, and move on if I can't find something obviously wrong. > > > >> Or provide some patch for the bot > >> to test? > > > >syzbot only can try patches after it finds a repro. So in this case, > >no, it can't try your patches. > > > >Hope the above clarifies things for you. > > Chiming in here as LKFT seems to be able to hit a nearby warning on > boot. > > The link below contains the full log as well as additional information > on the run. > > https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.13-rc2-= 232-g4800575d8c0b/testrun/26323524/suite/log-parser-test/test/exception-war= ning-cpu-pid-at-mmlist_lruc-list_lru_del/details/ > Thanks for the info, I'm trying to reproduce and checking the code. There were similar WARN_ON s some years ago and these WARN_ON was removed by commit 2788cf0c401c allowing nr_items to become a wrong value, but as that commit message mentioned, that should not be a problem. I added these back because the new lock_list_lru_of_memcg should ensure a stable list_lru, so they might help catch wrong usage. There could be some corner cases or synchronization issues that are not well considered for these sanity checks, I'm looking at it. An bold fix is just remove these WARN_ON as such wrong values might not be harmful. I'll do more checks and tests locally and report back. > -- > Thanks, > Sasha