From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 842DAE77197 for ; Mon, 6 Jan 2025 02:18:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 185496B0092; Sun, 5 Jan 2025 21:18:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 134716B0093; Sun, 5 Jan 2025 21:18:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3DF36B0095; Sun, 5 Jan 2025 21:18:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D67BB6B0092 for ; Sun, 5 Jan 2025 21:18:11 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6210344F4E for ; Mon, 6 Jan 2025 02:18:11 +0000 (UTC) X-FDA: 82975417182.28.3303392 Received: from mout01.posteo.de (mout01.posteo.de [185.67.36.65]) by imf19.hostedemail.com (Postfix) with ESMTP id 448EE1A000E for ; Mon, 6 Jan 2025 02:18:09 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=posteo.net header.s=2017 header.b=iuxMRQl7; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (imf19.hostedemail.com: domain of charmitro@posteo.net designates 185.67.36.65 as permitted sender) smtp.mailfrom=charmitro@posteo.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736129889; a=rsa-sha256; cv=none; b=Qh2ShUwdXToVUXJ+0Zcoxz8q03JCpx9mKuj20cr5UOsJNA7ASy2dpW35e9+jursAm69rXB gy5SUYqe0y2r/5IVhbEqHbW5nATUFCGcal8+u7eqODk8AlVmqZ5xUIfhq7cTF5YgQTtMIx +GhUiWdexeiWGBSucc0QDcYCWDyKG+w= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=posteo.net header.s=2017 header.b=iuxMRQl7; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (imf19.hostedemail.com: domain of charmitro@posteo.net designates 185.67.36.65 as permitted sender) smtp.mailfrom=charmitro@posteo.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736129889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bPSyNstbMUAqKIIXNEqwhrBYFBWIgfbU2LDlnWojdug=; b=MOdBiqzHH+vOzOR47m3hMIUrjEuKePcyJDIogk1TeMFH/8ZpPSFGIIIAAHnumeosEXWcxt 3P1Wbk8jAJLleR2rYgW/AY47sqhY2o9a28WgDvbjqlwk8iOR52s8KgipPG5RV6PjGL5C94 AuslZr+psjeeGb0qO0aFrBrMo4t7c3w= Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id E09EE240027 for ; Mon, 6 Jan 2025 03:18:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1736129886; bh=xI9wBPRL1ZswrBHSrh4zl8QeXnnZbiq6TgczobAPdd4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: From; b=iuxMRQl7fFMT8rnL/N/9+/cQel7FF5tEsvViraBxpV05uRdbjS3AnvA9o1M2Uh+9r /tqMSe9czgpDYkrsf9rEQdy7j9DHuzIWc2x55M04kQchrJLsDdBHftMmDRotIilCTx MjFF5C9a46CdnwWdwBr3n65KUnSs7ddvQQwhVotBQWW79EatkY2kccuQHSP+iiLE8i l0Gklf4hK8PAW9ygmaof8VeOpm/5HIesprt9yqk+bCctfjGS0WHlnvYB6p9wQvodof I86D1USzeIo7jA7EFsJyfz6vUr9pIH+iXxTdD2BZxzUffDVwREnTaPKTHV30DN2/7J 8Cm6RV5f4zMxw== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4YRHs93tHsz6txx; Mon, 6 Jan 2025 03:18:05 +0100 (CET) From: Charalampos Mitrodimas To: Koichiro Den Cc: Lorenzo Stoakes , linux-mm@kvack.org, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] vmstat: disable vmstat_work on vmstat_cpu_down_prep() In-Reply-To: <2q7ge6cgzeowqffyn6w6ed4trhaaumv5ubdgud2tsoolen7wpw@4akuomhbacyh> (Koichiro Den's message of "Sat, 4 Jan 2025 13:00:17 +0900") References: <20241221033321.4154409-1-koichiro.den@canonical.com> <2q7ge6cgzeowqffyn6w6ed4trhaaumv5ubdgud2tsoolen7wpw@4akuomhbacyh> Date: Mon, 06 Jan 2025 02:18:04 +0000 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Stat-Signature: 5yaiwiwx4ng5cmgch3krf47gzk1iq69s X-Rspam-User: X-Rspamd-Queue-Id: 448EE1A000E X-Rspamd-Server: rspam08 X-HE-Tag: 1736129889-883107 X-HE-Meta: U2FsdGVkX18xy1+mBmCNP0ixQaC3CBV2WWLNJ6kmBOmJsLyYKI7mnV0MlPzB7hzkGN9JHRB5r/SSFh4q5Slf05xednI1Pl6ET4Ggn02X2gQPcJ50JQTyxH/qXoAM3wdDxDZhc1LVc4M2RllEzCbaPM6kPhpDuVVQmr8UwBETpNccR+l2ko071cwcx65vUJicJ99ndmJMrOlqnafhNUAsk/VhgPXBUvkd2XLWQXtJ+i4LLGpg0MkYrcOIlLBzT2f6m/z5W/F3z/Fn4sGhCB0b/KHU6g0hY9yDxBv+boUVMlyophsKpUl4MpnJ8YXhyI/JKXVFnHYM0wuLhQtcHaq/u+9NTzC8OPCMAr7NFsD3yvLLwYE1ScVYsw+W2AjBiEtkhaatzC7elwE/Si/JtPs7791MoKtUU3jMQILl9YksH3m8CfTKqZyAFp18z+hE9QJuqYIGVs6C3vOotKoyaVMZVXouwqC61gMcL4zcaB+N4wehfF5f5YNU4CghAmEP4VMyJjZ785KDXcq1Jq3CkdQ/mV5xn8u/dR8tmbzNcUOQtOMK4mtPyhdnq+xKD8+KtTKMMrWndfncZIUDWtOSVbdT4ypizIlROIRFfTQnCSuejjsKusoMuf2BgOMMPAszmluT6FiGF3NKEeSxzuKNKw/I4P7Jev3V2Emhu3JkwMDnkITcRO3dCT2bIzRAlnwVEDv3uajXJsiQNiV3qKfiY7vSv9sXDF5pm+pcWGP27h6RV/M2/+n4ZqWiMZpZGlQtCbtoXNaatKdUvoynMgvzOMVMsbJFUJnLDAtyNDwcDxxN4CVIhSR5WsFaPyp4oreh/tNiwj05f/21HiwVhDwUox3GKLLTnUMAof1h4M/Jzbszb/VR/mk2zGIa94BwGQxi6OfsAPnzDbbmHltgeOmH77ylzLRSVlJiFMO7V+J8r4p3dBTo66QjrPW7Y7Vm0OpRBcjkbCsQlEFb9IPJ6Ea+4+t VNfeid5x RdQiHg3jMCV0eaYg/IU1Gf0ovogbpNQ5TsjECp4XYRJRmIy4KhSNzohxvOvV7QxFZ966Boc0w8on1tk4qUd/192Fnp7ZtXE5+y5o0DjqMzh5Fmy4eWqXJr/+PDuyQsEDAl2ehgCiKVMAEH6ZypQ13PWdD/Q53nGH5plzhhbEnK/b8yAodijDxKEW/47MeuDiqdnSJpRTKNkYV9l5TcecmYYKOxViRbdN2Kg/HYnGBeGppQ2aNveFoGhgxDZ/Fz0khO4YyUpXJrR7kk5a8siyFgVvOHnBXarS3GgTlZ9yE6Zzzpc03g8BngJKVq+QpzDmCSLPJ2sSidvKG0juQQHyJVOu9bFmln8xEqretVwuvCJ4DxmNJe6MlM0bPnNO4/cgU3Z7FBXM9oyrna96JV4ealWryNRhD/ZqVVGMunYqjGXe+z2m1pWRKrs62rg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Koichiro Den writes: > On Fri, Jan 03, 2025 at 11:33:19PM +0000, Lorenzo Stoakes wrote: >> On Sat, Dec 21, 2024 at 12:33:20PM +0900, Koichiro Den wrote: >> > Even after mm/vmstat:online teardown, shepherd may still queue work for >> > the dying cpu until the cpu is removed from online mask. While it's >> > quite rare, this means that after unbind_workers() unbinds a per-cpu >> > kworker, it potentially runs vmstat_update for the dying CPU on an >> > irrelevant cpu before entering atomic AP states. >> > When CONFIG_DEBUG_PREEMPT=y, it results in the following error with the >> > backtrace. >> > >> > BUG: using smp_processor_id() in preemptible [00000000] code: \ >> > kworker/7:3/1702 >> > caller is refresh_cpu_vm_stats+0x235/0x5f0 >> > CPU: 0 UID: 0 PID: 1702 Comm: kworker/7:3 Tainted: G >> > Tainted: [N]=TEST >> > Workqueue: mm_percpu_wq vmstat_update >> > Call Trace: >> > >> > dump_stack_lvl+0x8d/0xb0 >> > check_preemption_disabled+0xce/0xe0 >> > refresh_cpu_vm_stats+0x235/0x5f0 >> > vmstat_update+0x17/0xa0 >> > process_one_work+0x869/0x1aa0 >> > worker_thread+0x5e5/0x1100 >> > kthread+0x29e/0x380 >> > ret_from_fork+0x2d/0x70 >> > ret_from_fork_asm+0x1a/0x30 >> > >> > >> > So, for mm/vmstat:online, disable vmstat_work reliably on teardown and >> > symmetrically enable it on startup. >> > >> > Signed-off-by: Koichiro Den >> >> Hi, >> >> I observed a warning in my qemu and real hardware, which I bisected to this commit: >> >> [ 0.087733] ------------[ cut here ]------------ >> [ 0.087733] workqueue: work disable count underflowed >> [ 0.087733] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4313 enable_work+0xb5/0xc0 I also encountered this in my QEMU Debian installation. >> >> This is: >> >> static void work_offqd_enable(struct work_offq_data *offqd) >> { >> if (likely(offqd->disable > 0)) >> offqd->disable--; >> else >> WARN_ONCE(true, "workqueue: work disable count underflowed\n"); <-- this line >> } >> >> So (based on this code) presumably an enable is only required if previously >> disabled, and this code is being called on startup unconditionally without >> the work having been disabled previously? I'm not hugely familiar with >> delayed workqueue implementation details. >> >> [ 0.087733] Modules linked in: >> [ 0.087733] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc4+ #58 >> [ 0.087733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.3-1-1 04/01/2014 >> [ 0.087733] RIP: 0010:enable_work+0xb5/0xc0 >> [ 0.087733] Code: 6f b8 01 00 74 0f 31 d2 be 01 00 00 00 eb b5 90 0f 0b 90 eb ca c6 05 60 6f b8 01 01 90 48 c7 c7 b0 a9 6e 82 e8 4c a4 fd ff 90 <0f> 0b 90 90 eb d6 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 >> [ 0.087733] RSP: 0018:ffffc900000cbe30 EFLAGS: 00010092 >> [ 0.087733] RAX: 0000000000000029 RBX: ffff888263ca9d60 RCX: 0000000000000000 >> [ 0.087733] RDX: 0000000000000001 RSI: ffffc900000cbce8 RDI: 0000000000000001 >> [ 0.087733] RBP: ffffc900000cbe30 R08: 00000000ffffdfff R09: ffffffff82b12f08 >> [ 0.087733] R10: 0000000000000003 R11: 0000000000000002 R12: 00000000000000c4 >> [ 0.087733] R13: ffffffff81278d90 R14: 0000000000000000 R15: ffff888263c9c648 >> [ 0.087733] FS: 0000000000000000(0000) GS:ffff888263c80000(0000) knlGS:0000000000000000 >> [ 0.087733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 0.087733] CR2: 0000000000000000 CR3: 0000000002a2e000 CR4: 0000000000750ef0 >> [ 0.087733] PKRU: 55555554 >> [ 0.087733] Call Trace: >> [ 0.087733] >> [ 0.087733] ? enable_work+0xb5/0xc0 >> [ 0.087733] ? __warn.cold+0x93/0xf2 >> [ 0.087733] ? enable_work+0xb5/0xc0 >> [ 0.087733] ? report_bug+0xff/0x140 >> [ 0.087733] ? handle_bug+0x54/0x90 >> [ 0.087733] ? exc_invalid_op+0x17/0x70 >> [ 0.087733] ? asm_exc_invalid_op+0x1a/0x20 >> [ 0.087733] ? __pfx_vmstat_cpu_online+0x10/0x10 >> [ 0.087733] ? enable_work+0xb5/0xc0 >> [ 0.087733] vmstat_cpu_online+0x5c/0x70 >> [ 0.087733] cpuhp_invoke_callback+0x133/0x440 >> [ 0.087733] cpuhp_thread_fun+0x95/0x150 >> [ 0.087733] smpboot_thread_fn+0xd5/0x1d0 >> [ 0.087734] ? __pfx_smpboot_thread_fn+0x10/0x10 >> [ 0.087735] kthread+0xc8/0xf0 >> [ 0.087737] ? __pfx_kthread+0x10/0x10 >> [ 0.087738] ret_from_fork+0x2c/0x50 >> [ 0.087739] ? __pfx_kthread+0x10/0x10 >> [ 0.087740] ret_from_fork_asm+0x1a/0x30 >> [ 0.087742] >> [ 0.087742] ---[ end trace 0000000000000000 ]--- >> >> >> > --- >> > v1: https://lore.kernel.org/all/20241220134234.3809621-1-koichiro.den@canonical.com/ >> > --- >> > mm/vmstat.c | 3 ++- >> > 1 file changed, 2 insertions(+), 1 deletion(-) >> > >> > diff --git a/mm/vmstat.c b/mm/vmstat.c >> > index 4d016314a56c..0889b75cef14 100644 >> > --- a/mm/vmstat.c >> > +++ b/mm/vmstat.c >> > @@ -2148,13 +2148,14 @@ static int vmstat_cpu_online(unsigned int cpu) >> > if (!node_state(cpu_to_node(cpu), N_CPU)) { >> > node_set_state(cpu_to_node(cpu), N_CPU); >> > } >> > + enable_delayed_work(&per_cpu(vmstat_work, cpu)); >> >> Probably needs to be 'if disabled' here, as this is invoked on normal >> startup when the work won't have been disabled? >> >> Had a brief look at code and couldn't see how that could be done >> however... and one would need to be careful about races... Tricky! >> >> > >> > return 0; >> > } >> > >> > static int vmstat_cpu_down_prep(unsigned int cpu) >> > { >> > - cancel_delayed_work_sync(&per_cpu(vmstat_work, cpu)); >> > + disable_delayed_work_sync(&per_cpu(vmstat_work, cpu)); >> > return 0; >> > } >> > >> > -- >> > 2.43.0 >> > >> > >> >> Let me know if you need any more details, .config etc. >> >> I noticed this warning on a real box too (in both cases running akpm's >> mm-unstable branch), FWIW. > > Thank you for the report. I was able to reproduce the warning and now > wonder how I missed it.. My oversight, apologies. > > In my current view, the simplest solution would be to make sure a local > vmstat_work is disabled until vmstat_cpu_online() runs for the cpu, even > during boot-up. The following patch suppresses the warning: > > diff --git a/mm/vmstat.c b/mm/vmstat.c > index 0889b75cef14..19ceed5d34bf 100644 > --- a/mm/vmstat.c > +++ b/mm/vmstat.c > @@ -2122,10 +2122,14 @@ static void __init start_shepherd_timer(void) > { > int cpu; > > - for_each_possible_cpu(cpu) > + for_each_possible_cpu(cpu) { > INIT_DEFERRABLE_WORK(per_cpu_ptr(&vmstat_work, cpu), > vmstat_update); > > + /* will be enabled on vmstat_cpu_online */ > + disable_delayed_work_sync(&per_cpu(vmstat_work, cpu)); > + } > + > schedule_delayed_work(&shepherd, > round_jiffies_relative(sysctl_stat_interval)); > } > > If you think of a better solution later, please let me know. Otherwise, > I'll submit a follow-up fix patch with the above diff. Can't think of a better solution myself but this fixes the issue. Thanks! > > Thanks. > > -Koichiro C. Mitrodimas