From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D618BE77188 for ; Sat, 4 Jan 2025 04:00:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D288B6B0082; Fri, 3 Jan 2025 23:00:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD85E6B0088; Fri, 3 Jan 2025 23:00:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B78FC6B0089; Fri, 3 Jan 2025 23:00:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 959806B0082 for ; Fri, 3 Jan 2025 23:00:27 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 17706AFA29 for ; Sat, 4 Jan 2025 04:00:27 +0000 (UTC) X-FDA: 82968417294.08.43495B7 Received: from smtp-relay-internal-0.canonical.com (smtp-relay-internal-0.canonical.com [185.125.188.122]) by imf21.hostedemail.com (Postfix) with ESMTP id C62501C0006 for ; Sat, 4 Jan 2025 04:00:24 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b="nxoZS/sC"; dmarc=pass (policy=none) header.from=canonical.com; spf=pass (imf21.hostedemail.com: domain of koichiro.den@canonical.com designates 185.125.188.122 as permitted sender) smtp.mailfrom=koichiro.den@canonical.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735963225; a=rsa-sha256; cv=none; b=1bB9vCBANobAMoDkIJVXON0rThGAFR93bfRc20d2JrehCmmBoNGLop3iu2F5F8oWp+bKW7 xMafyxUg2B4XeUogMt6Ry1DwKwRv+hoUKXeFedU7Szei10hJtA89N9MndZ8JBoNpsRJoVT 799akhg7MxvlJfwJ+VKFvaIkivhCpAk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b="nxoZS/sC"; dmarc=pass (policy=none) header.from=canonical.com; spf=pass (imf21.hostedemail.com: domain of koichiro.den@canonical.com designates 185.125.188.122 as permitted sender) smtp.mailfrom=koichiro.den@canonical.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735963225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=En56cyjnsFkndGuih8IbN4845mhQel3SMwCtQplkdxY=; b=qH3+96VUle1wSBfTMmkZe+dQ2IwGlWovjDXZIDTArOK8HpApzyICjW+JRPs/c/Se2bv612 T1ieFbp6UbithRmGh7ExFs9SDaYdERevnscib+v6u571TG+SP86nZQyg4O6g1/vXeWQXl6 EJFLBST7V/vyRWyxQih859vKDVEqDzY= Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 24A333F689 for ; Sat, 4 Jan 2025 04:00:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1735963221; bh=En56cyjnsFkndGuih8IbN4845mhQel3SMwCtQplkdxY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=nxoZS/sCBzfBJQG4DKwKRcsg8EjjDe7rWrzwv1Wr3GvVv4Sg+SGsqPQRGQAKlgQ6B c4uOCF9MiQDgEhSqyhNdEKkSm3D2Pxfn1iylhToRZrYCUWij5nUZX2fR+dPG8A7C3h 0eSFRGYjUbVI6r/3nrJZ2DDmdMYWTJmaXff3pH3c5UUCGrFcKxuohqTUYxMqkJADQ/ Y8woEAr+3Mf/Huy1oo0Yt5taexnaOeED0fWOo9Eeh1U6SfNxvNfxD22uTS3O9ANeok PKZV9xng7JXvphjKj4JSDeTmI4tcAJ8+Dp5gwWWpVPRx6x4rcXvX9XpYVx6TlX4WxA Nd1nhVbYVY9/A== Received: by mail-pl1-f198.google.com with SMTP id d9443c01a7336-21631cbf87dso156938775ad.3 for ; Fri, 03 Jan 2025 20:00:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735963219; x=1736568019; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=En56cyjnsFkndGuih8IbN4845mhQel3SMwCtQplkdxY=; b=sN32JkA+Z5A8H18C0rwvtbjsEbk3VPKpjrHbkMYzid4So6YSBltdI+xLMg7k6jIj5L zfC+/OfewPwqk2wd1ozJAU8Lz0vSGcfV0GkL4069h+picbu2rVDzaKq4fkRU7h03YlLE DKFU2NSo4RX2/RfnymCOO791G34IXgP1oUMOe+LHrq2dXF/IbTcrRr1pqKGzXZhmo9mj u+9ME6el/hLcTKGC4hhbsThjJ7qb5lFZ7+8y74N/veSs+H8/4bLW9OG0w9JVkoD2rIDL iVpT0UEUiNXkafkNu44WL/e8GgHkv1Gv/HU033zFiqQdWuj6QFxbj2LyEqgDU6MF28X/ 5j3w== X-Gm-Message-State: AOJu0YyXhdRamA63W6YWc/9AXx35fnDaR0ytuUh+8NJhjzjZ8UCPo5S0 QgjXqiJFEuXqCVT0B+DnetvHXOAx8QJqOf8koyH+mjYzd9mOD/mvKh9wc5lTmNrew/qhzOIIlar cj9By7/s/WFNWjhUf/Ldx+q1apDRejMOEIXpbWwNrrvDYJLNM/bsWmHSaZCetc9J1 X-Gm-Gg: ASbGncvjnm5uIb9jsWJSAfhJaNUFEiS6bBqo6QusC9doioXTMgjhcaWvUyhKqJSauaj qfWjXeme8P0U6QwWnx+55XIaSB7vV8PqW+bxhKWvu9T4JAg0AjNtld7bFsUCV3077wSss8dDekI hTBFyMfBdzeGdNw9IoCeAhV6v/tA7zRw1xwWr+3YjdFqE+LsranWmNN8Ww4pHp0Ex1hRIAjYXBe tuX3HSTzkN82uIRT5mE1T0+FGuCo2UftALqK8vuBJefJatE4ppVAXNwOg== X-Received: by 2002:a17:903:41c1:b0:215:6426:30a5 with SMTP id d9443c01a7336-219e70c01f7mr804432555ad.40.1735963219294; Fri, 03 Jan 2025 20:00:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IEyIdDeczagLwS6XQkKw51dtsFsFroQrIQ/o6teITFoj+y2QmbWlWJMT4gRqVHHcB5Hy8AmKg== X-Received: by 2002:a17:903:41c1:b0:215:6426:30a5 with SMTP id d9443c01a7336-219e70c01f7mr804432295ad.40.1735963218928; Fri, 03 Jan 2025 20:00:18 -0800 (PST) Received: from localhost ([240f:74:7be:1:344b:9a3:23c2:a577]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc964a73sm252065195ad.45.2025.01.03.20.00.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2025 20:00:18 -0800 (PST) Date: Sat, 4 Jan 2025 13:00:17 +0900 From: Koichiro Den To: Lorenzo Stoakes Cc: linux-mm@kvack.org, akpm@linux-foundation.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] vmstat: disable vmstat_work on vmstat_cpu_down_prep() Message-ID: <2q7ge6cgzeowqffyn6w6ed4trhaaumv5ubdgud2tsoolen7wpw@4akuomhbacyh> References: <20241221033321.4154409-1-koichiro.den@canonical.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C62501C0006 X-Stat-Signature: d3hiz3wx1tqu6dotaanfbeefxqu6dhkz X-Rspam-User: X-HE-Tag: 1735963224-281342 X-HE-Meta: U2FsdGVkX18ANaGcjCMtTAE49hzAeMOcw3dnYt6/+BeuQiXyxMFTwlBXh3+xyFFFqQgT2lDidXqdFS5HBz+3XaoZ/6BsXRQAAPcCrkvRaEfGIzxK+ycf70GGeN5dDrKsP8AvqUdDJPG/twlEIkPkAI6AcUAUaMv2Ka0ZA+sq6olBzLQrFMhcFhV2cFbv2zEm6y9yfynZonjhOEBPHZ358ONuiaON9XsFE05wdgSsUaEgGQt2i9VOL4Y0fNz9kceOCuYPHvF6SyHFLTqfHQEsmtaGCmTCQNJMjhLsxBcVLbmNAEn213kbrBgpkq6M4xd/BDwGSIFxE0ZtlluCvD2iHXmF/TRBTtNHp/wnnDgy0ogstG2LNPS5cpUx45G9ugYB6tDpMsk+T8iG6eKY2ngqN1KplwqT5m7onYmJ6NEVX/DohIWoRMlm8LYG0h4lrxwnNZ8DWYy2DK3q3qVk3op/TPglEqT9e+EP0TiCijkFb3clSZvBRXZbUXk6k6paiqg2Tp8h6rcJXjTukMRovUdX+dLvT7NsIGfbed7s2V85APQh3wpcRgCLZbdk9UAiomFRespVTFfXCPOGReFr/CW7DhUqmUZzim66BbPTkB3Oq1R2itbxL/pYLX/VygPmBwUmV3WoSFk+yVVBltQeAWqWfoaAqG0/D0FkoGNuh8zzGEtv9KsMNOvqaylgZ0kUelYt4hh8IaUcfY97+uqFYX4YoDq67h+0zPcsV77ZGVXORkeL2o7PU6WA/O7wHIJIeGy2FnBut7IuCOViJUqeLjr9+aa9yrfea2qxa8pL/2Sk9CULZnv2K3wwgTzXZn1KKn47sw0KG1+if/vLWcjWXYzr9l+qLS5q4eJaIJh1kOZ63pz5Np11Behu+YvOtupxqUp1wERzdba53Y5BhOHPVydPedbyPaFXlmgh7RChCGs19M0ILH3kJb0jvDs7c5MFgIJh37reAi7aq7WSYtgz7Yz mbyrdXtE Hd8zhnQ5xzlFz9JuUHvZyH9NOU3OWdSDcH/ZhKWEbTbeMyzrW2te31FZj/dxrI1Axd1eqiRRhO4MkXYLVTw+k7+POGGKk/vskpZ2c3s0JvSPyx32Xx53RA48VfAKZdOk2uPkDuxIAdSpASfAo9mdudojO0R/XzwJ3vXsS3D4CGmQsjbOeqTXZqungFxDR5qz5v4GKoYXb/8VOZbUajiahNatmjv+eNK6J52bSOKCKvMpEK6vpIQv02rYHKcSgzcBVJA4Lic8c+5Uw8r3FQB9QIJbqr0XiaIr7d+msb7MDDKVugyCBJwS/G4nhpWlk4hBnWl74ZFzB0nAaOu2LBqONxsawS2dUrnhHHQUDT9Arr+msDdlOVsRT/DaTMVmEEz7W6983JC1QwavUrSqehTo6uPOUqBvIeMybu7KqTgnrtV1LbbJNNFB5M5fnMHTn1w4HYiIePa4xBQ4FpiHTyt5w9ZfRC8qQ9iFh0AapYaBVEUof1pSH9xROot6zMOn59KOfutCb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 03, 2025 at 11:33:19PM +0000, Lorenzo Stoakes wrote: > On Sat, Dec 21, 2024 at 12:33:20PM +0900, Koichiro Den wrote: > > Even after mm/vmstat:online teardown, shepherd may still queue work for > > the dying cpu until the cpu is removed from online mask. While it's > > quite rare, this means that after unbind_workers() unbinds a per-cpu > > kworker, it potentially runs vmstat_update for the dying CPU on an > > irrelevant cpu before entering atomic AP states. > > When CONFIG_DEBUG_PREEMPT=y, it results in the following error with the > > backtrace. > > > > BUG: using smp_processor_id() in preemptible [00000000] code: \ > > kworker/7:3/1702 > > caller is refresh_cpu_vm_stats+0x235/0x5f0 > > CPU: 0 UID: 0 PID: 1702 Comm: kworker/7:3 Tainted: G > > Tainted: [N]=TEST > > Workqueue: mm_percpu_wq vmstat_update > > Call Trace: > > > > dump_stack_lvl+0x8d/0xb0 > > check_preemption_disabled+0xce/0xe0 > > refresh_cpu_vm_stats+0x235/0x5f0 > > vmstat_update+0x17/0xa0 > > process_one_work+0x869/0x1aa0 > > worker_thread+0x5e5/0x1100 > > kthread+0x29e/0x380 > > ret_from_fork+0x2d/0x70 > > ret_from_fork_asm+0x1a/0x30 > > > > > > So, for mm/vmstat:online, disable vmstat_work reliably on teardown and > > symmetrically enable it on startup. > > > > Signed-off-by: Koichiro Den > > Hi, > > I observed a warning in my qemu and real hardware, which I bisected to this commit: > > [ 0.087733] ------------[ cut here ]------------ > [ 0.087733] workqueue: work disable count underflowed > [ 0.087733] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4313 enable_work+0xb5/0xc0 > > This is: > > static void work_offqd_enable(struct work_offq_data *offqd) > { > if (likely(offqd->disable > 0)) > offqd->disable--; > else > WARN_ONCE(true, "workqueue: work disable count underflowed\n"); <-- this line > } > > So (based on this code) presumably an enable is only required if previously > disabled, and this code is being called on startup unconditionally without > the work having been disabled previously? I'm not hugely familiar with > delayed workqueue implementation details. > > [ 0.087733] Modules linked in: > [ 0.087733] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc4+ #58 > [ 0.087733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.3-1-1 04/01/2014 > [ 0.087733] RIP: 0010:enable_work+0xb5/0xc0 > [ 0.087733] Code: 6f b8 01 00 74 0f 31 d2 be 01 00 00 00 eb b5 90 0f 0b 90 eb ca c6 05 60 6f b8 01 01 90 48 c7 c7 b0 a9 6e 82 e8 4c a4 fd ff 90 <0f> 0b 90 90 eb d6 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 > [ 0.087733] RSP: 0018:ffffc900000cbe30 EFLAGS: 00010092 > [ 0.087733] RAX: 0000000000000029 RBX: ffff888263ca9d60 RCX: 0000000000000000 > [ 0.087733] RDX: 0000000000000001 RSI: ffffc900000cbce8 RDI: 0000000000000001 > [ 0.087733] RBP: ffffc900000cbe30 R08: 00000000ffffdfff R09: ffffffff82b12f08 > [ 0.087733] R10: 0000000000000003 R11: 0000000000000002 R12: 00000000000000c4 > [ 0.087733] R13: ffffffff81278d90 R14: 0000000000000000 R15: ffff888263c9c648 > [ 0.087733] FS: 0000000000000000(0000) GS:ffff888263c80000(0000) knlGS:0000000000000000 > [ 0.087733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 0.087733] CR2: 0000000000000000 CR3: 0000000002a2e000 CR4: 0000000000750ef0 > [ 0.087733] PKRU: 55555554 > [ 0.087733] Call Trace: > [ 0.087733] > [ 0.087733] ? enable_work+0xb5/0xc0 > [ 0.087733] ? __warn.cold+0x93/0xf2 > [ 0.087733] ? enable_work+0xb5/0xc0 > [ 0.087733] ? report_bug+0xff/0x140 > [ 0.087733] ? handle_bug+0x54/0x90 > [ 0.087733] ? exc_invalid_op+0x17/0x70 > [ 0.087733] ? asm_exc_invalid_op+0x1a/0x20 > [ 0.087733] ? __pfx_vmstat_cpu_online+0x10/0x10 > [ 0.087733] ? enable_work+0xb5/0xc0 > [ 0.087733] vmstat_cpu_online+0x5c/0x70 > [ 0.087733] cpuhp_invoke_callback+0x133/0x440 > [ 0.087733] cpuhp_thread_fun+0x95/0x150 > [ 0.087733] smpboot_thread_fn+0xd5/0x1d0 > [ 0.087734] ? __pfx_smpboot_thread_fn+0x10/0x10 > [ 0.087735] kthread+0xc8/0xf0 > [ 0.087737] ? __pfx_kthread+0x10/0x10 > [ 0.087738] ret_from_fork+0x2c/0x50 > [ 0.087739] ? __pfx_kthread+0x10/0x10 > [ 0.087740] ret_from_fork_asm+0x1a/0x30 > [ 0.087742] > [ 0.087742] ---[ end trace 0000000000000000 ]--- > > > > --- > > v1: https://lore.kernel.org/all/20241220134234.3809621-1-koichiro.den@canonical.com/ > > --- > > mm/vmstat.c | 3 ++- > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/mm/vmstat.c b/mm/vmstat.c > > index 4d016314a56c..0889b75cef14 100644 > > --- a/mm/vmstat.c > > +++ b/mm/vmstat.c > > @@ -2148,13 +2148,14 @@ static int vmstat_cpu_online(unsigned int cpu) > > if (!node_state(cpu_to_node(cpu), N_CPU)) { > > node_set_state(cpu_to_node(cpu), N_CPU); > > } > > + enable_delayed_work(&per_cpu(vmstat_work, cpu)); > > Probably needs to be 'if disabled' here, as this is invoked on normal > startup when the work won't have been disabled? > > Had a brief look at code and couldn't see how that could be done > however... and one would need to be careful about races... Tricky! > > > > > return 0; > > } > > > > static int vmstat_cpu_down_prep(unsigned int cpu) > > { > > - cancel_delayed_work_sync(&per_cpu(vmstat_work, cpu)); > > + disable_delayed_work_sync(&per_cpu(vmstat_work, cpu)); > > return 0; > > } > > > > -- > > 2.43.0 > > > > > > Let me know if you need any more details, .config etc. > > I noticed this warning on a real box too (in both cases running akpm's > mm-unstable branch), FWIW. Thank you for the report. I was able to reproduce the warning and now wonder how I missed it.. My oversight, apologies. In my current view, the simplest solution would be to make sure a local vmstat_work is disabled until vmstat_cpu_online() runs for the cpu, even during boot-up. The following patch suppresses the warning: diff --git a/mm/vmstat.c b/mm/vmstat.c index 0889b75cef14..19ceed5d34bf 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -2122,10 +2122,14 @@ static void __init start_shepherd_timer(void) { int cpu; - for_each_possible_cpu(cpu) + for_each_possible_cpu(cpu) { INIT_DEFERRABLE_WORK(per_cpu_ptr(&vmstat_work, cpu), vmstat_update); + /* will be enabled on vmstat_cpu_online */ + disable_delayed_work_sync(&per_cpu(vmstat_work, cpu)); + } + schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } If you think of a better solution later, please let me know. Otherwise, I'll submit a follow-up fix patch with the above diff. Thanks. -Koichiro