From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C013FC0219D for ; Thu, 13 Feb 2025 14:38:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 032BE6B007B; Thu, 13 Feb 2025 09:38:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7D0A6B0082; Thu, 13 Feb 2025 09:37:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D44126B0083; Thu, 13 Feb 2025 09:37:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B6E016B007B for ; Thu, 13 Feb 2025 09:37:59 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 753BF141039 for ; Thu, 13 Feb 2025 14:37:59 +0000 (UTC) X-FDA: 83115175878.05.1EEE94D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 0D1DD40013 for ; Thu, 13 Feb 2025 14:37:56 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B8ZcgXgq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of gmonaco@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=gmonaco@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739457477; a=rsa-sha256; cv=none; b=Giw2rf+rpSRDiFBUMeAZDqNG9ZHRPE/qEEqY87dIbBTz6uwMPc/Zu3IPRWwuZRiGA9LOFx J1tIQeU3Ue4x4TARYV7E2jqyhdu9Wb2jflG0ru6VgsuzJPilTezToB1dZ+318fk6/SMMdI HqdArVJ+Y7qljLgbq4HB3TGjztX7jP4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B8ZcgXgq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of gmonaco@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=gmonaco@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739457477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QMsIkRsLOsVXj850WlfngMWu2xehoAAjf9EtoK8GX7I=; b=XwaYGazrRme+CKBIoRNuJg3xcZtViJdGVrorVaMmb4U+A2kkQMKaX4wq2LDgLzy9xlM7T+ rwF3Tizp9aY26oi0NaBp115wG2wQ4MPdmLSZurnx68xN2U0MAzyOXWWpjCC95ueW/w8z5A NdabtgD3SbWFEyunyaJFTEaHhuU1hPA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739457476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=QMsIkRsLOsVXj850WlfngMWu2xehoAAjf9EtoK8GX7I=; b=B8ZcgXgqpvOyLaGsa7I2Q6WTvPTPSD475YRj54MH8m+VZqrTGJqgEvyLL+Zhk2kK0RC3k/ Ux9IJacDUYlS8rxO1BwbY+YQCLZyhZ0+wQ2jToNcf+Rqw1dCNoSnWL9ZVlbGHL/9Y535ZJ 5XxpS2fpqZbYos4HsiRdN9uhX+zroQI= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-583-HSJjeORaP8iUzOJqmv_IWA-1; Thu, 13 Feb 2025 09:37:51 -0500 X-MC-Unique: HSJjeORaP8iUzOJqmv_IWA-1 X-Mimecast-MFC-AGG-ID: HSJjeORaP8iUzOJqmv_IWA Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-38dc88ed9c0so540536f8f.1 for ; Thu, 13 Feb 2025 06:37:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739457470; x=1740062270; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QMsIkRsLOsVXj850WlfngMWu2xehoAAjf9EtoK8GX7I=; b=XgcsdWYu/011kZ3YauV7cDcYUKEgeNVEwDHMJXw3NrjwITxqxvICdGDJU2GzepwX8G yV9Wj/SyEiYceKOf178r5pOTe2EMFDzZMOFJNo3r+cTaFm+1YMnwzF0cfWrvashOhzKO 4AKcdl3JIclWKj6854Esl7OmODIfo/UQGAO25Kbxcg4tyK6KzE2wNRIUHyYLS20ON7dE +l2wiluElwpX/UKQetFT5DubyYX66WCWf0altsOHEqVNNk0vZl0Fmz7BwQVlE3qVgLn+ XdkpB4y58X6dmB/97SAijyg2YF4oIbdasohuPWHjFVQcG3KYpY+UQZ3Yk3aaCw+gusjd gtOA== X-Gm-Message-State: AOJu0Ywx78dSjqxmZLxOIpVtsAezKpVtVSKNpSaHDpxT1MFSMZcNDxe6 ZPJpj0hbOeoTRe6wf0aSMsF1Sq4DnVI75ywPP+ukqWndZhKYv1jUJvo8d19p5QA2eyrRDbL0lVG /r33uSrYqiaSuMGxYmTvf0a1a1uLGZ1c+skgN/5Z9fA2zB82J X-Gm-Gg: ASbGncsy/CTofk5EN6sSENFYun+KlUgmHVWaLjm7V8EuIEMUilTa2SuzsBRzliXz3po BPS5JFhcSW6/RfaCrVpZ1G5OYoh6T+5EUh/1pAJ6zgzwKecko5+ifnFKA6OZI8gyMyRNZBPRi4O naQrsqq7nJyTNOxiPNGiKt+mNtI/8KlrakrRWsJSYL8h5LkMD97i+dbXnib42O5WgZix5xN7Tbt pGWbpCggNQjGocgFSfxF2I9+QT8Zc2HU6WUoMGCYlbpe4AUg1MuvClT1gD3ijldgTDFlLUC3q/n Hf1P6cT3g+9Mb1oy0hH+eCUtbdwEXCg= X-Received: by 2002:a5d:648f:0:b0:38f:277a:4eb3 with SMTP id ffacd0b85a97d-38f277a5258mr2651023f8f.8.1739457470387; Thu, 13 Feb 2025 06:37:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IHDTqe4qsYtkDP00ujYm7DVHNEfDKKsf9UNHm5PPWdZ5xhGE4IKSeM3/RqB5gJfr7CQu4AvKA== X-Received: by 2002:a5d:648f:0:b0:38f:277a:4eb3 with SMTP id ffacd0b85a97d-38f277a5258mr2650985f8f.8.1739457469974; Thu, 13 Feb 2025 06:37:49 -0800 (PST) Received: from gmonaco-thinkpadt14gen3.rmtit.csb ([185.107.56.42]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38f258b44a7sm2119604f8f.12.2025.02.13.06.37.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Feb 2025 06:37:49 -0800 (PST) Message-ID: <35fe8e74229af24f45954dd27789363dd5c2f8b8.camel@redhat.com> Subject: Re: [PATCH v6 2/3] sched: Move task_mm_cid_work to mm delayed work From: Gabriele Monaco To: Mathieu Desnoyers , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, aubrey.li@linux.intel.com, yu.c.chen@intel.com, Andrew Morton , Ingo Molnar , Peter Zijlstra , Ingo Molnar , "Paul E. McKenney" , Shuah Khan Date: Thu, 13 Feb 2025 15:37:47 +0100 In-Reply-To: <1a295a1e-08da-4684-81be-9539773a1c94@efficios.com> References: <202502131405.1ba0803f-lkp@intel.com> <17bda9071b6962414f61668698fa840501819172.camel@redhat.com> <1a295a1e-08da-4684-81be-9539773a1c94@efficios.com> Autocrypt: addr=gmonaco@redhat.com; prefer-encrypt=mutual; keydata=mDMEZuK5YxYJKwYBBAHaRw8BAQdAmJ3dM9Sz6/Hodu33Qrf8QH2bNeNbOikqYtxWFLVm0 1a0JEdhYnJpZWxlIE1vbmFjbyA8Z21vbmFjb0ByZWRoYXQuY29tPoiZBBMWCgBBFiEEysoR+AuB3R Zwp6j270psSVh4TfIFAmbiuWMCGwMFCQWjmoAFCwkIBwICIgIGFQoJCAsCBBYCAwECHgcCF4AACgk Q70psSVh4TfJzZgD/TXjnqCyqaZH/Y2w+YVbvm93WX2eqBqiVZ6VEjTuGNs8A/iPrKbzdWC7AicnK xyhmqeUWOzFx5P43S1E1dhsrLWgP User-Agent: Evolution 3.54.3 (3.54.3-1.fc41) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: zXSSvklNveH-NJ2bqGTd2y36N4cUpRoU-tDG4xiA5ZY_1739457470 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 0D1DD40013 X-Rspamd-Server: rspam12 X-Stat-Signature: junffg1phbuzebyj4kcwh9er39mqwg3f X-HE-Tag: 1739457476-186912 X-HE-Meta: U2FsdGVkX18yM1ZLSyt2hymzAVqzRUjWJNjZ7HaO3lE9RxRG+WZ/E+t5d/j0CQMAXVm32WoSpJ6G44unT4hFrS35yZRcVYiT5zycZhEgrjnztA3ss4hVykl1gutzlkE9thJyGaiFB+TGXc53qYrz5YAC/3RbZRE3SkNBDhBW9SkZ7S8rnDOYpjP8HX3AwQHHYZN+R7KOISrbjXku2AtkdkmZ6F63EgefaUXUsfvzx8JJ0QSJ2XAo+H+GR78ndbogW51uW3Y4wqn6fAApkR/2CipqaYPRf/yk2omkIF1BRRpyz3NiPvo4rFw8XFRBMde9iqNameiGAtSdg3eClivEb7D+Jeyd+6G/zas7So4WRMq2EDjhDb+WBWzeXX/fXrBk4XRvCtn9BdUi4/vd1XNHrzCoWGhFcbmU6BL1xNmrHpTXJyTKIrSNr36skV+HPsmCCmtgcbsuSId5lJukwFqQxL5W4wzs1tsuISBrdR6iJDA1ryhS6si1zDM3Bk1vF5qoE/Wa9cHiBpLpQQr+uBhFUynWySwvb6C3CDqZXnmGT4EneBkb2zMCo9oZUtfBQ5ouNnM0lyqrkFqXkRaMOBDzpfV3V3G5sGXUEbY+M1npgNHkwOSMBxKYgIkk9xLRPTsK3Rj+rsHjCKVqbBGnLP1gI2IYdzjQjdo3fV84FN9Cfg5kGYxcTWa6TjO63rVVFGNX9HRvD/H1G86Xhycm5g5ltKoznnO3ik4UJvyqH7sI/SoNpQvQ3rz1/8UDli0dzG52uSaRgQ9/uOrT+R1eh1ckcRPgH55RC6hHoy5/OYSN8ZcGIK8QFrUHIsvw1PJSk0MKgYKbowkRlSOdwq9cW8Q3lw6ZZ15Iclvrt036O0aByepgTZdUlOzwqrbR/qozl/dzvzCrrhC8k7IUoD1wqu95KHabMotEe3Y9QfPOMrFrxtsJYyN5LZSn5mo+0baQekS7L6Vze8DLUckJ4fZPyLU uAuhpv/P +Z7Po8I7nUiyrmSN2sZ4B4Gm/k/Cj3I0G3nceHMWs/ZIs/6HgD8dNhkfWxC/kDWGO0hezoQeDsO1jwNJbvflqKVr4AYGcCYoJ+oBWck+zdH/X2bhMHKOSU5QmaqMmqJl6fjcR0cGRnUINvCgwfXpjjo1Jvuk/yHVFEOLs0n1rd8wjxNkAT3JfLnXNWd6m47m0HL+bTwlM4x/3vL6wXqHisJzSoln1jrf0aP3qahUJ/N0ihOQpOVmAP/6y9eio+lIZxMvJF9wzJq9THeUdbO5xdBnVtbEf2I+gCMPMFhdc62WBOlisEoqmgSlDxjuYG88ytHyUZNlNH2G5qbhD7S35fp7i02r44UANIpdCL7Licfs2250aTr2iUAlxGPQKrPG19SxL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 2025-02-13 at 08:55 -0500, Mathieu Desnoyers wrote: > On 2025-02-13 08:25, Gabriele Monaco wrote: > > On Thu, 2025-02-13 at 14:52 +0800, kernel test robot wrote: > > > kernel test robot noticed > > > "WARNING:at_kernel/workqueue.c:#__queue_delayed_work" on: > > >=20 > > > [=C2=A0=C2=A0=C2=A0 2.640924][=C2=A0=C2=A0=C2=A0 T0] ------------[ cu= t here ]------------ > > > [ 2.641646][ T0] WARNING: CPU: 0 PID: 0 at > > > kernel/workqueue.c:2495 > > > __queue_delayed_work (kernel/workqueue.c:2495 (discriminator 9)) > > > [=C2=A0=C2=A0=C2=A0 2.642874][=C2=A0=C2=A0=C2=A0 T0] Modules linked i= n: > > > [=C2=A0=C2=A0=C2=A0 2.643381][=C2=A0=C2=A0=C2=A0 T0] CPU: 0 UID: 0 PI= D: 0 Comm: swapper Not > > > tainted > > > 6.14.0-rc2-00002-g287adf9e9c1f #1 > > > [=C2=A0=C2=A0=C2=A0 2.644582][=C2=A0=C2=A0=C2=A0 T0] Hardware name: Q= EMU Standard PC (i440FX + > > > PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 > > > [ 2.645943][ T0] RIP: 0010:__queue_delayed_work > > > (kernel/workqueue.c:2495 (discriminator 9)) > >=20 > > There seem to be major problems with this configuration, I'm trying > > to > > understand what's wrong but, for the time being, this patchset is > > not > > ready for inclusion. >=20 > I think there is an issue with the order of init functions at boot. >=20 > poking_init() calls mm_alloc(), which ends up calling mm_init(). >=20 > The WARN_ON() is about a NULL wq pointer, which I suspect happens > if poking_init() is called before workqueue_init_early(), which > allocates system_wq. >=20 > Indeed, in start_kernel(), poking_init() is called before > workqueue_init_early(). >=20 > I'm not sure what are the init order dependencies across subsystems > here. > There is the following order in start_kernel(): >=20 > [...] > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mm_core_init(); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 poking_init(); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ftrace_init(); >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* trace_printk can be e= nabled here */ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 early_trace_init(); >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Set up the sched= uler prior starting any interrupts (such > as the > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * timer interrupt)= . Full topology setup happens at > smp_init() > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * time - but meanw= hile we still have a functioning > scheduler. > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sched_init(); >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (WARN(!irqs_disabled(= ), > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "Interrupts were enabled *very* early, fixin= g > it\n")) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 local_irq_disable(); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 radix_tree_init(); > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 maple_tree_init(); >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Set up housekeep= ing before setting up workqueues to allow > the unbound > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * workqueue to tak= e non-housekeeping into account. > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 housekeeping_init(); >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Allow workqueue = creation and work item > queueing/cancelling > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * early.=C2=A0 Wor= k item execution depends on kthreads and > starts after > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * workqueue_init()= . > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 workqueue_init_early(); > [...] >=20 > So either we find a way to reorder this, or we make sure > poking_init() > does not require the workqueue. >=20 > Thanks, >=20 > Mathieu >=20 Nice suggestion! That seems the culprit.. >From the full dmesg of the failure I've seen there's also a problem with disabling the delayed work synchronously, since mmdrop cannot sleep if we are not in PREEMPT_RT. I'm trying to come with some satisfactory solution for both, ideally: 1. the delayed work is not needed in early boot, we may have a better place where to start it 2. we can cancel the work asynchronously on mmdrop and abort it if the pcpu_cid is null, but it seems racy, perhaps there's a better place for that too Thanks, Gabriele