From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79761C46467 for ; Mon, 16 Jan 2023 16:59:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D4626B0073; Mon, 16 Jan 2023 11:59:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 884066B0074; Mon, 16 Jan 2023 11:59:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74B796B0075; Mon, 16 Jan 2023 11:59:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 64CE76B0073 for ; Mon, 16 Jan 2023 11:59:30 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0A3D8AAC8D for ; Mon, 16 Jan 2023 16:59:30 +0000 (UTC) X-FDA: 80361273300.04.9311B21 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 454DA20004 for ; Mon, 16 Jan 2023 16:59:28 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673888368; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5C/pXQj60xjLVCUn07cewonvr7ibeGOwDHeRdhIJopE=; b=B16G0Znr5EC1EnmlLMZuZngyAFPVx+99ByQ0PlSPRthdNv+Css/Wrv0XN7LIsXHEK3NMbt Bf0O8CmdtlJnp5RjTpmkn5KXNdNMDAdsUiLGv5v9zH5HK8EdN7v18H3Y1YRMm5t/T6Omqy 0CrpZIROLc8J1Ws9C8uBWyCV1kEN2RA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673888368; a=rsa-sha256; cv=none; b=aKD/zPYO1u9HeHJx7ArtPAgM2561IhSoQc8NsMXUOOfX80YrAMchmnczIk6tZ7FljDx4Za 3gbjEbLZZGHGbmNaeNeh094DnU7aEntd58YxnBUgQcsHmtQfYVLdYat3U1yfLvdiz+nz/+ 3tmDic1NnJ+NJ/SR20EV5dSiD+cfxY4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30C411570; Mon, 16 Jan 2023 09:00:09 -0800 (PST) Received: from FVFF77S0Q05N.cambridge.arm.com (FVFF77S0Q05N.cambridge.arm.com [10.1.35.162]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5B0403F67D; Mon, 16 Jan 2023 08:59:10 -0800 (PST) Date: Mon, 16 Jan 2023 16:59:04 +0000 From: Mark Rutland To: Peter Zijlstra Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: Re: [PATCH v3 00/51] cpuidle,rcu: Clean up the mess Message-ID: References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230112194314.845371875@infradead.org> X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: nejdkrdcruxbo63sjggixagmmd9tm6pj X-Rspamd-Queue-Id: 454DA20004 X-HE-Tag: 1673888368-991648 X-HE-Meta: U2FsdGVkX1/Fm1Ua9/EoL6IWrOZpWQLSu610dfGMg+m5tkiT5QMsvUkXu70/VeLkEJ6fLKTPTF1HS6DMEQwyNP+hVEUjyDinIe5qXinyMhtU+0uOEHRJxeHB+kFZIafuZKxf9UtPv/ihO+3qw9BieclnMj+A2DkLryC40S7CpvyQ9gDtfqUhjpjsHAh50+aSnYH8fVQhaS0h+2gHRXkS65yi66telHou6AcTUJJzVJ+oj594Gjdm/Gp61Ugq9JLxF3HmZG0KRRp8iDJQ67Z74e247FJC8w1m6AZjfoOTKi29OxFD+m1TK4ByoAT6VkNeFkXTphwO7CHJ2RakCeTlPpsGeDfYl+SPWqszBgl1PmeTrgBs5QlKbCmM0SnalyM4mR8u8BKDqQRPGO08QkdNS+ZHwuGD7SAvJLw4pm4fLdxkgBsTfumI11VnTV/caxQITO8LZABcsefuqnK5Gbql3Y2tLvM+DAtQpr2CEL+4dPq3S2Rrg79Pqpp0IATjhjIkWhh/0+PhYN8YiiHhLti0GaevHfO5Fod1omU719T2/3Cx0gNbpwL7CYCUZTVhjGGt2Qi/dFsbjiTsa3hAOexOsReBuBiP3RLYRzcOPzfqPV0viYBdUrpSThjDaQdr6HGsnfYJDs6QLmLUVNSEvbNAOiYCeh0vuLGNp+dBm1LlEDZ+sm2jbfkOBGuNGLDK6Chk64qR+K4ibHwn1AAtzRd8ReTnWol0mCvIK6ko8ZvEQoIce46mLBil3xfMn8Jzx6MZ82+dHqjYEw4VuRAOdPzt/tecfIV2VqZeXpEB52bfBzLMjiT40ltsgWXi9IZ+6cb/YMdPOY4Bh9ACMdTH4EsrnaLhoOhjdKaA4G7K+Gfl/Whedwb8FrCztFnLYLOEoU9X5AsnJvxtlZVxe5fjAbhDUDx9CiLm6/UICD+fbJX0yl6a9xns7wk8M+TvbW0ClHA4g1PnbyUvl1RhFUnSPY4 pB0DYWkO 2ojRm5Qj5JnVs5Y3t9odHtGy2Eo3fJqLjhAuO0F6N/lph9m9MKGr16Hz+xvUWfki3GLGNYhPgjHMVM7JIN7dJY3z8R+WasaEBF5cIxrLG+VMMpmDDsj+xML8J6f8eM/Df3uBkF3xTomtSGUM+jSxQemBp/ANLP3Mql+U6zp3xUw10m7KJ8M49Psxbv1/k5q7eYPSyYr1OuZtLQ2dj3ZNtHNJ2Nl/IMUoWNAFNnysNFqq6nBSQTGhiEWlMkSEr/8QHOmVi9JLzepB9E0USxcMyY1GzaqgHXnbfHhT9gPM7EIDIPmBBPOQ26S/A4jVG/3J5WnacJbIWVbySb/egEZpJym/skfeg7GMw5BTrlzMoymvwbFR2Uv5ZCb9nOZasaPv/zqsRhPpszMjwYmhdgtV5wBS84gUsed22KiAeCJC73e/9zHvb+vAD60FwDPQ6oB79dz9Je1iAvCyJvVgXPu7/Gv+UIhNXkd+R+JBklSV2p5OLaJLjmqzGWhM/X4Zj6rFrPRhzVY+pBRnEM/rl/K9IQOj2Mde13np+Ou/GmlWCaFTQPY7uoTSdDjRITout+DQV+xFNXgKr3Cn2sAGVYEPi96T7FvcW9orl6CG1412TbY9P7BL84DTtG2BFcdNnhCCC6R0H2KQXgaluwH/odN0XSLnTBxcVyTuP8oeRZVw3X1/EyNjIlITrvw0a5523mHV4bE6ftwgidoy855s+QrKBVJv80sfio2mLd1Ciidjso2QSAE+TetxltrqdTHZ9QyQ0Y3QVC9RzkHr6FFAWAJDcSziDByUHWXi5JbTeOW5mdmhR0jL5r9uorkb7T2BX81w4d5IcDOrEdFCEXoeIvhjVfW2s+PZI9z8m2Hf1Ok8br1p74S0lED75yfayrhFuY2r+4WbjeLOiq9aS39lvZCd2OEbjxt91aoYMWD1RYDyuAfRncX/z5U8MZ05979pm6MRJqRSNRYkYbcdh4XE/sc24QeyQsqim /R48oroG HrYWx3E4VAkGZLq8yB6vIZpnJhxIE0q4k22qG+4dIQN5jCJfqjy3ZOMfj0XFFdW/OQgjMjnXGfYvfaPUs1SrXWDAVtvmSNRSLDsiHa1Wlfj9jbMquwn+IKnzt8ENWmev+eUmJkCOzutadV295ZAsCg4M2BBQSPdkaSxavZqmQv2VkUEH85pZzTDaDD+/iA/I0G5qzXC8PyEh5iZgDCtG0vHkyfNTgPhuCw7M+2xOYcGgFRAK9rNicwtzRLXWD8nWSrxqL94XInp53doOcPiJFwr97UcSdeQ5RsJRTxI/N/l75qJXqWqnc6tJGQKasX2VBjWFkmESVFj3Xd+N7GCQkVS3QYH8i1luftc+mx/ILmK3dm2aobEvJ3Dz5yXpz2nDKHQF0J+LGFvqcDlYiccy09Tg1PK/8pmW33rCII860UFog1J1owiW/jzw24M4oiEBpFHvtN5SziiR13mlfQRGYOny10DD+N0pr35PeqrwCo1YL1aZ//WXABvwYgwoxJ/sp7AVXOg8zioXa4Zn25JORvgs4QsIfXW1BRuagBtAF3nrZpPjFQdCjtznLdgj+H67A3TUY9sSThj2cNJx2J0NCPRvLtd+bj7PLr8qyDBszND5sbwO97qjucWY5d2oEor6LEc1weZup0tAJzZOhfoPUdL7BluOvlnNLZnaxRWxcAWfLdz/TtRtF+K/b36lm38Oz7aYai2NoFsvt90+Ap1KPZF+wuLYThr2ldY2MtW+cIokTymWOQ8RUkKYhEkzm+v4OZE9PM7DlExUQn40lTN965NdSv9tmmVgkMK/0e9b3SNPRIbheWyBpGTRzHjNSOl8fyDMgta3f/CMOpGK3r3VHj7XdK7BYStzzNyE2iXmQZlfMkIfxhuC61MOw7OjBVAI3NvuGhChZSCwa5wo3KAJ283gUTMmOk9TrvVJS3iNN59UUbBXHS5XV7aI0QixM9FjrhtkHmb/0lE7R2z8uWYBeUckJD6bh AOb9IMZJ CSkD/o0k4RIolLmQVLiLnDRpcM9sM+BMe0FwAR8KHc93mq11NX/tCQ25tb6V67Dy2p/x1WKhpYsDd1sQ/N2DnT5SdzTGELV1LqRjUmAqSmS2RiGAFHHkXrIbt4/In1E9p7LPTN39XA8rrZk6oFknNbBs1pAZYbNuIVTnMiOt1qlhD1NqX7C5aZenDNQj30sGLsmPBHmAeQVOcArYtE+SLVCQuUcJzz9Y5EuSfbDJcufQZt7y2reE5x2FkecVqMnQ93DvN6oyiqBKd1XzkA1gSqWXvXSQZR7J80S/sd4Eg9J2QSiZrxmQ5erj3XkmESg1bJW1mh6kwnW3B63K2/4On0/Sw2HwfWKYg6w3f6mYL8M2an00/mrEfkSAPXmb/oWWhNao2TXKeizEH3/ZkoZweZbJazWqulSLqp+HjzxXzDX9zB0pRyyARo7+G308AV4IOp2JP/0zZInfmxuaYXQlooS7iTrOtdKH6AebQ3hhAq0L6IznzUoov/qlsEIZJPlI+3LIgeJ7QvDt5A6jlu5IXuSZ+7xV7LYpH/B/brlFvbJeROgNOmagxlXXc4qksz965OglGR3guXxluSs0aCYs63dRVWzlD4JKcxZdiRLXLlq/mo9hPVN+7vY21o9T5JUw6B1hSqybCy8vW5sTZeIlUOycnKJxiW7uP6x+YkUXDA3YhudzeiYuD+n5+1cN91cqR3Lzt3GBANesfdMM6moH+f5m70S9Xv7u3+cWOS+ARnlT6fAstT+q7UxEJE/XgvzcTT0tykeYflPyvYnsc8gXJJ9XqmRCUTDeamikUmwvqztQ0P/S5GHrF/9/JwWB71b1hKCPr7240CwypAsUvfxAA/zaMbFN8HOyeSedqhA5wmUN24v8I6BPTv9tz/9hNAB78StxT+AmVRxwLHQxJIHEyGdh/BnRlfibvP2vEqEvMXt0fPpVvMLQlc6pw942HuAMOnai1xycvMfK3zsnAi1ryUdf3aeDo PNE7P2+z sQdTpvNsLx5oiU6bCcR0599VnK+tbxo1WAXMz0PP5rdXW13wQzjx+n4shDJ9X5aKDgnVZL3ORkJKapjbywpVJI7Zg02MZnq0l384yq3cDyviP2IWpkx1m9TO98oNE59quEI1y5c69hHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jan 12, 2023 at 08:43:14PM +0100, Peter Zijlstra wrote: > Hi All! Hi Peter, > The (hopefully) final respin of cpuidle vs rcu cleanup patches. Barring any > objections I'll be queueing these patches in tip/sched/core in the next few > days. I'm sorry to have to bear some bad news on that front. :( I just had a go at testing this on a Juno dev board, using your queue.git sched/idle branch and defconfig + CONFIG_PROVE_LOCKING=y + CONFIG_DEBUG_LOCKDEP=y + CONFIG_DEBUG_ATOMIC_SLEEP=y. With that I consistently see RCU at boot time (log below). | ============================= | WARNING: suspicious RCU usage | 6.2.0-rc3-00051-gced9b6eecb31 #1 Not tainted | ----------------------------- | include/trace/events/ipi.h:19 suspicious rcu_dereference_check() usage! | | other info that might help us debug this: | | | rcu_scheduler_active = 2, debug_locks = 1 | RCU used illegally from extended quiescent state! | no locks held by swapper/0/0. | | stack backtrace: | CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.2.0-rc3-00051-gced9b6eecb31 #1 | Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II May 16 2021 | Call trace: | dump_backtrace.part.0+0xe4/0xf0 | show_stack+0x18/0x30 | dump_stack_lvl+0x98/0xd4 | dump_stack+0x18/0x34 | lockdep_rcu_suspicious+0xf8/0x10c | trace_ipi_raise+0x1a8/0x1b0 | arch_irq_work_raise+0x4c/0x70 | __irq_work_queue_local+0x48/0x80 | irq_work_queue+0x50/0x80 | __wake_up_klogd.part.0+0x98/0xe0 | defer_console_output+0x20/0x30 | vprintk+0x98/0xf0 | _printk+0x5c/0x84 | lockdep_rcu_suspicious+0x34/0x10c | trace_lock_acquire+0x174/0x180 | lock_acquire+0x3c/0x8c | _raw_spin_lock_irqsave+0x70/0x150 | down_trylock+0x18/0x50 | __down_trylock_console_sem+0x3c/0xd0 | console_trylock+0x28/0x90 | vprintk_emit+0x11c/0x354 | vprintk_default+0x38/0x4c | vprintk+0xd4/0xf0 | _printk+0x5c/0x84 | lockdep_rcu_suspicious+0x34/0x10c | printk_sprint+0x238/0x240 | vprintk_store+0x32c/0x4b0 | vprintk_emit+0x104/0x354 | vprintk_default+0x38/0x4c | vprintk+0xd4/0xf0 | _printk+0x5c/0x84 | lockdep_rcu_suspicious+0x34/0x10c | trace_irq_disable+0x1ac/0x1b0 | trace_hardirqs_off+0xe8/0x110 | cpu_suspend+0x4c/0xfc | psci_cpu_suspend_enter+0x58/0x6c | psci_enter_idle_state+0x70/0x170 | cpuidle_enter_state+0xc4/0x464 | cpuidle_enter+0x38/0x50 | do_idle+0x230/0x2c0 | cpu_startup_entry+0x24/0x30 | rest_init+0x110/0x190 | arch_post_acpi_subsys_init+0x0/0x18 | start_kernel+0x6f8/0x738 | __primary_switched+0xbc/0xc4 IIUC what's happenign here is the PSCI cpuidle driver has entered idle and RCU is no longer watching when arm64's cpu_suspend() manipulates DAIF. Our local_daif_*() helpers poke lockdep and tracing, hence the call to trace_hardirqs_off() and the RCU usage. I think we need RCU to be watching all the way down to cpu_suspend(), and it's cpu_suspend() that should actually enter/exit idle context. That and we need to make cpu_suspend() and the low-level PSCI invocation noinstr. I'm not sure whether 32-bit will have a similar issue or not. I'm surprised no-one else who has tested has seen this; I suspect people haven't enabled lockdep and friends. :/ Thanks, Mark.