From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE82AC63798 for ; Tue, 24 Nov 2020 03:00:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B9B420870 for ; Tue, 24 Nov 2020 03:00:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WEqloDj3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B9B420870 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D1906B005D; Mon, 23 Nov 2020 22:00:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 35C116B006E; Mon, 23 Nov 2020 22:00:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FB6D6B0070; Mon, 23 Nov 2020 22:00:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id E4A1C6B005D for ; Mon, 23 Nov 2020 22:00:27 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 934FA3638 for ; Tue, 24 Nov 2020 03:00:27 +0000 (UTC) X-FDA: 77517808494.29.patch83_0c0f6f22736a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 71F30180868CA for ; Tue, 24 Nov 2020 03:00:27 +0000 (UTC) X-HE-Tag: patch83_0c0f6f22736a X-Filterd-Recvd-Size: 12246 Received: from mail-qv1-f68.google.com (mail-qv1-f68.google.com [209.85.219.68]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Tue, 24 Nov 2020 03:00:26 +0000 (UTC) Received: by mail-qv1-f68.google.com with SMTP id u23so9973512qvf.1 for ; Mon, 23 Nov 2020 19:00:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9dXI24aC8NvP/kqZdopSzP2LewH/N8Z0aB66LTLcHvE=; b=WEqloDj3pQkHX/rykrKshZfsbcCFrb7Nnf00c3YqN5OaW4DiwvjYgmOEv/T/Jeir9Y 52cpDiHcVrElzwhPz4rHypEcCfMDSIQ3obrclpYdIx57IeRgdY3gWjbY+NgxnT6WxdvT ypjzMSO1oFqLdVQ6YBjlT87lUiNGuM2pCijcLekmYKBSlzFR9ivbGjMfJ8NdvoZZzzTW L2vhfbZVf7ouhvYHOUTHRx6Pvt7XrrjHU/8Y9bKrK5EHQ+UHrlU5of2XVVRpExMeCaON NI16vHytRRMrKcTlEimaI6cyyYJvmL1wm2qU1YH68vUAI082poVvdaqOANqoINvZcZbw aDCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9dXI24aC8NvP/kqZdopSzP2LewH/N8Z0aB66LTLcHvE=; b=HtKeRJ9CJyjUIJVoUCSdxxlNSYIQBKiGBqmpreZL5LihY6VSZOz5iWdzTNHIciqqR2 NqnnK09NoNZGr2YjshIN8rC4dWX4RaldksUtlKrFBLQmtPsM4M26cYM+21S02oJeHk8P EYeQGnYETa0gX2kj7nc2v7rjm6Yj0Mt5YzP1C4TanQzeBGw4U9a7cVOYgxuXWtyQVoZj Dw0pIAzL8et5CgFRk7MZ1Pfl5oC39weMEYMS2pjM8UiHp9B+CzTUeoHsB0OZlSGX8qs+ T63nv8YZmHUzjKqG6zv0mvVblvFcFdC0npCfNSWSv3aQxFcdEhsbgFWSvfhMzS5m5Ll+ y4qw== X-Gm-Message-State: AOAM532D+fRl8COlRHdxhcDf5pqzarIz4x2wqS7z5QAEVizCxpqrvwBe dGOh7M74FOqL2Z9L15Z/EvY= X-Google-Smtp-Source: ABdhPJxH7jj3FF+5RuwZ9GNlWj2c8UxLx+AfCz81AZzM1Ew0KXCJd7hKJ4QfbZ8o59GXirU6bTTphg== X-Received: by 2002:a05:6214:1150:: with SMTP id b16mr2543732qvt.46.1606186826260; Mon, 23 Nov 2020 19:00:26 -0800 (PST) Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com. [66.111.4.228]) by smtp.gmail.com with ESMTPSA id h8sm7294688qka.117.2020.11.23.19.00.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Nov 2020 19:00:25 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id 657D527C0054; Mon, 23 Nov 2020 22:00:22 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 23 Nov 2020 22:00:22 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrudegjedgkeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeeuohhquhhn ucfhvghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrth htvghrnhepvdelieegudfggeevjefhjeevueevieetjeeikedvgfejfeduheefhffggedv geejnecukfhppedufedurddutdejrddugeejrdduvdeinecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqhhunhdomhgvshhmthhprghuthhh phgvrhhsohhnrghlihhthidqieelvdeghedtieegqddujeejkeehheehvddqsghoqhhunh drfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvgdrnhgrmhgv X-ME-Proxy: Received: from localhost (unknown [131.107.147.126]) by mail.messagingengine.com (Postfix) with ESMTPA id 816D83064AB3; Mon, 23 Nov 2020 22:00:20 -0500 (EST) Date: Tue, 24 Nov 2020 10:59:45 +0800 From: Boqun Feng To: Steven Rostedt Cc: Marco Elver , "Paul E. McKenney" , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Mark Rutland , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan , Thomas Gleixner Subject: Re: [PATCH] kfence: Avoid stalling work queue task without allocations Message-ID: <20201124025945.GG286534@boqun-archlinux> References: <20201113175754.GA6273@paulmck-ThinkPad-P72> <20201117105236.GA1964407@elver.google.com> <20201117182915.GM1437@paulmck-ThinkPad-P72> <20201118225621.GA1770130@elver.google.com> <20201118233841.GS1437@paulmck-ThinkPad-P72> <20201119125357.GA2084963@elver.google.com> <20201120142734.75af5cd6@gandalf.local.home> <20201123152720.GA2177956@elver.google.com> <20201123112812.19e918b3@gandalf.local.home> <20201123134227.6df443db@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201123134227.6df443db@gandalf.local.home> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Steven, On Mon, Nov 23, 2020 at 01:42:27PM -0500, Steven Rostedt wrote: > On Mon, 23 Nov 2020 11:28:12 -0500 > Steven Rostedt wrote: > > > I noticed: > > > > > > [ 237.650900] enabling event benchmark_event > > > > In both traces. Could you disable CONFIG_TRACEPOINT_BENCHMARK and see if > > the issue goes away. That event kicks off a thread that spins in a tight > > loop for some time and could possibly cause some issues. > > > > It still shouldn't break things, we can narrow it down if it is the culprit. > > [ Added Thomas ] > > And that's just one issue. I don't think that has anything to do with the > other one: > > [ 1614.162007] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: > [ 1614.168625] (detected by 0, t=3752 jiffies, g=3529, q=1) > [ 1614.170825] rcu: All QSes seen, last rcu_preempt kthread activity 242 (4295293115-4295292873), jiffies_till_next_fqs=1, root ->qsmask 0x0 > [ 1614.194272] > [ 1614.196673] ================================ > [ 1614.199738] WARNING: inconsistent lock state > [ 1614.203056] 5.10.0-rc4-next-20201119-00004-g77838ee21ff6-dirty #21 Not tainted > [ 1614.207012] -------------------------------- > [ 1614.210125] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. > [ 1614.213832] swapper/0/1 [HC0[0]:SC0[0]:HE0:SE1] takes: > [ 1614.217288] ffffd942547f47d8 (rcu_node_0){?.-.}-{2:2}, at: rcu_sched_clock_irq+0x7c0/0x17a0 > [ 1614.225496] {IN-HARDIRQ-W} state was registered at: > [ 1614.229031] __lock_acquire+0xae8/0x1ac8 > [ 1614.232203] lock_acquire+0x268/0x508 > [ 1614.235254] _raw_spin_lock_irqsave+0x78/0x14c > [ 1614.238547] rcu_sched_clock_irq+0x7c0/0x17a0 > [ 1614.241757] update_process_times+0x6c/0xb8 > [ 1614.244950] tick_sched_handle.isra.0+0x58/0x88 > [ 1614.248225] tick_sched_timer+0x68/0xe0 > [ 1614.251304] __hrtimer_run_queues+0x288/0x730 > [ 1614.254516] hrtimer_interrupt+0x114/0x288 > [ 1614.257650] arch_timer_handler_virt+0x50/0x70 > [ 1614.260922] handle_percpu_devid_irq+0x104/0x4c0 > [ 1614.264236] generic_handle_irq+0x54/0x78 > [ 1614.267385] __handle_domain_irq+0xac/0x130 > [ 1614.270585] gic_handle_irq+0x70/0x108 > [ 1614.273633] el1_irq+0xc0/0x180 > [ 1614.276526] rcu_irq_exit_irqson+0x40/0x78 > [ 1614.279704] trace_preempt_on+0x144/0x1a0 > [ 1614.282834] preempt_schedule_common+0xf8/0x1a8 > [ 1614.286126] preempt_schedule+0x38/0x40 > [ 1614.289240] __mutex_lock+0x608/0x8e8 > [ 1614.292302] mutex_lock_nested+0x3c/0x58 > [ 1614.295450] static_key_enable_cpuslocked+0x7c/0xf8 > [ 1614.298828] static_key_enable+0x2c/0x40 > [ 1614.301961] tracepoint_probe_register_prio+0x284/0x3a0 > [ 1614.305464] tracepoint_probe_register+0x40/0x58 > [ 1614.308776] trace_event_reg+0xe8/0x150 > [ 1614.311852] __ftrace_event_enable_disable+0x2e8/0x608 > [ 1614.315351] __ftrace_set_clr_event_nolock+0x160/0x1d8 > [ 1614.318809] __ftrace_set_clr_event+0x60/0x90 > [ 1614.322061] event_trace_self_tests+0x64/0x12c > [ 1614.325335] event_trace_self_tests_init+0x88/0xa8 > [ 1614.328758] do_one_initcall+0xa4/0x500 > [ 1614.331860] kernel_init_freeable+0x344/0x3c4 > [ 1614.335110] kernel_init+0x20/0x16c > [ 1614.338102] ret_from_fork+0x10/0x34 > [ 1614.341057] irq event stamp: 3206302 > [ 1614.344123] hardirqs last enabled at (3206301): [] rcu_irq_exit_irqson+0x64/0x78 > [ 1614.348697] hardirqs last disabled at (3206302): [] el1_irq+0x80/0x180 > [ 1614.353013] softirqs last enabled at (3204216): [] __do_softirq+0x630/0x6b4 > [ 1614.357509] softirqs last disabled at (3204191): [] irq_exit+0x1cc/0x1e0 > [ 1614.361737] > [ 1614.361737] other info that might help us debug this: > [ 1614.365566] Possible unsafe locking scenario: > [ 1614.365566] > [ 1614.369128] CPU0 > [ 1614.371747] ---- > [ 1614.374282] lock(rcu_node_0); > [ 1614.378818] > [ 1614.381394] lock(rcu_node_0); > [ 1614.385997] > [ 1614.385997] *** DEADLOCK *** > [ 1614.385997] > [ 1614.389613] 5 locks held by swapper/0/1: > [ 1614.392655] #0: ffffd9425480e940 (event_mutex){+.+.}-{3:3}, at: __ftrace_set_clr_event+0x48/0x90 > [ 1614.401701] #1: ffffd9425480a530 (tracepoints_mutex){+.+.}-{3:3}, at: tracepoint_probe_register_prio+0x48/0x3a0 > [ 1614.410973] #2: ffffd9425476abf0 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_enable+0x24/0x40 > [ 1614.419858] #3: ffffd94254816348 (jump_label_mutex){+.+.}-{3:3}, at: static_key_enable_cpuslocked+0x7c/0xf8 > [ 1614.429049] #4: ffffd942547f47d8 (rcu_node_0){?.-.}-{2:2}, at: rcu_sched_clock_irq+0x7c0/0x17a0 > [ 1614.438029] > [ 1614.438029] stack backtrace: > [ 1614.441436] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.0-rc4-next-20201119-00004-g77838ee21ff6-dirty #21 > [ 1614.446149] Hardware name: linux,dummy-virt (DT) > [ 1614.449621] Call trace: > [ 1614.452337] dump_backtrace+0x0/0x240 > [ 1614.455372] show_stack+0x34/0x88 > [ 1614.458306] dump_stack+0x140/0x1bc > [ 1614.461258] print_usage_bug+0x2a0/0x2f0 > [ 1614.464399] mark_lock.part.0+0x438/0x4e8 > [ 1614.467528] mark_held_locks+0x54/0x90 > [ 1614.470576] lockdep_hardirqs_on_prepare+0xe0/0x290 > [ 1614.473935] trace_hardirqs_on+0x90/0x370 > [ 1614.477045] el1_irq+0xdc/0x180 > [ 1614.479934] rcu_irq_exit_irqson+0x40/0x78 > [ 1614.483093] trace_preempt_on+0x144/0x1a0 > [ 1614.486211] preempt_schedule_common+0xf8/0x1a8 > [ 1614.489479] preempt_schedule+0x38/0x40 > [ 1614.492544] __mutex_lock+0x608/0x8e8 > > > The above has: > > preempt_schedule_common() { > trace_preempt_on() { > > el1_irq: > handle_arch_irq { > irq_enter(); > [..] > irq_exit(); > } > bl trace_hardirqs_on > > > I wonder if the lockdep logic got confused on ARM64 by the rework done to > lockdep and tracing with respect to irq entry / exit. > I'm also staring at this problem and another thing caused my attention is that there is a line like the following after the lockdep splat: [...] BUG: scheduling while atomic ... , which means preemption count has some inconsistency too. Given this, a possible case cause this is that we got preempted inside a rcu_node lock critical section (I know, this is quite impossible, but preemption count and lockdep data are maintained quite separately, so it's unlikely they are broken at the same time...) Will continue to look into this. Regards, Boqun > Or maybe there's an rcu_node leak lock that happened somewhere? > > -- Steve