From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67790C2D0E4 for ; Fri, 20 Nov 2020 18:17:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A62482242B for ; Fri, 20 Nov 2020 18:17:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wFr1zLBj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A62482242B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DDF096B0036; Fri, 20 Nov 2020 13:17:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D66FD6B005C; Fri, 20 Nov 2020 13:17:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2E576B005D; Fri, 20 Nov 2020 13:17:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id 8E1F16B0036 for ; Fri, 20 Nov 2020 13:17:46 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2979D1EE6 for ; Fri, 20 Nov 2020 18:17:46 +0000 (UTC) X-FDA: 77505604932.16.vein12_110adb62734d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 0A530100E690C for ; Fri, 20 Nov 2020 18:17:46 +0000 (UTC) X-HE-Tag: vein12_110adb62734d X-Filterd-Recvd-Size: 7045 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Fri, 20 Nov 2020 18:17:45 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id p8so10974034wrx.5 for ; Fri, 20 Nov 2020 10:17:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=giLwGu45+mkpAyteW5JjczsUwt2xu5kU0h/C4LFwJ2c=; b=wFr1zLBjOF9rnmBu4dONY85QJfuQdrC1q3VGOLs6wv+27rz4ELtVLvqWNT1EQmPJeP b/DEveUrWtbf3ClEGVeEX8NeFHuqDwvPQUo6bXUjsr3Py7DHCLZcDEfbtgE1+aawCH5B eUzNpMiKo70UyssLqrE034oG+ojnU48jMeUGXB/OuuAU8XcWzvbKyHmTDJBrA768ALkL 7BxCHw5a7bNl87a62XUxg1ioHwTlgDYAOiJFzkeFpMcufbTTMi90cj6T6DmFBgQxYn7S NJNCMDJeTh99hFoO/EBvtqS5im23Jav3qGAWiHWir13cnYDaqmT3kKCNWKOLz7Gvc/Go ua/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=giLwGu45+mkpAyteW5JjczsUwt2xu5kU0h/C4LFwJ2c=; b=n9r4GWKI1Gh7q95OWMBnKt95rRwzQyibY1tBybFMI0y37I2Z9BJhe6LAsXkNee56JF 58ylCxxdrBrpKygTdxlNkkaHysk16WVjfJTpo2mkNhHus5nShQ3Ltzf0w7u+/UwWG78B oaGahPC9oQkWTBNgCQrMdf62yxN0JUKgpYtXCeRLTG4M+maE5s+swXDAtMzdt2LMloAh mOTVU2ko9gJmH19iGoMY08Zt0O5RV18XuS9sgdeHq0Yqx9Fqgf8B7BsoWQeEv7dEkIYx bWeh8YPuFvYqctV69d0g8SGAg6s+KMJFu3eQuoZYBRU2pGXyJfXMeHJj/CJUzVYcq9sB FM5g== X-Gm-Message-State: AOAM530cOJpbu0+l/V1skyfJbDKBpfBKDDmgwQZK7SbNWmYiUG+vDUgp 62Nvb77HcmR24ZBfmQfNvfDdBA== X-Google-Smtp-Source: ABdhPJypfvUfZgF5ojloWzuTAX8p2n/TOFSkwnjUALuGsK1Ip6DPAbzOUC423R8qr2cVO7IcyikW0Q== X-Received: by 2002:adf:9e4c:: with SMTP id v12mr16806903wre.22.1605896264123; Fri, 20 Nov 2020 10:17:44 -0800 (PST) Received: from elver.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) by smtp.gmail.com with ESMTPSA id g11sm6243435wrq.7.2020.11.20.10.17.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 10:17:43 -0800 (PST) Date: Fri, 20 Nov 2020 19:17:37 +0100 From: Marco Elver To: Steven Rostedt Cc: "Paul E. McKenney" , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Mark Rutland , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra , Tejun Heo , Lai Jiangshan , linux-arm-kernel@lists.infradead.org Subject: Re: linux-next: stall warnings and deadlock on Arm64 (was: [PATCH] kfence: Avoid stalling...) Message-ID: <20201120181737.GA3301774@elver.google.com> References: <20201118225621.GA1770130@elver.google.com> <20201118233841.GS1437@paulmck-ThinkPad-P72> <20201119125357.GA2084963@elver.google.com> <20201119151409.GU1437@paulmck-ThinkPad-P72> <20201119170259.GA2134472@elver.google.com> <20201119184854.GY1437@paulmck-ThinkPad-P72> <20201119193819.GA2601289@elver.google.com> <20201119213512.GB1437@paulmck-ThinkPad-P72> <20201120141928.GB3120165@elver.google.com> <20201120102613.3d18b90e@gandalf.local.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201120102613.3d18b90e@gandalf.local.home> User-Agent: Mutt/1.14.6 (2020-07-11) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Nov 20, 2020 at 10:26AM -0500, Steven Rostedt wrote: > On Fri, 20 Nov 2020 15:19:28 +0100 > Marco Elver wrote: > > > None of those triggered either. > > > > I found that disabling ftrace for some of kernel/rcu (see below) solved > > the stalls (and any mention of deadlocks as a side-effect I assume), > > resulting in successful boot. > > > > Does that provide any additional clues? I tried to narrow it down to 1-2 > > files, but that doesn't seem to work. > > > > Thanks, > > -- Marco > > > > ------ >8 ------ > > > > diff --git a/kernel/rcu/Makefile b/kernel/rcu/Makefile > > index 0cfb009a99b9..678b4b094f94 100644 > > --- a/kernel/rcu/Makefile > > +++ b/kernel/rcu/Makefile > > @@ -3,6 +3,13 @@ > > # and is generally not a function of system call inputs. > > KCOV_INSTRUMENT := n > > > > +ifdef CONFIG_FUNCTION_TRACER > > +CFLAGS_REMOVE_update.o = $(CC_FLAGS_FTRACE) > > +CFLAGS_REMOVE_sync.o = $(CC_FLAGS_FTRACE) > > +CFLAGS_REMOVE_srcutree.o = $(CC_FLAGS_FTRACE) > > +CFLAGS_REMOVE_tree.o = $(CC_FLAGS_FTRACE) > > +endif > > + > > Can you narrow it down further? That is, do you really need all of the > above to stop the stalls? I tried to reduce it to 1 or combinations of 2 files only, but that didn't work. > Also, since you are using linux-next, you have ftrace recursion debugging. > Please enable: > > CONFIG_FTRACE_RECORD_RECURSION=y > CONFIG_RING_BUFFER_RECORD_RECURSION=y > > when enabling any of the above. If you can get to a successful boot, you > can then: > > # cat /sys/kernel/tracing/recursed_functions > > Which would let me know if there's an recursion issue in RCU somewhere. To get the system to boot in the first place (as mentioned in other emails) I again needed to revert "rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled", as otherwise would run into the deadlock. That used to still result in stall warnings, except when ftrace's recursion detection is on it seems. With that, this is what I get: | # cat /sys/kernel/tracing/recursed_functions | trace_selftest_test_recursion_func+0x34/0x48: trace_selftest_dynamic_test_func+0x4/0x28 | el1_irq+0xc0/0x180: gic_handle_irq+0x4/0x108 | gic_handle_irq+0x70/0x108: __handle_domain_irq+0x4/0x130 | __handle_domain_irq+0x7c/0x130: irq_enter+0x4/0x28 | trace_rcu_dyntick+0x168/0x190: rcu_read_lock_sched_held+0x4/0x98 | rcu_read_lock_sched_held+0x30/0x98: rcu_read_lock_held_common+0x4/0x88 | rcu_read_lock_held_common+0x50/0x88: rcu_lockdep_current_cpu_online+0x4/0xd0 | irq_enter+0x1c/0x28: irq_enter_rcu+0x4/0xa8 | irq_enter_rcu+0x3c/0xa8: irqtime_account_irq+0x4/0x198 | irq_enter_rcu+0x44/0xa8: preempt_count_add+0x4/0x1a0 | trace_hardirqs_off+0x254/0x2d8: __srcu_read_lock+0x4/0xa0 | trace_hardirqs_off+0x25c/0x2d8: rcu_irq_enter_irqson+0x4/0x78 | trace_rcu_dyntick+0xd8/0x190: __traceiter_rcu_dyntick+0x4/0x80 | trace_hardirqs_off+0x294/0x2d8: rcu_irq_exit_irqson+0x4/0x78 | trace_hardirqs_off+0x2a0/0x2d8: __srcu_read_unlock+0x4/0x88 Thanks, -- Marco