From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CC46C2D0A3 for ; Thu, 12 Nov 2020 12:49:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2D93216C4 for ; Thu, 12 Nov 2020 12:49:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DtSbpKCy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2D93216C4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D904D6B006E; Thu, 12 Nov 2020 07:49:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D1D3F6B0070; Thu, 12 Nov 2020 07:49:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C58666B0071; Thu, 12 Nov 2020 07:49:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 97E636B006E for ; Thu, 12 Nov 2020 07:49:14 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 349BF33CD for ; Thu, 12 Nov 2020 12:49:14 +0000 (UTC) X-FDA: 77475746628.29.stem08_3011f9027306 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 18A50180868E4 for ; Thu, 12 Nov 2020 12:49:14 +0000 (UTC) X-HE-Tag: stem08_3011f9027306 X-Filterd-Recvd-Size: 8111 Received: from mail-oo1-f68.google.com (mail-oo1-f68.google.com [209.85.161.68]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Thu, 12 Nov 2020 12:49:13 +0000 (UTC) Received: by mail-oo1-f68.google.com with SMTP id c25so1263392ooe.13 for ; Thu, 12 Nov 2020 04:49:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/zzVT0v3gmep680s+NflLICQeHQjw0uUukpoJ+hjBrc=; b=DtSbpKCyJJlwdxbWSfe0mwiJTWQ79UKf/WmXV7b9uNhKBwdsDZRrmG2xoCpBl6Kp3j 8ae0sk0MxeZLjTdjiiYAnhQGr7R2W8Rb7kFoGg5DxQzY/+cFfHgkDgFnfw6QTy9pT/8i oPXDHPDZrPUwPm0U3tGNw2qdt/0v28MsVQB33rsNyEeqJA3MqHlsXecSAk32RXAhCXbM uwZUt2VpW8p4LghTLwYSWxtKXma6FJvJ+Z5aeqZal8qb7/fNSJzZybJI8yKuckhmtqN7 yUiS+W+bOhnxGSbX48a2akomtCU9Qtu0ld0YS6p4FhSusuBPKldZxmdKciDBgX8gA497 LxFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/zzVT0v3gmep680s+NflLICQeHQjw0uUukpoJ+hjBrc=; b=sCZYNu6mtv5QUiVcC2p/dIfzrE6+Ol98KuGxdK9+N3E0SC42rfNtuO2iL4j1PMmemO cA3z0rt7limgSa3HUHWqUx4zmDkNmIfU1Rak3O/soIY5cl1EB2YLQkYy+9WGjPIfXT7o h8GiYHsDiIEAAbIqbZHd+rTNzo0QVGD38sODl4HUNe42OD9ccl024qdjvviMZhtWJATM Xa9mOmKx6ZV30jqY6+WgxaeW3oig6l15Rl9MjAWlHV1rYvOjpCshp6nnCg+vRvshAelR +Uw9aeSefcNzh3a/SDl5aL9zcuMW3+RpejzgGbnztW6So+N7+Ue22ofPZIW1TXu4mMTD kiSQ== X-Gm-Message-State: AOAM5311ORfPZwKJgWI3f9uZjmMPAUsdtbiqscK6SjF7jzXMmzjdgBZU nzHj8vToan0V1tHQcpukYbXEHmFq2y7PCIl2cAGx/g== X-Google-Smtp-Source: ABdhPJy4T/4JR0HsAHFI4mQwdqG3dv1YlLSLp/cBeR6FlVgRLJAq1qC/K03YTVySQnpytt2zTK43iORmhePWvN2Qx4Y= X-Received: by 2002:a4a:d886:: with SMTP id b6mr1622933oov.14.1605185352811; Thu, 12 Nov 2020 04:49:12 -0800 (PST) MIME-Version: 1.0 References: <20201110135320.3309507-1-elver@google.com> <20201111133813.GA81547@elver.google.com> <20201111130543.27d29462@gandalf.local.home> <20201111182333.GA3249@paulmck-ThinkPad-P72> <20201111183430.GN517454@elver.google.com> <20201111192123.GB3249@paulmck-ThinkPad-P72> <20201111202153.GT517454@elver.google.com> <20201112001129.GD3249@paulmck-ThinkPad-P72> In-Reply-To: <20201112001129.GD3249@paulmck-ThinkPad-P72> From: Marco Elver Date: Thu, 12 Nov 2020 13:49:00 +0100 Message-ID: Subject: Re: [PATCH] kfence: Avoid stalling work queue task without allocations To: "Paul E. McKenney" Cc: Steven Rostedt , Anders Roxell , Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Jann Horn , Mark Rutland , Linux Kernel Mailing List , Linux-MM , kasan-dev , rcu@vger.kernel.org, Peter Zijlstra Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 12 Nov 2020 at 01:11, Paul E. McKenney wrote: > > On Wed, Nov 11, 2020 at 09:21:53PM +0100, Marco Elver wrote: > > On Wed, Nov 11, 2020 at 11:21AM -0800, Paul E. McKenney wrote: > > [...] > > > > > rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled > > > > > > > > Sadly, no, next-20201110 already included that one, and that's what I > > > > tested and got me all those warnings above. > > > > > > Hey, I had to ask! The only uncertainty I seee is the acquisition of > > > the lock in rcu_iw_handler(), for which I add a lockdep check in the > > > (untested) patch below. The other thing I could do is sprinkle such > > > checks through the stall-warning code on the assumption that something > > > RCU is calling is enabling interrupts. > > > > > > Other thoughts? > > > > > > Thanx, Paul > > > > > > ------------------------------------------------------------------------ > > > > > > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h > > > index 70d48c5..3d67650 100644 > > > --- a/kernel/rcu/tree_stall.h > > > +++ b/kernel/rcu/tree_stall.h > > > @@ -189,6 +189,7 @@ static void rcu_iw_handler(struct irq_work *iwp) > > > > > > rdp = container_of(iwp, struct rcu_data, rcu_iw); > > > rnp = rdp->mynode; > > > + lockdep_assert_irqs_disabled(); > > > raw_spin_lock_rcu_node(rnp); > > > if (!WARN_ON_ONCE(!rdp->rcu_iw_pending)) { > > > rdp->rcu_iw_gp_seq = rnp->gp_seq; > > > > This assert didn't fire yet, I just get more of the below. I'll keep > > rerunning, but am not too hopeful... > > Is bisection a possibility? I've been running a bisection for past ~12h, and am making slow progress. It might be another 12h, but I think it'll get there. > Failing that, please see the updated patch below. This adds a few more > calls to lockdep_assert_irqs_disabled(), but perhaps more helpfully dumps > the current stack of the CPU that the RCU grace-period kthread wants to > run on in the case where this kthread has been starved of CPU. Thanks, I will apply that after the bisection runs. > Thanx, Paul > > ------------------------------------------------------------------------ > > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h > index 70d48c5..d203ea0 100644 > --- a/kernel/rcu/tree_stall.h > +++ b/kernel/rcu/tree_stall.h > @@ -189,6 +189,7 @@ static void rcu_iw_handler(struct irq_work *iwp) > > rdp = container_of(iwp, struct rcu_data, rcu_iw); > rnp = rdp->mynode; > + lockdep_assert_irqs_disabled(); > raw_spin_lock_rcu_node(rnp); > if (!WARN_ON_ONCE(!rdp->rcu_iw_pending)) { > rdp->rcu_iw_gp_seq = rnp->gp_seq; > @@ -449,21 +450,32 @@ static void print_cpu_stall_info(int cpu) > /* Complain about starvation of grace-period kthread. */ > static void rcu_check_gp_kthread_starvation(void) > { > + int cpu; > struct task_struct *gpk = rcu_state.gp_kthread; > unsigned long j; > > if (rcu_is_gp_kthread_starving(&j)) { > + cpu = gpk ? task_cpu(gpk) : -1; > pr_err("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%#lx ->cpu=%d\n", > rcu_state.name, j, > (long)rcu_seq_current(&rcu_state.gp_seq), > data_race(rcu_state.gp_flags), > gp_state_getname(rcu_state.gp_state), rcu_state.gp_state, > - gpk ? gpk->state : ~0, gpk ? task_cpu(gpk) : -1); > + gpk ? gpk->state : ~0, cpu); > if (gpk) { > pr_err("\tUnless %s kthread gets sufficient CPU time, OOM is now expected behavior.\n", rcu_state.name); > pr_err("RCU grace-period kthread stack dump:\n"); > + lockdep_assert_irqs_disabled(); > sched_show_task(gpk); > + lockdep_assert_irqs_disabled(); > + if (cpu >= 0) { > + pr_err("Stack dump where RCU grace-period kthread last ran:\n"); > + if (!trigger_single_cpu_backtrace(cpu)) > + dump_cpu_task(cpu); > + } > + lockdep_assert_irqs_disabled(); > wake_up_process(gpk); > + lockdep_assert_irqs_disabled(); > } > } > }