From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0156EC64E7C for ; Thu, 22 Oct 2020 22:27:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7BDD4223C7 for ; Thu, 22 Oct 2020 22:27:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7BDD4223C7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DC9666B0071; Thu, 22 Oct 2020 18:27:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D01E76B0072; Thu, 22 Oct 2020 18:27:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA1A76B0073; Thu, 22 Oct 2020 18:27:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0219.hostedemail.com [216.40.44.219]) by kanga.kvack.org (Postfix) with ESMTP id 86DFF6B0071 for ; Thu, 22 Oct 2020 18:27:24 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2C70E181AEF07 for ; Thu, 22 Oct 2020 22:27:24 +0000 (UTC) X-FDA: 77400998808.04.dog03_2d144a227254 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 00B9F80073CF for ; Thu, 22 Oct 2020 22:27:23 +0000 (UTC) X-HE-Tag: dog03_2d144a227254 X-Filterd-Recvd-Size: 10741 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Thu, 22 Oct 2020 22:27:22 +0000 (UTC) IronPort-SDR: NJVFeigZHbIq2FWw8VMzyCguuH/4i59f7MNSCD9yPAZyLVAb3nA98TSm1mp/NU5q3264KMQ1JC 71iFdzLxW2fA== X-IronPort-AV: E=McAfee;i="6000,8403,9782"; a="185293784" X-IronPort-AV: E=Sophos;i="5.77,404,1596524400"; d="scan'208";a="185293784" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2020 15:27:21 -0700 IronPort-SDR: ay6jcPxJebLGWrOYdNxv1McOYP32pqKcqe+sQMqbMBZuQO46WeTfpvuDcojj9VnxaOQNS2JtMi +3Zc/AvXDsQQ== X-IronPort-AV: E=Sophos;i="5.77,404,1596524400"; d="scan'208";a="523289113" Received: from iweiny-desk2.sc.intel.com (HELO localhost) ([10.3.52.147]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2020 15:27:21 -0700 From: ira.weiny@intel.com To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Andy Lutomirski , Peter Zijlstra Cc: Ira Weiny , x86@kernel.org, Dave Hansen , Dan Williams , Andrew Morton , Fenghua Yu , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Subject: [PATCH 06/10] x86/entry: Move nmi entry/exit into common code Date: Thu, 22 Oct 2020 15:26:57 -0700 Message-Id: <20201022222701.887660-7-ira.weiny@intel.com> X-Mailer: git-send-email 2.28.0.rc0.12.gb6a658bd00c9 In-Reply-To: <20201022222701.887660-1-ira.weiny@intel.com> References: <20201022222701.887660-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Lockdep state handling on NMI enter and exit is nothing specific to X86. = It's not any different on other architectures. Also the extra state type is no= t necessary, irqentry_state_t can carry the necessary information as well. Move it to common code and extend irqentry_state_t to carry lockdep state= . Signed-off-by: Thomas Gleixner Signed-off-by: Ira Weiny --- arch/x86/entry/common.c | 34 ------------------------------- arch/x86/include/asm/idtentry.h | 3 --- arch/x86/kernel/cpu/mce/core.c | 6 +++--- arch/x86/kernel/nmi.c | 6 +++--- arch/x86/kernel/traps.c | 13 ++++++------ include/linux/entry-common.h | 24 +++++++++++++++++++++- kernel/entry/common.c | 36 +++++++++++++++++++++++++++++++++ 7 files changed, 72 insertions(+), 50 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 870efeec8bda..18d8f17f755c 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -209,40 +209,6 @@ SYSCALL_DEFINE0(ni_syscall) return -ENOSYS; } =20 -noinstr bool idtentry_enter_nmi(struct pt_regs *regs) -{ - bool irq_state =3D lockdep_hardirqs_enabled(); - - __nmi_enter(); - lockdep_hardirqs_off(CALLER_ADDR0); - lockdep_hardirq_enter(); - rcu_nmi_enter(); - - instrumentation_begin(); - trace_hardirqs_off_finish(); - ftrace_nmi_enter(); - instrumentation_end(); - - return irq_state; -} - -noinstr void idtentry_exit_nmi(struct pt_regs *regs, bool restore) -{ - instrumentation_begin(); - ftrace_nmi_exit(); - if (restore) { - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(CALLER_ADDR0); - } - instrumentation_end(); - - rcu_nmi_exit(); - lockdep_hardirq_exit(); - if (restore) - lockdep_hardirqs_on(CALLER_ADDR0); - __nmi_exit(); -} - #ifdef CONFIG_XEN_PV #ifndef CONFIG_PREEMPTION /* diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idten= try.h index b2442eb0ac2f..247a60a47331 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -11,9 +11,6 @@ =20 #include =20 -bool idtentry_enter_nmi(struct pt_regs *regs); -void idtentry_exit_nmi(struct pt_regs *regs, bool irq_state); - /** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points * No error code pushed by hardware diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/cor= e.c index 1c08cb9eb9f6..eb3338c0bbc1 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -1983,7 +1983,7 @@ void (*machine_check_vector)(struct pt_regs *) =3D = unexpected_machine_check; =20 static __always_inline void exc_machine_check_kernel(struct pt_regs *reg= s) { - bool irq_state; + irqentry_state_t irq_state; =20 WARN_ON_ONCE(user_mode(regs)); =20 @@ -1995,7 +1995,7 @@ static __always_inline void exc_machine_check_kerne= l(struct pt_regs *regs) mce_check_crashing_cpu()) return; =20 - irq_state =3D idtentry_enter_nmi(regs); + irq_state =3D irqentry_nmi_enter(regs); /* * The call targets are marked noinstr, but objtool can't figure * that out because it's an indirect call. Annotate it. @@ -2006,7 +2006,7 @@ static __always_inline void exc_machine_check_kerne= l(struct pt_regs *regs) if (regs->flags & X86_EFLAGS_IF) trace_hardirqs_on_prepare(); instrumentation_end(); - idtentry_exit_nmi(regs, irq_state); + irqentry_nmi_exit(regs, irq_state); } =20 static __always_inline void exc_machine_check_user(struct pt_regs *regs) diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index 4bc77aaf1303..bf250a339655 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -475,7 +475,7 @@ static DEFINE_PER_CPU(unsigned long, nmi_dr7); =20 DEFINE_IDTENTRY_RAW(exc_nmi) { - bool irq_state; + irqentry_state_t irq_state; =20 /* * Re-enable NMIs right here when running as an SEV-ES guest. This migh= t @@ -502,14 +502,14 @@ DEFINE_IDTENTRY_RAW(exc_nmi) =20 this_cpu_write(nmi_dr7, local_db_save()); =20 - irq_state =3D idtentry_enter_nmi(regs); + irq_state =3D irqentry_nmi_enter(regs); =20 inc_irq_stat(__nmi_count); =20 if (!ignore_nmis) default_do_nmi(regs); =20 - idtentry_exit_nmi(regs, irq_state); + irqentry_nmi_exit(regs, irq_state); =20 local_db_restore(this_cpu_read(nmi_dr7)); =20 diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 3c70fb34028b..bffbbe29fc8c 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -405,7 +405,7 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif =20 - idtentry_enter_nmi(regs); + irqentry_nmi_enter(regs); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); =20 @@ -651,12 +651,13 @@ DEFINE_IDTENTRY_RAW(exc_int3) instrumentation_end(); irqentry_exit_to_user_mode(regs); } else { - bool irq_state =3D idtentry_enter_nmi(regs); + irqentry_state_t irq_state =3D irqentry_nmi_enter(regs); + instrumentation_begin(); if (!do_int3(regs)) die("int3", regs, 0); instrumentation_end(); - idtentry_exit_nmi(regs, irq_state); + irqentry_nmi_exit(regs, irq_state); } } =20 @@ -864,7 +865,7 @@ static __always_inline void exc_debug_kernel(struct p= t_regs *regs, * includes the entry stack is excluded for everything. */ unsigned long dr7 =3D local_db_save(); - bool irq_state =3D idtentry_enter_nmi(regs); + irqentry_state_t irq_state =3D irqentry_nmi_enter(regs); instrumentation_begin(); =20 /* @@ -907,7 +908,7 @@ static __always_inline void exc_debug_kernel(struct p= t_regs *regs, regs->flags &=3D ~X86_EFLAGS_TF; out: instrumentation_end(); - idtentry_exit_nmi(regs, irq_state); + irqentry_nmi_exit(regs, irq_state); =20 local_db_restore(dr7); } @@ -925,7 +926,7 @@ static __always_inline void exc_debug_user(struct pt_= regs *regs, =20 /* * NB: We can't easily clear DR7 here because - * idtentry_exit_to_usermode() can invoke ptrace, schedule, access + * irqentry_exit_to_usermode() can invoke ptrace, schedule, access * user memory, etc. This means that a recursive #DB is possible. If * this happens, that #DB will hit exc_debug_kernel() and clear DR7. * Since we're not on the IST stack right now, everything will be diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index 474f29638d2c..47f9a0658acf 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -342,7 +342,10 @@ void irqentry_exit_to_user_mode(struct pt_regs *regs= ); =20 #ifndef irqentry_state typedef struct irqentry_state { - bool exit_rcu; + union { + bool exit_rcu; + bool lockdep; + }; } irqentry_state_t; #endif =20 @@ -402,4 +405,23 @@ void irqentry_exit_cond_resched(void); */ void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state)= ; =20 +/** + * irqentry_nmi_enter - Handle NMI entry + * @regs: Pointer to currents pt_regs + * + * Similar to irqentry_enter() but taking care of the NMI constraints. + */ +irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs); + +/** + * irqentry_nmi_exit - Handle return from NMI handling + * @regs: Pointer to pt_regs (NMI entry regs) + * @irq_state: Return value from matching call to irqentry_nmi_enter() + * + * Last action before returning to the low level assmenbly code. + * + * Counterpart to irqentry_nmi_enter(). + */ +void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t ir= q_state); + #endif diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 0a1e20f8d4e8..5cc2e4174d7c 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -398,3 +398,39 @@ noinstr void irqentry_exit(struct pt_regs *regs, irq= entry_state_t state) rcu_irq_exit(); } } + +irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) +{ + irqentry_state_t irq_state; + + irq_state.lockdep =3D lockdep_hardirqs_enabled(); + + __nmi_enter(); + lockdep_hardirqs_off(CALLER_ADDR0); + lockdep_hardirq_enter(); + rcu_nmi_enter(); + + instrumentation_begin(); + trace_hardirqs_off_finish(); + ftrace_nmi_enter(); + instrumentation_end(); + + return irq_state; +} + +void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t ir= q_state) +{ + instrumentation_begin(); + ftrace_nmi_exit(); + if (irq_state.lockdep) { + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(CALLER_ADDR0); + } + instrumentation_end(); + + rcu_nmi_exit(); + lockdep_hardirq_exit(); + if (irq_state.lockdep) + lockdep_hardirqs_on(CALLER_ADDR0); + __nmi_exit(); +} --=20 2.28.0.rc0.12.gb6a658bd00c9