From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44411C4707F for ; Tue, 25 May 2021 21:03:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC13861430 for ; Tue, 25 May 2021 21:03:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC13861430 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 652246B0036; Tue, 25 May 2021 17:03:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 601DD6B006C; Tue, 25 May 2021 17:03:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42E516B006E; Tue, 25 May 2021 17:03:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 0BB476B0036 for ; Tue, 25 May 2021 17:03:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A5B19907D for ; Tue, 25 May 2021 21:03:01 +0000 (UTC) X-FDA: 78180978162.23.67C2683 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf06.hostedemail.com (Postfix) with ESMTP id 0E1D0C0042E3 for ; Tue, 25 May 2021 21:02:54 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 6DDC26140F; Tue, 25 May 2021 21:03:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1621976580; bh=klxS3qDE45PcoyEfedj1p8MFJaSVmaOXxKg6fz+dHKI=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=Va9P31DucEeNVjKtTrQjIx2PQfmM66FIsANZHvhBXxynLbMZvtoOJydKlhyVrFXUs JC2fja1A1l9yjWXKPi5g/jhoWUtw2Gj5P3eeAr5ZCtViSf3JVcc1GAwOMR/J5QgR42 ytXCGw6l7SnjVbcYxTn/MVu5OnlV8dsy+U86js6LSjS3UehUKYoPYY4p2yB7ziWOXV 4oQfLhyfNhRplTg8ZVUi4Wco2oLHiCuZGdcS+oonp4L2IMt/CaICRCwTovj9Bo6/FR YDNABdf0uhREZDIoHTFs+fsLJ5kn/hW31Ljs00mCHy+xtwyxJ7cGKNuiog7GRh1QHO JT1/0OJ+cPCow== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 35BAB5C037F; Tue, 25 May 2021 14:03:00 -0700 (PDT) Date: Tue, 25 May 2021 14:03:00 -0700 From: "Paul E. McKenney" To: Oliver Sang Cc: Thomas Gleixner , John Stultz , Stephen Boyd , Jonathan Corbet , Mark Rutland , Marc Zyngier , Andi Kleen , Feng Tang , Xing Zhengjun , Chris Mason , LKML , Linux Memory Management List , lkp@lists.01.org, lkp@intel.com Subject: Re: [clocksource] 8e614d5b58: WARNING:at_kernel/time/clocksource-wdtest.c:#wdtest_func.cold Message-ID: <20210525210300.GU4441@paulmck-ThinkPad-P17-Gen-1> Reply-To: paulmck@kernel.org References: <20210505143616.GC9038@xsang-OptiPlex-9020> <20210505180312.GM975577@paulmck-ThinkPad-P17-Gen-1> <20210507171259.GA236800@paulmck-ThinkPad-P17-Gen-1> <20210525073655.GE7744@xsang-OptiPlex-9020> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210525073655.GE7744@xsang-OptiPlex-9020> Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Va9P31Du; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf06.hostedemail.com: domain of "SRS0=jYIl=KU=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 198.145.29.99 as permitted sender) smtp.mailfrom="SRS0=jYIl=KU=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0E1D0C0042E3 X-Stat-Signature: kfsfbi1h7dqcq37gpjya99nze7m8wca7 X-HE-Tag: 1621976574-804450 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 25, 2021 at 03:36:55PM +0800, Oliver Sang wrote: > hi Paul, > > On Fri, May 07, 2021 at 10:12:59AM -0700, Paul E. McKenney wrote: > > On Wed, May 05, 2021 at 11:03:12AM -0700, Paul E. McKenney wrote: > > > On Wed, May 05, 2021 at 10:36:16PM +0800, kernel test robot wrote: > > > > > > > > > > > > Greeting, > > > > > > > > FYI, we noticed the following commit (built with gcc-9): > > > > > > > > commit: 8e614d5b58992e722f07de7c2426f2c44668092b ("clocksource: Provide kernel module to test clocksource watchdog") > > > > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master > > > > > > > > > > > > in testcase: boot > > > > > > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G > > > > > > > > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): > > > > > > > > > > > > +-------------------------------------------------------------------------+------------+------------+ > > > > | | bdbd9c673e | 8e614d5b58 | > > > > +-------------------------------------------------------------------------+------------+------------+ > > > > | WARNING:at_kernel/time/clocksource-wdtest.c:#wdtest_func.cold | 0 | 11 | > > > > | RIP:wdtest_func.cold | 0 | 11 | > > > > +-------------------------------------------------------------------------+------------+------------+ > > > > > > Might it be useful to address the lockdep issues that preceded this splat? > > > > > > Leaving that aside, the system appears to still be booting. There are > > > RCU CPU stall warning messages later on, and then the system hangs more > > > than six minutes while still booting, presumably due to the large number > > > of self-tests and debug options enabled. > > > > > > The intent is that the clocksource-wdtest tests run after boot has > > > completed. One approach would be to test it using modprobe after boot > > > has completed. In addition, the clocksource-wdtest module is not designed > > > to handle CPU overload conditions, and making it do so would reduce the > > > effectiveness of the test. > > > > > > I suggest setting clocksource-wdtest.holdoff=N, where "N" is in seconds > > > and is large enough that boot has completed. Alternatively, use modprobe > > > to activate this module from userspace after boot has completed. > > > > > > What I do is just set CONFIG_TEST_CLOCKSOURCE_WATCHDOG=y in an ordinary > > > rcutorture run, if that helps. > > > > All that aside, does the patch below help in your environment? > > sorry for late. > I applied below patch to f4c6b34ee1 ("clocksource: Provide kernel module to test clocksource watchdog") > still found the issue on both f4c6b34ee1 and patched kernel. > > attached dmesg.xz FYI Hey, I had to ask, and thank you for testing this. If I am reading dmesg.xz correctly, the system is still in the process of booting on a single CPU, and thus the watchdog might be being delayed for a very long time. I was hoping that waiting a minute would suffice, but apparently not. My next suggestion is to wait until the system boots, then use "modprobe clocksource-wdtest" to run the test from a shell. Or to test it via rcutorture, as discussed earlier. Other thoughts? Thanx, Paul > [ 157.028293] ------------[ cut here ]------------ > [ 157.031055] WARNING: CPU: 0 PID: 207 at kernel/time/clocksource-wdtest.c:154 wdtest_func.cold+0x6a/0x117 > [ 157.034366] Modules linked in: > [ 157.036812] CPU: 0 PID: 207 Comm: wdtest Tainted: G S W 5.13.0-rc1-00006-g2aaf396b0698 #1 > [ 157.040112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 > [ 157.043283] EIP: wdtest_func.cold+0x6a/0x117 > [ 157.045833] Code: 00 69 55 00 a1 24 eb ea da 83 c4 10 85 c0 74 02 0f 0b a1 04 32 9a d9 39 1d b0 0f 9a d9 0f 93 c1 83 e0 40 0f 94 c2 38 d1 74 02 <0f> 0 > b 85 c0 74 32 b8 c0 31 9a d9 e8 ca d7 3a fb b8 19 00 00 00 c7 > [ 157.052700] EAX: 00000040 EBX: 00000000 ECX: 00000001 EDX: 00000000 > [ 157.055738] ESI: d8abf847 EDI: d8abf836 EBP: c680ff84 ESP: c680ff78 > [ 157.058770] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010202 > [ 157.061854] CR0: 80050033 CR2: 00000000 CR3: 1a780000 CR4: 000406d0 > [ 157.064864] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > [ 157.067797] DR6: fffe0ff0 DR7: 00000400 > [ 157.070414] Call Trace: > [ 157.072676] kthread+0xf6/0x140 > [ 157.074915] ? wdtest_ktime_read+0x80/0x80 > [ 157.077140] ? kthread_insert_work_sanity_check+0x60/0x60 > [ 157.079472] ret_from_fork+0x1c/0x28 > [ 157.081558] irq event stamp: 533 > [ 157.083404] hardirqs last enabled at (541): [] console_unlock+0x4c5/0x600 > [ 157.085717] hardirqs last disabled at (550): [] console_unlock+0x435/0x600 > [ 157.087921] softirqs last enabled at (368): [] __do_softirq+0x2f0/0x44b > [ 157.090071] softirqs last disabled at (567): [] call_on_stack+0x45/0x60 > [ 157.092111] ---[ end trace 0b325ef1942d2bda ]--- > > > > > > If so, I can adjust so that my testing gets done quickly and yours > > avoids false-positive failures. > > > > Thanx, Paul > > > > ------------------------------------------------------------------------ > > > > diff --git a/kernel/time/clocksource-wdtest.c b/kernel/time/clocksource-wdtest.c > > index 01df12395c0e..0d8542f8b1d2 100644 > > --- a/kernel/time/clocksource-wdtest.c > > +++ b/kernel/time/clocksource-wdtest.c > > @@ -149,7 +149,7 @@ static int wdtest_func(void *arg) > > s = ", expect clock skew"; > > pr_info("--- Watchdog with %dx error injection, %lu retries%s.\n", i, max_cswd_read_retries, s); > > WRITE_ONCE(wdtest_ktime_read_ndelays, i); > > - schedule_timeout_uninterruptible(2 * HZ); > > + schedule_timeout_uninterruptible(60 * HZ); > > WARN_ON_ONCE(READ_ONCE(wdtest_ktime_read_ndelays)); > > WARN_ON_ONCE((i <= max_cswd_read_retries) != > > !(clocksource_wdtest_ktime.flags & CLOCK_SOURCE_UNSTABLE));