From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 579AEC433EF for ; Mon, 11 Oct 2021 15:16:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF10760EE9 for ; Mon, 11 Oct 2021 15:16:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DF10760EE9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 527C56B0071; Mon, 11 Oct 2021 11:16:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AF3B900002; Mon, 11 Oct 2021 11:16:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3774A6B0073; Mon, 11 Oct 2021 11:16:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0248.hostedemail.com [216.40.44.248]) by kanga.kvack.org (Postfix) with ESMTP id 222636B0071 for ; Mon, 11 Oct 2021 11:16:43 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D6AF3181D6078 for ; Mon, 11 Oct 2021 15:16:42 +0000 (UTC) X-FDA: 78684508644.38.0881CAA Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) by imf27.hostedemail.com (Postfix) with ESMTP id 8706270148F4 for ; Mon, 11 Oct 2021 15:16:42 +0000 (UTC) Received: by mail-ot1-f43.google.com with SMTP id v2-20020a05683018c200b0054e3acddd91so17581610ote.8 for ; Mon, 11 Oct 2021 08:16:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=etYgg+h2O5V3SwL5gD13pIM27MsV6K0L3L1sqF1wYTE=; b=el38wyOt6M4eOeshwmiPoV7jMG5ZX6uvIFaNnvNoSJYlaEEJhW2pZPAWYzAg454+LD 1E5js8LOfImu5A1wtJiHp6RfkDoqcMXVa7wnE66jUxr7VQNkJBG/hDq/rfVVcSj6oUGO 2uDquzaB0K4L2mod1iI7TG7b8zmA/0AMXCqQmdBYvGrF9Qvp71qj8/O2IKRhH6ijvE2l vB+AoW3OuMTrolDnGAfnkKMHWAXmWmZHenMyNVlQXP7zuXY8OgTCulNHtvsYjOnGEex5 RNguP7gYMbN75QetPxH3xlevB6ew5ulzPIyVe6V311Fa09+4UsjP4cpUxUzJqRJyppIb pNKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=etYgg+h2O5V3SwL5gD13pIM27MsV6K0L3L1sqF1wYTE=; b=SC30Ze/++gISZZhivCwAL8axC5MgeO2kK21BYJptsfHVFGw6n38mO9kN/lyNvf7zq+ 6ApkiJzKfLMDIVY7U4tc2kGexpxr5ft6V9V4luh3rRcRb1KZ8IQG75WfKgkV5iahtMGG rwd0fwUIm+8Fl77vokrcTlFM7SwmLbn5tDZM6GY+Nn/tv2oYWbpfoKHjauC86x61jZy2 qBvEkjbrmz+8wrm7zQAyXLDoNyEcScRbdCPO+aiqrJaK/GQyXBHob4vX4FzHluCMzYSY Ihjnd/lN3tzhSPGL8ImDAJU9IYAMyeG0kfA8Q8qlBpN9a5w4kiHg8TdrU3e3GtdJ5NQW a/Aw== X-Gm-Message-State: AOAM532nFF7AjBJAPd2w9wwFTB95/gEm9QoO+qbT9I9ZGYMcmi8Qz5Sm zVPrFeWAdJWQZS9P+DbC1si5WyXJyV1qS1t/N4/E0g== X-Google-Smtp-Source: ABdhPJy3IwrhJ7udHn/k4mW8pPjAGJ7Y5Fqmen34KA6kKWe57E+fesp4mtWBLidlT1RKOa4qSiZheM5CxtVOr2PEVAc= X-Received: by 2002:a05:6830:402c:: with SMTP id i12mr7503223ots.319.1633965401529; Mon, 11 Oct 2021 08:16:41 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dmitry Vyukov Date: Mon, 11 Oct 2021 17:16:30 +0200 Message-ID: Subject: Re: BUG: soft lockup in __kmalloc_node() with KFENCE enabled To: Andrea Righi Cc: Marco Elver , Alexander Potapenko , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8706270148F4 X-Stat-Signature: d9z9xbt3dhiua89gu1xbzqut6o4861us Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=el38wyOt; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of dvyukov@google.com designates 209.85.210.43 as permitted sender) smtp.mailfrom=dvyukov@google.com X-HE-Tag: 1633965402-625705 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 11 Oct 2021 at 17:11, Andrea Righi wrote: > > On Mon, Oct 11, 2021 at 05:00:15PM +0200, Dmitry Vyukov wrote: > > On Mon, 11 Oct 2021 at 16:42, Andrea Righi wrote: > > > > > > On Mon, Oct 11, 2021 at 12:03:52PM +0200, Marco Elver wrote: > > > > On Mon, 11 Oct 2021 at 11:53, Andrea Righi wrote: > > > > > On Mon, Oct 11, 2021 at 11:23:32AM +0200, Andrea Righi wrote: > > > > > ... > > > > > > > You seem to use the default 20s stall timeout. FWIW syzbot uses 160 > > > > > > > secs timeout for TCG emulation to avoid false positive warnings: > > > > > > > https://github.com/google/syzkaller/blob/838e7e2cd9228583ca33c49a39aea4d863d3e36d/dashboard/config/linux/upstream-arm64-kasan.config#L509 > > > > > > > There are a number of other timeouts raised as well, some as high as > > > > > > > 420 seconds. > > > > > > > > > > > > I see, I'll try with these settings and see if I can still hit the soft > > > > > > lockup messages. > > > > > > > > > > Still getting soft lockup messages even with the new timeout settings: > > > > > > > > > > [ 462.663766] watchdog: BUG: soft lockup - CPU#2 stuck for 430s! [systemd-udevd:168] > > > > > [ 462.755758] watchdog: BUG: soft lockup - CPU#3 stuck for 430s! [systemd-udevd:171] > > > > > [ 924.663765] watchdog: BUG: soft lockup - CPU#2 stuck for 861s! [systemd-udevd:168] > > > > > [ 924.755767] watchdog: BUG: soft lockup - CPU#3 stuck for 861s! [systemd-udevd:171] > > > > > > > > The lockups are expected if you're hitting the TCG bug I linked. Try > > > > to pass '-enable-kvm' to the inner qemu instance (my bad if you > > > > already have), assuming that's somehow easy to do. > > > > > > If I add '-enable-kvm' I can triggering other random panics (almost > > > immediately), like this one for example: > > > > > > [21383.189976] BUG: kernel NULL pointer dereference, address: 0000000000000098 > > > [21383.190633] #PF: supervisor read access in kernel mode > > > [21383.191072] #PF: error_code(0x0000) - not-present page > > > [21383.191529] PGD 0 P4D 0 > > > [21383.191771] Oops: 0000 [#1] SMP NOPTI > > > [21383.192113] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15-rc4 > > > [21383.192757] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014 > > > [21383.193414] RIP: 0010:wb_timer_fn+0x44/0x3c0 > > > [21383.193855] Code: 41 8b 9c 24 98 00 00 00 41 8b 94 24 b8 00 00 00 41 8b 84 24 d8 00 00 00 4d 8b 74 24 28 01 d3 01 c3 49 8b 44 24 60 48 8b 40 78 <4c> 8b b8 98 00 00 00 4d 85 f6 0f 84 c4 00 00 00 49 83 7c 24 30 00 > > > [21383.195366] RSP: 0018:ffffbcd140003e68 EFLAGS: 00010246 > > > [21383.195842] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004 > > > [21383.196425] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9a3521f4fd80 > > > [21383.197010] RBP: ffffbcd140003e90 R08: 0000000000000000 R09: 0000000000000000 > > > [21383.197594] R10: 0000000000000004 R11: 000000000000000f R12: ffff9a34c75c4900 > > > [21383.198178] R13: ffff9a34c3906de0 R14: 0000000000000000 R15: ffff9a353dc18c00 > > > [21383.198763] FS: 0000000000000000(0000) GS:ffff9a353dc00000(0000) knlGS:0000000000000000 > > > [21383.199558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [21383.200212] CR2: 0000000000000098 CR3: 0000000005f54000 CR4: 00000000000006f0 > > > [21383.200930] Call Trace: > > > [21383.201210] > > > [21383.201461] ? blk_stat_free_callback_rcu+0x30/0x30 > > > [21383.202692] blk_stat_timer_fn+0x138/0x140 > > > [21383.203180] call_timer_fn+0x2b/0x100 > > > [21383.203666] __run_timers.part.0+0x1d1/0x240 > > > [21383.204227] ? kvm_clock_get_cycles+0x11/0x20 > > > [21383.204815] ? ktime_get+0x3e/0xa0 > > > [21383.205309] ? native_apic_msr_write+0x2c/0x30 > > > [21383.205914] ? lapic_next_event+0x20/0x30 > > > [21383.206412] ? clockevents_program_event+0x94/0xf0 > > > [21383.206873] run_timer_softirq+0x2a/0x50 > > > [21383.207260] __do_softirq+0xcb/0x26f > > > [21383.207647] irq_exit_rcu+0x8c/0xb0 > > > [21383.208010] sysvec_apic_timer_interrupt+0x7c/0x90 > > > [21383.208464] > > > [21383.208713] asm_sysvec_apic_timer_interrupt+0x12/0x20 > > > > > > I think that systemd autotest used to use -enable-kvm, but then they > > > removed it, because it was introducing too many problems in the nested > > > KVM context. I'm not sure about the nature of those problems though, I > > > can investigate a bit and see if I can understand what they were > > > exactly. > > > > This looks like just a plain bug in wb_timer_fn, not something related > > to virtualization. > > Do you have this fix? > > https://syzkaller.appspot.com/bug?extid=aa0801b6b32dca9dda82 > > Yes, it looks like I have this: > > d152c682f03c block: add an explicit ->disk backpointer to the request_queue Then there is another bug in wb_timer_fn I guess... Don't know if this is the same or something else: https://lore.kernel.org/lkml/CAHbLzkrdGva2dzO36r62LKv_ip5trbMK0BO3vCeSBk2_7OE-zA@mail.gmail.com/ There also were some data races in this function: https://groups.google.com/g/syzkaller-upstream-moderation/search?q=wb_timer_fn