From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 943A9C433F5 for ; Mon, 11 Oct 2021 07:10:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2614360F24 for ; Mon, 11 Oct 2021 07:10:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2614360F24 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=canonical.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id AAA946B006C; Mon, 11 Oct 2021 03:10:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8DAE6B0071; Mon, 11 Oct 2021 03:10:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97100900002; Mon, 11 Oct 2021 03:10:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0061.hostedemail.com [216.40.44.61]) by kanga.kvack.org (Postfix) with ESMTP id 8AEDD6B006C for ; Mon, 11 Oct 2021 03:10:35 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 32D87181AEF23 for ; Mon, 11 Oct 2021 07:10:35 +0000 (UTC) X-FDA: 78683283630.24.F8A44CC Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) by imf29.hostedemail.com (Postfix) with ESMTP id 9C4EA901E59B for ; Mon, 11 Oct 2021 07:10:34 +0000 (UTC) Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id E6D4E3F324 for ; Mon, 11 Oct 2021 07:10:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1633936232; bh=h1SUzCf91Jwpb9fhrX0NRvLQL7EyCkzdP0gPLLjD4b8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=aWQClpvl6MpmGv9YTLKRHj5GGnXK8s2u8T7VdhdRFsL4lf0cgATAFNQrEoL/mNSqF dZSqBHS8N7J1UAHIKMT18lUVqy0jHk/UPvA7+XYn0BtUKRakQS3jhHDP38S3z5tJR/ UaR80i0SutViYWZOQZXIoZ0gIFjjkCsVVyxkQleFOY1rZ7CEepHknROdCEo762lzXf KP8u9T7L7A8DX0LGQ5vEZhZTF0Fhh/ill2mmcP91WDMwUenWGinlMRvBoNM0aK7Ynb xKaVm3oOHYlI7Apf7s3v+L+/XdY2Vc595yBTG39QnyaTNCZN648cJLZmrqTUHw1+K3 V53T2LWmy11WQ== Received: by mail-ed1-f69.google.com with SMTP id z23-20020aa7cf97000000b003db7be405e1so4102092edx.13 for ; Mon, 11 Oct 2021 00:10:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=h1SUzCf91Jwpb9fhrX0NRvLQL7EyCkzdP0gPLLjD4b8=; b=OSL6V46LTJB+HA7lB/HDJdxhPNmGQ9r3AF86uKYBefiUeGCzXpRIvL96uuodrRj9dt jbsvpur9r/y/BW8m5eEvWC4H6Kkn+OI09mu4iqODW3+vTY6LvWu0Vkh4aloZpRhZGh6B 6HDbWgThOK2kSg2UrXC2gUyvB2oBsiTogy0VMb4LOmwv797NTVN1S90Es7fZSVWL5k4p meXwLacgmqaOtSpg73GmSvmOQGbeTIEO0FaH6xbTo6gwM45OLvURS6BZlX0Yqo1adqhs zT0LCQWNol9/4EjJcfXPofUKhCZ8eKUQGkv7TP+WaYVwaacVSaoUUUaL8pjRxGqGqa6W z37g== X-Gm-Message-State: AOAM532TTyil7w11eVQAQRCq3gBXBkdZxzMSOf0JS4ueMoi0yIQgOAjc z5ZDL9N8lBufXtgJYvjHzG7XYJ6K1CuQb/yV5F3g5HfxbiC0MW0btAGo7kxD2JG9RY5+JxiuWLV d4xBprnJ5WEGrb12WZNKnwW5+GRGY X-Received: by 2002:a17:906:c009:: with SMTP id e9mr23871351ejz.509.1633936232561; Mon, 11 Oct 2021 00:10:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzaYokr373n9+pvypy7SyEpzzJy/337+0O1svYG2JXNNbPWvZ57+ivNRR1KCyf2hS8d9lVTrg== X-Received: by 2002:a17:906:c009:: with SMTP id e9mr23871340ejz.509.1633936232329; Mon, 11 Oct 2021 00:10:32 -0700 (PDT) Received: from localhost ([2001:67c:1560:8007::aac:c1b6]) by smtp.gmail.com with ESMTPSA id y19sm2560185edd.39.2021.10.11.00.10.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Oct 2021 00:10:31 -0700 (PDT) Date: Mon, 11 Oct 2021 09:10:30 +0200 From: Andrea Righi To: Marco Elver Cc: Alexander Potapenko , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: BUG: soft lockup in __kmalloc_node() with KFENCE enabled Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 9C4EA901E59B X-Stat-Signature: tf7g5kqpj4xrabwi58pqskmnbxbqdjy9 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b=aWQClpvl; dmarc=pass (policy=none) header.from=canonical.com; spf=pass (imf29.hostedemail.com: domain of andrea.righi@canonical.com designates 185.125.188.123 as permitted sender) smtp.mailfrom=andrea.righi@canonical.com X-HE-Tag: 1633936234-52828 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 11, 2021 at 08:48:29AM +0200, Marco Elver wrote: > On Mon, 11 Oct 2021 at 08:32, Andrea Righi wrote: > > On Mon, Oct 11, 2021 at 08:00:00AM +0200, Marco Elver wrote: > > > On Sun, 10 Oct 2021 at 15:53, Andrea Righi wrote: > > > > I can systematically reproduce the following soft lockup w/ the latest > > > > 5.15-rc4 kernel (and all the 5.14, 5.13 and 5.12 kernels that I've > > > > tested so far). > > > > > > > > I've found this issue by running systemd autopkgtest (I'm using the > > > > latest systemd in Ubuntu - 248.3-1ubuntu7 - but it should happen with > > > > any recent version of systemd). > > > > > > > > I'm running this test inside a local KVM instance and apparently systemd > > > > is starting up its own KVM instances to run its tests, so the context is > > > > a nested KVM scenario (even if I don't think the nested KVM part really > > > > matters). > > > > > > > > Here's the oops: > > > > > > > > [ 36.466565] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [udevadm:333] > > > > [ 36.466565] Modules linked in: btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear psmouse floppy > > > > [ 36.466565] CPU: 0 PID: 333 Comm: udevadm Not tainted 5.15-rc4 > > > > [ 36.466565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 > > > [...] > > > > > > > > If I disable CONFIG_KFENCE the soft lockup doesn't happen and systemd > > > > autotest completes just fine. > > > > > > > > We've decided to disable KFENCE in the latest Ubuntu Impish kernel > > > > (5.13) for now, because of this issue, but I'm still investigating > > > > trying to better understand the problem. > > > > > > > > Any hint / suggestion? > > > > > > Can you confirm this is not a QEMU TCG instance? There's been a known > > > issue with it: https://bugs.launchpad.net/qemu/+bug/1920934 > > > > It looks like systemd is running qemu-system-x86 without any "accel" > > options, so IIUC the instance shouldn't use TCG. Is this a correct > > assumption or is there a better way to check? > > AFAIK, the default is TCG if nothing else is requested. What was the > command line? This is the full command line of what systemd is running: /bin/qemu-system-x86_64 -smp 4 -net none -m 512M -nographic -vga none -kernel /boot/vmlinuz-5.15-rc4 -drive format=raw,cache=unsafe,file=/var/tmp/systemd-test.sI1nrh/badid.img -initrd /boot/initrd.img-5.15-rc4 -append root=/dev/sda1 rw raid=noautodetect rd.luks=0 loglevel=2 init=/lib/systemd/systemd console=ttyS0 selinux=0 SYSTEMD_UNIT_PATH=/usr/lib/systemd/tests/testdata/testsuite-14.units:/usr/lib/systemd/tests/testdata/units: systemd.unit=testsuite.target systemd.wants=testsuite-14.service systemd.wants=end.service And this is running inside a KVM instance (so a nested KVM scenario). -Andrea