From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E06BC433EF for ; Mon, 11 Oct 2021 06:32:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1611360C41 for ; Mon, 11 Oct 2021 06:32:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1611360C41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=canonical.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 66AC4900002; Mon, 11 Oct 2021 02:32:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61B106B0071; Mon, 11 Oct 2021 02:32:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 509BC900002; Mon, 11 Oct 2021 02:32:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0100.hostedemail.com [216.40.44.100]) by kanga.kvack.org (Postfix) with ESMTP id 4250E6B006C for ; Mon, 11 Oct 2021 02:32:11 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id CF885181AEF15 for ; Mon, 11 Oct 2021 06:32:10 +0000 (UTC) X-FDA: 78683186820.37.530CF5F Received: from smtp-relay-internal-0.canonical.com (smtp-relay-internal-0.canonical.com [185.125.188.122]) by imf07.hostedemail.com (Postfix) with ESMTP id 4BB541001C8B for ; Mon, 11 Oct 2021 06:32:10 +0000 (UTC) Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 887DA3FFFE for ; Mon, 11 Oct 2021 06:32:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1633933928; bh=lwPipFh22ocDNcUOEMMzIfuiqy6HWce+IYzbCS/BeoI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=nxEy/w5rIvrdRUi7ndV/Za9X7t0GqOASEwWzu0aKNlcm5kJW2IVyTUdi3xui0PHim QJWdfnpLWu1I0WzoGi7m/NhVPlVZkXpDyvA6+YYREuNli4GxpfqeHCwZAEMVs+w0dD VjnNGA3rHNCB6vajf6Hu83MslwVYaoD0RLka8HIgaONMjtllYKvbt3CDbjxXP7mfPz zVHB6fC0blGV7KgBLwd1h9q7lRkeEtkKF53dAcPHkp1FZ6wFHYMC2fNp7R99iUXwIr 5R9dhl9WgSoqUo2FzJEkGHuB81g8RGL9bJ7rGlFmAWYPsFIpKLDJ8nvEwH6kEZS881 S6purO0WyoUqQ== Received: by mail-ed1-f72.google.com with SMTP id t18-20020a056402021200b003db9e6b0e57so664255edv.10 for ; Sun, 10 Oct 2021 23:32:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=lwPipFh22ocDNcUOEMMzIfuiqy6HWce+IYzbCS/BeoI=; b=JwPor8wAqw4m+ESqyUoQyKF2J2z/ygE/d62llNmd/uzixGCAEK0zBV7yfWAoLrNoTi BKjIUKAyLhP/DJxL2JqL9MX3Kgh6K/0B8tMQMTk0y0yeGHD+kk1hDXU/6mMGzFPyM/+4 Ya79X0aE5bpBGRzYyAm3mEpZNNp1HJzQM1Q6fpql8yY5Ki6j5EWE7toTaAgX0neBPFqT zpDb9Ayzq1hQPugj1ZRF9eDdiQnlGhux4ZZDs9kYkD1zUT85HLppCUDx3fcLsotUYKDI f4J58OMf6vlOrnlYiPW3eqtHsExa15GNgVzAf7qW01CbwC5G4w+HEsK8Dsne5Ft7+ry+ NfHQ== X-Gm-Message-State: AOAM531/a2HrvS1NwCb1Dsa9SxH8buwzix0JhoWYpncIC7dOK9tgEf2V XHXQknRzEXUeq93wXWyPBHoeb9d64dcpFGFfecT77ib7l3g2oE7HcsUccg7G8tnDOQEjuJanABH XdcLOU/VsOz7hUgBeAXu3r3XAuElz X-Received: by 2002:a17:907:334f:: with SMTP id yr15mr7244671ejb.8.1633933928135; Sun, 10 Oct 2021 23:32:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyiQVas22w3YLNrLfxsQfnioldXrMyoqDjEYmIYtoIDG9IU242B0gbfD03+5OA9YzDxoQlL+Q== X-Received: by 2002:a17:907:334f:: with SMTP id yr15mr7244636ejb.8.1633933927912; Sun, 10 Oct 2021 23:32:07 -0700 (PDT) Received: from localhost ([2001:67c:1560:8007::aac:c1b6]) by smtp.gmail.com with ESMTPSA id o15sm2945745ejj.10.2021.10.10.23.32.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Oct 2021 23:32:07 -0700 (PDT) Date: Mon, 11 Oct 2021 08:32:05 +0200 From: Andrea Righi To: Marco Elver Cc: Alexander Potapenko , Dmitry Vyukov , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: BUG: soft lockup in __kmalloc_node() with KFENCE enabled Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4BB541001C8B X-Stat-Signature: pm1aci9g893b9tm4999ocyz1iwcsk951 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=canonical.com header.s=20210705 header.b="nxEy/w5r"; dmarc=pass (policy=none) header.from=canonical.com; spf=pass (imf07.hostedemail.com: domain of andrea.righi@canonical.com designates 185.125.188.122 as permitted sender) smtp.mailfrom=andrea.righi@canonical.com X-HE-Tag: 1633933930-740222 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 11, 2021 at 08:00:00AM +0200, Marco Elver wrote: > On Sun, 10 Oct 2021 at 15:53, Andrea Righi wrote: > > I can systematically reproduce the following soft lockup w/ the latest > > 5.15-rc4 kernel (and all the 5.14, 5.13 and 5.12 kernels that I've > > tested so far). > > > > I've found this issue by running systemd autopkgtest (I'm using the > > latest systemd in Ubuntu - 248.3-1ubuntu7 - but it should happen with > > any recent version of systemd). > > > > I'm running this test inside a local KVM instance and apparently systemd > > is starting up its own KVM instances to run its tests, so the context is > > a nested KVM scenario (even if I don't think the nested KVM part really > > matters). > > > > Here's the oops: > > > > [ 36.466565] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [udevadm:333] > > [ 36.466565] Modules linked in: btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear psmouse floppy > > [ 36.466565] CPU: 0 PID: 333 Comm: udevadm Not tainted 5.15-rc4 > > [ 36.466565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 > [...] > > > > If I disable CONFIG_KFENCE the soft lockup doesn't happen and systemd > > autotest completes just fine. > > > > We've decided to disable KFENCE in the latest Ubuntu Impish kernel > > (5.13) for now, because of this issue, but I'm still investigating > > trying to better understand the problem. > > > > Any hint / suggestion? > > Can you confirm this is not a QEMU TCG instance? There's been a known > issue with it: https://bugs.launchpad.net/qemu/+bug/1920934 It looks like systemd is running qemu-system-x86 without any "accel" options, so IIUC the instance shouldn't use TCG. Is this a correct assumption or is there a better way to check? > > One thing that I've been wondering is, if we can make > CONFIG_KFENCE_STATIC_KEYS=n the default, because the static keys > approach is becoming more trouble than it's worth. It requires us to > re-benchmark the defaults. If you're thinking of turning KFENCE on by > default (i.e. CONFIG_KFENCE_SAMPLE_INTERVAL non-zero), you could make > this decision for Ubuntu with whatever sample interval you choose. > We've found that for large deployments 500ms or above is more than > adequate. Another thing that I forgot to mention is that with CONFIG_KFENCE_STATIC_KEYS=n the soft lockup doesn't seem to happen. Thanks, -Andrea