From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7F5FC54EBD for ; Thu, 12 Jan 2023 08:23:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E0418E0003; Thu, 12 Jan 2023 03:23:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3902E8E0001; Thu, 12 Jan 2023 03:23:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2584E8E0003; Thu, 12 Jan 2023 03:23:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 15B268E0001 for ; Thu, 12 Jan 2023 03:23:57 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D2A80C0192 for ; Thu, 12 Jan 2023 08:23:56 +0000 (UTC) X-FDA: 80345458872.12.9F9BFED Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf04.hostedemail.com (Postfix) with ESMTP id 433694000F for ; Thu, 12 Jan 2023 08:23:55 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BtyBdfPy; spf=pass (imf04.hostedemail.com: domain of yuzhao@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673511835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IG/lXuBi4W6bnR7FT/EfwptVnjmC9kTL2vImM/29XUc=; b=VsfkNlAiNh5NHZ/qnyBSrCpRz4L7w9Lj+LZe+i7zWsuP+l0H1EDhbseMlsiI7OrCTA1d/y N45V/tSOa/08hJVaGOV/LweSiKa0/xbt+spYqt7UGkCIcDwdw0/5WhbXFLqVq2woun4+OO +RT+BB9hPg8jle5wuVRUDC5u27ohG4A= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BtyBdfPy; spf=pass (imf04.hostedemail.com: domain of yuzhao@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673511835; a=rsa-sha256; cv=none; b=whgO1jvn4HxBL1nkWTICHemzztsgzc6i9jinqQoPOsmQnLfZVoFxlPxeD/+vWcvKPT1CRa hQwX6n5cEGsAZ5BeBCQ4Q+f9QsJbGxi5v7C6YytDWo9cPWOyzIhJJDOWhNXH9YlbJkF+KI YGJx7jF8+AAhvD4d2XE5lPOEczJ235k= Received: by mail-yb1-f176.google.com with SMTP id o75so17849008yba.2 for ; Thu, 12 Jan 2023 00:23:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IG/lXuBi4W6bnR7FT/EfwptVnjmC9kTL2vImM/29XUc=; b=BtyBdfPy7rq6ANcxjxuZcWUkW0PzQqpxPNP5gSH+U5v3D7Z8NJ2VgKNIOme6aymNP4 2CzNQKk7aKEInGLp07mD3Vtk2QbhpvcBlgdYHlxd4PjkKDbQJIFTaWJHVFV2F5kxQwgy dJpBUXRjvVam0h11QcBrq1NswA/SrewqD//v9uTpdu9igB6kJJCh53F/k0puaRBEWpjA zoL5tsgmLasj1Wx5oqhyrxCb5J6dRQMxJLIdxbNCSjpltGSF1HqWkzDz4/biCAbxI6KK 0cpMA4hH4qIg8PAv+rizRQlxa/zq3CpwTWma2uz+20WvBTzjHJ5cKAGn4COGOXiZRG9I ZItA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IG/lXuBi4W6bnR7FT/EfwptVnjmC9kTL2vImM/29XUc=; b=kEWhXknM6PrjhA7XIbwNmS7QyhkK/xmXIgYPYW0ZKNF6qvxNw4K1v+pokgNK1KqC5D K/yI71BbyiNpHeEVk422pPw8GB72nMovgAfCtFhXadN8r2rLSUxvjLSu/cDrnps4IaGX SyYDhoL5+dDVmkVZt7vjdtpH69ptKLB6PZT71P55npdCZMdZUnltFaHxSHNZOsQFrWsf hZzLs25/FV8xV+qzbFkWXmgGTDomRDpMfvPOYUSeB2Gd1Ktk5pA4YJVdXTeGK7pt9OAc 09V3uScE7oDvnbQLN8m8aY7gqt/xa2q/CoHrb6llvAoRgytggmSm922AYFPQEBjrDCz3 6+mw== X-Gm-Message-State: AFqh2kpO5/XKQVFQwkRvXTRNRf9hUwaD5Cy+7UAqeWw8L1DvNR8ABZJb K1cBHC/SkGTEIdbYHsDyE2ZT2R1iseOL90/bWhSSRQ== X-Google-Smtp-Source: AMrXdXsmmJiY8FKovMiOWbRsrx15VJu0Wp4I8ApmSa2FHOQC74u6+SLogHjTvt9IVADl1tflz7b8tjYRtMbQWKhszPU= X-Received: by 2002:a25:eb0b:0:b0:73e:1951:3e92 with SMTP id d11-20020a25eb0b000000b0073e19513e92mr2672672ybs.388.1673511834231; Thu, 12 Jan 2023 00:23:54 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Yu Zhao Date: Thu, 12 Jan 2023 01:23:17 -0700 Message-ID: Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) To: Pedro Falcato , Liam Howlett Cc: Jiri Slaby , Paolo Bonzini , kvm@vger.kernel.org, Andrew Morton , mm , Michal Hocko , Vlastimil Babka , shy828301@gmail.com Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 433694000F X-Stat-Signature: 19qb7dcmafqwkhcmg5z9d9jotsypghym X-HE-Tag: 1673511835-947626 X-HE-Meta: U2FsdGVkX18WQQPD+vjHcgDEuWdVGUeaP/LrbPnJ3vu2LjS1eV38l24S/uPdurziEs2UtQ1FbVzvmk7EukGIaTl8KSa4G0bpfmyMdVl+8REWSk8UwV7l+oxpX67jZQv6FC76+QtEH5IONVs38Z8snl1ARZsRarQCH6ZdXZ35mzV9rf3rZ/fxVIwH3i9kSucniyJVBkYTaJJLE3w0pBnjwNRSbcpN9C9smsIJkJLJ3bIw0oMPaNTAssuQzyct2XvFtsSZojetHqJVaW8hoToG550tBwAOCoDnw3UYK/CZKdQO2VctdjEYsIt4d5eNoNYLmh6n8Ky+tbBJwmjvQKTN6hur+FuBdvdwxpdLEEjyPOR7eXGoN3Be5ZVLeuE2QV9K+3renVELh64tJBGXfC8JVw9IpO8t+1QfQaDUsqVaP2hE5Lh/+rlXmI3mB46QX4f74Z+L2lDCktuVFpXoFdHnHnx28S8xm6KuehGLwHpM7KXVLHr90pPCsMHBlR0cTXQMXxmIZTUJVyGJfXgPewtIDd/hLBGRx3tNYnWRKt94Bi13djNdQ7W5t5414rskEo/FnZ5GDGhPUmwuCPODq7EWapXJSFhP86cIJYoenOjMnl6XjOddPtxrL1o7D9jLrvupYRPX8tMHegZ8Ei342LbjiPWynTkcvBtHrWZRwTpvFaoSac8/fGV6pBX4ej3EsNOv5scr27xUV9dUtphVf5rTQ6uY5I3vyFHLhmKsyJM3HNz/2bTvlwZCcBdJNVkAHIdw6UF4ERpKrcm15qGkZO3pMgn8mwYLnYpeCvZlZuN39iH+YTanb+d1gcx05B8z9m1qVJEdM1/dQtlfLxT+VTj9ZGy77Sm6D4Ok5+NodefNO9wPye+ABjUdIbdL3fvVt1coktoJwCtt2eXWKUEPg6Dp1pVo+CHQwi/hhrw25Gub8hdFZYloql2LTCvY3TZqhECsKNblUG+dCP87hEyXBgB +iCnw0l1 jiH2pDm/nVj/7uhM9sKIQwxwgOAD1NauPAsIAEEdqsurHS1wKHc6LJ0Ingtqqh2OHRv9XNJv0cvw3JXhmq0TwVdN3oD750WVBEsFN1XsHhLGP3VHGMVutocj3IFWY/kMcKfGm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 11, 2023 at 5:37 PM Pedro Falcato wrote: > > On Wed, Jan 11, 2023 at 8:00 AM Jiri Slaby wrote: > > > > Hi, > > > > after I updated the host from 6.0 to 6.1 (being at 6.1.4 ATM), my qemu > > VMs started stalling (and the host at the same point too). It doesn't > > happen right after boot, maybe a suspend-resume cycle is needed (or > > longer uptime, or a couple of qemu VM starts, or ...). But when it > > happens, it happens all the time till the next reboot. > > > > Older guest's kernels/distros are affected as well as Win10. > > > > In guests, I see for example stalls in memset_orig or > > smp_call_function_many_cond -- traces below. > > > > qemu-kvm-7.1.0-13.34.x86_64 from openSUSE. > > > > It's quite interesting that: > > $ cat /proc//cmdline > > is stuck at read: > > > > openat(AT_FDCWD, "/proc/12239/cmdline", O_RDONLY) = 3 > > newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 > > fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 > > mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > > 0) = 0x7f22f0487000 > > read(3, ^C^C^C^\^C > > > > too. So I dumped blocked tasks (sysrq-w) on _host_ (see below) and > > everything seems to stall on mmap_read_lock() or > > mmap_write_lock_killable(). I don't see the hog (the one actually > > _having_ and sitting on the (presumably write) lock) in the dump though. > > I will perhaps boot a LOCKDEP-enabled kernel, so that I can do sysrq-d > > next time and see the holder. > > > > > > There should be enough free memory (note caches at 8G): > > total used free shared buff/cache > > available > > Mem: 15Gi 10Gi 400Mi 2,5Gi 8,0Gi > > 5,0Gi > > Swap: 0B 0B 0B > > > > > > I rmmoded kvm-intel now, so: > > qemu-kvm: failed to initialize kvm: No such file or directory > > qemu-kvm: falling back to tcg > > and it behaves the same (more or less expected). > > > > Is this known? Any idea how to debug this? Or maybe someone (I CCed a > > couple of guys who Acked mmap_*_lock() shuffling patches in 6.1) has a > > clue? Bisection is hard as it reproduces only under certain unknown > > circumstances. > > Hi, > > I just want to chime in and say that I've also hit this regression > right as I (Arch) updated to 6.1 a few weeks ago. > This completely ruined my qemu workflow such that I had to fallback to > using an LTS kernel. > > Some data I've gathered: > 1) It seems to not happen right after booting - I'm unsure if this is > due to memory pressure or less CPU load or any other factor > 2) It seems to intensify after swapping a fair amount? At least this > has been my experience. > 3) The largest slowdown seems to be when qemu is booting the guest, > possibly during heavy memory allocation - problems range from "takes > tens of seconds to boot" to "qemu is completely blocked and needs a > SIGKILL spam". > 4) While traditional process monitoring tools break (likely due to > mmap_lock getting hogged), I can (empirically, using /bin/free) tell > that the system seems to be swapping in/out quite a fair bit > > My 4) is particularly confusing to me as I had originally blamed the > problem on the MGLRU changes, while you don't seem to be swapping at > all. > Could this be related to the maple tree patches? Should we CC both the > MGLRU folks and the maple folks? I don't think it's MGLRU because the way it uses mmap_lock is very simple. Also you could prevent MGLRU from taking mmap_lock by echo 3 >/sys/kernel/mm/lru_gen/enabled, or you could disable MGLRU entirely by echoing 0 to the same file, or even at build time, to rule it out. (I assume you turned on MGLRU in the first place.) Adding Liam. He can speak for the maple tree.