From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 941FCC46467 for ; Wed, 11 Jan 2023 08:31:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F8CA8E0002; Wed, 11 Jan 2023 03:31:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A8DF8E0001; Wed, 11 Jan 2023 03:31:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 072018E0002; Wed, 11 Jan 2023 03:31:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E95FF8E0001 for ; Wed, 11 Jan 2023 03:31:25 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B84EC1606A1 for ; Wed, 11 Jan 2023 08:31:25 +0000 (UTC) X-FDA: 80341848930.10.D7ED115 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf11.hostedemail.com (Postfix) with ESMTP id 206764000A for ; Wed, 11 Jan 2023 08:31:23 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b="cmuaLl7/"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf11.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673425884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oub3vFhESlSmnZ3siI+/gjjg/Dwj1NqJPNyjl/9KfIQ=; b=vy9aJE9V4wUhT8hmjn1C+IzyOSXLNq/AgWMqxmkL9HnJhNGY+vYXAQWulVdswE7E1507U1 XQ4ORLgajWjNsJFt2WHJwv5HlaCQc2p2d3MFNS9ZB+yIGCRwiIWEyrL/S2iWAhbycHx9HI +0vf4bQBN2aXeRo6vREkLC49q/jbrq8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b="cmuaLl7/"; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf11.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.28 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673425884; a=rsa-sha256; cv=none; b=mverhOhpmFNbV0Jir9WvnV9OzQfWp02/n1C5sPElLxlUq7LTVB4Bj/kkjpdAAqmRD4pRaL jhEEGJMD+zUOAxiFFxnVl0HcT/XnmWa911dDr4whQp2UHir68eYr6f14+cdRyCz+mmz0Mw uZbuJ/LGL44di77Vxg2z8TwRzvroCmk= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7394916F67; Wed, 11 Jan 2023 08:31:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1673425882; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=oub3vFhESlSmnZ3siI+/gjjg/Dwj1NqJPNyjl/9KfIQ=; b=cmuaLl7/52pfKxXUc/CBXEiqKnPqLpQWCowccOGASSIbzhEdDU1PcadMlGNfelOEeS253n ieQeV/pM1Hoiy2NrnFHJWhXDGxSQSohTJtBMOF6FbkZpu4vJMh4i2+Kn9jgKphqnyao8NA Al0ZewKR17ODODZJfuuTf6aiYiT5Oag= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 5215113591; Wed, 11 Jan 2023 08:31:22 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 9cEiEdpzvmOURgAAMHmgww (envelope-from ); Wed, 11 Jan 2023 08:31:22 +0000 Date: Wed, 11 Jan 2023 09:31:21 +0100 From: Michal Hocko To: Jiri Slaby Cc: Paolo Bonzini , kvm@vger.kernel.org, Andrew Morton , mm , yuzhao@google.com, Vlastimil Babka , shy828301@gmail.com Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 206764000A X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: fi1mum1q6si1bfhmmk1di638dr37dc3m X-HE-Tag: 1673425883-647882 X-HE-Meta: U2FsdGVkX18+YmxkBNZUIY2SIYy9d61ss4LR5QFv1mYVYDUD8ANGe9X8xq43bDWdROzeRqrXjtPSyYrB8pP3CNNMDLdzcKrwbih3xovEmn69rAhmHWnX0+3y5CDnX2hXLH0XzXntPEBbDeG0EKjhbQV176O89O4t8JesCWNhNknJUf0U4vWS1C4M2F49LPz+3m4/NXGTPiS5BWL1MZaqdI7dPfpyLHjf4k0a2mTT1XbuWWFMYzqOUb2vsg8PfzWcXOI9HLwGtHaq3qIQeLG4lp8K8+pGOHN1QewnfWZVejg2XF/bCvsdVqjnAnXRnmh4egxE1M+ra32JYh7JJvb7sZG8OkV6vmKJex3ODOjZu07hzt5ArR+xshxtLPQdaOT9L94N4BBQwtSPGUPpXGVYyNutZI7m68eOMfEN8D92gAZ5RAuKK7Z9fpEn7ecPEQFEHqkc+9TN/h50hLFoKh1GZApfuYq3O95kacQzbVdG/hS+yBSmkqQng7yYr9vMCgjILfKBVfUyIkdGiJzEbMOv/83oO0ydVnm8L30cYwYjgMAlhrEgCkCNnFjJTaQnZ9c6YTcUAEe2AeSAm4timfbHKR5qN3kqNk+a1hFFtiwbWj1qONhvx1WErnkwqaK6F7HXnBW04tn/HnMuc6LRih86b7tthJ0jgDtywaebU53DICMVN3CqPEUXeIpaAN01eEgvodlpRwXo5CzXXfreZBLxwx7lG59mBTrJZ2e+Qjm/JHxXGknn3LlJhsBlAx/Bhsp+ZiRdyDAhH4+uXZDA1eLkVW5GTzUaJlPFA6MDzo4+lIdazN8WPnYYY1kEDhL4lOXC4nYbBSuQP1u/M6Zhp7efxUIrCMruimHDH1eDmf0Fzo/SNOQfn5NHWVgXR2UBxCY/ovTk53IDtrXvZwKsjsIfSfq0vNo+5qp3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 11-01-23 09:00:04, Jiri Slaby wrote: > Hi, > > after I updated the host from 6.0 to 6.1 (being at 6.1.4 ATM), my qemu VMs > started stalling (and the host at the same point too). It doesn't happen > right after boot, maybe a suspend-resume cycle is needed (or longer uptime, > or a couple of qemu VM starts, or ...). But when it happens, it happens all > the time till the next reboot. > > Older guest's kernels/distros are affected as well as Win10. > > In guests, I see for example stalls in memset_orig or > smp_call_function_many_cond -- traces below. > > qemu-kvm-7.1.0-13.34.x86_64 from openSUSE. > > It's quite interesting that: > $ cat /proc//cmdline > is stuck at read: > > openat(AT_FDCWD, "/proc/12239/cmdline", O_RDONLY) = 3 > newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0 > fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 > mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f22f0487000 > read(3, ^C^C^C^\^C This will require mmap_lock for read as well. And that is blocked because there is a pending writer as well. [...] > Is this known? Any idea how to debug this? I would recommend taking a crashdump when the host is in that state and examine the state of the blocked lock. Dump will hopefully give you more information about potential holder of the lock. If it is blocked on writer then you should get the owner task. If it is in read mode then this can get more tricky because the exact owner might be different from the recorded one. Anyway the full list of tasks and their backtraces could tell more. HTH -- Michal Hocko SUSE Labs