From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B0CCC54EBD for ; Thu, 12 Jan 2023 10:35:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E79B38E0002; Thu, 12 Jan 2023 05:35:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E29448E0001; Thu, 12 Jan 2023 05:35:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF3288E0002; Thu, 12 Jan 2023 05:35:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BEE338E0001 for ; Thu, 12 Jan 2023 05:35:30 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 985E91A0C4A for ; Thu, 12 Jan 2023 10:35:30 +0000 (UTC) X-FDA: 80345790420.11.C2E0497 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf10.hostedemail.com (Postfix) with ESMTP id B8E32C001E for ; Thu, 12 Jan 2023 10:35:28 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=H4VWT2yB; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=g5So4v2c; dmarc=none; spf=pass (imf10.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673519729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BUVmg7NGHndbDWIy/H+0rPDMMgMi+epK1rYp+m3mbJw=; b=WcPokKwehm9MiXFdepVzjww/SfjN5ML61OGbNwLEe8PNbfZDw5MRtnEKceN48R1JwuoB8P xxokWHngW43ANMYHxS2wTZxqRBgE+YnsphzIx7SjFnhQOhxeUVFK7PMfItPe7MJGHxR42o 4fcqGvAC3X8C0apbjRR1htx3wwYm7VQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=H4VWT2yB; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=g5So4v2c; dmarc=none; spf=pass (imf10.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673519729; a=rsa-sha256; cv=none; b=jrrm5nxdUJtk8OFArO+u6EFhwUgh0l3Ddr9uhaHtCuysdxEpvbdk630KOVJuKagXiPHSk+ aGkScGlHavc3EhvkHvH59FRNnS/uAfK3mXdBRI1gXh7f3dRJ2G/e9W0y4tPCKK4FFCCGen gs33lVYi+n9w8MBdEMv4bQpGKk08yYM= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E518B3FB9C; Thu, 12 Jan 2023 10:35:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1673519726; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BUVmg7NGHndbDWIy/H+0rPDMMgMi+epK1rYp+m3mbJw=; b=H4VWT2yBKVYJ62gR93K6HDFZJXMsEbUT23XNl0/W2/sVjYoZDBfEnGGjqBFNuqxPgcyWs3 ZAL4rRKRENtJ2GgsvA84qTnsBaHoH+xMsBssHofM02cIksVw/KTm24W83f9AP1tRVqTxO+ eq2vLv3H9PLX3xjMu/dE08/2Hpoibxo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1673519726; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BUVmg7NGHndbDWIy/H+0rPDMMgMi+epK1rYp+m3mbJw=; b=g5So4v2cWk+0WhY3mWEnt5Ya2kZDUof+QmIWtB1RxKLJy/fPs2MeXUEio7fV6qvgcCDM7z boEPGm6qf2ae44Bg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B7F8613776; Thu, 12 Jan 2023 10:35:26 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 1vUxLG7iv2MmKQAAMHmgww (envelope-from ); Thu, 12 Jan 2023 10:35:26 +0000 Message-ID: <1d278b52-5c75-1f95-adf8-3e9c699cf6ca@suse.cz> Date: Thu, 12 Jan 2023 11:35:26 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) Content-Language: en-US To: Maxim Levitsky , Jiri Slaby , Pedro Falcato Cc: Paolo Bonzini , kvm@vger.kernel.org, Andrew Morton , mm , yuzhao@google.com, Michal Hocko , shy828301@gmail.com, Mel Gorman References: <7aa90802-d25c-baa3-9c03-2502ad3c708a@kernel.org> <6d989ca2330748ed682c81fc5f43e054a70e70a8.camel@redhat.com> From: Vlastimil Babka In-Reply-To: <6d989ca2330748ed682c81fc5f43e054a70e70a8.camel@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B8E32C001E X-Stat-Signature: o8qrw3t3rdn5taxz8py4ftzhdqakcgnc X-HE-Tag: 1673519728-125400 X-HE-Meta: U2FsdGVkX1+LgdF2DX357AioN7qtVR4XeCh6BTtbz3WRVYBXOlimQVt751L3lagKY2oBg7V3f2llSXTumTC2SNqURK8GmxdjXZ12Emszo9UbmYT1GOliJ8Vc1+gH+dos0AcJNcl3zmsaqR0fmLkfclU1pdtTUfvdp/w0QkacRy+EBJnTVEkxBt65qsIXrBwWtMABpyV+x+VagPiLBTqWmU9lkzEutT9/3Eaa7NguxrimosJoLiESTWS8n0EwRzmAtFfpVv7UKVv9cybY4UOvjLAy1HNAlH7nYVmJodwO/pwcFt5DeE5/kQnk1FEb0xI3S7ZNW8YO8K5D62Dl9vlRY5ITIhIhpUi2ZrUAfQGoG1RvrRFCMVQP3GYcQEV5//58I1e7mBYQrXxLRoTM6TCySQ9vcbbubq4pILBNDpSO8zYKP/9g7k5bcnmC0qcaKjfOIqC1rNSyy1soOm907J2F/QZs+UBc2t5Y5pSLO8impd3e4hSMbIR2x6eSCQ8g30fm3CNl+ytLEjJRF095V8VtISi6CKfyZ1mqreRrGvalMvrdbsiB+znW4tWsoIgbjqYFYHffBUoeTlnMLJaD1WMG2AtlKtXHsTYzURjgev7y5ZBNs4OxorPlTMOJMNxCuahmbVGH2xXVt8AgpmskcTjaqjGFc6efysT1IPvnRDgeFEKXi8mHHa5u8o/RI3dbNZOHo4Bqaw2Hn6slqqwcjRB5UXDmJ2tfS1pkjVqua5PmPkdkc7+CEy2YOR2KRmWisVVAM3uF4Gi+HTmR8bcQ7Z/84Q7s2OLNcxsfnim/US7xMfged7NtujtdPw5snnnC7q6f+jdo0RAupeUIeIHHMVVb4fhfkNvEtIy9Az/5pmdl0Ywiy7fLpj8qm5HmZr81cyy1Lf3EcZCrd4p9XaD4l/gj/Dp/4dRYrj/eat/NmoGtXnc1OdODUuPNETEiqfJ36jxH0hJqERPzISJX+tqV2Z+ HJUjs8TL c8nNeri4z9hlCZDK69ox8Vz0vsBctrD21VzAxcOrH0tghAIaWIMrhmdKKug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 1/12/23 11:31, Maxim Levitsky wrote: > On Thu, 2023-01-12 at 07:07 +0100, Jiri Slaby wrote: >> >> I have rebooted to a fresh kernel which 1) have lockdep enabled, and 2) >> I have debuginfo for. So next time this happens, I can print held locks >> and dump a kcore (kdump is set up). >> >> regards, > > It is also possible that I noticed something like that on 6.1: > > For me it happens when my system (also no swap, 96G out which 48 are permanetly reserved as 1G hugepages, > and this happens with VMs which don't use this hugepage reserve) > is somewhat low on memory and qemu tries to lock all memory (I use -overcommit mem-lock=on) > > Like it usually happens when I start 32 GB VM while having lot of stuff open in background, but > still not nearly close to 16GB. > As a workaround I lowered the reserved area to 32G. > > I also see indication that things like htop or even opening a new shell hang quite hard. > > What almost instantly helps is 'echo 3 | sudo tee /proc/sys/vm/drop_caches' > e.g that makes the VM start booting, and unlocks everything. Note it's possible that temporary+recoverable things like that have a different cause caused by compaction caught in a loop, as discussed here: https://lore.kernel.org/all/a032e3bf-8470-095a-2262-791f5678e590@suse.cz/ > Best regards, > Maxim Levitsky >