From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 313E5C3DA42 for ; Wed, 17 Jul 2024 11:29:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F0716B00A3; Wed, 17 Jul 2024 07:29:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A11B6B00A4; Wed, 17 Jul 2024 07:29:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 667826B00A5; Wed, 17 Jul 2024 07:29:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 492E26B00A3 for ; Wed, 17 Jul 2024 07:29:48 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E0F2B1415C6 for ; Wed, 17 Jul 2024 11:29:47 +0000 (UTC) X-FDA: 82349024814.03.DB455EB Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf05.hostedemail.com (Postfix) with ESMTP id 0454C100016 for ; Wed, 17 Jul 2024 11:29:45 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aKeJ0+yE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721215753; a=rsa-sha256; cv=none; b=GCk0kN0xqPIfctQiFYyzs5b/kqDXDGLSGcUY/JB+FZYa59pCt+JjGMSDhU7zKrfurIFOoW NLz9+XMcJx5bVCQHROcCGYAQITPK8kjRS1LCxW5oJnHpEZIt8HF04vj/kv0vMGmpwRcjdo Bcqmy2xl+We56Ub4oBB5WMeo0og7L5E= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aKeJ0+yE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721215753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TzVI1EFi6ALaN0XOENS1zraS9tfSh0A/9XOpn/vT/ZY=; b=eZlUIgeYPH3XjOx9rNM025+PBCXBHxknvdKRipeeinCa6VPZuaaXhn5LcONRvcJKklbtRL TQR3++60n94/tRB7A6+LuYHasObBYfHLrHthLqmzkXa/uWxz+QRiLoTg6LbYOw0W11oprL c+REOON6cd9nLFzg8zMhi3PxoiSts5c= Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a77c1658c68so700288266b.0 for ; Wed, 17 Jul 2024 04:29:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721215784; x=1721820584; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=TzVI1EFi6ALaN0XOENS1zraS9tfSh0A/9XOpn/vT/ZY=; b=aKeJ0+yEmAvaW+z0YYjfJkDkH9Gnq2kRTiHvyVWO+c1fED3MkFjW3Flgnp0sl62BwZ V4k3kYDRSRe4Be0KJOd18F802+Ncfx0W+RM9CLcBnW46IhRY6YDifPtUbRZk8xrObzKG l5ddMIOItztwLlUtfi/Ho2nc77zyo+HrKhXw3qUokY+PX7lUgR/CqjaWvf+wUoyZZE3o 3WzXw/2/czKsrD/A8nlOD64CeMikWFUxZ70Z4NJCKoEYmnE7jRxIOmyipXtFO7Adq+/y Zic/CxCXatj4Z2nlQCDe7TlcT0VtmvhI5TXiwSsmDRtp7raw1iR8zBCt5V/CNR94M81I aNhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721215784; x=1721820584; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TzVI1EFi6ALaN0XOENS1zraS9tfSh0A/9XOpn/vT/ZY=; b=T2kA1OFNYL+88sB4zROQbc/r3cpmXZCHQItahzw8FcPmq3OeU3xfqS+GBk7Zy7wa8u qdo4ob6rRRWCH1k62Wq/fz8OFcn33U9SP0gnYinC5xgtm/5cAocGu6YPFoJRSMG3Hyt7 KNqqPFAJg3pCZXWW1shR9DnU8YOBtpPYV8/lIm+BJh4ZabUE8UHBUjwEk7A3GdgQE4oa vpvyYCIkx3q9ZyBKEuhElOl+U79J65A4Dh+DAEj4n3u2NR6U/EVOowIGbDlCjZoFXx6Q Kh5I6Mr/9Zyg0aywe0ivNbRX+OXBvFZ0ScLxbqGoVBrLs/lz/k77V5kEg3CheTESE0ON ehRg== X-Forwarded-Encrypted: i=1; AJvYcCVkkeZFIStl2ynICMeQ9e/WFKPjq0jaDMvYMbDmLg7PRuGnR3gZcCnZtcgB0/PvcvVn8ZUYf5JOaiz7y8aPUin3MlY= X-Gm-Message-State: AOJu0Yzw2rXxbF2HzrLkargJ8yAja/5bAJifZgWNjKv/J+XJeU7yjyg1 eQ7N2qrGTToSx0SatavAGUb35RI/aiEv8ORdhSxcf/Asz6edFNIQGGVoAGIh1Wk24krrLhj9g/L EbOBYXW+XUuyvvt1nKBSVW740Bn0= X-Google-Smtp-Source: AGHT+IGZ45vk6STtbL9xDLsKaQi2+D8pgY/I0jmfzUSTDFYODXC0DIYb58IA1z/3Vcs03IpW+4zu4qLJoie/U3dy/A8= X-Received: by 2002:a17:906:547:b0:a72:8d2f:8594 with SMTP id a640c23a62f3a-a7a01147a55mr97637366b.27.1721215784008; Wed, 17 Jul 2024 04:29:44 -0700 (PDT) MIME-Version: 1.0 References: <3128c3c0-ede2-4930-a841-a1da56e797d7@suse.cz> In-Reply-To: <3128c3c0-ede2-4930-a841-a1da56e797d7@suse.cz> From: Mateusz Guzik Date: Wed, 17 Jul 2024 13:29:31 +0200 Message-ID: Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system To: Vlastimil Babka Cc: Bharata B Rao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, yuzhao@google.com, kinseyho@google.com, Mel Gorman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0454C100016 X-Stat-Signature: qskdub7edhfe6edg3zg8exu6fbngyqcb X-Rspam-User: X-HE-Tag: 1721215785-493710 X-HE-Meta: U2FsdGVkX185pWF8kaNj66kjJUn4ztW0N7Ef7fm5Tx7ycwU3uDSq4wW3bMviHGt1OypKUJ8tLAcq0wSjx6Dm1rjxuqHVKtA+l5+2OWLa5DlcAEun1Erl0MYOAASMkABGv8xJO2RrK/DfXzYOI235oKobsx3xlhNhcTjZSkaOQOk8OD7PNo8zqyVdhDyBY/D0X4IAb/lbdmwLiAA8iPMpy1usIrPkXrbgtt1RRynsfHQiyNha6MwvQN4GD7sX6zdV2EN+gFtmuoqq77Tdpskvk1T/P1SOIobQBsx4UE4afXczy/hcs6h3LgbIE9HYM9FGaWbweZ6AfIH6FX9ipQ22o2nfmx8Tq8XS6OJKrOjBEYMfxDfc/4KQIjPdo9o40xUmdKrR/e946fiXR/+ra6D9jAmaTtUhBuyjwFazAaolXNwFdDVxbCVWQDWWRlIcYU1z5VlX0xADjosZc1CiVZFTq4jIvVN88IjTiuTdeuDyC/1hXEG7Is7eIczh6Le8KJegraubFYJYFbeo1rMASyN6aaoxQUrdpFlkYb+EFjEm4SnfUnk671BgWJPSMRX/oqsO4lhqPMUnwqbiJbx8cIf7Bu9vyj9AnkEB3jdNAqY/zNa+dnNpiY1Cd68QzfE0ZcUmfYMxzrpOhc1KCR8JcwG6ZZTWbmUC6tQiaBsiPSUWsDBR4jK3753Wtk/jrXZ1ugHgqHYUEGs0UaedMVbBfFbtIagwr5hEs5wuK5NyvcVDsdP4EfEsWa3Dm6ClZJW8QXeet1VEluVkr87n5Xh/CtFEnJcadajub1YYl0J+ZDeuW1g2Yc8ReodUaKkQs5/Y8g4j85ymnRQASwoqIulsTHQowOjDny15rjKhqQHAstUCF53fm2o7x1cU1Jlrtdx+TtdRvK0i4tFScevkzT9F0ontsMMuqCyVU+ekdJNned+qWPsYJBgfpVt+gxhXb8ImGurWauq5rB9sRPACktcoesa La9+3ihr vg65bQ8p8yFe9wS73lOsJgDcXW7XFYwHtZUhtXTc7KWY3xgZLP0+BR64lW4+x4pQdwGX83om3/4vbq0wIInfNF/pXjGsl2n/CtwHVL//ytF0zEu+Tdbf0cj71sTKiuLMGNLc4kH26MadkgfpvlSsg3OqILuEzjA/XJ0EWSL2l9Nxj+R3PxZE+p6yGPU+TYmDJv3faxNLtIBthpDBW4Y4YM95FUVhsogtfzahpTJvNMrl5n+t8pzYyN1qIl4P8/7UnQqaCoOtvcBnLoy4rVauCg50jt6KOXAowS1LtN2crBIy8hSH71SplNG8Kx4ie/bipYBYB X-Bogosity: Ham, tests=bogofilter, spamicity=0.297086, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 17, 2024 at 11:42=E2=80=AFAM Vlastimil Babka w= rote: > > On 7/3/24 5:11 PM, Bharata B Rao wrote: > > The general observation is that the problem usually surfaces when the > > system free memory goes very low and page cache/buffer consumption hits > > the ceiling. Most of the times the two contended locks are lruvec and > > inode->i_lock spinlocks. > > [snip mm stuff] There are numerous avoidable i_lock acquires (including some only showing up under load), but I don't know if they play any role in this particular test. Collecting all traces would definitely help, locked up or not, for example: bpftrace -e 'kprobe:queued_spin_lock_slowpath { @[kstack()] =3D count(); }' -o traces As for clear_shadow_entry mentioned in the opening mail, the content is: spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); __clear_shadow_entry(mapping, index, entry); xa_unlock_irq(&mapping->i_pages); if (mapping_shrinkable(mapping)) inode_add_lru(mapping->host); spin_unlock(&mapping->host->i_lock); so for all I know it's all about the xarray thing, not the i_lock per se. -- Mateusz Guzik