From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4152C3DA60 for ; Wed, 17 Jul 2024 16:44:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 737016B0088; Wed, 17 Jul 2024 12:44:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E5F86B0089; Wed, 17 Jul 2024 12:44:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5ADBA6B0098; Wed, 17 Jul 2024 12:44:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3B9B06B0088 for ; Wed, 17 Jul 2024 12:44:25 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DC3EC40BE3 for ; Wed, 17 Jul 2024 16:44:24 +0000 (UTC) X-FDA: 82349817648.27.450DB6C Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf25.hostedemail.com (Postfix) with ESMTP id 082B4A0020 for ; Wed, 17 Jul 2024 16:44:22 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aQH9+IWQ; spf=pass (imf25.hostedemail.com: domain of kmanaouil.dev@gmail.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=kmanaouil.dev@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721234632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cFfYluthlwA2RVy6qc5qWtiAoAbqX1RAJ5c9inBp4sc=; b=evjiRqPLKOFTLXpZsy67FABnYBbAJJMBVzsC2DEvoNw14/F3ozSQpNhfyeWH96TYHEfKcy 0D9zCfnHsm9yk2MkDWgjXCKheAG+sUxa5e9Fx3mYxvag7/Akwu1Rl0lYM87NtY/xY/Bi1h ntxx9OkH4MisKWrJ460aXcp3q1YxY7w= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aQH9+IWQ; spf=pass (imf25.hostedemail.com: domain of kmanaouil.dev@gmail.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=kmanaouil.dev@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721234632; a=rsa-sha256; cv=none; b=lEeokp6/3vuNL6rxL1kmtHFN/oUxJFUmKdNOYEePvSTUmrBK/pg8uuSYO6qPx+by4xrPdK 9/DFGVXR8lYTt39iJ7CeMsWkhAQnvMpNeg1+WZACXxMfrUQyT8C56oFp+X1RrQgDNXK8mn As2/ExUholQ1LxHhNa3ubt2MCX5aIfM= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-4279dadb7b1so3541485e9.0 for ; Wed, 17 Jul 2024 09:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721234661; x=1721839461; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=cFfYluthlwA2RVy6qc5qWtiAoAbqX1RAJ5c9inBp4sc=; b=aQH9+IWQ5+Y74qQAPjkHFiqdpeU0I28yHdxvaUhg6ADwbx62rdBRvDLbULAddtuBVv WN9nQVhuQfO0OttBpgLF6XXTjgGME5mL/l6G/ae0kj7AmY5g+M+6YB+mwej6nRlExHL/ EaGOc3OrCL46qq0W5Mm/DSqLBwB3FPOqZWKw2XKGKtdIK0b4s7aeXRkbXsKLR7E1b3iY XuAs4FFuJbtKcv1OdooFFZEssLw2VqZwSxvZfe2WdgzOK4rxG4CZ2XOV+CyH6hhlvxlv DjET+EnElQbE/Z+Dp9LsJvy9KfyDfX/u5OHdxv3WQp1WA6zOsMEuWsLIqdtsE4Tnpaq8 QglA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721234661; x=1721839461; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=cFfYluthlwA2RVy6qc5qWtiAoAbqX1RAJ5c9inBp4sc=; b=TDdJGgd9AZzMVIB+mXLxbV2Vu6OKa7YudtdeEXs+Atg9n1zolbF0jrOQsdSA9tUef9 k3W4VtCWoUyq/nz18q/7jJRbwJ9Hy+TS3+AvyycqLaW83ZkUeH5hyCjej+rV5Ydwwc46 jEVToWLXcIviko9yeQRir27Q5VMXIUZjm4nd7jKrmuFsQkZZJ2L4Dpr1EwRKvHWqrjSN moa+XfL9v3C4WhZsJXZpkpT69hXDJnhn1Z7KHhBb2vHZsMpdIdsPyZCmJpNK6CLSnpS+ mAUIxUAKbZQxajUPr4QcaaWXp/+DtoPrv/djslCw4gwzkWVKHMMF7iSwQlbu2I8Nv3R+ D96Q== X-Forwarded-Encrypted: i=1; AJvYcCV1/pfRYaY/i3WSLF3KgxwA0yEFuDtNRLBLM6NduA5ZkYwNn+9FFWBS/qPQpqXqjX1sMfbyASjj7hKFgDMNodcccmA= X-Gm-Message-State: AOJu0Yw2JMlpJ1qmoS9K+8/Ts2VWBsbyDEl9TdHT5AReytnouLR+OhzP FvW12pOzgUBbri6exHemX6AqiEAjengTr0P8EnQHOw9kC6usgKFS X-Google-Smtp-Source: AGHT+IHM0649GqAgF7poVfYwbNHNNEIpOsttmVLHcOptNhloT99i/92dTKzOliYqvCTpM53GD6fg+w== X-Received: by 2002:a05:6000:188f:b0:366:e4b4:c055 with SMTP id ffacd0b85a97d-36831735017mr929595f8f.7.1721234661091; Wed, 17 Jul 2024 09:44:21 -0700 (PDT) Received: from ed.ac.uk ([2001:630:3c1:90:1614:6de0:61c7:40b0]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-368422c0297sm886129f8f.28.2024.07.17.09.44.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 09:44:20 -0700 (PDT) Date: Wed, 17 Jul 2024 17:44:18 +0100 From: Karim Manaouil To: Bharata B Rao Cc: Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, yuzhao@google.com, kinseyho@google.com, Mel Gorman , Mateusz Guzik Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system Message-ID: References: <3128c3c0-ede2-4930-a841-a1da56e797d7@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: ikd9oo44ax48d76yzi8dwmet58dxorwi X-Rspam-User: X-Rspamd-Queue-Id: 082B4A0020 X-Rspamd-Server: rspam02 X-HE-Tag: 1721234662-112639 X-HE-Meta: U2FsdGVkX18i91OLzhmW2+PSqaQuJtsoYk9VVExgr8FGu2iZWS4NLUouZ6gqUchPhN2D3+OW1w0ZR2ktKoiWoA/98GAStMwIoweJEQOo4iaCxNSHM0/HXSA4fZizTX0QQSx76bCkN3SC17CRnNb4Ohb7ZPphTemJS/00Nb4/CiCjd1WjDURs0jak87EumM6i00nBEQqWsazESLAXXEDGpL3/q2Ykqp7xcz6HA012cCRzOJR9XIeQJv+JbdoarDFiE3IMoLZljoRFYDni59Nj36zjBwgWlrXZIwDzfWuzzvVJFpUlAQl/Drs+05cAvje4A8TpplpZRquYqVmKEHbrLad/xuamvK1mEwTob+LCZCNrVwm4rrfmArqmbJcvd4IQgdASdukv9eNEbf8YfKBkTzwmudE9e6JgG2Pmc8klU+eDiIEpJ20ntvWps2/QD7TZ57+k3usIZaw9cxunvcSfIwwiDMQwwn1F7EZIBo5KVSVul0cvVYohmbi34plfNaVUfVyxkUC8mj/UBM57l1MLj33iGL9H3DsU5J2mJnT5vlCxxNILYqyR4DNDQ0gXuIzJLLVPfjyS637V/YeCJVbjtONi63el+ojf7o3ukhFvn+5OYWNnDJArX3JbtH4n+AHqo5881UjpwUrXYhkHX6lz/gAnBwTIb8RGVd1E5/bKmlHKSjW4m7O5tQ7mn8ZUWx6LOUFBRU1nzjw7MG5ZcF0lR+zmjgjRly9F3st34NS65J0xEo3VCQ1wNQJEaH4pImmyjtQ2c3f1/6WF7aqV69AQjsvOUeeTzNg9YOy0HJWUz0AdE3iX7K/8GGR6iA02T2EQi9rGUTK2KCz9EMahkS9WZZ/RWQiZNeStOQw2LFkRoohsC8A6acfmwXpkmGlRvIAcYdnIpPwqpNKDpnZ4ej2IUQBS9WmWLSZZJqUjS8Eb0H9pfV9sTRQe5boRzFwKEt8l/Z/AqV0Sb3lKLoVGIPj Ba8zNUUR zvTvIYvACf0v69jL0RF9dtyI2XTfr2dgB+YosaOhQZ5Ik6LbAQNmnWCUmzdaQ8SGXw7q0JV8A7+RxVCO4HWrgttnV1VqWIw5B5jKBEv98tYwt20c2tfJambwCyRUlN7wg6znX5ho7Gzfyvf3H0AhrVYgaJ0qSQPvRsl440oeNiuMX0jWHK6qb6A6HXJWeNSJwoTKvdJ62ypO/sBeoKF1nyFpQ0SYyqRXHe8w7lFa2RYdhCk5uIIgv7xX7uAt0QPTS3viXE/dUqTiXHTuwzdjsR5xAsHlWQ0peBCOKd0Z81JlbpFgz8YZvF/EUQlizNhZpBRE8dcoB/Fi5IHaFmF2NpezuCOa32C/JWba07tnEJkEWVJbIhQAyqbp+dYnJsflFValnyn/yl59HWk5VVdAo2c7w8ez8Ww9SqmvOA+xCVS/8B/A/QzPXVXHMQn6X6WtRmKk0AN2SIcBgGJ0L9Ew5iIoXA1g5N8WN6GyHcyA7AtpqIsUXEQ/w34cx7rt/qAxb+fbZDVMBqh8MAX53lf7OXJh9Lw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 17, 2024 at 04:01:05PM +0530, Bharata B Rao wrote: > On 17-Jul-24 3:12 PM, Vlastimil Babka wrote: > > On 7/3/24 5:11 PM, Bharata B Rao wrote: > > > Many soft and hard lockups are seen with upstream kernel when running a > > > bunch of tests that include FIO and LTP filesystem test on 10 NVME > > > disks. The lockups can appear anywhere between 2 to 48 hours. Originally > > > this was reported on a large customer VM instance with passthrough NVME > > > disks on older kernels(v5.4 based). However, similar problems were > > > reproduced when running the tests on bare metal with latest upstream > > > kernel (v6.10-rc3). Other lockups with different signatures are seen but > > > in this report, only those related to MM area are being discussed. > > > Also note that the subsequent description is related to the lockups in > > > bare metal upstream (and not VM). > > > > > > The general observation is that the problem usually surfaces when the > > > system free memory goes very low and page cache/buffer consumption hits > > > the ceiling. Most of the times the two contended locks are lruvec and > > > inode->i_lock spinlocks. > > > > > > - Could this be a scalability issue in LRU list handling and/or page > > > cache invalidation typical to a large system configuration? > > > > Seems to me it could be (except that ZONE_DMA corner case) a general > > scalability issue in that you tweak some part of the kernel and the > > contention moves elsewhere. At least in MM we have per-node locks so this > > means 256 CPUs per lock? It used to be that there were not that many > > (cores/threads) per a physical CPU and its NUMA node, so many cpus would > > mean also more NUMA nodes where the locks contention would distribute among > > them. I think you could try fakenuma to create these nodes artificially and > > see if it helps for the MM part. But if the contention moves to e.g. an > > inode lock, I'm not sure what to do about that then. > > See below... > > > > > > > > > > 3) AMD has a BIOS setting called NPS (Nodes per socket), using which a > > > socket can be further partitioned into smaller NUMA nodes. With NPS=4, > > > there will be four NUMA nodes in one socket, and hence 8 NUMA nodes in > > > the system. This was done to check if having more number of kswapd > > > threads working on lesser number of folios per node would make a > > > difference. However here too, multiple soft lockups were seen (in > > > clear_shadow_entry() as seen in MGLRU case). No hard lockups were observed. > > These are some softlockups seen with NPS4 mode. > > watchdog: BUG: soft lockup - CPU#315 stuck for 11s! [kworker/315:1H:5153] > CPU: 315 PID: 5153 Comm: kworker/315:1H Kdump: loaded Not tainted > 6.10.0-rc3-enbprftw #12 > Workqueue: kblockd blk_mq_run_work_fn > RIP: 0010:handle_softirqs+0x70/0x2f0 > Call Trace: > > __irq_exit_rcu+0x68/0x90 > irq_exit_rcu+0x12/0x20 > sysvec_apic_timer_interrupt+0x85/0xb0 > > > asm_sysvec_apic_timer_interrupt+0x1f/0x30 > RIP: 0010:iommu_dma_map_page+0xca/0x2c0 > dma_map_page_attrs+0x20d/0x2a0 > nvme_prep_rq.part.0+0x63d/0x940 [nvme] > nvme_queue_rq+0x82/0x210 [nvme] > blk_mq_dispatch_rq_list+0x289/0x6d0 > __blk_mq_sched_dispatch_requests+0x142/0x5f0 > blk_mq_sched_dispatch_requests+0x36/0x70 > blk_mq_run_work_fn+0x73/0x90 > process_one_work+0x185/0x3d0 > worker_thread+0x2ce/0x3e0 > kthread+0xe5/0x120 > ret_from_fork+0x3d/0x60 > ret_from_fork_asm+0x1a/0x30 > > > watchdog: BUG: soft lockup - CPU#0 stuck for 11s! [fio:19820] > CPU: 0 PID: 19820 Comm: fio Kdump: loaded Tainted: G L > 6.10.0-rc3-enbprftw #12 > RIP: 0010:native_queued_spin_lock_slowpath+0x2b8/0x300 > Call Trace: > > > > _raw_spin_lock+0x2d/0x40 > clear_shadow_entry+0x3d/0x100 > mapping_try_invalidate+0x11b/0x1e0 > invalidate_mapping_pages+0x14/0x20 > invalidate_bdev+0x40/0x50 > blkdev_common_ioctl+0x5f7/0xa90 > blkdev_ioctl+0x10d/0x270 > __x64_sys_ioctl+0x99/0xd0 > x64_sys_call+0x1219/0x20d0 > do_syscall_64+0x51/0x120 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x7fc92fc3ec6b > > > The above one (clear_shadow_entry) has since been fixed by Yu Zhao and fix > is in mm tree. > > We had seen a couple of scenarios with zone lock contention from page free > and slab free code paths, as reported here: https://lore.kernel.org/linux-mm/b68e43d4-91f2-4481-80a9-d166c0a43584@amd.com/ > > Would you have any insights on these? Have you tried enabling memory interleaving policy for your workload? Karim PhD Student Edinburgh University